On Biostatistics and Clinical Trials

How to interpret odds ratios that are smaller than 1?

2012-02-12T16:55:00.001-05:00

Here is a question posted on the web about the interpretation of odds ratios that are less than 1.

"I know that OR estimates= 1 mean that both groups/categories have the same odds. I also know that if OR estimates are greater than 1, e.g, 1.24 for Young vs. Old persons, then I can say: Young people have 24% increase in the odds of living in an apartment than older people. Or, I also know I can say, for example, for an OR of 0.322 Non-White vs. White, that the odds of Whites are 1/.322 = about 3 times higher than those of Non-Whites, to live in a house they own. Now, how would I say the odds are of a NON-White person in the example above, to live in a house they own? Is is 1-.322=.678 less likely, with respect to odds, to live in a house they own? Or, similarly, they have 67.8% lower odds to live in a house they own? "

If we have to say the odds for a Non-White person, we may say "Non Whites have odds .322 times as great as those of Whites".

In an article "When can odds ratio misled?", Davies et al stated:

"the odds of an event is the number of those who experience the event divided by the number of those who do not. It is expressed as a number from zero (event will never happen) to infinity (event is certain to happen). Odds are fairly easy to visualise when they are greater than one, but are less easily grasped when the value is less than one. Thus odds of six (that is, six to one) mean that six people will experience the event for every one that does not (a risk of six out of seven or 86%). An odds of 0.2 however seems less intuitive: 0.2 people will experience the event for every one that does not. This translates to one event for every five non-events (a risk of one in six or 17%). "

Another webblog described the issue in interpreting the odds ratio that is less than one.

"When you are interpreting an odds ratio (or any ratio for that matter), it is often helpful to look at how much it deviates from 1. So, for example, an odds ratio of 0.75 means that in one group the outcome is 25% less likely. An odds ratio of 1.33 means that in one group the outcome is 33% more likely."

In an article "The odds ratio: calculation, usage, and interpretation" in Biochemia Medica, the author clear suggest converting the odds ratio to be greater than 1 by arranging the higher odds of the evnet to avoid the difficulties in interpreting the odds ratio that is less than 1.

“An OR of less than 1 means that the first group was less likely to experience the event. However, an OR value below 1.00 is not directly interpretable. The degree to which the first group is less likely to experience the event is not the OR result. It is important to put the group expected to have higher odds of the event in the first column. It is not valid to try to determine how much less the first group’s odds of the event was than the second group’s. When the odds of the first group experiencing the event is less than the odds of the second group, one must reverse the two columns so that the second group becomes the first and the first group becomes the second. Then it will be possible to interpret the difference because that reversal will calculate how many more times the second group experienced the event than the first. If we reverse the columns in the example above, the odds ratio is: (5/22)/(45/28) = (0.2273/1.607) = 0.14 and as can be seen, that does not tell us that the new drug group died 0.14 times less than the standard treatment group. In fact, this arrangement produces a result that can only be interpreted as “the odds of the first group experiencing the event is less than the odds of the second group experiencing the event”. The degree to which the first group’s odds are lower than that of the second group is not known.”

In practice, when dealing with the odds ratio less than 1, when possible, I almost always try to reverse the column or recode the response variable to get the odds ratio larger than 1 before I do an interpretation. It is easier for people (especially non-statisticians) to understand the odds ratio with the value greater than 1.

In an example below, the treatment group is actually less effective in terms of the response.

Treatment	Failure (0)	Success (1)
No (0)	21	30
Yes (1)	32	17

The following SAS code can be easily used to calculate the odds ratio:
Data test; input Trt resp count; datalines;
1 1 17
1 0 32
0 1 30
0 0 21
;
proc logistic data=test descending; weight count; model resp=trt;
run;

From the SAS outputs, we get the odds ratio of 0.372, which indicates that the treatment group has odds 0.372 times lower compared to the non-treatmetn group in terms of the success. The interpretation is somewhat difficult to understand.

The program can be easily revised to calculate the odds ratio of failure rate, which gives an odds ratio of 1/0.372 = 2.689. The odds ratio can be intepretated as "the odds of achieve the success in non-treatment group is 2.689 times higher than that in treatment group".

proc logistic data=test; weight count; model resp=trt;
run;

In SAS PROC Logistic, with descending option, probability modeled is response=1 (success); without descending option, probability modeled is response=0 (failure);

Design and Analysis of Bioequivalence Studies for Highly Variable Drugs

2012-02-05T23:54:00.000-05:00

For bioequivalence studies, it is often for us to show the average bioequivalence by declaring the bioequivalence if the 90% confidence interval of the geometric least squares mean ratio is within 80-125%. The associated study design is typically 2x2x2 cross over design with reasonable sample size (for example, 12 subjects, 24 subjects,…) if the within subject variable is not so big. This approach has been outlined in several FDA’s guidelines:

- Statistical Approaches to Establishing Bioequivalence

- Bioavailability and Bioequivalence Studies for Orally Administered Drug Products — General Considerations

- Food-Effect Bioavailability and Fed Bioequivalence Studies

Recently, there are a lot of discussions about the bioequivalence studies for a product with high variability (high variable drugs). Highly Variable Drugs refer to the type of drugs with higher within subject variability and is Defined as one for which the root mean square error (RMSE) from the ANOVA bioequivalence analysis > 0.3 for either AUC or Cmax.

For highly variable drugs, if we employ the common study design, the required sample size will be very large, which will cause the ethic concerns to implement such studies.

FDA had several advisory committee meeting in discussing this issue. The most recent meetings were in 2004 and 2009. In 2009 meeting, the slide presentation by Dr Conner from FDA summarized the development in dealing with this issue and FDA’s position (see slide presentation “Bioequivalence Methods for Highly Variable Drugs and Drug Products”).

Among various approaches to address the bioequivalence issue for highly variable drugs, reference-scaled average BE approach has been suggested. This approach requires less subjects in the study, but with replicated treatment design such as three-period, reference- replicated, crossover design with sequences of TRR, RTR, & RRT or four-period design with sequences of TRTR and RTRT. The replicated crossover designs were also discussed in FDA guidance “Statistical Approaches to Establish Bioequivalance”, but was for dealing with the carryover effects. Here, the replicated crossover designs are for dealing with highly variable drugs.

The implementation of the reference-scaled average BE approaches have been detailed and discussed in FDA guidance (draft) many publications. The most relevant ones are:

- FDA Guidance on Progesterone (2011)

- Sample Sizes for Designing Bioequivalence Studies for Highly Variable Drugs by Endrenyi and Tothfalusi (2012)

- Bioequivalence of Highly Variable Drugs: A Comparison of the Newly Proposed Regulatory Approaches by FDA and EMA by Karalis et al (2011)

Recording the outcome for AE/SAE when multiple events contribute to Death

2012-01-28T12:47:00.000-05:00

In clinical trials, death event may occur. According to CDISC CDASH standards, death should not be recorded as an adverse event (AE) or serious AE, but should be recorded as the outcome of the event. The condition that resulted in the death should be recorded as the AE/SAE.

In the case of multiple AE/SAEs contributing to the fatal (death) outcome, there seem to be two different ways in recording the AE outcome. There is no clear regulatory guideline in detail about this situation. The most closely related guideline may be from the ICH E2B where it states:

B.2.i.8 Outcome of reaction/event at the time of last observation
recovered/resolved
recovering/resolving
not recovered/not resolved
recovered/resolved with sequelae
fatal
unknown

User Guidance:
In case of irreversible congenital anomalies the choice, not recovered/not resolved should be used.
Fatal should be used when death is possibly related to the reaction/event. Considering the difficulty of deciding between "reaction/event caused death" and "reaction/event contributed significantly to death", both were grouped in a single category. Where the death is unrelated, according to both the reporter and the sender, to the reaction/event, death should not be selected here, but should be reported only under section B.1.9.

In practice, one way is to record multiple SAEs and record 'fatal' for each of these SAEs. The drawback of this approach is to have multiple SAEs with 'fatal' outcome for the same subject while subject can only die once.

Another way is to identify one SAE as the principal cause of the death. In this case, only will one SAE have the outcome recorded as Fatal. The subject can only die once so it makes sense to record ‘fatal’ as the outcome for the principal event The question is that what the appropriate outcome should be for other SAEs that may also contribute to the death event. If there is an 'Ongoing' option in the list of AE outcomes, the appropriate choice may be 'ongoing' indicating that the SAE is ongoing during the time of death. If there is no choice of 'ongoing' in the AE outcome list (as specified in E2B above), the most appropriate choice seems to be "not recovered/not resolved" indicating that the SAE is still not resolved during the time of death. For AE stop time, the principal SAE with fatal outcome should be the time of death. For other SAEs contributing to death, the stop time may be appropriately recorded as 'ongoing' instead of recording the death time as the SAE stop time.

Serious Adverse Events (SAE) vs Severe Adverse Events

2011-12-02T15:14:00.000-05:00

Professionals who are new to the clinical trial field are often confused with the concept of 'Serious Adverse Events (SAEs)' and 'Severe Adverse Events". Severity is not synonymous with seriousness. SAE is based on patient/event outcome or action criteria usually associated with events that pose a threat to a patient's life or functioning. Seriousness (not severity) serves as a guide for defining regulatory reporting obligations. In other words, the SAEs need to be filfill additional reporting process (reported to corporate global drug safety group or pharmacovigilence group, regulatory authorities, EC/IRBs). Severe AE is one class of AEs with severity (old term intensity) classified as 'severe'. Severe AE is one of the AE classifications – AE severity (other classifications are relationships/causality).

The FDA defines a serious adverse event (SAE) as one when the patient outcome is one of the following:

Death
Life-threatening
Hospitalization (initial or prolonged)
Disability - significant, persistent, or permanent change, impairment, damage or disruption in the patient's body function/structure, physical activities or quality of life.
Congenital anomaly
Requires intervention to prevent permanent impairment or damage

On the other hand, Severity of an AE is a point on an arbitrary scale of intensity of the adverse event in question. The terms "severe" and "serious" when applied to adverse events are technically very different. They are easily confused but can not be used interchangeably, require care in usage.

A headache is severe, if it causes intense pain. There are scales like "visual analog scale" that help us assess the severity. On the other hand, a headache is not usually serious (but may be in case of subarachnoid haemorrhage, subdural bleed, even a migraine may temporally fit criteria), unless it also satisfies the criteria for seriousness listed above. Similarly, a severe rash is not likely to be an SAE. However, mild chest pain may result in a day’s hospitalization and thus is an SAE.

Classifications of the AE sevirity often include the following:

Mild: Awareness of signs or symptoms, but easily tolerated and are of minor irritant type causing no loss of time from normal activities. Symptoms do not require therapy or a medical evaluation; signs and symptoms are transient.
Moderate: Events introduce a low level of inconvenience or concern to the participant and may interfere with daily activities, but are usually improved by simple therapeutic measures; moderate experiences may cause some interference with functioning
Severe: Events interrupt the participant’s normal daily activities and generally require systemic drug therapy or other treatment; they are usually incapacitating

The guidelines for AE severity assessment is based on:

Other sources about this topic may be useful:

Reporting/recording the Serious Adverse Events (SAE) vs. Adverse Event (AE) Outcomes

2011-11-27T11:20:00.000-05:00

In clinical trials, the serious adverse event reporting is critical to the safety assessment and to fulfill the regulatory requirements. The criteria for defining an SAE have been documented in many regulatory guidelines. However, in clinical trial implementation, the confusion could arise whether or not an event should be reported as an SAE or outcome of an SAE. Misinterpretation of the regulatory guidelines could cause in the inappropriate reporting of SAEs.

Acording to ICH E2A “CLINICAL SAFETY DATA MANAGEMENT: DEFINITIONS AND STANDARDS FOR EXPEDITED REPORTING”

                  A serious adverse event (experience) or reaction is any untoward medical occurrence that at any dose:                          * results in death,
                         * is life-threatening,
                             NOTE: The term "life-threatening" in the definition of "serious" refers to an event in which the patient was at
                             risk of death at the time of the event; it does not refer to an event which hypothetically might have caused death
                             if it were more severe.
                         * requires inpatient hospitalisation or prolongation of existing hospitalisation,
                         * results in persistent or significant disability/incapacity, or
                         * is a congenital anomaly/birth defect.
FDA website has provided a little bit more detail descriptions on SAE
"What is a Serious Adverse Event?

An adverse event is any undesirable experience associated with the use of a medical product in a patient. The event is serious and should be reported to FDA when the patient outcome is:
Death

Report if you suspect that the death was an outcome of the adverse event, and include the date if known.
Life-threatening

Report if suspected that the patient was at substantial risk of dying at the time of the adverse event, or use or continued use of the device or other medical product might have resulted in the death of the patient.
Hospitalization (initial or prolonged)

Report if admission to the hospital or prolongation of hospitalization was a result of the adverse event.

Emergency room visits that do not result in admission to the hospital should be evaluated for one of the other serious outcomes (e.g., life-threatening; required intervention to prevent permanent impairment or damage; other serious medically important event).
Disability or Permanent Damage

Report if the adverse event resulted in a substantial disruption of a person's ability to conduct normal life functions, i.e., the adverse event resulted in a significant, persistent or permanent change, impairment, damage or disruption in the patient's body function/structure, physical activities and/or quality of life.
Congenital Anomaly/Birth Defect

Report if you suspect that exposure to a medical product prior to conception or during pregnancy may have resulted in an adverse outcome in the child.
Required Intervention to Prevent Permanent Impairment or Damage (Devices)

Report if you believe that medical or surgical intervention was necessary to preclude permanent impairment of a body function, or prevent permanent damage to a body structure, either situation suspected to be due to the use of a medical product.
Other Serious (Important Medical Events)

Report when the event does not fit the other outcomes, but the event may jeopardize the patient and may require medical or surgical intervention (treatment) to prevent one of the other outcomes. Examples include allergic brochospasm (a serious problem with breathing) requiring treatment in an emergency room, serious blood dyscrasias (blood disorders) or seizures/convulsions that do not result in hospitalization. The development of drug dependence or drug abuse would also be examples of important medical events."

The standard coding dictionary for adverse events is MedDRA (Medical Dictionary for Regulatory Activities). The guidance document MedDRA® TERM SELECTION: POINTS TO CONSIDER gives clear explanation how death and other patient outcomes should be handled.

3.2 – Death and Other Patient Outcomes

Death, disability, and hospitalization are considered outcomes in the context of safety reporting and not usually considered ARs/AEs. Outcomes are typically recorded in a separate manner (data field) from AR/AE information. A term for the outcome should be selected if it is the only information reported or provides significant clinical information.

(For reports of suicide and self-harm, see Section 3.3).

3.2.1 Death with ARs/AEs

Death is an outcome and not usually considered an AR/AE. If ARs/AEs are reported along with death, select terms for the ARs/AEs. Record the fatal outcome in an appropriate data field.

3.2.4 Other patient outcomes (non-fatal)

Hospitalization, disability and other patient outcomes are not generally considered ARs/AEs.

There are many other examples in terms of recording the outcome instead of AE/SAE. Adverse events represent the untoward medical event, not the intervention to treat that event. For example, if a subject has appendectomy, the AE is appendicitis not the surgical procedure; if a subject has an limb amputation, the AE is the cause for amputation (perhaps, the worsening of the ischemia in the peripheral artery) and limb amputation should be reported as the outcome of the AE/SAE; If a patient is hospitalized due to congestive heart failure, congestive heart failure should be reported SAE and hospitalization should be reported as an outcome for congestive heart failure.
We should also be aware that not every hospitalization will have an associated SAE to be reported. Any AE leading to hospitalization or prolongation of hospitalization meets ONE of the followings should not be reported as SAE.
A hospitalization admission is pre-planned (ie, elective or scheduled surgery arranged prior to the start of the study). European Commission’s guidelines on medical devices “CLINICAL INVESTIGATIONS: SERIOUS ADVERSE EVENT REPORTING “ indicated that a planned hospitalization for pre-existing condition, or a procedure required by the Clinical Investigation Plan, without a serious deterioration in health, is not considered to be a serious adverse event.
A hospitalization admission is clearly not associated with an AE (eg, social hospitalization for purposes of respite care). If a patient wants to be stay in the hospital during the drug treatment because of the fear that something bad could happen, this should not be reported as SAE just because of the hospital stay if nothing else happens
According to these definitions, the events with outcome of death, hospitalization, disability or permanent damage, congenital anomaly/birth defect, … should be reported as SAE while death, hospitalization, disability or permanent damage, congenital anomaly/birth defect…should be reported as the outcome of the corresponding SAE. To be crystal clear, the Death, Hospitalization should not be reported as SAE and the causes leading to the death and hospitalization should be reported as SAE.

Studentized residual for detecting outliers

2011-11-24T00:23:00.000-05:00

Last time, I discussed the outliers and a simple approach of Dixon’s Q test for detecting a single outlier. When there are multiple outliers, we can detect the outliers using the standard deviation (for data that is normal distributed) or using percentiles (for the skewed data). A box plot may be useful to visually check the data for potential outliers.

In regression setting, there are several approaches in detecting the outliers. One of the approaches is to utilize the ‘standardized residual’ or ‘studentized resitual’. In linear regression, an outlier is an observation with large residual. In other words, it is an observation whose dependent-variable value is unusual given its values on the predictor variables.

The studentized residual is the quotient resulting from division of a residual by an estimate of its standard deviation. Just like the standard deviation, the studentized residual is very useful in detecting the outliers. For values outside the 3, 4, or 5 times standard deviation, we may have reasonable doubt that the values are outliers. In regression setting, observed values outside 3, 4, or 5 times the studentized residual are the targets for outliers.

In SAS, two regression procedures can be easily utilized to compute the studendized residual for detecting outliers. PROC REG and PROC GLM. The studentized residual is labelled as RSTUDENT in Output statement. Other regression procedure (such as PROC MIXED) also compute studentized residual as part of Influence test.

output out=newdata rstudent=xxx;
Further readings:

Regression with SAS - Regression Diagnostics
SAS version 9.3 PROC REG
SAS version 9.3 PROC GLM

Outliers in clinical trial, Dixon's Q test for a single outlier

2011-11-12T09:07:00.000-05:00

In clinical trials, we deal with the outlier issue differently from other fields. During the clinical trial, for the suspected ‘outliers’, every effort should be taken to query the investigator sites, to repeat measures, or to re-test the samples in order to get the correct information. Typically those suspected ‘outliers’ can be clarified during the data cleaning process. It is just not very common to throw away the data (even it is suspected to be ‘outlier’) in clinical trials. In one of pharmacokinetics studies, I did have to deal with the suspected outliers (we used the term ‘exceptional value’ instead of ‘outliers’). After the sample re-test, we still had one value very high. Instead of throwing away this exceptional value, we had to perform the analysis with and without this exceptional value.

In one of the presentations by a FDA officer, the term ‘outliers’ vs anomalous are used.

Outlier subjects may be “real” results and are therefore very valuable in making a correct BE conclusion
Anomalous results are data that are not correct due to some flaw in study conduct or analysis

In many situations, it is very difficult to know for sure whether or not an exceptional value is a outlier or an anomalous result.

In ICH E9 "Statistical Principles for Clinical Trials", the handling of outliers was discussed in the section of "missing values and outliers".

5.3 Missing Values and Outliers

Missing values represent a potential source of bias in a clinical trial. Hence, every effort should be undertaken to fulfil all the requirements of the protocol concerning the collection and management of data. In reality, however, there will almost always be some missing data. A trial may be regarded as valid, nonetheless, provided the methods of dealing with missing values are sensible, and particularly if those methods are pre-defined in the protocol. Definition of methods may be refined by updating this aspect in the statistical analysis plan during the blind review. Unfortunately, no universally applicable methods of handling missing values can be recommended. An investigation should be made concerning the sensitivity of the results of analysis to the method of handling missing values, especially if the number of missing values is substantial.

A similar approach should be adopted to exploring the influence of outliers, the statistical definition of which is, to some extent, arbitrary. Clear identification of a particular value as an outlier is most convincing when justified medically as well as statistically, and the medical context will then often define the appropriate action. Any outlier procedure set out in the protocol or the statistical analysis plan should be such as not to favour any treatment group a priori. Once again, this aspect of the analysis can be usefully updated during blind review. If no procedure for dealing with outliers was foreseen in the trial protocol, one analysis with the actual values and at least one other analysis eliminating or reducing the outlier effect should be performed and differences between their results discussed.

I was recently asked for help to test an outlier for the data from a lab experiment (not a clinical trial).

The titer for the same sample was measured for 20 times. The titer is 25 for 7 times, 125 for 12 times. However, for one time, the title is 625. Is there any way to test (statistically) whether the titer of 625 is an outlier?

Titer	25	125	625
N	7	12	1

There is a simple test for outlier called Dixon's Q-test. Dixon’s Q-test calculates the Q value that is the ratio of the Gap (the difference between the extreme value and the immediately adjacent value) and the Range (the difference between the extreme value and the maximal or minimal value)

In the case above, the titer value needs to be log-transferred first, therefore, with Log10 data transfer, data will be listed as the following (in order):

1.39794 1.39794 1.39794 1.39794 1.39794 1.39794 1.39794 2.09691 2.09691 2.09691
2.09691 2.09691 2.09691 2.09691 2.09691 2.09691 2.09691 2.09691 2.09691 2.79588

The gap = 2.79588 - 2.09691 = 0.69897
The range = 2.79588 - 1.39794 = 1.39794
The Q value = 0.69897 / 1.39794 = 0.5

The Q value will then be compared with the critical value. The critical value can be found at difference web sources or from the original paper. The critical value for N=(7+12+1) = 20 is 0.342.
Since Q value is larger than 0.342, we can reject 2.79588 and conclude that the original value 625 (log-transferred value of 2.79588) is a outlier.

If we use a Log5 data transfer, the calculation will be easier and conclusion is the same.

This approach can only be used for detecting a single outlier. If there are more than one values in 625 titer group, Dixon's Q test will not be an appropriate approach.

Typically, identifying of the outliers is against a continuous variable (ie, the data is continuous). The data above contains many ties (due to the design). Therefore, the results from the Dixon’s Q-test needs to be interpreted in caution. The determination of the outliers should always be based on the understanding of the experimental data.

For further reading about the outlier issues:

Story of Xigris (Protein C) for Sepsis

2011-10-29T22:34:00.000-04:00

Xigris, also called Drotrecogin Alfa (Activated) or Protein C, was the only approved drug for severe sepsis indication and it was withdrawn from the market last week (Oct 25, 2011). In a recently completed clinical trial (PROWESS-SHOCK trial), Xigris failed to show a survival benefit. Due to the early controversies over Xigris’s approval and the continuous debate on Xigris’s risk benefit, PROWESS-SHOCK trial has been under watch since its start. The study design, statistical analysis plan, and unblinding plan have all been published way before the completion of the trial.

A decade ago, prior to the approval of Xigris for sepsis indication, the risk-benefit had been debated quite a bit. Xigris was know to be linked to the increased risk of serious bleeding in patients. there was "controversy surrounds both the drug study itself and the FDA approval," wrote NEJM editor-at-large Richard P. Wenzel, MD in 2002. FDA held the anti-infective advisory committee meeting for Xigris in treating sepsis. The FDA approved the drug despite the advisory committee's split vote (10 to 10) due to concerns about the validity of the claimed efficacy and safety findings on the basis of a single trial. At that time, Xigris was approved based on a single pivotal trial (PROWESS trial) that was also stopped early for efficacy. At that time, the FDA reviewers certainly believed that Xigris was beneficial and could save a lot of lives.

PROWESS trial has been the model for other clinical trials in Sepsis even though the PROWESS trial itself has been criticized for changes in the protocol during the trial. According to the NEJM article by H. Shaw Warren, MD, from Massachusetts General Hospital in Boston, and fellow consultants to the FDA, the study protocol changed during the PROWESS trial, shifting the study population composition toward patients with less severe underlying disease and more acute infectious illnesses. Other changes included use of a different placebo and elimination of protein C deficiency status as a primary variable. Around the same time, Lilly began producing the drug using a new master cell bank. Cumulative mortality curves suggest an improvement in protective efficacy of Xigris after these changes were made.

Subsequent trials have now shown that Xigris has no benefit and has unfavorable risk-benefit profiles. The ADDRESS trial (published in 2005) showed the absence of a beneficial treatment effect, coupled with an increased incidence of serious bleeding complications. The result indicates that Xigris should not be used in patients with severe sepsis who are at low risk for death, such as those with single-organ failure or an APACHE II score less than 25. Now the PROWESS-SHOCK trial further confirmed that the risk of bleeding outweigh the benefit in reducing the mortality – unfortunately it is a decade after Xigris has been on the market.

The market practice for Xigris has also been criticized. Several years ago, there were a lot of talks about Lilly’s influence on a committee in defining the sepsis treatment guidelines which was in favor of using Xigris.

Retrospectively, we can have something to learn from the Xigris story: 1) a single pivotal trial may be insufficient in confirming the treatment benefit; 2) change the protocol during the trial could have bias to the trial results; 3) stop a trial for efficacy may be risky.

New drugs for life-threatening disease such as sepsis are desperately needed, however, to demonstrate the benefit of any drug in the complicated sepsis treatment is a challenging task. The diversities in sepsis treatment in various institutes make the clinical trials in sepsis very difficult and the sample size for sepsis trials need to be sufficiently large to show the benefit.

A medical joke to share

2011-10-27T20:22:00.000-04:00

Not sure where the origin is. It is circulated quite a bit.

Best friends graduated from medical school at the same time and decided that, in spite of two different specialties, they would open a practice together to share office space and personnel.

Dr. Smith was the psychiatrist and Dr. Jones was the proctologist; they put up a sign reading: "Dr. Smith and Dr. Jones: Hysterias and Posteriors". The town council was livid and insisted they change it.

So, the docs changed it to read: "Schizoids and Hemorrhoids". This was also not acceptable, so they again changed the sign. "Catatonics and High Colonics" - No go.

Next, they tried "Manic Depressives and Anal Retentives" - thumbs down again. Then came "Minds and Behinds" - still no good. Another attempt resulted in "Lost Souls and Butt Holes" - unacceptable again! So they tried "Analysis and Anal Cysts" - not a chance. "Nuts and Butts" - no way. "Freaks and Cheeks" - still no good. "Loons and Moons" - forget it.

Almost at their wit's end, the docs finally came up with: "Dr. Smith and Dr. Jones - Specializing in Odds and Ends". Everyone loved it.

Will Electronic Data Capture be always better than Paper-CRF?

2011-10-15T07:22:00.000-04:00

The traditional way to do the clinical trial data management is to use the paper based case report forms (CRFs). The blank paper CRFs are distributed to the investigator sites. The investigator or study coordinator fills out the CRFs. CRFs will then be monitored and collected from the investigator sites. CRFs will subsequently be handled by a centralized group - data management group where the activities include the clinical database building, data entry, data cleaning, data clarification,...

The industry trend has been gradually moving away from the paper-based CRFs and moving toward to the electronic data capture (EDC). In EDC world, the database was built prior to the study start (significant longer leading time prior to the study study is needed) . The data will be directly entered into the database by the investigator site (investigator or study coordinator). EDC has been touted by many vendors as the preferred way for conducting clinical trials: getting the data fast, saving timeline, saving cost, minimizing data transcription errors... While this is generally true, it is not universal.

In some situations, the trial using the traditional paper-based CRFs is a better way than EDC. For example, in a clinical trial for a rare disease, there are many investigator sites and each site may only enroll very few subjects or not enroll any subject. The EDC will not be an efficient way in data collection. Many site staff will be trained on EDC and never have chance to enroll any patient into the study and never have a chance to use EDC. When a site finally has a chance to enroll a subject, the initial training on using EDC may be a distant memory.

The EDC trial is not always cheap. With EDC trial, significant cost could be spent on the EDC system hosting and EDC system help desk support. Imagining a slow enrollment trial running for 7-8 years, the cost for hosting EDC system and providing the help desk support will be too much comparing to a paper-based study.

While EDC is a trend, the adoption of EDC is not universal. In some situations, the traditional paper CRFs may be better.

Confidence Interval for Difference in Two Proportions

2011-09-13T17:46:00.000-04:00

In many clinical trials, the outcome is binomial and a 2 x 2 table can be constructed. The analysis can be based on the difference in two proportions (treatment group vs. control group). SAS Proc Freq can be used to obtain the difference between the proportions and the asymptotic confidence interval can be calculated for the difference between two proportions. The formula is (p1-p2) +/- Z(alpha/2)*sqrt((p1*q1/n1)+p2*q2/n2)).
However, the asymptotic confidence interval produced by PROC FREQ requires a somewhat large sample size (say cell counts of at least 12) - this is the case at least for SAS version up to 9.2. For moderately small sample size, it is better to use the formula provided in Fleiss (1981, page 29) Stokes (2000, page 29-30) where the confidence interval is adjusted by 0.5*(1/n1 + 1/n2) - therefore a little wider. The confidence interval directly from SAS Proc FREQ is a little narrower than those using the formula. In practice, the statistician needs to make the choice which one to use in calculating the confidence interval for difference in proportions depending on the sample size situation.

Fleiss, JL (1981) Statistical Methods for Rates and Proportions. New York: John Wiley & Sons, Inc.
Stokes, Davis, and Kock (2000) Categorical Data Analysis using the SAS System, 2nd edition

The example from Stocks book can be implemented in SAS using the following SAS codes:

data respire2;
input treat $ outcome $ count @@;
datalines;
test    f 40
test    u 20
placebo f 16
placebo u 48
;

*** the confidence interval directly from SAS PROC FREQ;
proc freq order=data;
weight count;
tables treat*outcome / riskdiff;
run;

*** the confidence interval calculated from the formula (See section 2.4 Difference in Proportions
     in Stokes et al 'Categorical Data Analysis Using the SAS System' 2nd edition;
proc freq data=respire2 order=data;
    weight count;
    tables treat/noprint out=tots (drop=percent rename=(count=bign));
run;

proc freq data=respire2;
    weight count;
    tables treat*outcome/noprint out=outcome (drop=percent);
    run;

proc sort data=tots;
by treat;
run;

proc sort data=outcome;
    by treat;
run;

data prop;
    merge outcome tots;
    by treat;
    if treat='test' then p1=count/bign;
    if treat='placebo' then p2=count/bign;
run;

data prop1(rename=(count=count1 bign=bign1)) prop2(rename=(count=count2 bign=bign2));
     set prop;
     if treat='test' then output prop1 ;
     if treat='placebo' then output prop2;
run;

data proportion;
merge prop1(drop= p2 treat) prop2(drop = p1 treat);
run;

***Calculate the difference in proportions, SE, and 95% confidence interval using formula by Fleiss;
data cal;
set proportion;
    variance=(p1*(1-p1)/(bign1)) + (p2*(1-p2)/(bign2));
    diff=(p1-p2);
    lower=(diff - ((1.96*(sqrt(variance)) + .5*(1/bign1 + 1/bign2))));
    upper=(diff + ((1.96*(sqrt(variance)) + .5*(1/bign1 + 1/bign2))));
    se=(sqrt(variance));

run;

proc print;
format p1 p2 variance diff lower upper se 5.3;
run;

Is it time to change the clinical monitoring practice in clinical trials?

2011-09-09T10:52:00.000-04:00

In industry, the current monitoring practice relies on ‘on-site monitoring’ and 100% source data verification (on all data fields). This process is very costly and is one of the main reasons that the clinical trials now become so expensive. This process is really the most conservative interpretation of ICH E-6 guidance on
Guideline for Good Clinical Practice and the 1988 FDA’s “Guidance for the Monitoring of Clinical Investigations”. These guidance only require “the sponsor should ensure that the trials are adequately monitored” and leave the door open in terms of the frequency of the monitoring and the approaches of the clinical monitoring. In industry, the conduct of the clinical trials are highly regulated. Sponsors are usually take the most conservative approaches no matter how costly these approaches are.

Will ‘on-site monitoring’ be really effective? Will 100% source data verification really be needed? Should we identify the new ways to conduct the cost-effective clinical monitoring?

Last month, FDA withdrew its 1988 guidance on “Guidance for the Monitoring of Clinical Investigations” and issued its draft guidance “Oversight of Clinical Investigations - A Risk-based Approach to Monitoring”. The newly issued guidance suggested it is acceptable to use alternative approaches (such as remote monitoring, centralized monitoring, risk-based monitoring). The guidance also suggested that the source data verification should be focused on critical fields (key efficacy and safety variables) and less than 100% source data verification on less important fields may be acceptable. The guidance gives a clear signal that the Sponsors are encouraged to explore the cost-effective ways to conduct the clinical monitoring instead of solely relying on the on-site monitoring.

If this guidance gets implemented, we may expect the increasing role of statisticians in clinical monitoring, especially the centralized monitoring. Currently, statisticians will identify the issues in the late stage of the clinical trials when statisticians or statistical programmers start to perform the data analyses. The new guidance says:

“…notably, the advancement in EDC systems enabling centralized access to both trial and source data and the growing appreciation of the ability of statistical assessments to identify clinical sites that require additional training and/or monitoring.”

“Centralized monitoring is a remote evaluation carried out by sponsor personnel or representatives (e.g., data management personnel, statisticians, or clinical monitors) at a location other than the site(s) at which the clinical investigation is being conducted.”

Odds ratio and risk ratio in clinical trials #2

2011-08-21T23:35:00.000-04:00

In my previous article, I discussed the odds ratio and risk ratio (or relative risk ratio). In clinical trials with binary outcome, both odds ratio and relative risk ratio are used. Since the clinical trials are similar to the cohort studies in epidemiology field, it seems to be more reasonable to use relative risk ratio in clinical trials. However, the odds ratio may be more commonly used in practice. This may be due to the fact that the odds ratio can be easily modeled using logistic regression. This could also be due to the fact that the odds ratio is typically larger than relative risk ratio that may be desired by the researcher.

For a non-inferiority or equivalence trials with binary outcome, one may desire to have a smaller standard error, therefore a narrower confidence interval – in this case, the relative risk ratio may be better than odds ratio.

Wikipedia gives an excellent comparison between relative risk ratio and odds ratio.

In an article “How can I estimate relative risk in SAS using proc genmod for common outcomes in cohort studies?”, the calculation of odds ratio, relative risk ratio, and their confidence intervals are illustrated.

Using SAS Proc Genmod, both odds ratio, relative risk ratio, and their confidence intervals can be easily calculated:

For odds ratio:

Proc genmod data = xxx descending;
    class treatment;
model outcomevariable = treatment
                                                  / dist = binomial link = logit;
estimate 'Beta' treatment 1 -1/ exp;
run;

Here, the “link=logit” can be omitted since the logit link function is default when distribution is binomial.

For relative risk ratio,
proc genmod data = xxx descending;
    class treatment;
    model outcome = treatment
                              / dist = binomial link = log;
    estimate 'Beta' treatment 1 -1/ exp;
run;

Here, the “link=log” can NOT be omitted since the log link function is NOT default when distribution is binomial.

Relative risk ratio can also be estimated using poisson regression especially when the event ratio is small.

Proc genmod data = eyestudy;
    class id;
    model outcome = treatment
                          / dist = poisson link = log;
    repeated subject = id/ type = unstr;
    estimate 'Beta' treatment 1 -1/ exp;
run;

Here, the “link=log” can be omitted since the log link function is default when distribution is poisson.

There are several advantages of using Proc Genmod to calculate the odds ratio and risk ratio. Adjusted odds ratio and adjusted relative risk ratio can be easily calculated when there are continuous or categorical covariates. The model can be easily modified to fit the longitudinal data.

Proc Logistic can be used for calculating the odds ratio (and the confidence interval) and can adjust for continuous or categorical covariates. However, Proc Logistic can not be used for calculating the relative risk ratio.

proc logistic;
     model outcome = treatment;
run;

Proc FREQ can be used for calculating the odds ratio and relative risk ratio (and asymptotic confidence interval) using /cmh option. For adjusted odds ratio or risk ratio, only the categorical covariate can be used.

proc freq order=data;
    tables covariate*treatment*response / CMH;
run;

In the output, the odds ratio will be explicitly indicated while relative risk ratio will be labeled as “col1 risk” or “col2 risk”.

There is no regulatory guidance forcing the use of odds ratio or relative risk ratio. However, in clinical trials, if we compare the ratio of two proportions (eg the proportion of success in treated group vs. the proportion of success in control group), relative risk ratio seems to be better. Relative risk ratio resemble the hazard ratio in may aspects.

In FDA’s guidance “Diabetes Mellitus — Evaluating Cardiovascular Risk in New Antidiabetic Therapies to
Treat Type 2 Diabetes” The calculation of risk ratio is suggested. The guidance indicated
"Sponsors should compare the incidence of important cardiovascular events occurring with the investigational agent to the incidence of the same types of events occurring with the control group to show that the upper bound of the two-sided 95 percent confidence interval for the estimated risk ratio is less than 1.8. This can be accomplished in several ways. The integrated analysis (meta-analysis) of the phase 2 and phase 3 clinical trials described above can be used. Or, if the data from all the studies that are part of the meta-analysis will not by itself be able to show that the upper bound of the two-sided 95 percent confidence interval for the estimated risk ratio is less than 1.8, then an additional single, large safety trial should be conducted that alone, or added to other trials, would be able to satisfy this upper bound before NDA/BLA submission. Regardless of the method used, sponsors should consider the entire range of possible increased risk consistent with the confidence interval and the point estimate of the risk increase. For example, it would not be reassuring to find a point estimate of 1.5 (a nominally significant increase) even if the 95 percent upper bound was less than 1.8.”

In a presentation by Dr Bob O’Neill “Non -Inferiority Clinical Trials Some key statistical issues and concepts” he suggested that Log (Hazard ratio) or Log(relative risk) is preferred when determining the non-inferiority margin

In statistical review for Maxipime (cefepime hydrochloride) NDA, the risk ratio and 95% confidence interval are used. In an article “Relative risks of reported serious injury and death associated with hemostasis devices by gender”, the risk ratio were reported.

However, there are also many cases of using odds ratio instead of risk ratio in clinical trials. Some examples are:

In summary, while both odds ratio and risk ratio can be used in clinical trials, risk ratio should be given the adequate emphasis in comparing the ratio of two proportions (between two treatment groups). In non-inferiority clinical trials, the risk ratio and its confidence interval are preferred.

FDA's new guidance on device approval process and device clinical trial

2011-08-18T21:03:00.001-04:00

For a long time, device makers have been complaining about the FDA's device approval process. I personally heard a lot of talks that the device approval process is easier than the approval of the drug and the biological product. The requirements for clinical trials in device approval has lower standard comparing to clinical trials in drugs and biological products. FDA device division (CDRH) has the loose criteria for the product approval. In response to the critics, FDA now releases two new draft guidance on August 15, 2011 and is seeking to educate industry on device approval.

The first guidance “Factors to Consider when Making Benefit-Risk Determinations in Medical Device Premarket Review” explains the agency's approval process for diagnostic and therapeutic devices, specifically:

How the agency weighs the benefits and risks of a device
How the agency assess the seriousness of a disease or condition
How many people would use the device if approved
The availability of other devices approved to treat the same condition

It is interesting enough that in examples used in this guidance, several clinical trials were mentioned as flawed or with unreliable data, however, FDA would approve the device anyway. This is just another reflection that CDRH indeed has lower data quality standard comparing to CDER for drugs and CBER for biological products.

In the second draft document “Design Considerations for Pivotal Clinical Investigations for Medical Devices”, the FDA laid out its expectations for clinical trials for medical devices. The agency stated that it looks for a study to provide reasonable assurance that the device is safe and effective.

Back in July 2011, FDA issued a guidance on “in Vitro Companion Diagnostic Devices” where the In Vitro Companion diagnostic device (IVD companion diagnostic device) is defined as an in vitro diagnostic device that provides information that is essential for the safe and effective use of a corresponding therapeutic product. The guidance intended to accomplish the following:

Define in vitro companion diagnostic device
Explain the need for FDA oversight of IVD companion diagnostic devices
Clarify that, in most circumstances, if use of an IVD companion diagnostic device is essential for the safe and effective use of a therapeutic product, the IVD companion diagnostic device and therapeutic product should be approved or cleared contemporaneously by FDA for the use indicated in the therapeutic product labeling
Provide guidance for industry and FDA staff on possible premarket regulatory pathways and FDA’s regulatory enforcement policy
Describe certain statutory and regulatory approval requirements relevant to therapeutic product labeling that stipulates concomitant use of an IVD companion diagnostic device to ensure safety and effectiveness of the therapeutic product

No matter what the device is, if clinical trials are required, the statistical analyses can be based on the FDA guidance “Statistical Guidance on Reporting Results from Studies Evaluating Diagnostic Tests” that was issued in 2007.

In Device field, not all approval requires clincial trials. If clinical trials are required, they do not have to be interventional. If clinical trials are interventional, they do not have to be randomized, controlled. If clinical trials are randomized, controled, they do not have to be double-blinded. Some clinical trials in device field may be using the samples (for example blood samples) from patients with no intervention performed. The blood samples could be historical retains or prospectively collected. The use of the blood samples for device clinical trial still needs the informed consent from the patient.

Equipoise and Lack of Equipoise in Randomized Clinical Trials

2011-08-13T13:20:00.001-04:00

According to Merriam-Webster dictionary, the word "Equipoise" means a state of equilibrium. In clinical trial, the concept of 'clinical equipoise' means that there is genuine uncertainty over whether a treatment will be beneficial. In other words, in randomized controlled clinical trials, there should be substantial uncertainty or there is no clear evidence that one treatment arm is particularlly better or worse. Clinical equipoise provides the ethical basis for medical research involving patients assigned to different treatment arms of a clinical trial - it is unethical to assign a subject to an inferior arm if the lack of equipoise exists and if there is substantial evidence that one treatment is better or worse than another treatment.

In the real word, when we plan a randomized, controlled clinical trial, 'lack of equipoise' may often exist. This is especially true in late stage clinical trials. In late stage clinical trials, there typically be some evidences about the benefit and treatment effect of the experimental drug from early phase I or phase II clinical trials. Our sample size calculation is based on the assumed treatment effect of the experimental drug.

We recently published a paper in the Journal of Neurology "Challenges of clinical trial design when there is lack of clinical equipoise: use of a response-conditional crossover design" where we discussed a situation of 'lack of equipoise' and the use of a response-conditional crossover design to ease the concern about the lack of equipoise in clinical trial design. Several small trials had suggested that IVIg is beneficial in treating the disease CIDP - lack of equipoise. However, in the absence of an approved treatment for this indication, gaining regulatory approval for the use of IVIg in this indication required the conduct of large-scale, placebo-controlled confirmatory trials. Using the response-conditional crossover design, we eased the concern about subjects exposed to the 'perceived' inferior arm (Placebo in this case). The results indicated that we minimized subject's exposure to the inferior treatment arm.

Interpretation of 'clinical equipoise' may be different among clinicians, investigators, patients, clinical trial sponsors, and regulatory agencies. Evidences of treatment effects from small-scale clinical trials may be thought as real evidence for clinician and patients, but not for regulatory agencies.

When we bring the overall benefit/risk into the picture for assessing the 'clinical equipoise', it may be difficult to determine whether or not a clinical equipoise exist or not. A new treatment may have been demonstrated beneficial in efficacy, but with great uncertainty in safety.

Data Query on Diary and Patient Reported Outcome?

2011-07-14T23:13:00.001-04:00

We keep hearing that for patient report outcome (PRO) including patient diary data, there is no query or data clarification required and the diary data is source data and can not be queried. For example, in a paper titled “How to clean up dirty data in Patient reported outcomes”, it said “The investigator is not allowed to query any of the patient's answers which leads in general to a lot of dirty data.”. Is it true that the diary or patient reported outcome can not be queried at any circumstance no matter how horrible the data quality is? It is not true and this popular perception is just not true.

Data clarification or data query is an essential process in clinical data management to ensure that the questionable data are corrected. According to EMA reflection paper ON EXPECTATIONS FOR ELECTRONIC SOURCE DOCUMENTS USED IN CLINICAL TRIALS, “Data clarification is part of the process to ensure complete data and the clarification process is one of the processes underlining the need for the maintenance of the audit trail.”

In a clinical study with paper-based case report form, the data clarification is typically issued by the data managers to the investigation sites. The investigator then reviews the issues to provide the responses to the data query. Data managers or clinical monitors can not directly make the changes to the data without issuing query and getting approval from the investigation site. In a clinical study with electronic data capture (EDC), the data clarification/query process is built into the EDC system. The data is entered at the investigation site. The query is issued by data managers or study monitors within EDC system. The investigator will then provide the responses to the query also within EDC system and make the data corrections.

The process becomes vague for patient reported outcome (PRO) or patient diary (no matter it is on the paper or electronic). The key difference for PRO data is that the data is directly recorded or entered by the subject or patient. There is a perception that no matter how poorly the data is, there is no data clarification or query process for PRO data or diary data.

In many clinical trials, the study endpoints rely on the collection of the information provided by the patient (daily symptoms, daily activities, quality of life,…). For example, for clinical studies on urinary incontinence, the primary efficacy endpoint may be the frequency of urinary incontinence episodes (UIE) per week as determined from patient daily diary. For clinical studies on female sexual dysfunction, the clinical endpoint may be the sexual events or encounters recorded daily by the study subjects using diaries. .

I used to work on a clinical trial with irritable bowel syndrome indication where the study endpoints were collected by a touch-tone telephone system (IVR – interactive voice response system). Efficacy parameters were symptom relief, abdominal discomfort or pain, bloating, stool frequency, stool consistency, straining and urgency recorded through IVR on daily basis by the study subjects. For stool frequency, the subject were asked “how many bowel movements did you have today?” As a statistician, I had to check the outliers before analyzing the data. I found some entries with the number of bowel movements being extremely high (55, 66). When I discussed these impossible numbers with the study manager, I was told that patient diary could not be queried. I pointed out that if these obvious data errors could not be corrected, the study data would be severely compromised. Later, we identified many more entries with duplicate numbers (such as 11, 22, 33, 44, 55, 66). After inquiring to the study subjects, it was found that the subject pressed the telephone number key twice for 1, 2, 3, 4, 5, and 6. This is just an example showing that the diary data could be easily recorded wrongly in the database and the query for accuracy is necessary.

FDA’s guidance on “Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims” touched on ensuring the quality of PRO data. In discussion about the ePRO, FDA guidance indicated the concern about clinical investigator inability to maintain and confirm electronic PRO data accuracy. It said “the data maintained by the clinical investigator should include an audit trail to capture any changes made to the electronic PRO data at any point in time after it leaves the patient’s electronic device.” Furthermore, the guidance indicated the concern about ability of any entity other than the investigator (and/or site staff designated by the investigator) to modify the source data”. These statements inexplicitly imply that the diary data can be queried and modified by the clinical investigator (and/or site staff designated by the investigator) and the clinical investigator has responsibility to ensure the completeness and accuracy of the diary data.

Adaptive Licensing for Drug Approval

2011-07-07T22:55:00.000-04:00

From this year’s Drug Information Association (DIA) annual conference in Chicago, I learned a new concept of “Adaptive Licensing”. According to http://www.bnid.org/node/6921, “in the past five years, a wave of proposals for reform of drug licensing has emerged in the EU, US, and Canada under the labels of staggered approval, adaptive licensing, managed entry and progressive authorization. Through iterative phases of information gathering followed by regulatory evaluation and correction, these approaches seek to align licensing decisions on market access of drugs with emerging information on benefits and harms of drugs as actual used. It is hoped that this approach will provide patients with earlier access to innovative drugs to address unmet medical needs, better management of known risks, and improved detection of unanticipated adverse effects that emerge in use. “

The current drug licensing process is called ‘phased approach’ which requires the sponsors to conduct a series of clinical trials from phase I to phase III to establish the safety/tolerability and to confirm the efficacy before the regulatory agencies can consider the approval for marketing authorization of a product. This phased approach and the purpose of each phase of the clinical trial are discussed in this free web article. Recently, with the adaptive design concept, we are trying to break the traditional phased approach. The seamless phase II/III studies or seamless phase I/II studies have been much discussed and debated. Even with adaptive design, before a drug can be approved, a series of clinical trials are still required. Let’s now call “learning and confirming”: from early trial for learning to late stage trial for confirmation of the safety and efficacy with Adequate and well-controlled clinical study(ies) (A&WC).

With the phased drug approval approach, there will be a magic moment during the drug approval process. This magic moment is the regulatory reviewer’s action date (or decision date) - The date tells when a regulatory action, such as an original or supplemental approval, takes place. The regulatory agencies will be based on the review of data from pre-marketing clinical studies to make a decision of approving or not approving a production for market authorization. If the efficacy and safety have been demonstrated, the product is approved; if the efficacy and safety have not been sufficiently demonstrated, additional clinical studies may be requested; if the efficacy and safety are not demonstrated, the application of market authorization will be denied.

The adaptive licensing is trying to remove this magic moment from the drug approval process and instead considers the drug licensing as a continuous process. Whether or not a product should be authorized for marketing depends on the risk-benefit ratio. When the benefit outweighs the risk, the product should be approved; when the risk outweighs the benefit, the product should not be approved. For an approved product, if the benefit/risk ratio becomes unfavorable, the product should be withdrawn from the market. I liken this continuous adaptive drug licensing process to the p-value assessment in statistics. The magic moment is like the magic number of p=0.05 (p-value should be a continuous number and p=0.051 (not significant) may not be different from p=0.049 (significant)).

It looks like EU is pioneering in implementing the adaptive licensing. EMA has started to work on Adaptive Licensing to Reach Roadmap Goals (Roadmap to 2015) . However, in US, there is a similar program called “Accelerated Approval of New Drugs for Serious or Life-Threatening Illnesses”. With this program, “FDA may grant marketing approval for a new drug product on the basis of adequate and well-controlled clinical trials establishing that the drug product has an effect on a surrogate endpoint that is reasonably likely, based on epidemiologic, therapeutic, pathophysiologic, or other evidence, to predict clinical benefit or on the basis of an effect on a clinical endpoint other than survival or irreversible morbidity. Approval under this section will be subject to the requirement that the applicant study the drug further, to verify and describe its clinical benefit, where there is uncertainty as to the relation of the surrogate endpoint to clinical benefit, or of the observed clinical benefit to ultimate outcome. Postmarketing studies would usually be studies already underway. When required to be conducted, such studies must also be adequate and well-controlled. The applicant shall carry out any such studies with due diligence.”

However, the recent debates on whether or not Avastin (Bevicuzmab) should be withdrawn from the market for breast cancer demonstrated how difficult to implement this program. Based on the established rule, if a drug is approved through ‘accelerated approval’, the approval is conditional and can be withdrawn from the market if the drug is later showed to be ineffective or with unfavorable benefit/risk ratio. Avastin in breast cancer case just showed how difficult to withdraw a product due to ineffectiveness. In even worse situation, the drug companies may not fulfill the commitment to finish the follow-up studies. There have been several cases of marketing withdrawal due to safety concerns (for example, Vioxx), but it seems to be more difficult to withdraw a product from the market due to the efficacy concern.

Adaptive licensing may be the future direction for drug approval process; however, many issues need to be considered and resolved before this new process can be implemented.

What is the impact of adaptive licensing on patent expiration?
What is the tipping point the conditional approval can be granted?
How to assess the benefit/risk ratio with sound scientific / statistical approaches (ie avoiding subjective assessment in benefit/risk ratio)?
What if the committed follow-up studies never completed?
How to deal with the difficulties to withdrew a product when other factors (for example, emotional effect in Avastin case) are kicked in?

Is blinded study really blinded? - assessment of blinding / unblinding in clinical trials

2011-06-18T18:42:00.000-04:00

Randomization and blinding are critical components of the clinical trial from the start (design) to the end. Randomized, controlled, and double blinded trial (RCT) has been the ideal clinical trial design. Inappropriate randomization and blinding (or potential unblinding) affect the integrity for the clinical trial. If the patient or investigator is aware of the treatment assignment, there will be conscious or unconscious biases in assessing efficacy, safety, or patient-reported outcome. With available software and computer programs, generating randomization schedule is relatively easy. Ten years ago, I wrote a paper on “Generating randomization schedule using SAS programming” to show how easily the randomization can be generated. With the interactive response technologies (IRT) including interactive voice response system (IVRS) and interactive web response system (IWRS), implementation of the randomization can also be easily managed. However, maintaining the blinding during the study may not be as easy as we thought.

I still remember the time when I was one of the randomization team members in PPD. After we generated the randomization schedule, we had to put the randomization schedule into an envelope and sealed with signatures. Then we had to put the envelope into a locked security box in a secured randomization room. In order to get the randomization schedule, at least two statisticians had to be present in order to open the security box.

While the actual randomization schedule is locked and secured, the randomization information or treatment assignment concealment can still be compromised by what happened at the site, how the patient and investigator guess the treatment assignment, and how the unblinded personnel communicate with the blinded team members.

There are many factors that can cause the potential unblinding. Here are some examples:

Guess treatment assignment by the experience of adverse events and side effects. Suppose an intravenously administered drug can cause more headaches than Placebo, a patient with headache may guess he/she is on treatment group and not on Placebo. While this guess may not be 100% accurate, majority of patients may guess their treatment assignment correctly. In a book by Chow et al, ‘Design and analysis of clinical trials: concepts and methodologies’, an example about challenge in maintaining blinding was described “beta-blocker (e.g., pro-pranolol) have specific pharmacologic effects such as lowering blood pressure and the heart rate and distinct adverse effects such as fatigue, nightmares, and depression. Since blood pressure and heart rate are vital signs routinely evaluated at every visit in clinical trials, if a drug such as propranolol is known to lower blood pressure and the heart rate, then preservation of blindness is a huge challenge and seems almost impossible” In a large scale study (BHAT study), at the conclusion of the trial, patients, investigators, and clinical coordinators were asked to guess the patient’s treatment assignment, 79.9%, 69.6%, and 67% of patients, investigators, and clinical coordinators respectively guessed correctly the patient was on Propranolol and 42.8%, 58.6, and 70.6% of patients, investigators, and clinic coordinators respectively guessed correctly that the patient was on Placebo.

Guess treatment assignment by improvement or no improvement in efficacy. If there is a prior knowledge that an treatment is effective (lack of equipoise), the investigator or patient can guess which treatment the patient is on based on the lack of effect.

Guess treatment assignment by knowing the blood concentration of the drug or analytes. If a treatment is for augmentation purpose, a patient could have his/her blood sample tested to know whether or not the concentration for augmented drug is increased or not, then guess which treatment group he/she is on.

In double-blinded studies, there are always some unblinded groups. These groups could include global drug safety for safety monitoring, laboratories that measure drug concentration or biomarkers, study drug supplies, site unblinded pharmacist… all of these groups could potentially reveal the treatment assignment to other study team unintentionally.

For a study with DMC that involved a third party to prepare the unblinded data for DMC, treatment concealment could potentially be compromised during the information exchange with the blinded study team. This is critical for studies with adaptive designs where the patient data needs to be constantly reviewed and analyzed. An interesting example was discussed by Janet Witts regarding an awkward situation in an adaptive design where the DMC knew the event rate by treatment assignment and the sponsor didn’t.

There is a dilemma when we develop the informed consent form. On the one hand, we are required to put into the informed consent form as much information as we can. On the other hand, the more information we put into the informed consent form, the more likely we enable the patients to guess their treatment assignment (based on their experience of side effects or perceived efficacy).

Ideally, in double-blind trial, it is a good practice to evaluate for both the subjects and investigators whether or not blinding / masking has been preserved. However, in real world, it is rare in double-blinded clinical trials to include formal assessment about how good the blinding has been preserved. If assessment of blinding becomes a routine, I think that many studies will show that subjects / investigators guessed correctly more frequently than they should have done by chance alone. Part of the reason this assessment has not been done often is perhaps the difficulty to explain the study results if the blinding is found to be compromised. It will be extremely difficult to assess the magnitude of the impact on the safety and efficacy evaluation if the blinding / treatment assignment concealment is compromised.

Further readings:

Assuring that double-blind is blind

Blinded trials taken to the test: an analysis of randomized clinical trials that report tests for the success of blinding

Blinding, unblinding, and the placebo effect: An analysis of patient’s guesses of treatment assignment in a double-blind clinical trial

Can keeping clinical trial participants blind to their study treatment adversely affect subsequent care?

Assessment of blinding in clinical trials

Concealing treatment allocation in randomised trials

Bland-Altman Plot for Assessing Agreement

2011-06-15T22:22:00.000-04:00

Bland-Altman plot is a scatter plot of variable means plotted on the horizontal axis and the differences plotted on the vertical axis which shows the amount of disagreement between the two measures (via the differences) and lets you see how this disagreement relates to the magnitude of the measurements.

When I was in graduate school, the statistical analysis of microarray data just started to be a hot topic. In collaboration with Dr Rick Song, we looked at the microarray data and wrote a manuscript titled “On Graphical Presentation and Quantitative Analysis of cDNA Microarray Data” and we presented in JSM. In this manuscript, we proposed to use Bland-Altman plot. In clinical trials, I have not got a chance to apply this approach, but I do often see articles using the Bland-Altman plot. For example, an article titled “Using the Bland–Altman method to measure agreement with repeated measures” from British Journal of Anaesthesia.

When data is appropriate, Bland-Altman plot can be a handy tool to use. It is worth relaying the paragraphs from our original paper on graphical presentation of micro-array data using Bland-Altman plot.

“Graphical presentation is usually the first step for data analysis of microarray data. In the case without duplication (this is typical in microarray experiment), scatter plots will be drawn and then a regression line drawn through the data. This helps the eye in gauging the degree of agreement between two measurements and also may help us to identify the "outliers" that represent the differentially expressed genes in microarray experiment.

In clinical medicine, to assess agreement between two methods of clinical measurement, Bland and Altman proposed to plot the difference between the methods (A-B) against the mean (A+B)/2[12,13,14,15]. This approach has been extensively used in medical research for assessing measurement error and comparing different measurements for the same quantity. Bland and Altman’s method can be also applied to the microarray data. We can plot (Rm-Gm) against (Rm+Gm)/2 (figure2 above).

Calculating or plotting a regression line is not our focus as we are not concerned with the estimated prediction of one color intensity by another but with the theoretical relationship of equality and deviations from it.

There are several advantages for presenting the microarray data using Brand and Altman’s approach:

The plot of difference against mean allows us to investigate any possible relationship between the discrepancies and the true value. The plot will also show clearly any extreme or outlying observations. If two different samples are used in the experiment, these extreme or outlying observations could indicate the differentially expressed genes. It is often helpful to use the same scale for both axes when plotting differences against mean values. This feature helps to show the discrepancies in relation to the size of the measurement.

Brand and Altman's method makes it easier for us to estimate the precision of the estimated limits of agreement between two color intensities. We want a measure of the agreement that is easy to estimate and to interpret for a measurement on the color intensity of an individual gene. An obvious starting point is the difference between measurements by the two channels on the same gene. There may be a consistent tendency for one channel to exceed the other. This is called calibration factor and can be estimated by the mean difference. There will also be variation about this mean, which we can estimate by the standard deviation of the differences. These estimates are meaningful only if we can assume that calibration factor and variability are uniform throughout all genes.”

More references on Bland-Altman Plot:

Restructure FDA's Drug Review Process?

2011-06-03T21:11:00.000-04:00

Last week, I had a chance to listen to a speech by Dr Scott Gottlieb. While he touched several topics in related to the health care reforms, I was specifically interested in his discussion on restructuring FDA’s drug approval process.

Dr Gottlieb gave a lot of insights within FDA and analyzed the root cause of the very long and inefficient FDA drug review process.

Following his speech, I located his paper “SHOULD FDA RESTRUCTURE ITS DRUG REVIEW PROCESS?” from FDLI’s website. A lot of his analyses are so true and to the point.

For example, he elaborated why FDA adopt a matrix management structure for its review program.

“Prior to FDA’s adoption of a matrix management structure for its drug review program, agency scientists were organized largely around the clinical areas in which they worked (oncology, cardio-renal, antiviral, etc). This therapeutically focused structure had some advantages, but also led to some of its own challenges. FDA’s adoption of a matrix structure was aimed at solving some of these problems.

For one thing, sponsors complained that the advice they received about disciplines like biostatistics or clinical pharmacology varied (sometimes significantly) across different therapeutic divisions. Statisticians in one clinical division would be interpreting certain principles of statistics or evaluating a particular protocol design in a manner different than statisticians inside another therapeutic division.

These discrepancies still occur. But it is believed that the matrix organizational structure cuts down on this sort of conflict.

Grouping all of the statisticians or pharmacologists inside the same office increases opportunities for comparable training and cross-calibration on key principles. CDER management pulled the first group of the review divisions—the chemists—in 1995. The impetus was differences in pharmaceutical quality requirements being maintained among the different clinical divisions. Ultimately, having the chemists organized as a single group fostered the development of consistent standards. It also enabled FDA to negotiate the standards established by the International Conference on Harmonization (ICH).

Another reason for establishing the matrix was to improve morale. FDA remains a very “physician centered” culture, but was much more so prior to adoption of the matrix. Staff who lacked medical degrees or who weren’t the clinical reviewers on an application sometimes complained that they felt marginalized in the review process. As one statistician told me, “we were treated like second-class citizens.” Specialists from non-clinical disciplines like statistics also complained that remaining immersed in a single therapeutic area didn’t give them the breadth of experience that they needed for their own professional development.

Similarly, the organization of scientific personnel by therapeutic area was also seen as an impediment to their continued training in their chosen disciplines. For example, statisticians came together for the equivalent of grand rounds or other kinds of shared learning experiences. But these kinds of cross-training opportunities were challenging because staff were ultimately accountable to their divisions. The shared training opportunities weren’t prioritized. Efforts were also made to rotate non-clinical experts across different therapeutic areas. But the challenges endured.”

He then went on discussing the issues with FDA’s weak matrix management structure.

“But in practical terms, the weak matrix means that FDA project managers have limited dominion over key aspects of the review. Key disciplines involved in the review aren’t accountable to the project manager, or the division director. It is a system where there are few management carrots and no sticks. This weak structure also makes it harder to organize collaborative projects or even team meetings. The project manager doesn’t have strong authority when it comes to managing the collaboration between the different scientists involved in a drug’s review.”

He also mentioned the quality of the FDA review scientist and issues with FDA’s policy to allow very flexible working hours and work-from-home schedule. The current FDA drug review team is loosely organized and inefficient in many aspects. However, it is not easy to make big changes to the current process.

Differences between SOPs and WPs

2011-05-30T16:42:00.000-04:00

Working in industry setting (versus academic), we all understand that it is critical for us to be familiar with the Standard Operating Procedures (SOPs) / Working Procedures (WPs) and follow these procedures. This is even more obvious in pharmaceutical and drug development industry. An employee could be fired not because of incompetence, but because of violating these procedures. A couple of days ago, I attended ChineseAssociation for Science and Technology Annual Convention, one session was to discuss about the requirements for starting up a new company. Several local entrepreneurs elaborated the intellectual property (IP), business plan, funding, a loyalty team, and so on. Nobody mentioned these procedures. But I think that it is equally important to have a set of necessary SOPs and WPs in order to start a new company, especially a service-type company in highly regulated drug development field such as contract research organization. Scientists working in the academic setting often find the difficulties when they try to land a position in industry mainly because they lack the ‘industry experience’ – one critical part is that they lack the experience in following up the strict industry SOPs. If you interview a candidate who does not even understand the term ‘SOP’, you know that your candidate will have a long way to adapt to the industry setting.

While there are definitions for Standard Operating Procedures (SOPs), it is hard to define Working Procedures (WPs), not to mention the differences between SOP and WP.

Standard Operating Procedure is established procedure to be followed in carrying out a given operation or in a given situation. In clinical trial setting, the SOP is defined in ICH E6 as “detailed, written instructions to achieve uniformity of the performance of a specific function. “

The issue may be with the wording ‘detailed’ in this definition. The common understanding within the industry is that SOP needs to be written in a little bit high level and the working procedure provides the detailed steps according to SOP. If the strict definition of SOP from ICH E6 is followed, there perhaps should not be any working procedures. All working procedures should be called SOPs. There is really no defined boundary for separate the SOPs and WPs.

My personal opinion is that SOPs are the procedures we must follow, typically on high level, with GCP compliance while WPs are procedures we should follow, typically in more detail, for internal standardization and efficiency. For example, in biostatistics, a SOP must be in place to require the independent validation for statistical outputs. A WP may be created to detail how independent validation should be implemented.

Here are some observations about SOPs and WPs:

1) Within FDA, the term SOPP (standard operating procedure and policies) is used for SOPs and WPs.

http://www.fda.gov/BiologicsBloodVaccines/GuidanceComplianceRegulatoryInformation/ProceduresSOPPs/default.htm

2) There are two papers containing good discussions about SOP:

3) There seems to be no formal definition ‘working procedure’. Different companies may call this differently and different people may have different understanding for ‘working procedure’.

There are some discussions on the web:

Having SOPs and WPs in place is one thing; having everybody trained on SOPs and WPs is another thing. If SOPs and WPs are written in great detail like step-by-step instructions, the procedures may become a burden to be followed. This could in turn cause issues in SOP compliance when there is an audit.

Last week, we had a group meeting during the lunch time and we ordered some food from Jason’s Deli. It was good to see that the delivery guy brought a Bagset Checklist which listed the items and quantities for each item. Unfortunately, the guy clearly did not follow their procedure and did not check the check list. When he delivered, he realized that he forgot to bring ‘bulk chips’. He had to go back to pick up this missed item. Had he follow the procedure and check his Bagset Checklist, he would have not missed any item. In this case, the consequence of the non-compliance is minor, but in clinical trials, the consequence of the non-compliance could sometimes be very significant.

I-Spy 2 Trial - a study with fancy name and fancy design

2011-05-14T00:21:00.001-04:00

If you see the term “I-spy”, you would typically think something related to movie, kids game or TV series. Recently, I-spy has been linked to a novice clinical trial design in breast cancer.

For a short description about this study, clinicaltrials.gov is a good place. In clinicaltrials.gov, I-SPY 2 TRIAL is listed as “Neoadjuvant and Personalized Adaptive Novel Agents to Treat Breast Cancer”. For more detail descriptions about this study, you should check out the following websites:

Barker et all (2009) I-SPY 2: an adaptive breast cancer trial design in the setting of neoadjuvant chemotherapy, Nature Translational Medicine
Ispy2.org – I-Spy 2 clinical trial website
I-Spy 2 Advocate Website – contains many presentations
An IT bundle to support adaptive trials with novel agents and emerging biomarkers

This study got a lot of publicities. For example, in FDA’s ‘advancing Regulatory Science for Public Health’, it specifically mentioned the I-Spy 2 trial,

“Personalized treatment for cancer

The “I-SPY 2 TRIAL,” launched in March 2010, represents a groundbreaking new clinical trial model that will help scientists quickly and efficiently test the most promising drugs in development for women with higher risk, rapidly growing breast cancers. During the trial, drugs in development are individually targeted to the biology of each woman’s tumor using specific genetic or biological markers, known as “biomarkers.” By applying an innovative trial design, researchers will use data from one set of patients’ treatments to treat other patients – more quickly eliminating ineffective treatments and drugs. The I-SPY 2 trial was developed under the Biomarkers Consortium, a unique public-private partnership that includes the FDA, the National Institutes of Health, and major pharmaceutical companies, led by the Foundation for the National Institutes of Health. “

The claimed advantages for this trial design are:

Utilized Personalized Medicine
Uses genetic or biological marker (“biomarkers”) from individual patients’ tumors to screen promising new treatments, identifying which treatments are most effective in specific types of patients
Improved Efficiency (fewer patients, less time, and fewer resources)

Enable researchers to use early data from one set of patients to guide decisions about which treatments might be more useful for patients later in the trial, and eliminate ineffective treatments more quickly Enable the development of more informed, smaller phase III trials

So what is this trial design? Can this trial really achieve its purpose? Here is what I would say:

I-Spy 2 is a phase II study, anything generated from this study will need to be confirmed in Phase III confirmatory studies.
I-Spy 2 is a government-sponsored study and supported by Foundation of NIH, Biomarkers Consortium, NCI, and others. It may never happen if it is an industry-sponsored study.
I-Spy 2 is academic study and not a study for product licensure. If it is study for product licensure, there may be difficulties for study design to be accepted by regulatory agencies.

I-Spy 2 design is complicated and it employs Bayesian Adaptive, Response Adaptive Randomization (play-the-winner approach), Biomarker Adaptive, Stopping and adding treatment arms. The algorithm and stopping rules are complicated and the assumptions / basis for generating these algorithms / stopping rules may eventually be demonstrated incorrect.
I-Spy 2 is a randomized, open label study. It would be more difficult to implement if this is a double-blinded study
I-Spy 2 got it fame due to its use of ‘personalized medicine’, ‘adaptive design’ – all fit into the hot areas
I-Spy 2 is a multi-center, NOT multi-national study. It would be much more challenging if this is a multi-national study

Although I-Spy 2 got a lot of attentions, not everyone is convinced with this design. The study is closed watched – let’s see the fate once it is completed.

Mobile phone text message used in clinical trial?

2011-04-29T21:14:00.000-04:00

Recent discussions with my friends in China surprised me a little bit. They are way ahead in applying the advanced technology in public health area. They used the mobile phone text message to promote the smoking cessation. They recently applied research grant to study the effect of mobile phone text message in improving the maternal and child care. For pregnant women or new mothers, the mobile phone text message is used to send reminders, maternal care tips, child care tips,...

There have been some publications demonstrating the effectiveness of text message in these public health promotion areas. For example, a study in New Zealand demonstrated that smoking cessation using mobile phone text messaging is as effective in Maori as non-Maor. The newscientist.com reported that text messages double young smokers' quit rates.There are

It is natural to think that such text message approach can be used in clinical trials. When I search the clinicaltrials.gov, I can find many clinical trials using text message mostly for improving the adherence of clinical visits or adherence of drug taking.

In my opinion, effectiveness of using text message in clinical trials depends on the study population and the country the study is conducted. In China, if the study is conducted in urban areas, mobile phone text message could be very effective because 1) almost everyone has mobile phone; 2) people use text message more often than actual calling. In the United States, for general population, text message may not be a good approach because some family may still rely on residence-line phone instead of cell phone. Even though they have cell phone, they rarely use text message on daily basis. If a study is conducted in high school students or college students, text message could be an effective tool since all students like to use text messages.

Emerging Statistical Issues in the Conduct and Monitoring of Clinical Trials

2011-04-16T12:09:00.000-04:00

This Wednesday, I had a chance to attend “University of Pennsylvania Annual Conference on Statistical Issues in Clinical Trials”. The topic for this year is “Emerging Statistical Issues in the Conduct and Monitoring of Clinical Trials”. The number of participants was just right in size and the conference was organized pretty well.

In terms of the topics, there are some of them I like and some of them I don’t like. The presentation slides will eventually be posted on conference’s website, however, I would like to give one or two sentences commenting on each topic.

“Sample size estimation incorporating disease progression” – the key issue is the adequacy of the study endpoint. A good endpoint will incorporate the impact of the disease progression.

“Hurdles and future work in adaptive designs” – it is good to hear the discussion about the hurdles, caveats of the adaptive designs. Still very often, a lot of people only talk about the advantages of adaptive designs – too good to be true. Similarly, a recent article "a once-rare type of clinical trial that violates one of the sacred tenets of trial design is taking off, but is it worth the risk? " from The-Scientist magazine gave some objective assessments on implementing the adaptive designs.

“Predicting accrual in ongoing trials” – utilizing the complicated statistical model to predict the accrual is a waste of time. Accrual in ongoing clinical trials is 95% clinical operations issues, 5% related to statistics. Is it worth to modeling the accrual?

“New incentive approaches for adherence” – money incentives including lottery is a sensitive topic and ethic issue could follow no matter it is incentive for adherence or for study visit compliance. Money incentives are different depending on participants’ social economic status (family income). $100 lottery may be very incentive to some, but not to others.

“Efficient source dada verifications in cancer trials” – I always thought that all data fields had to be 100% source data verified. It is not entirely true in large scale trials in oncology or in studies with cardiovascular endpoint. In industry, we are rather conservative.

“Estimation of effect size in trials stopped early” – trials stopped early due to efficacy is not very common and should not be encouraged. Difficulty in estimating the effect size still exists for trials stopped early.

“Accounting in analysis for data errors discovered through sampling” – Unreliable data or large % of missing data is always a concern, even for observational studies. Statistical approach may not be a good option. When data is garbage, the results we draw from the data will also be garbage – so called ‘garbage in, garbage out’ no matter which statistical model is utilized to address the data issues.

“Some practical issues in the evaluation of multiple endpoints” – It is so correct that we should play down the importance of differentiating ‘primary endpoint’, ‘secondary endpoint’, ‘tertiary endpoint’,… Multiple comparison has been expanded so much and is everywhere now (co-primary, primary and secondary, co-secondary, secondary superiority test after non-inferiority test, interim analysis, meta analysis, ISE,…). Are we overdoing this?

Sparse sample and population pharmacokinetics

2011-04-09T10:35:00.001-04:00

In drug development, it is necessary to understand the pharmacokinetics profiles (or time concentration profiles) of the experimental drug and calculate the pharmacokinetic (PK) parameters (Area Under the Curve – AUC, Clearance – CL, or Volume of distribution –Vd). These PK parameters can provide the estimate of the dose exposure and assist in the decision on dose timing and dose interval. In order to calculate the PK parameters, we typically need a serial of blood samples at multiple time points (usually more than 6) after the drug administration. In some situations, it is not feasible or not practical to obtain these many blood samples. The obvious example is in pediatric studies where it is not feasible to obtain multiple blood samples due to the blood volume restriction. The specimen may not just be blood samples. If the PK is conducted using other specimens, it is usually difficult to obtain multiple PK samples. For example, we could obtain middle ear fluid (MEF) sample to determine the antibiotic drug concentration in the ear and bronchoalveolar lavage (BAL) to determine the drug exposure in the lung. It is not practical to obtain multiple samples for these special specimens due to the safety concern.

When very few samples are available for each patient, we call it ‘sparse sampling’. With sparse data, we would need to employ a

Population PK

approach to estimate the PK parameters, describe the PK profile, or do PK/PD modeling. The use of population PK during the drug development has been steadily increasing. Regulatory agencies have issued several guidance on the use of population pharmacokinetics.

à FDA guidance on Population Pharmacokinetics

à FDA guidance General Considerations on Pediatric PK

à FDA guidance Pharmacokinetics in Pregnancy

à EMEA Guidance on the Role of Pharmacokinetics in the Development of Medicinal Products in the Paediatric Population

à EMEA Guidance on Reporting the Results of Population Pharmacokinetic Analyses

There are different sparse sample designs. Below are some of the sparse sample designs I have seen.

An example of sparse sampling at fixed time points is described in a paper by Vogelmeier et al. They used BAL fluid sample to study the intrapulmonary half-life of aerosolized product in Normal Volunteers”.

For BAL fluid samples, it is not feasible to obtain serial samples at all six time points (at screening, 0.5, 6, 12, 24, and 36 h). Therefore, in this study, “each volunteer underwent two BALs. The first lavage was done in the screening phase with an interval of between 3 and 7 d before inhalation of the drug. The volunteers were randomly assigned to one of five groups with the second lavage following 0.5, 6, 12, 24, or 36 h after aerosol administration. Each of the groups consisted of six individuals…”

Subjects in group 1 contributed two BAL samples at Screening and at 0.5 hours after inhalation.

Subjects in group 2 contributed two BAL samples at Screening and at 6 hours after inhalation.

Subjects in group 3 contributed two BAL samples at Screening and at 12 hours after inhalation.

Subjects in group 4 contributed two BAL samples at Screening and at 24 hours after inhalation.

Subjects in group 5 contributed two BAL samples at Screening and at 36 hours after inhalation.

With subjects from all five groups combined, a overall picture of the PK profiles over 24 hours after inhalation could be described. Original paper provided only the summary analysis. Nowadays, the data could be further analyzed using nonlinear mixed model from population PK model with software such as NONMEM.

In FDA guidance on Population Pharmacokinetics, an example was provided for estimating the AUC using sparse data (1-2 middle ear fluid samples per subject) in pediatric subjects.

“The penetration of drug X into middle ear fluid (MEF) was investigated using population PK analysis with sparse data (1-2 samples per subject) obtained from 36 pediatric patients (2 months to 2.0 years of age) who underwent clinical therapy with drug X. The estimated area under the concentration-time curve (AUC) that was above the minimum inhibitory concentration (MIC) (AUCMIC) and the half-life of drug X are 12.5 ug.hr/ml and 6.1 hours in MEF, respectively, vs. 23.7 ug.hr/ml and 3.2 hours in plasma, respectively….”

With this short description, we don’t know if MEF samples are taken from subjects at various times or fixed times. However, non-linear mixed model must have been used for analyzing the data.

FDA’s guidance on population pharmacokinetics states, “the full population PK sampling design is sometimes called experimental population pharmacokinetic design or full pharmacokinetic screen. When using this design, blood samples should be drawn from subjects at various times (typically 1 to 6 time points) following drug administration. The objective is to obtain, where feasible, multiple drug levels per patient at different times to describe the population PK profile. This approach permits an estimation of pharmacokinetic parameters of the drug in the study population and an explanation of variability using the nonlinear mixed-effects modeling approach. “

If a full population PK sampling design is used, the sampling scheme will be something like below. The different subject could contribute different number of samples at various times.

Subject number	Blood sampling time (t)	concentration at time t Ct
001	Predose	xxx
001	24 hours post dose	xxx
002	Predose	xxx
002	8 hours post dose	xxx
002	12 hour post dose	xxx
003	Immediately postdose	xxx
003	5 hour post dose	xxx
004	4 hour post dose	xxx
…

Then when non-linear mixed model such as NONMEM is used to fit the data to characterize the PK profile with PK parameter (such as AUC) = function of concentration (Ct) at time t.

In multiple dose studies, if the purpose is to characterize the PK profile at steady state, one could implement a strategy of splitting the number of samples into different dose intervals.

Suppose we need 8 serial blood samples (t1 to t8) to calculate AUC and the dose interval is weekly, we can have these 8 samples split into 4 dose cycles. For each subject, we would only take two samples for each dose cycle. At steady state, for each subject, we expect PK profile after each repeat dose is not much different; the concentration at day 1 after repeat dose #1 would be similar to the concentration at day 1 after repeat dose #4, and so on. In this case, we would be able to calculate AUC for each subject with 8 samples from four dose intervals (instead of 8 samples from one dose interval over 7 days). The drawback is that the study period would be longer.

Use of SF-36 in Clinical Trials

2011-03-11T16:21:00.000-05:00

The SF-36 is a multi-purpose, short-form health survey with 36 questions. SF-36 is one of the most popular instruments for generic health surveys and it can be used across age, disease, and treatment group, and are appropriate for a wide variety of applications. Conversely to generic health surveys, disease specific health surveys are focused on a particular condition or disease. In clinical trials, SF-36 remains as one of the most common instruments for assessing the Health Related Quality of Life (HQOL), especially in diseases where there is no valid disease-specific tool.

SF-36 yields an 8-scale profile of functional health and well-being scores (so called domain scores) as well as psychometrically-based physical and mental health summary measures [physical component summary (PCS) and mental component summary (MCS)] and a preference-based health utility index (question #2).

The mapping from the original questions -> 8 domains -> PCS or MCS is sketched in the diagram below. Notice that only 35 out of 36 questions are used in this diagram. The question 2 asks about the general health status and does not contribute to the calculation of domain scores and component summaries. A good use of question 2 is to use its responses as anchor in identifying the minimal clinically important difference (MCID). In one of our publications in J Neurol Neurosurg Psychiatry, we indeed used this approach to identify the MCID.

For these 36 questions, the response categories vary depending on the question. The response categories range from 2 (yes, no) to 6 (all of the time, most of the time, a good bit of the time, some of the time, a little of the time, none of the time). Therefore, in order to calculate the domain score, a scoring method or algorithm has to be employed. For PCS and MCS, the calculation will be based on equations with coefficients from the regression models generated from the General Healthy Populatoin. In US, it is the Healthy General US Population. If different healthy population is used, the factor score coefficients for the Z_scores will be different and PCS and MCS values will be different.

The details about scoring method can be found at QualityMetric’s website. The scoring and calculation of component summaries require the programming. Some of the example programs (but not validated) can be found from the web:

Some questions and answers on using SF-36 in clinical trials:

Q: Is SF-36 free for using in clinical trials?

A: It is not free. License has to be obtained for using in industry-sponsored clinical trials. See qualitymetric website for detail.

Q: Why do we have question #2 that is used in calculation of any domain score and component summary?

A: It can be used as an assessment of general health status and also as an anchor for identifying MCID.

Q: Which general health population should be used for norm-based scoring?

A: The advantage of norm-based scoring is to facilitate the comparisons. If a study is a US domestic study, General Healthy US Population should be used. If it is an international study, the country-specific General Healthy Populations are preferred. SF-36 has been validated in many languages.

Q: What will be language to describe the statistical analysis plan for SF-36

A: For study protocol or for journal article statistical method section, analysis plan for SF-36 should be kept simple. In one of our publications on SF-36, we simply said:

"The corresponding physical component summary and mental component summary values for the randomized participants were calculated using the reported means, SDs, and factor score coefficients that came from the healthy general US population in 1990. A linear T-score transformation method was used so that both the physical component summary and the mental component summary scores were standardized with a range of 0 (lowest) to 100 (highest)"

Q: Could SF-36 be used in cost utility analysis?

A: No. SF-36 is not a utility score. However, Sf-36 can be converted to utility score (such as EQ-5D). See my previous blog

Q: Could we have one overall score for SF-36?

A: No. PCS and MCS have to be analyzed separately. You can not add PCS and MCS to have a single overall score.

Q: How to analyze the domain scores and component summaries?

A: Typically, 8 domain scores and 2 component summaries can be analyzed separately using analysis of variance or analysis of covariance or other methods such as repeat measurement depending on the study design.

A good approach in analyzing the SF-36 is to compare the each domain score with the General Healthy Population to show how much difference between the patients in the study and the General Healthy Population for pre-treatment and for end treatment visits. This approach was utilized in our SF-36 publication in Neurology.

Incidence Rate (IR) – How could this be wrongly calculated?

2011-03-03T22:12:00.001-05:00

I am very surprised to see how a simple concept of ‘incidence rate’ can be wrongly calculated in documents submitted to regulatory agencies (such as FDA). In a briefing document titled “Tiotropium (SPIRIVA): Pulmonary Allergy Drug Advisory Meeting – November 2009” submitted by a sponsor, there were wrong statements every where about the calculation of the incidence rate for safety variables.

For example, on page 50, it says “Incidence rates of adverse events were computed as the number of patients experiencing an event divided by the person-years at risk”; In Section 8.1.5 (Statistical methods), it says “For each event, an incidence rate (IR) was calculated from the number of patients with an event divided by the cumulative time at risk within a treatment group and expressed as patient-years.” In their summary tables, they footnoted “the number of patients with an event” (instead of the number of total events) was used in calculating the incidence rate. They never listed the total number of patient year (the denominator) for their Incidence rate calculation. In ‘Statistical method’ section, they even tried to justify the use of “the difference in incidence rate” because “most Tiotropium trials have significantly greater number of patients in the placebo group discontinuing the trial early compared to tiotropium treated patients.”

“Incidence Rate” is a basic concept from epidemiology studies and is calculated as the number of events divided by the number of patient years. According to free medical dictionary, “incidence rate is the probability of developing a particular disease during a given period of time; the numerator is the number of new cases during the specified time period and the denominator is the population at risk during the period. “ According to Wikipedia, “The incidence rate is the number of new cases per population in a given time period. When the denominator is the sum of the person-time of the at risk population, it is also known as the incidence density rate or person-time incidence rate. In the same example as above, the incidence rate is 14 cases per 1000 person-years, because the incidence proportion (28 per 1,000) is divided by the number of years (two). Using person-time rather than just time handles situations where the amount of observation time differs between people, or when the population at risk varies with time. Use of this measure implicitly implies the assumption that the incidence rate is constant over different periods of time, such that for an incidence rate of 14 per 1000 persons-years, 14 cases would be expected for 1000 persons observed for 1 year or 50 persons observed for 20 years.”

In an article by Marco et al “Incidence of Chronic Obstructive Pulmonary Disease in a Cohort of Young Adults According to the Presence of Chronic Cough and Phlegm”, the incidence rate is correctly defined for calculation.

“Incidence rates of COPD were estimated as the ratio of the number of new cases and the number of person-years at risk (per 1,000), which were considered equal to the length of the follow-up for each member of the cohort.”

The key is that if you calculate the ‘incidence rate’, your numerator must be ‘number of events’, not ‘number of patients with an event’. For events that can only occur once in a lifetime for a specific patient (such as cancer), there may not be much difference between ‘number of events” and “number of patients with an event”. However, for events occurr more than one time for a specific patient, “number of events” and “number of patients with an event” are very different concepts.

In Tiotropium briefing document, the correct calculation for incidence rate should be ‘number of events (AEs or COPDs)’ divided by ‘the patient year’. It was simply wrong when they used ‘number of patients with an event’ as the numerator in their calculation of incidence rate. Their justification for using the difference in incidence rate is just the opposite of their statement. If placebo group has more dropouts, their way of calculating the incidence rate will overestimate the rate for placebo group and underestimate the rate for Tiotropium group. This can be easily illustrated using an example below:

Assuming 10 patients in Tiotropium and 10 subjects in Placebo group, 5 patients in Tiotropium group and 5 patients in Placebo group had at least one COPD during the study. The incidence of COPD will be 5/10 = 50% in both groups. Suppose it is a one-year trial, all patients in Tiotropium group completed the one-year and all patients in Placebo group completed only 6 months. The patient year will be 10X1 = 10 for Tiotropium group and 10x0.5 = 5 for Placebo group. The incidence rates now become 5/10 = 50% in Tiotropium group and 5/5 = 100% in Placebo group – this is just simply wrong. In this case, when the patient year (or person year) is used as denominator, the numerator used in the calculation should be the number of events, not the number of patients with an event.

It is unfortunate this simple concept of ‘incidence rate’ has been wrongly calculated in Tiotropium studies. This wrong calculation may have been embedded in their paper published in prestigious New England Journal of Medicine.

If ‘number of patients with an event’ is used in the numerator, the denominator has to be the total number of patients (not the number of patient year). ‘Number of patients with an event’ divided by ‘number of total patients’ is called ‘incidence of events’ – this is a typical way when we summarize the adverse events in clinical trials.

Study Center Pooling Strategy in Multicenter Clinical Trials

2011-02-25T16:24:00.000-05:00

Pooling the study center for statistical analysis purpose is rather an old issue. However, we can still see the discussion o f study center pooling strategy or algorithm in the study protocol or the statistical analysis for multi-center clinical trials. When a clinical trial has multiple centers, study center or investigator site is usually included in the statistical analysis either by including as an exploratory variable in the model (for example ANOVA or ANCOVA) or by conducting the categorical analysis adjusted by study center (for example, Mantel-Haenszel test, Elteren's test, Wilcoxon rank sum test stratified by pooled center). However, there could be situation that some study centers have very few subjects and can not be directly included as a stand alone center for the analysis. In this situation, a pooling strategy is often employed to combine the small centers together. The reason for pooling the small centers instead of using center as random effect may be due to the factor that centers in the clinical trial are rarely a random sample of all possible centers. It is not uncommon to find the statistical analysis including pooled center in regulatory submission or in publications, for example, in NDA for Refludan (the analysis was stratified by pooled center) and in FDA advisory committee documents (… were analyzed using Wilcoxon rank sum test stratified by pooled center (centers that entered fewer subjects than a complete block were pooled by country)). Here are some of the example languages describing such pooling strategies:

“Statistical tests will be performed as two-sided tests and will be adjusted to the multi-centric design of the study. A center must have enrolled at least 8 subjects to be a standalone center in the analysis (centers enrolling less than 8 subjects will be pooled – will be done before the study unblinding”

“Study centers were pooled from largest to smallest until the pooled center had more than 5 subjects with post baseline data in each treatment group. No pooled center had more than 15% of the total number of subjects”

“The majority of study centers were small. A small center was defined as any center with <5 patients with postbaseline data in any treatment group, resulting in 5 large and 25 small centers. To avoid loss of information, small centers were pooled from largest to smallest until the pooled center had 5 patients in each treatment group. These centers were grouped into 11 pooled centers for the purpose of analysis."

In one of hypertension clinical trials, the pooling strategy is described as “To avoid loss of information, small centers (<5 per protocol patients) were pooled from largest to smallest until the pooled center had 5 per protocol patients in each treatment group. These centers were grouped into 19 pooled centers for the purpose of analysis. The pooling algorithm was predetermined before unblinding the data, and the pooling algorithm was described in the statistical analysis plan for the study. Considering the subjective nature of the pooling algorithm, albeit prespecified before completion of the study, an exploratory analysis was also performed with actual center as a fixed effect in contrast to pooled centers. This analysis did not change the inference.”

In a type 2 diabetes trial, a different pooling strategy was used “For all center stratified analyses, centers with <24 randomized and treated subjects were pooled on a geographical basis, independently of treatment identification.”

In a recent brief book for PDAC, the sponsor provided the detail pooling strategy for centers “Pooling algorithm for centers: For non-US sites, all investigative sites within a country with fewer than 10 randomized subjects will be combined into a single pooled site for analysis purposes. If a resulting pooled site still has fewer than 10 randomized subjects, then this pooled site will be further combined with the smallest unpooled site within that country. If there is not another unpooled site within that country, then the pooled site will be combined with the smallest pooled site from another country. This pooling process will continue until there are at least 10 randomized subjects in each pooled site. For US sites, all investigative sites within a geographic region with fewer than 10 randomized subjects will be combined into a single pooled site for analysis purposes. If a resulting pooled site still has fewer than 10 randomized subjects, then this pooled site will be further combined with the smallest unpooled site within that region. If there is not another unpooled site within that region, then the pooled site will be combined with the smallest pooled site from another region within the US. This pooling process will continue until there are at least 10 randomized subjects in each pooled site.”

As we can see from the examples above, the cut point for center pooling (5, 8, 10, or 24) is really arbitrary and there is no scientific basis for choosing one or another. The decision on the cut point may be based on the distribution of the number of subjects across centers.

Center pooling strategy could sometimes be questioned by the regulatory reviewers. For example, in BLA review of Rebif, FDA reviewer had concerns about the pooling strategy “The sponsor’s study center pooling strategy: Per the pre-specified strategy in the sponsor’s statistical analysis plan (SAP), pooling of study centers for inclusion of center as a main effect in analyses was to have been based on geographic considerations for small centers. In fact, the pooling strategy actually used was data driven which is problematic. NOTE: There were 56 participating centers from 9 countries. The smallest recruiting center had 3 subjects, 2 centers contributed 4 subjects, and 5 centers contributed 6 subjects each. The remaining centers contributed between 6 – 24 subjects each (CSR, Table 3, pp. 65-66). This reviewer performed analyses of major efficacy endpoints based on strict geographic pooling of centers into 3 groups (US, Canada, and Europe) as well as un-pooled analyses (not including the center effect). In addition, descriptive analyses for individual centers were also performed for the primary and major secondary efficacy endpoints. The sponsor’s positive statistical findings were found to be robust based on these analyses.”

In Biopharmaceutical Report (Summer 1998), Paul Gallo wrote an article titled “Practical Issues in Linear Models Analyses in Multicenter Clinical Trials” which contained a section discussing “construction of composite centers”. The caveats of using the composite centers are also discussed in the paper.

“In performing unweighted analyses, a practice of defining artificial “pooled” or “composite” centers is often employed; that is, data from different centers are treated in the analysis as if they came from the same center. A number of small centers may be combined, or one or more small centers may be combined with a larger center. This practice attempts to minimize the large variance inflation and data instability of unweighted analyses when there are very small centers. Composites may be constructed to the extent of eliminating empty cells to ensure that treatment effects are estimable in models containing interaction terms. More commonly, this is done to achieve some minimum cell size felt to appropriately limit the influence of individual observations; values around 5 are often chosen. ”

Arbitrarily pooling the centers sometimes does not make sense at all. This is exactly true when the centers with small number of enrolled subjects are pooled even though these centers are scattered in totally unrelated geographic regions or countries. When pooled center is used and the statistically significant center effect is detected, the interpretation of the results is difficult. Instead of the center pooling purely based on the number of enrollees, the geographic distribution of centers should be considered. In many cases, instead of pooling centers by the number of enrollees, we could use country and geographic region in the analysis. In one of our multi-national clinical trials, we grouped centers by geographic region as North American, South American, Eastern Europe, Western Europe, and Eastern Asia. The strategy worked very well.

If possible, we could use the random effect model to include the study site / center as random effect to avoid the center pooling. We could also use a center weighting strategy that is similar to the Meta analysis where centers with more subjects are given more weights.

Guidelines for Blood Volumes in Clinical Trials (Especially in Pediatric Clinical Trials)

2011-02-08T00:00:00.002-05:00

Nowadays, the clinical study protocols are becoming more and more complicated and require more and more blood sample draws for various purposes. The blood samples are needed for testing the hematology, chemistry, immunogenicity (for biological products), biomarkers (for diagnostic or other purpose), pharmacogenomics,… In some clinical trials, additional blood samples (sample retains) may be drawn for future studies (even though we may not know what the future study will be). If the study has the component of pharmacokinetics, the many more samples (series blood samples) will be drawn within short period to characterize the pharmacokinetic profile, estimate the total drug exposure (AUC), and calculate other pharmacokinetic parameters.

With increasing in the number of blood draws or the blood volumes, the ethic issue often arises, especially in clinical trials with children.

US FDA and EMA do not really regulate the maximum blood volume that can be drawn from a subject during the clinical trials. The requirements for limiting the blood sample volume may come from the National Institute of Health (NIH), American Academy of Pediatrics, World Health Organization (WHO), and European Union (EU) and are typically enforced by the ethic bodies such as Institute Review Board (IRB) and Ethics Committee (EC). The requirements on blood volume during the clinical trials may be different depending on the country and local IRB.

The blood volume drawn for pharmacokinetic studies in pediatric population is specifically a concern and has been discussed extensively. Stephen RC Howie (2010) reviewed blood sample volumes in child health research: review of safe limits in Bulletin of the World Health Organization (BLT) . WHO also has its guidelines on drawing blood: best practices in phlebotomy. The guidelines are not specifically for clinical trials, rather for general blood donations. The guidelines contain specific technical requirements for blood drawn in pediatric and neonatal subjects.

In US, Code of Federal Regulations has a specific chapter (Part 46) to discuss protection of human subjects and the chapter contains a subpart D to address additional Protections for Children Involved as Subjects in Research. While there is no specific requirement on the limit of blood volume, the CFR indicated that the research involves no more than minimal risk to the subjects and IRB should take into account the purposes of the research and the setting in which the research will be conducted and should be particularly cognizant of the special problems of research involving vulnerable populations, such as children, prisoners, pregnant women, mentally disabled persons, or economically or educationally disadvantaged persons. Similarly, American Academy of Pediatrics has its policy on Guidelines on Ethical Conduct of Studies to Evaluate Drugs in Pediatric Populations. The policy requires “…with the growing number of pediatric drug studies, IRBs need to be familiar with the various research-design methods that inimize risk to the child. Examples include limiting research under some circumstances to pharmacokinetic and safety data, combining this approach with pharmacodynamic data, and minimizing the volume of blood withdrawn through the use of sensitive assays, pediatricenabled laboratories, and population pharmacokinetic approaches"

National Institute of Health Clinical Center has a guideline M95-9: Guidelines for Blood Drawn for Research Purposes in the Clinical Center, however, the guideline is only accessible for NIH internal use.

Two articles from the web actually reflect the limit of blood volume in US.

How Much Blood is too Much Guideline including the maximum volumes from several institutes
Clinical Translational Research Center TraCS Institute Approved by: Administrative Director and Nursing Director SOP Title and #: Pediatric Blood Draw: includes a table for “NIH Pediatric Blood Volume for Research Guidelines”
Blood Sampling Guidelines from Partners Human Research Committee Blood volume taken from children must be less than 3 cc/kg body weight per 8 week period.
BLOOD VOLUME COLLECTED PER VENIPUNCTURE
The maximum volume for a single phlebotomy is:
Term newborn – Age 18: 3 mL/kg, up to a maximum of 150 mL
Age 18 – 85: Maximum of 150 mL
Age 86 and above: Maximum of 100 mL
Repeated Phlebotomy over Acute Periods: Where possible, blood collection over a 24- hour period should be limited to 5 mL/kg with balanced consideration of patient safety and clinical needs

In EU, there are specific guidelines on "ETHICAL CONSIDERATIONS FOR CLINICAL TRIALS ON MEDICINAL PRODUCTS CONDUCTED WITH THE PAEDIATRIC POPULATION"

"13.2 Volume of blood

Preterm and term neonates have very limited blood volume, are often anaemic due to age and frequent sampling related to pathological conditions. The fact that children, especially in this age group, receive blood transfusions (or iron or erythropoietin supplementation) should not be used as a convenience for increased volume or frequency for blood sampling.

The following blood volume limits for sampling are recommended (although are not evidence-based).
If an investigator decides to deviate from these, this should be justified. Per individual, the trial-related blood loss (including any losses in the manoeuvre) should not exceed 3 % of the total blood volume during a period of four weeks and should not exceed 1% at any single time. In the rare case of simultaneous trials, the recommendation of 3% remains the maximum. The total volume of blood is
estimated at 80 to 90 ml/kg body weight; 3% is 2.4 ml blood per kg body weight."
The guidelines on blood volume are usually based on the amount of blood in percentage of total blood volume (BLV). BLV varies depending on the age and body weight. A good reference for BLV for pediatrics can be found in pediatricareonline.com.

Edit check - a critical step to ensure the data quality during clincial trials

2011-01-28T22:18:00.000-05:00

In clinical trial, one critical task is to ensure that the data collected or data entered into the system / database is valid, correct, and logically sound. This task requires a data quality plan starting from designing a good study protocol -> developing efficient case report forms -> providing clear instructions for completing case report forms -> implementing electronic edit checks -> monitoring the study data / source data verification -> data clarification process -> data review process. One of the steps is to implement the electronic edit checks.
Edit check is a program instruction or subroutine that tests the validity of input in a data entry program. According to the CDISC clinical research glossary from Applied Clinical Trials, the edit check is defined as:

An auditable process, usually automated, of assessing the content of a data field against its expected logical, format, range, or other properties that is intended to reduce error. NOTE: Time-of-entry edit checks are a type of edit check that is run (executed) at the time data are first captured or transcribed to an electronic device at the time entry is completed of each field or group of fields on a form. Back-end edit checks are a type that is run against data that has been entered or captured electronically and has also been received by a centralized data store.

Electronic edit checks allow us to use the power of the computer to check for illogical, incomplete or inconsistent data. In clinical trial, one of the most important tasks facing clinical data management personnel is to produce the electronic Edit Checks specifications for a study. Developing the electronic edit check specification -- and processing the queries that result from them -- is arguably the most vital and time-consuming data cleaning activity data management personnel undertakes. The study statistician should always participate in the process of developing the electronic edit checks to ensure that the critical edit checks are included. Effectively implementing the edit check can prevent the illogical, incomplete, or inconsistent data from entering into the data capture system or data set, which will make the downstream data analyses much easier.

There are two types of edit checks:

Univariate edit checks (include range checks): these are the edit checks only applicable to a single field or single variable. For example, for subject weight, we can set up an edit check to ensure that the extreme or unlikely value not to be entered. Let’s say we set up a range check if a data entry is smaller than 90 lb or greater than 300 lb. For lung function test, we may set up an edit check for predicted FEV1 to be no less than 20% because it is unlikely to have someone with predicted FEV1 <20%. The univariate edit checks are usually run instantly during the time of data entry.

Multivariate edit checks (also called aggregate edit checks): these are the edit checks with more than one fields or variables involved. These edit checks cross check the entries across multiple fields / variables to ensure the data is logical and consistency. For example, if the entry on Gender field is ‘Male’, there should not be data for pregnancy test result field. If the reason for subject dropping out the study is entered as ‘adverse events’, there should be a corresponding entry in AE data set. Statistician can provide great inputs in identifying the multivariable edit checks. Some multivariate edit checks could involve the complicated algorithm and take considerable time to run. In this situation, the multivariable edit checks can be run at back-end at a specified interval (for example, 2 am at night).

One misunderstanding is to think that all data issues can be resolved by implementing the edit checks. Edit check is only one of the steps in the data cleaning process. Also, there should be balance in terms of the number of edit checks. Too many edit checks for non-critical fields could be very annoying for people who enter the data. This is especially true for clinical trials using electronic data capture (EDC) where the data entry responsibility is delegated to the investigator and study coordinators who may lose patient if there are too many pop-up messages during the data entry. For example, if the telephone number needs to be entered, an edit check to enforce the data entry to follow xxx-xxx-xxxx would be unnecessary (xxxxxxxxxx and 1xxxxxxxxxx should also be accepted) – this is an example I see in some of the web forms – very annoying).

Regulatory Guidance on Source Data in EDC Trials

2011-01-23T21:14:00.001-05:00

When we move toward the clinical studies using electronic data capture, the ‘source data’ or ‘source document’ has been an issue. Unlike the paper-CRF (case report form) based study, the source data in EDC study can be confusing and sometimes vague. If the data was directly entered into EDC system, the EDC system is the direct source and there is no another source to be verified against. This could be worrisome to some people. In a 2008 article, I talked about this issue.

Recently, both FDA and EMEA published the guidance on this issue. FDA’s guidance "Electronic Source Documentation in Clinical Investigations" was issued in December, 2010. EMEA issued its guidance last June and the guidance titled “Reflection paper on expectations for electronic source data and data transcribed to electronic data collection tools in clinical trials”.

The guidance titles seem to suggest that they are written for the data management functions, however, the discussions in these two guidelines are more relevant to the clinical sites and study monitors. Switching the clinical study from paper CRF to EDC is not just about the shift of the data entry from data management group to the clinical sites, it actually has impact on how the entire study is operated.

FDA's New Website for Industry

2011-01-11T00:43:00.000-05:00

Have you noticed the changes in the design of FDA website (http://www.fda.gov/) recently? Last August, I mentioned the FDA's initiatives on transparency. As part of FDA's continued push to increase transparency in an agency once notorious for making decisions behind closed doors, the FDA has launched a new Web-based resource that industry can use to keep abreast of the regulatory status for drugs, devices, food, and cosmetics. The new website is under http://www.fda.gov/ForIndustry/ and is supposed to provide a repository for industries to understand FDA's detail processes in submission, reviewing, approval, and surveillance of the regulated products, and even the processes for complaints (dispute resolution). The website includes the sections that are very pertinent to us working in the pharmaceutical industry:

Developing products for rare disease and conditions
Dispute resolution
Guidance documents
FDA eSubmitter
Data standards
FDA basics for industry

FDA basics for industry includes the kind of basic information about the regulatory process that is often requested by drug, device, and biologic companies and is aimed at improving communication between FDA and industry by making basic information about the regulatory process more accessible to industry in a user-friendly format.

The new website reflects the great improvement towards the transparency and is a great resource for professionals working in the drug development industry.

Also see:

Agreement Statistics and Kappa

2011-01-02T01:06:00.000-05:00

In clinical trial and medical research, we often have a situation where two different measures/assessments are performed on the same sample, same patient, same image,… the agreement needs to be calculated as a summary statistics. Depending on whether or not the measurement is continuous or categorical, the agreement statistics could be different. Lin L had a very nice overview for agreement statistics.

Specifically for categorical assessment, there are many examples where the agreement statistics is needed. In a clinical trial with imaging assessment, the same image (for example, CT Scan, arteriogram,…) can be read by different readers. For disease diagnosis, a new diagnostic tool (with advantage of less invasive or easier to implement) could be compared to an established diagnostic tool… Typically, the outcome measure is dichotomous (e.g., disease vs no disease, positive vs. negative…).

The choice of the methods of comparison is influenced by the existence and/or practical applicability of a reference standard (golden standard). If a reference standard (golden standard) is available, we can estimate sensitivity and specificity – ROC (receiver operation characteristics) analysis. If a reference standard is not available or there is no golden standard for comparison, we can not perform ROC analysis. Instead, we can assess the agreement and calculate the Kappa. This has been discussed in detail in FDA’s Guidance for Industry and FDA Staff “Statistical Guidance on Reporting Results from Studies Evaluating Diagnostic Tests”. For example, for comparing the assessment from two different readers, we would calculate Kappa, overall percent agreement, positive percent agreement, and negative percent agreement. We would not use ROC statistics and would not calculate the sensitivity and specificity.
If we would like to assess the agreement between the urine pregnancy test and the serum pregnancy test, we could use the ROC and calculate the sensitivity, specificity, positive predictive value, and negative predictive value since the serum pregnancy test could be considered as a reference standard or golden standard for pregnancy test.

Kappa Statistic(K) is a measure of agreement between two sources, which is measured on a binary scale (i.e. condition present/absent). K statistic can take values between 0 and 1.

Poor agreement : K < 0.20
Fair agreement : K = 0.20 to 0.39
Moderate agreement : K = 0.40 to 0.59
Good agreement : K = 0.60 to 0.79
Very good agreement : K =0.80 to 1.00

A good review article about Kappa Statistics is the one written by Karemer et al “Kappa Statistics in Medical Research”.

SAS procedures can calculate Kappa Statistics easily. Here is a list of papers:

Bootstrap and SAS

2010-12-27T13:49:00.000-05:00

In statistics, bootstrapping is a resampling technique used to obtain estimates of summary statistics. In clinical trials, bootstrapping technique could be a useful approach in obtaining the precision of an estimator. Most common application of the bootstrapping technique may be in obtaining the confidence interval for an estimator while the typical way of obtaining the confidence interval through the standard error approach is impossible or difficult.

Here are two examples that the bootstrapping technique needs to be implemented. The first example is for a manuscript. When we submitted our paper to European Respiratory Journal, one of the reviewer comments was a request for evaluating the internal consistency. The comment says “The statistical method is sample-based as it consists in a regression performed on this sample. Such a method needs at least evaluation for internal consistency (by measuring the regression correlation on a subsample then validating on another subsample or better by using bootstrap and jackknife methods).”

The second example is a request from the regulatory agency for calculating the 95% CI for % relative dif
ference. When there are two treatment means: A and B; % relative difference is defined as %RD= (A-B)/A. There may be other approaches in this case, but bootstrapping technique could come handy in calculating the 95% CI for %RD.

Bootstrap can be easily implemented in SAS and it contains three main steps: 1) resample the data from the observed data set (observed data is only one sample) – SAS Proc Surveyselect can serve this purpose 2) obtain the statistics (or estimator) by performing the analysis for each sample / resample 3) perform the summary statistics from the collection of the statistics or estimator.

Bootstrap is a suggested statistical approach for obtaining the confidence interval for individual and population bioequivalence criteria.

Some good references about how to do bootstrapping using SAS are included here:

How can I bootstrap estimates in SAS? From UCLA ATS
Introduction to bootstrap estimation by Wiley
A Practical Introduction to the Bootstrap Using the SAS System by Nancy Barker
Bootstrap 101: Obtain Robust Confidence Intervals For Any Statistic by Dave Miller et al
A tutorial on bootstrapping in SAS system by Paul Thompson
Don't Be Loopy: Re-Sampling and Simulation the SAS® Way by David Cassell or Bootstrapmania: Resampling the SAS way by the same author
Bootstrapping and SAS

Ten years ago, I had to use a SAS macro to do the bootstrap for my PhD dissertation. The macro is still there on SAS website.

Bootstrap technique has also been built into several SAS procedures (such as Proc Multtest, Proc MI).

When bootstrap is used in regression situation, 'Bootstrap Pairs' technique may be employed. Freedman (1981) proposed to resample directly from the original data: that is, to resample the couple dependent variable and regressor, this is called bootstrapping pairs. Bootstrap pairs is described in a paper by Flachaire. The SAS macro for bootstrapping discussed two main ways to do bootstrap resampling for regression models, depending on whether the predictor variables are random or fixed.If the predictors are random, you resample observations just as you would for any simple random sample. This method is usually called "bootstrapping pairs". If the predictors are fixed, the resampling process should keep the same values of the predictors in every resample and change only the values of the response variable by resampling the residuals.

Counting the study day

2010-12-12T22:44:00.000-05:00

For every clinical trial, we need to count the study day for calculating the follow-up visits and for assessing the temporal relationship between events. The study day starts with the day that the subject is randomized and receives the first dose of the study medication. Usually, the randomization date and the first dose of the study medication date are the same. In clinical study protocol, there should always be a ‘schedule of events’ or ‘schedule of evaluations’ table which defines the study procedures and the study visits. This table should include the study day.

There is one critical difference in counting the study days. The protocol could count the day of subject receiving the first dose of the study medication as “day 0” or “day 1”.

If the first dose date is counted as day 0, the day immediately after the first dose date will be counted as day 1 and the date immediately before will be counted as day -1. Therefore, the study day is counted continuously as … day -7, day -6, day -5, day -4, day -3, day -2, day -1, day 0, day 1, day 2,… In this case, for programming, the study day variable can be created using the formula:

The event/visit date – first dose date

The problem with this counting is in 'day 0'. People are used to calling the first day of the study medication as the 'day 1'.

If the first dose date is counted as day 1, the day immediately after the first dose date will be counted as day 2 and the date immediately before will be counted as day 0 – which is confusing. In practice, if the first dose date is counted as day 1, the day 0 will not be used in the study day counting. The date immediately before will be counted as day -1 (skipped day 0). Therefore, the study day is counted as: day -7, day -6, day -5, day -4, day -3, day -2, day -1, day 1, day 2,… For programming, the study day variable would be created using two separate formulas for predose and postdose visits.

For pre-dose:

the event/visit date – first dose date

For post-dose:

the event/visit date – first dose date + 1

Both of these approaches (counting including study day 0 or not including study day 0) are not wrong, but sometimes confusions can arise when we calculate the study day variable. Even for CDISC, there are disagreements in handling this between Submission Data Set Tabulation Model (SDTM) (not allowing study day 0) and Analysis Data Set Model (ADaM) (allowing study day 0).

The following clinical trial protocol templates indicate that the study day counting starts with day 0:

The following clinical trials indicate that the study day counting starts with day 1. There are more industry trials like this.

The unit used in counting the study day depends on the length of the clinical trials. For a trial with months and years in duration, instead of counting by day, it is more practical to count by week, month, or year. For example, for a clinical trial with three years treatment duration, the last treatment date would be three years away. If we count by day, it will be something like day 1095. Even worse, some people may apply the time window to this date to have the last treatment date 1095 +/- 7 days. Sound stupid, isn’t it?

Counting the study day correctly is important for study investigators/coordinators to avoid the protocol deviation. The Barnettinternational actually developed a tool to facilitate the study day /visit scheduling.

A conditional probability issue?

2010-11-29T22:41:00.000-05:00

There is a question and answer from 'AskMarilyn' at Parade.com. I copy the question and answer here since it is a probability issue.

Question: Four identical sealed envelopes are on a table. One contains a $100 bill. You select an envelope at random and hold it in your hand without opening it. Two of the three remaining envelopes are then removed and set aside, still sealed. You are told that they are empty. You are not given the choice of keeping the envelope you selected or exchanging it for the one on the table. What should you do? A) Keep your envelope; B) switch it; or C) it doesn't matter.

Marilyn said you should switch envelopes. Here's her reason: Imagine playing this game repeatedly. You start with a 25% chance of choosing the envelope with the cash. Then two empty ones are taken away on purpose. (Only someone with knowledge of the contents can inform you that sealed envelopes are empty.) so if the $100 bill is in any of the three unchosen envelopes - which it is 75% of the time - you'll get it by switching.

However, I would choose the answer C) it doesn't matter. This is a conditional probability issue. In the beginning, with all four envelopes sealed, the probability of choosing one envelope with $100 bill is 25%. When two envelopes are revealed not to contain the $100 bill, for the remaining two envelopes, each now has 50% probability with $100 bill in it. It doesn't matter if you keep the envelope on hand or switch it for the one on the table.

Using RevMan to Conduct the Meta Analysis

2010-11-20T16:40:00.001-05:00

RevMan (or Review Manager) is designed as a review tool to facilitate the literature review and the meta analyses by the Cochrane Collaboration Group. RevMan can be downloaded from website for free. It can be installed into your system without requiring the system administer privilege. Thousands of systematic reviews and meta analyses published on the Cochrane Library are performed using RevMan. These systemic reviews and meta analyses have been one of the leading resources in evidence-based medicine.

RevMan can be easily used by the medical researchers who are non-statisticians. For statisticians who work in the medical research area, RevMan is an easy tool to perform the meta analyses and generate the graphs (forest plot, funnel plot) in publication standard.

The statistical method and statistical model are described in the document Standard statistical algorithms in Cochrane reviews by Jon Deeks and Julian Higgins and Cochrane Handbook for Systemic Review of Interventions. For statistical models, both fixed model and random model are included in the RevMan. For random models, DerSimonian and Laird random-effects models are used. This is most common random effects model used in Meta Analysis.

RevMan 5 is extremely easy to use. Various tutorials, tips, webinars are provided in RevMan documentation website and The Cochrane Collaboration Open Learning Material. I find it is extremely useful to watch two webinars (especially the part 2 regarding the data and analyses. For

To perform a Meta analysis, RevMan is just a tool. There are a lot of works to be done prior to enter the information including data into the RevMan. Considerable time needs to be spent on the literature search. Since the data used in Meta analyses relies on the publications, some data needs to be converted first. For example, for outcomes measured in continuous variable, the published article may only provide the Standard Error or just the 95% confidence interval. The SE can be easily converted to the Standard Deviation by multiplying the square root of the sample size. If only the 95% confidence interval is available, the standard deviation can be approximated by normal approximation using upper bound = mean +/- 1.96 * SE.

Good Review Practice

2010-11-07T22:18:00.000-05:00

In previous article, 'regulatory science' is discussed. 'Good Review Practice' can be considered one aspect of the regulatory science. Here Good Review Practice is specifically refer to a “documented best practice” within CDER that discusses any aspect related to the process, format, content and/or management of a product review.

On the industry side, the sponsor needs to establish the standard operating procedures (SOP) and the working procedure documents (WPDs) to ensure the compliance of the regulatory guidance and GCP and to improve the efficiency. On the regulatory side, it is important to establish the good review practice to ensure that the same standard procedures are following during the review process for drug approval.

These good review practices could cover the review process in different areas: efficacy, safety, pregnancy, CMC,... They are supposed to be written for FDA reviewers, however, understanding the good review practice is also very helpful for sponsor to prepare the regulatory submission documents in a way that is amenable to the reviewers. The mis-communication between the sponsor and the regulatory could be minimized. All necessary information/analyses required per good review practice are included in the submission documents.

Below are some links related to good review practice:

Regulatory Science

2010-10-31T00:14:00.000-04:00

Regulatory Science is the science of developing new tools, standards, and approaches to assess the safety, efficacy, quality, and performance of all FDA-regulated products including drug, biological products, medical device and more. On February 24, 2010, FDA along with NIH launched its Advancing Regulatory Science Initiative (ARS) aim to accelerate the process from scientific breakthrough to the availability of new, innovative medical therapies for patients.

On October 6, 2010, the U.S. Food and Drug Administration unveiled an overview of initiatives to advance regulatory science and help the agency assess the "safety, efficacy, quality and performance of FDA-regulated products." And published its white paper ’Advancing Regulatory Science for Public Health - A Framework for FDA's Regulatory Science Initiative” The white paper outlines the agency's effort to modernize its tools and processes for evaluating everything from nanotechnology to medical devices to tobacco products.

In companion to the release of the white paper, FDA commissioner, Dr Hamburg gave a speech to the National Press Club in Washington, DC

In the white paper, the section I “Accelerating the Delivery of New Medical Treatments to Patients” has specific meaning to statisticians. “Adaptive design” was not specifically mentioned in the white paper, however, any approach or methodology in clinical trial design that can expedite the drug development process should be encouraged. The personalized medicine should also be encouraged.

Even though the regulatory science or regulatory affairs is critical in drug development field, the professionals working in the field are very diversified and come from variety of different backgrounds. Perhaps, you can only learn the regulatory science through the experience and on-job training. However, I do notice that USC has a graduate program in regulatory science. Considering that FDA is increasing its investment in regulatory science and the regulatory laws are getting more and more complicated, the graduates from this program should not have any difficulty in finding a job.

Missing data in clinical trials - the new guideline from EMEA and National Academies

2010-10-08T14:45:00.003-04:00

Missing data issues have been discussed and debated for many years. Handling of missing data in clinical trials has been recognized as an important issue not only for statisticians who analyze the data, but also for the clinical study team who conduct the study. While we are still waiting for FDA to issue its guidance on missing data in clinical trials, there are several guidelines published recently.

EMEA just issued its final rule of "Guideline on missing data in confirmatory clinical trials". This guideline provided the guidance on handling the missing data from the perspective of European regulatory authorities. Comparing to the FDA's guidance on non-inferiority and adaptive design, EMEA's missing data guidance is written in plain language and can be easily understood by the non-statisticians.

The recent trend is to discourage the use of LOCF and other single imputation methods (ie, replace the missing value with the last measured value, with averaged value, or with baseline value,...). It is noted that LOCF is mentioned as one of the single imputation methods in EMEA's guideline. The guideline acknowledged that "Only under certain restrictive assumptions does LOCF produce an unbiased estimate of the treatment effect. Moreover, in some situations, LOCF does not produce conservative estimates. However, this approach can still provide a conservative estimate of the treatment effect in some circumstances.". The guideline further elaborated that LOCF may be a good technique for studies (e.g. depression, chronic pain) where the condition is expected to improve spontaneously over time, but may not be conservative for studies (e.g. Alzeimer's disease) where the condition is expected to worsen over time.

In the United States, the Division of Behavioral and Social Sciences and Education under National Research Council of the National Academies have been working on a project "Handling missing data in clinical trials". The working group recently makes its draft report available. The draft report is titled "The prevention and treatment of missing data in clinical trials". I like the word 'prevention' in the title since it is critical to prevent or minimize the occurrence of missing data. Once the missing data has happened, there is no universal method to handle the missing data perfectly. The assumptions of MACR, MAR, and MNAR can never been fully verified.

Academies' report on missing data has a stronger language in discouraging the use of LOCF and other simple imputation approaches. The recommendation #10 stated "Single imputation methods like last observation carried forward and baseline observation carried forward should not be used as the primary approach to the treatment of missing data unless the assumptions that underlie them are scientifically justified."

So far, there is no official guideline from FDA regarding the missing data handling (even though the topic has been the perennial topic in almost all statistics conferences and workshops). Nevertheless, a presentation by Dr. O'Neill to the International Society of Clinical Biostatistics may give some insides.

Individual response vs. group response

2010-10-03T22:15:00.001-04:00

In clinical trials, the efficacy endpoints are often measured as continuous variables. The hypothesis tests are used to determine whether or not there are statistically significant differences between one group vs. another group. This is desired by the statisticians. However, for treating physicians, the treatment effect on group basis may not translate to the effect to an individual patient. When we move toward to the personalized medicine, the individual response may be more important than just the group response.

It is interesting that the individual response and individual assessment (or within patient analysis, intra-subject changes...) were greatly discussed in this year's FDA/Industry Statistics Workshop.

For patient reported outcome, the statistically significant group change does not necessarily imply a meaningful difference for individual patients. To provide meaningful interpretation of patient reported outcome intervention and treatment effects, there should be a responder definition to classify each individual subjects as responder or non-responder. The FDA guidance stated "Regardless of whether the primary endpoint for the clinical trial is based on individual responses to treatment or the group response, it is usually useful to display individual responses, often using an a priori responder definition (i.e., the individual patient PRO score change over a predetermined time period that should be interpreted as a treatment benefit). The responder definition is determined empirically and may vary by target population or other clinical trial design characteristics. Therefore, we will evaluate an instrument’s responder definition in the context of each specific clinical trial." The challenging issue is how to determine the cutpoint or benchmark for the definition of the responder. Several approaches have been proposed in the literature. We had actually implemented various approaches to determine the responder (or clinical meaningful difference) in a neurology disease. In the article, two of the anchors are used: one based on physician's assessment and one based on global assessment by the patient (question #2 in SF-35 instrument). It is interesting that the statistical approaches are employed to find the clinical meaningful difference.

Once the cutpoint (clinical meaningful difference) is decided, the continous variable will be dichotomized into responder and non-responder. The analysis will them be shifted from the parametric method (t-test, ANOVA, ANCOVA,...) to categorical data analysis method (chisquare, logistic regression, generalized linear model,...). Statistician will argue that by doing so, we lost a lot of efficiency in statistical testing. A paper by Snappin and Jiang titled "Responder analyses and the assessment of a clinically relevant treatment effect" just did this argument.

In the recently published EMEA "Guideline on missing data in confirmatory clinical trials", responder analysis was mentioned to have a benefit of handling the missing data. It stated:

"In some circumstances, the primary analysis of a continuous variable is supported by a responder analysis. In other circumstances, the responder analysis is designated as primary. How missing data are going to be categorised in such analyses should be pre-specified and justified. If a patient prematurely withdraws from the study it would be normal to consider this patient as a treatment failure. However, the best way of categorisation will depend on the trial objective (e.g. superiority compared to non-inferiority).

In a situation where responder analysis is not foreseen as the primary analysis, but where the proportion of missing data may be so substantial that no imputation or modelling strategies can be considered reliable, a responder analysis (patients with missing data due to patient withdrawal treated as failures) may represent the most meaningful way of investigating whether there is sufficient evidence of the existence of a treatment effect."

Within-patient analyses were brought up again in assessing benefit:risk. Currently, the benefit:risk assessment relies on separate marginal analyses. The efficacy (benefit) and safety (risk) are analyzed separately. The aggregation of the benefit:risk relies on the assessment of medical reviewers, not statisticians. The aggregate analyses of benefit:risk are typically qualitative, not quantitative with significant subjectivity. With within-patient analyses, each patient was assessed for benefit and risk before performing the group comparison for treatment effect. One of these approaches is so called Q-Twist (The quality-adjusted time without symptoms of disease or toxicity of treatment) where the toxicity or safety information is incorporated into the efficacy assessment for each patient before any group comparison. The paper by Sherrill et al is one of these examples.

Do we really need to statistical soluation for everything?

2010-09-26T23:02:00.000-04:00

I came back from last week's FDA/Industry Statistics Workshop with more questions than answers. While the theme for this year's workshop is on risk benefit assessment, the old regular issues such as multiplicity, missing data, meta analysis, adaptive design, subgroup analysis,...are still the hot topics. For both the new (risk benefit assessment) and the old topics, there are more questions being raised and for many, there are no clear answer to these questions.

For adaptive design and non-inferiority clinical trials, FDA issued the draft guidance early this year; however, both guidance were written more like a technical report for statistician, and unlikely to be understood by the non-statisticians. For non-inferiority design, more questions were raised about the subjectivity / objectivity in determining the non-inferiority margin. For risk-benefit assessment, perhaps, we have to rely on the medical experts in the specific therapeutic area to make their subjective judgment based on the separate marginal analyses of Benefit (efficacy) and Risk (Safety) instead of different weighted modeling approaches. Perhaps, there is no simple mathematical and statistical solution for the benefit risk assessment. I believe that the advisory committee members were making subjective judgments based on their experience in voting in favor of or against a product for benefit and risk assessment - like Jury's verdict.

it is not a good thing that as statisticians, we come up with some complicated statistical methodology which we can not explain well to the non-statisticians. Eventually, we may need to go back to the basics to follow the KISS (Keep it simple) principle. Several years ago, the complicated and bad math that nobody could really understand caused the financial crisis. A working paper, Computational complexity and informational asymmetry in financial products, Sanjeev Arora, Boaz Barak, Markus Brunnermeier, Rong Ge. sheds some light on the complex mathematical models upon which credit default obligations and other derivatives are based.

Number of Events vs. Number of Subjects with Events

2010-09-19T09:49:00.000-04:00

Often in clinical trial safety data analysis, people are confused with the basic concept of "number of events" vs "number of subjects with events". Obviously, the number of events counts the events (event level) while number of subjects counts the subjects (subject level).

Using adverse event (AE) summary as an example,the difference between “the number of AEs” and “the number of subjects with AEs” sometimes may not be very obvious for some people. For the number of AEs, since the same subject can have more than one adverse events, we can not really calculate the percentage since the numerator and denominator could be any number. It is a mistake if you divide the number of events with the number of subjects under a treatment arm. You could have an unreasonably large percentage (sometimes larger than 100%).

For the number of subjects with AEs, we always count by subject. If a subject has more than one AEs, it will be counted one once. Therefore, the numerator (the number of subjects with AEs) is always smaller than the denominator (number of subjects exposed). We can calculate the percentage and the percentage should always be less than 100%. We can this percentage as 'incidence of AEs'. The following table (extracted from a document in FDA's website) is an example of AE presentation (counted by subjects).

The statistical summary tables for adverse events are often constructed to present the both total # of AEs and # of subjects with AEs (or precisely the number of subjects with at least one AEs). However, in the table, there will be no percentage calculated for total # of AEs. If the readers are not clear about the concept of "# of AEs" vs "# of subjects with AEs), they could question the correctness of the summary table. Very often, they might count the # of subjects with AEs and compare with the # of AEs and find discrepancies (sure there will be discrepancies). The reason? some subjects must have more than one AEs.

In some situations, we can indeed calculate the rate, proportion for # of Events (number of adverse events). For example, for total number of AEs, we could calculate how many of these events are mild, moderate, severe. You could see this information presented in package insert for some approved drugs on the market. We could also calculate the incidence rate of AEs by using the total number of AEs as numerator and total number of infusions, total number of dose distributed, or total number of person years as denominator. In these situations, we should always understand what the numerator is and what the denominator is. For a good presentation of the statistical summary table, the numerator and denominator used for calculation should be specified in the footnote. A few years ago, I saw a commercial presentation comparing a company's product safety with other competitive products. When they calculate the AE frequency for their product, they use (total number of AEs) / (the total number of doses). When they calculate the AE frequency for other competitive products, they use (total number of AEs) / (total number of subjects). Since each subject receives more than one doses, their calculation of the AE frequency for their product is markedly lower. However, this trick is wrong and unethical.

Restricted randomization, stratified randomization, and forced randomization

2010-09-12T15:38:00.000-04:00

Randomization is a fundamental aspect of randomized controlled trials (RCT). When we judge a quality of a clinical trial, whether or not it is a randomized trial is a critical point to consider. However, there are different ways in implementing the randomization and some of the terminologies could be very confusing, for example, 'restricted randomization', 'stratified randomization', and 'forced randomization'.

Without any restriction, the randomization is called 'simple randomization' where there is no block, no stratification applied. Simple randomization will usually not be able to achieve the exact balance of the treatment assignments if the # of randomized subjects are small. In contrary, the restricted randomization refer to any procedure used with random assignment to achieve balance between study groups in size or baseline characteristics. The first technique for restricted randomization is to apply the blocks. Blocking or block randomization is used to ensure that comparison groups will be of approximately the same size. Suppose we are planning to randomize 100 subjects to two treatment groups, with simple randomization, if we enroll entire 100 subjects, we may have approximately equal number of subjects in one of the treatment groups. However, if we enroll a small amount of subjects (for example 10 subjects), we may see quite some deviation from equal assignments and there may not be 5 subjects in each treatment arms. With the application of blocking (block size=10), we can ensure that with every 10 subjects, there will be 5 to each treatment arm.

Stratified randomization is used to ensure that equal numbers of subjects with one or more characteristic(s) thought to affect the treatment outcome in efficacy measure will be allocated to each comparison group. The characteristics (stratification factor) could be patient's demographic information (gender, age group,...) or disease characteristics (baseline disease severity, biomarkers,...). If we conduct a randomized, controlled, dose escalation study, the dose cohort itself can be considered as a stratification factor. With stratification randomization, we essentially generate the randomization within each stratum. # of strata depends on the number stratification factors used in randomization. If we implement 4 randomization factors with each factor having two levels, we will have a total of 16 strata, which means that our overall randomization schema will include a total 16 portions of the randomization with each portion for a stratum. In determining the # of strata used in randomization, the total number of subjects need to be considered. Overstratification could make the study design complicated and might also be prone to the randomization error. For example, in a stratified randomization with gender as one of the stratification factor, a male subject could be mistakenly entered as female subject and a randomization number from female portion instead of male portio nof the randomization schema could be chosen. This may have impact on the overall balance in treatment assignment as we originally planned. A paper by Kernan et al had an excellent discussion on stratified randomization.

One of the misconception about the stratification is that equal number of subjects are required for each stratum. for example, when we talk about randomization stratified by gender (male and female), people will think that we would like to have 50% of male and 50% of female subjects in the trial. This is not true. What we need is to (assuming 1:1 randomization ratio) have 50% of subjects randomized to each treatment arm in male subjects and in female subjects. This issue has been discussed in one of my old articles.

The forced randomization is another story and it basically to force the random assignment to deviate from the original assignment to deal with some special situation. For example, in a randomized trial with moderate and severe degree of subjects, we may put a cap on the # of severe subjects to be randomized. When the cap is achieved, the severe subjects will not be randomized any more, but the moderate subjects can still be randomized. We could enforce a cap for # of subjects at a specific country/site or limit the number of subjects for a specific treatment arm to be randomized at a particular country/site. The forced randomization is usually required to deal with the operation issues and is implemented through IVRS or IWRS. Too much forced randomization will neutralize the advantages of the randomization.

all three terms (restricted, stratified, and forced randomization) belong to the fixed sample size randomization in contrary to the dynamic randomization in adaptive designs.

Immunogenicity and its impact in clinical trials

2010-09-05T00:25:00.002-04:00

There is a shift in drug development field from the chemical compounds to the biological products - protein products and a shift from traditional pharmaceutical companies to biotechnology companies. For those who work on clinical trials on biological products, the term 'immunogenicity' must be a familiar term. Immunogenicity is the ability of a particular substance, such as an antigen or epitope, to provoke an immune response. In other words, if our drug is protein product, immunogenecity is the ability of the protein to induce humoral and/or cell-mediated immune responses. Immunogenicity testing is a way to determine whether patients are producing antibodies to biologics that can block the efficacy of the drugs. The development of anti-drug antibodies can also cause allergic or anaphylactic reactions, and/or induction of autoimmunity.

Several workshops have been organized to discuss the immunogenicity issues. Regulatory agencies are developing the guidelines to give industry the directions to incorporate the immunogenicity testing into the clinical development program for biological products.

Since the immunogenicity testing relies on the assay to measure the immune responses, the results of immunogenicity depend on which assay is used for testing. Therefore, the development of immunogenicity assay is also a critical issue in immunogenicity testing. Using an ultra sensitive assay could detect many false positives in immunogenicity testing. Using a less sensitive assay could under-estimate the immune response.

The following collection regarding immunogenicity testing should provide a good resource for this topic.

If a company develops a follow-on biologicals, a generic form of biological product or a copycat of a biotechnology product, immunogenicity testing is typically one of the critical points that need to be addressed. The regulatory, therefore, issued the guidelines on the immunogenicity testing when developing the follow-on biologicals.

Transparency - a change about drug approval information on FDA's website

2010-08-29T10:41:00.000-04:00

I recently noticed that for the approval of the new drugs (NDA) and biological products (BLA), the information about the approval process was published on FDA's website and in very timely fashion. Just a year ago, the FDA review/approval process regarding a new product was still not transparent to the public. We may be able to find some information on label, approval letter, and SBA (summary basis of approval); however, there were typically months or years after the approval.

Now, for the new approvals, not only the label, approval letter, and SBA, but also reviews from different perspectives (medical, statistical, pharmacology, enrivomental, CMC,...) as well as the administrative documents and correspondence between the FDA and the sponsor may be posted on FDA's website. Also listed or published is the list of FDA's officers who participated in the review and the decision making. The individuals from the sponsor's side may also be listed in some documents or correpondence.

This is obviously the outcome of FDA's initiative on transparency. "In June 2009, Food and Drug Administration (FDA) Commissioner Dr. Margaret Hamburg launched FDA's Transparency Initiative and formed an internal task force to develop recommendations for making useful and understandable information about FDA activities and decision-making more readily available to the public, in a timely manner and in a user-friendly format."

To feel these changes, we can just take a look at two products recently approved by FDA: one by CDER and one by CBER. Don't forget to visit "Administrative Document(s) and Correspondence" or "Approval History, Letters, Reviews, and Related Documents".

This is a good sign that FDA's drug approval process is being demystified and moving to the transparency.

LOCF, BOCF, WOCF, and MVTF

2010-08-17T23:34:00.001-04:00

In clinical trials, subjects are usually followed up for a period of time with multiple measurements / assessments at various time points. It is very common that some subjects will discontinue from the study early due to the reasons like 'lost to follow-up', 'withdraw consent', 'adverse events',...

With intention to treat population, imputation technique is needed to deal with the early termination subjects. While the fancy technique such as multiple imputation may be more statistically sound, some practical imputation techniques may be more popular. Here are some of them that I have used.

LOCF (last observation carried forward): this is probably the most common technique used in the practice in handling the missing data (especially for continuous measures). This is also the technique mentioned in ICH E9 "Statistical principles for clinical trials". It states "...Imputation techniques, ranging from the carrying forward of the last observation to the use of complex mathematical models, may also be used in an attempt to compensate for missing data..."
LOCF can be easily implemented in SAS. See a SUGI paper titled "The DOW (not that DOW!!!) and the LOCF in Clinical Trials"

BOCF (baseline observation carried forward): this approach may be more conservative if the symptoms are gradually improving over the course of the study. I used this technique in several clinical trials testing the analgesic drug (pain killer) in dental surgery patients. At the baseline right after the dental surgery, the pain scale is the worst. With time, the pain intensity is supposed to decrease. In this situation, BOCF technique is more conservative than LOCF. There is a web article to look at the feature of BOCF. BOCF along with LOCF and a modified BOCF are discussed in a most recent FDA advisory committee on Cymbalta for the Treatment of Chronic Pain

WOCF (Worst observation carried forward): this approach is the most conservative comparing to LOCF and BOCF. This technique has been used in analgesia drug as well as the trials with laboratory results as endpoint. For example, WOCF technique is mentioned in FDA Summary on Durolane.

LOCF, BOCF, and WOCF are handy technique for continuous measures. For a trial with endpoint as dichotomous variable (success vs failure; responder vs. non-responder),a technique called MVTF can be used. MVTF stands for missing value treated as failure. For example, this technique is mentioned in Statistical Review of NDA 21-385 in dermatology indication. In one of studies I participated, we employed the same technique (even though we did not use the term MVTF) to treat all subjects who discontinued from the study early as non-responders. This is a very conservative approach. The treatment effect may be neutralized a little bit during the implementation of this technique.

There are many other techniques used in the practice. Some of them may be just different terms for the same technique. In FDA Executive Summary Prepared for the
July 30, 2010 meeting of the Ophthalmic Devices Panel P080030, the following imputation techniques are mentioned.

Last Observation Carried Forward (LOCF) analysis

Best Reasonable Case analysis
Worst Reasonable Case analysis
Non-Responder analysis
Best Case analysis
Worst Case analysis

in practice, it is typically to employ at least two techniques in handling the missing data. This is part of so called 'sensitivity analysis'.

it must be pointed out that these practical missing data handling techniques have no statistical basis and have been criticized by many professionals especially in academic setting. These techniques that seem to be very conservative, may not be conservative in some situations.

Since LOCF is a technique used most, the critics are usually centered on the comparison of LOCF and other model based techniques (for example, Mixed-Effect Model Repeated Measure (MMRM) model). Some of the comparisons and discussions can be found at:

For continuous endpoints in longitudinal clinical trials, a good strategy may be to employ the mixed model or MMRM as the primary analysis and one of XOCFs as the sensitivity analysis. This is exactly the strategy used in NDA submission on Cymbalta for the Treatment of Chronic Pain

R-square for regression without intercept?

2010-08-07T01:14:00.000-04:00

Sometimes, simple linear regression may not be very simple. One of the issues is to decide whether or not to fit the regression with the intercept or without the intercept. For regression without intercept, the regression line goes through the origin. for regression with intercept, the regression line does not go through the origin.

In clinical trials, we may need to fit the regression models about the drug concentration vs. dose; AUC vs. trough concentration,...Regression with or without a intercept relies on the scientific background, not purely the statistics. Using the drug concentration vs dose as an example, if there is no endogenous drug concentration, a regression model without intercept makes sense. If there is a endogenous drug concentration, a regression model with intercept will be more appropriate - when there is no dose given, the drug concentration is not zero.

In some situation, regression models are purely data-driven or empirical. Choosing a model with or without an intercept may not be easy to decide. We recently had a real experience in this. With the same set of data, we fitted the models with intercept and without intercept. We thought we could judge which model was better by comparing the R-square values - an indicator for goodness of fit. Surprisely, the models without intercept were always much better than the models with intercept by comparing the R-squares. However, when we thought twice about this, we realized that in this situation, the R-square was no longer a good indicator of the goodness of fit.

The problem is that the regression model without intercept will always give a very high R-square. This is related to the way how the sum of squares are calculated. There are two excellent articles discussing this issue.

Comparing treatment difference in slopes?

2010-08-07T00:39:00.000-04:00

In regulatory setting, can we show the treatment difference by comparing the slopes between two treatment groups?

In a COPD study (e.g., a two arm, parallel group with primary efficacy variable measured at baseline and every 6 months thereafter), one can fit the random coefficient model and compare the treatment difference between two slopes. Also we can compare the treatment difference in terms of change from baseline to the endpoint (the last measure).

To test the difference in slopes, we would need to test whether or not the treatment*time interaction term is statistically significant. The assumption is that at the beginning of the trial, the intercept for both groups are the same - both groups start at the same level. Then if the treatment can slow the disease progression, the treatment group should show a smaller slope comparing with the placebo group. If all patients are followed up to the end of the study, if the slopes are different, the endpoint (change from baseline) analysis should also be statistically different. However, if the sample size is not sufficiently large, the results could be inconsistent by using slope comparison approach vs. endpoint analysis approach. For a given study, the decision has to be made which approach is considered as the primary endpoint. If we analyze the data using both approaches, we will then need to deal with the adjustment for multiplicity issue.

I used to make a comment saying "some regulatory authorities may prefer the simpler endpoint analysis"; I was then asked to provide the references to suport this statement. I did quite extensive research, but could not find any real relevant reference. However, by reviewing 'statistical reviews' in the BLA and NDA in US, it is very rare to see any product approval based on the comparison of the slopes. Many product approvals are based on the comparison of 'change from baseline'.

Every indication has its own accepted endpoints so the tradition takes precedence. For example, in Alzheimer's disease, there is a movement to look at differences in slopes, but this is based on trying to claim disease modification. Similarly, in the COPD area, some products are based on disease modification, the treatment differnces can be shown by comparing the differences in slopes between treatment groups.

It seems to be true that that the slope model (random coefficient model) may be preferred in academic setting, but endpoint approach - change from baseline (with last value carried forward) may be more practical in the industry setting.

From the statistical point of view, the slope approach makes a lot of sense, however, we need to be cautious about some potential issues: 1. In some efficacy measures, there might be some type of plateau. If the plateau is reached prior to the end of the study, there will be a loss of power comparing slopes.2. If the slope comparison is used as the primary efficacy measure, the # of measurements per year on the primary efficacy variable is relevant. One may think that the more frequent measures will increase the power to show the treatmetn difference in slopes. The question arise when designing the study: should we choose a shorter trial with more frequent measures? or should we choose a longer trial with less frequent measures?

Biosimilars - Generic Version of Biological Drugs

2010-06-13T15:07:00.000-04:00

The health reform legislation that was recently signed into law contains a provision that creates a pathway to enable the US Food and Drug Administration (FDA) to approve biosimilars - generic versions of biologic drugs. Unlike generic small molecule drugs, which is the synthetic chemical compounds, the complexity of biologic drugs makes it questionable whether a generic company could produce an identical biologic product. WSJ recently reported that Merck decided to end efforts to copy Amgen's blockbuster Aranesp anemia drug, which showed how the emerging field of developing generic versions of biotechnology therapies won't be easy to enter. For small molecule chemical compounds, once the patent is expired, the generic companies can start to manufacture the same ingredients (active pharmaceutical ingredients (APIs)) for the generic version of the brand products. The approval of the generic version typically requires only the bioavailability/bioequivalence tests in healthy volunteers. For biological products, it is typically the large proteins with 3-D or 4-D structures. The same protein with different 3-D structure may have different functions. This makes the biosimilars very difficult to copy - similarity does not mean the same therapeutic effects. When we deal with the proteins, we will always need to deal with the issues of immunogenicity. Some of the biological products (for example the plasma-derived products) can not be tested in the healthy volunteers. If a bioequivalence study is required, it is typically done in real patients, which makes the trial much more expensive than typical bioavailability/bioequivalence trial in healthy volunteers.

Further reading:

Some clarifications on Non-inferiority (NI) Clinical Trial Design

2010-05-29T08:44:00.000-04:00

I noted that FDA recently issued its draft guidance on non-inferiority clinical trial design. Two weeks ago, I attended the DIA webinar "understand the primary challenges facing non-inferiority studies" that featured the presentations by Drs Bob Temple, Bob O'Neill, and Ed Cox from FDA. Several issues that often prevent us from thinking about the use of NI trial are now clarified:

1. Can we use an active control when the active control is not approved for the indication, but used as standard care (off-label)?

This was answered in section V of the draft guidance. "The active control does not have to be labeled for the indication being studied in the NI study, as long as there are adequate data to support the chosen NI margin. FDA does, in some cases, rely on published literature and has done so in carrying out the meta-analyses of the active control used to define NI margins. "

2. When literatures are used to support the choice of NI margin, what if the endpoints are different from various historical studies?

in Section V of the draft guidance, it says "...among these considerations are the quality of the publications (the level of detail provided), the difficulty of assessing the endpoints used, changes in practice between the present and the time of the studies, whether FDA has reviewed some or all of the studies, and whether FDA and sponsor have access to the original data. As noted above, the endpoint for the NI study could be different (e.g., death, heart attack, and stroke) from the primary endpoint (cardiovascular death) in the studies if the alternative endpoint is well assessed".

3. What if there is no historical clinical trial that directly compare the active control vs. Placebo?
We would typically think that if this is a situation, NI study design is not an option any more because there is no way to estimate the NI margin (precisely M1 margin). However, Dr. Ed Cox presented an example during the webinar to estimate the non-inferiority margin in an indirect way. While there is no clinical trial directly comparing the active control with Placebo, we can still estimate the treatment effect of active control by search for evidence separately for active control group and for Placebo group. For example, in anti-infective area, a lot of antibiotics have been used for many years (perhaps even before FDA is formed). There might never been a formal clinical trial to show that certain antibiotics are better than placebo. Now in pursuing a new antibiotics for indication, a placebo controlled study is not ethical (since other antibiotics products are the standard care). In order to conduct a NI study, it is challenging in choosing the NI margin. The suggestion from Dr Ed Cox's presentation is to derive estimate of effect of active control over placebo by:

Estimate the placebo response rate or the response rate if untreated
Estimate the response rate in the setting of "inadequate" or "inappropriate" therapey
Estimate the response rate of the active control therapy from literatures with active control therapy information

A recent paper in 'Drug Information Journal" detailed the similar approach. See the link below for the paper "Noninferiority margin for clinical trials of antibacterial drugs for Nosocomial Pneumonia". FDA's guidance on "Community-Acquired Bacterial Pneumonia: Developing Drugs for Treatment" also has a section for non-inferiority margin issue in antibacterial drug.

Notice that for each estimation, a collection of literatures are analyzed and often the meta analysis is required. The meta analysis may require the random effect estimate. In Dr Cox's presentation, DerSimonian Larid random effects estimates is used. This approach is described in the original paper as well as many books on meta analysis (for example, "meta analysis of controlled clinical trials" by Anne Whitehead). My colleague wrote the SAS program for this approach. A SAS paper compared the results from DerSimonian approach with the results from SAS Proc Mixed and NLMixed.

Hodges-Lehmann Estimator

2010-05-16T16:30:00.000-04:00

According to Wikipedia, "the Hodges–Lehmann estimator is a method of robust estimation. The principal form of this estimator is used to give an estimate of the difference between the values in two sets of data. If the two sets of data contain m and n data points respectively, m × n pairs of points (one from each set) can be formed and each pair gives a difference of values. The Hodges–Lehmann estimator for the difference is defined as the median of the m × n differences.
A second type of estimate which has also been called by the name "Hodges–Lehmann" relates to defining a location estimate for a single dataset. In this case, if the dataset contains n data points, it is possible to define n(n + 1)/2 pairs within the data set, allowing each item to pair with itself. The average value is calculated for each pair and the final estimate of location is the median of the n(n + 1)/2 averages.(Note that the two-sample Hodges–Lehmann estimator does not estimate the difference of the means or the difference of the medians (it estimates the median of the differences, which, if the underlying distributions are asymmetric, is a different quantity), while the one-sample Hodges–Lehmann estimator does not estimate either the mean or the median.)"

I first time heard this estimator was in a pharmacokinetic bioequivalence study where we had to compare the Tmax between two groups. Typically, we don't need to compare the Tmax between treatment groups since the bioequivalence is typically based on AUC (area under plasma-concentration curve) and/or Cmax (maximum concentration). Assessment of t_max was mandatory only if

either a clinical claim was made (e.g., rapid onset like for some analgetics),
or based on safety grounds (e.g., IR nifedipine).

Tmax is the time to reach the maximum concentration (Cmax) after the drug administration. Tmax data is certainly not following the normal distribution and is usually taking only several pre-specified the sampling time point (depending on how many time points are specified in obtaining the PK profile).In this case, a distribution free non-parametric test needs to be used. Hodges-Lehmann estimator can fit into this situation. In addition to Tmax, Hodges-Lehmann can also be used to test the difference for Thalf (t1/2).

In old days, we have to write the SAS program by ourselves. In the latest version of SAS 9.2, Proc NPAR1WAY can be used for calculating the Hodges-Lehmann estimator and its confidence interval. See Hodges-Lehmann Estimation of Location Shift for details about the calculation and an example of "Hodges-Lehmann Estimation" from SAS website.

With HL statement and Exact HL statement in SAS Proc NPAR1WAY,Hodges-Lehmann estimator (location shift) can be estimated and its confidence intervals (asymptotic (Moses) for large sample and Exact in small sample situation) are provided. However, SAS procedure does not provide the p-value. The p-value may be obtained from Wilcoxin Rank Sum test.

Stronger Bioequivalence Standard?

2010-05-09T14:26:00.000-04:00

In April, 2010, "the Pharmaceutical Science and Clinical Pharmacology Advisory Committee (104715) (PSCPAC)" discussed the issues related to the bioequivalence standard.

"The statistical analysis and acceptance criteria seem to be the most confusing aspects of regulatory bioequivalence evaluation. The current statistical analysis, the two one-sided tests procedure, is a specialized statistical method that is capable of testing for “sameness” or equivalence between the two comparator products. The pharmacokinetic parameters, calculated from the bioequivalence study data, area under the plasma concentration-time curve, (AUC) and maximum plasma concentration (Cmax) represent the extent and rate of drug availability, respectively. All data is log-transformed and the analysis of variance (ANOVA) is used to calculate the 90% confidence intervals of the data for both AUC and Cmax. To be confirmed as bioequivalent, the 90% confidence intervals for the test (generic product) to reference (marketed innovator product) ratio must fall between 80 to 125%. This seemingly unsymmetrical criteria is due to the logtransformation of the data."

However, this one-size-fits-all approach may not be adequate for all pharmaceutical products. One category of the pharmaceutical products is called "critical dose (CD) drugs". CD drugs are also called "narrow therapeutic index (NTI) drugs" and are medicines for which comparatively small differences in dose or concentration may lead to serious therapeutic failures and/or serious adverse drug reactions. It is reasonable to assume that a more stringent bioequivalence criteria should be employed to ensure the safety of the product.

According to the voting results from advisory committee, advisory committee agreed that CD drugs are a distinct group of products; the FDA should develop a list of CD drugs; and the current BE standards are not sufficient for CD drugs.

The FDA proposes that in addition to 80-125% criteria based on 90% confidence interval, a limit of 90-111% on the geometric mean (point estimate) of all BE parameters (i.e., Cmax, AUC0-t, AUC0-∞) is added to the more stringent bioequivalence criteria. However, this proposal was not agreed by the advisory committee. Panelists commented that the scientific basis for the proposed limit of 90-111% was not justified. Some members specified that they did not favor use of Cmax in the proposal, but likely would have been swayed if it focused solely on AUC.

To claim the bioequivalence, should both AUC and Cmax meet the bioequivalence criteria? While regulatory guidance mentioned that AUC and Cmax are typically parameters for evaluating bioequivalence, there is no guidance formerly requiring that both AUC and Cmax have to be demonstrated. In some situation, Cmax may not be applicable in showing the bioequivalence. For example, when comparing the drug giving in different administration routes (intravenous vs subcutaneous), equivalence in AUC could be established while equivalence in Cmax could not be established.

CDISC beyond the data

2010-05-02T22:58:00.000-04:00

CDISC stands for the Clinical Data Interchange Standards Consortium. I have always been thinking that the CDISC is about the data standard and data structure and has nothing to do with the protocol, case report form and so on.

However, for the last several years, CDISC has expanded its reach into the entire flow of the clinical trial. For each step in clinical trial, there is its counterpart in CDISC standard.

Protocol: Protocol representation model (PRM)
Case Report Form: The Clinical Data Acquisition Standards Harmonization (CDASH)
Data management data set: The Study Data Tabulation Model (SDTM)
Analysis data set: Analysis data model (ADaM)

The protocol representation model is pretty new and is just recently released. PRM is actually now a subdomain of the BRIDG model. PRIDG stands for Biomedical Research Integrated Domain Group and is a collaborative effort engaging stakeholders from the Clinical Data Interchange Standards Consortium (CDISC), the HL7 Regulated Clinical Research Information Management Technical Committee (RCRIM TC), the National Cancer Institute (NCI) and its Cancer Biomedical Informatics Grid (caBIG®), and the US Food and Drug Administration (FDA).

For each clinical trial, the study protocol is the key. The study protocol is typically a text document and is developed from the protocol template. The protocol is considered as a document, not a data. PRM is trying to change this.

The PRM is NOT a specific protocol template; rather, when a template is designed to meet the purposes of a given organization or study type, the use of the PRM common elements will enable and facilitate information re-use without constraining the design of the study or the style of the document. The PRM elements have been found to be typical across study protocols, but they do not reflect either a minimum or a maximum set of elements.

There are four major components of the PRM v1.0—that is, four major areas of a protocol that the elements are related to:

Clinical Trial/Study Registry: Elements related to the background information of a study, based on the requirements from WHO and Clintrials.gov. Examples of elements in this area include Study Type, Registration ID, Sponsors, and Date of First Enrollment.

Eligibility: Elements related to eligibility criteria such as minimum age, maximum age, and subject ethnicity.

Study Design Part 1: Elements related to a study’s experimental design, such as Arms and Epochs.

Study Design Part 2: Elements related to a study’s Schedule of Events and Activities.

It is envisioned that with PRM, the key elements of the protocol can be considered as data strings and can be stored in the data set and can be re-used. The statistical analysis plan can be easily developed by importing the key elements from the protocol. However, to make all companies to follow this standard will take time. There may be a lot of challenges in implementing this standard. This standard needs to be endorsed by the medical writers and medical directors (not the data managers and statisticians) who actually develop the study protocol.

China's Regulations on Drug and Biological Products Registration

2010-04-18T11:57:00.000-04:00

In China, drug and biological products are regulated by SFDA (国家食品药品监督管理局). sFDA is a counterpart of US FDA. Its Center for Drug Evaluation (药品审评中心) regulates both drug and biological products (sort of combination of US FDA's CDER and CBER divisions).

Law and Guidance:

Drug Administration Law: Dec 2001

All clinical trials should be pre-approved by sFDA
All clinical trials should be carried out by qualified investigators
Detailed procedures and technical data should be submitted

Regulations for Implementation of the Drug Administration Law: Sep 2002

Good Clinical Practice: Aug 2003 (In Chinese)

Statistical Guidelines for Clinical Trials of Drugs and Biologics: Mar 2005 (in Chinese)

Pharmacokinetics and Bioequivalence: 2005 (化学药物制剂人体生物利用度和生物等小等效性研究技术指导原则)

Toxicology: 2005 (化学药物长期毒性试验技术指导原则)

Hong Kong; GCP for Proprietary Chinese Medicines: Feb 2004 (PDF669KB)

Drug Registration Regulation (药品注册管理办法): Jul 2007 and its appendices

It is interesting that in its appendices, there are requirements for sample size. For a new or an imported drug applications, the sample size should meet the statistical requirement and the minimal cases required. For category I and II (new drugs), the minimal cases required (trial group exposure): 20-30 for Phase I, 100 for Phase II, 300 for Phase III, 2000 for Phase IV. For category III and IV (imported drugs), trials should have at least 100 pairs. In the event of more than one indication, cases for each main indication shall be at least 60 pairs.

Further reading:

Chen, Feng, Chen, Qiguang, Yu, Hao, Chen, Jie, Hsu, Jason (2008). Current statistical requirements for pharmaceutical clinical trials In China. Drug Information Journal 42, pp. 321-330.

When to Finalize the Statistical Analysis Plan (SAP)?

2010-04-11T12:04:00.000-04:00

Recently, a group of statisticians in Linkedin.com (presumabally all working in drug development industry) discussed the following posted questions:
"A client wants me to prepare final SAP shortly after protocol and CRFs are finalized for a Phase 3 trial, to submit to FDA prior to start of study. I find this unusual. Any experience doing so? When?"

There are responses like "I do not see why SAP need to be finalized until it is time to lock the database and unblind"; "Why do you want to wait? What will you learn or gain by waiting?"...

First of all, let's look at the ICH guidance (E9 Statistical Principles for Clinical Trials):

"The statistical analysis plan may be written as a separate document to be completed after finalising the protocol. In this document, a more technical and detailed elaboration of the principal features stated in the protocol may be included. The plan may include detailed procedures for executing the statistical analysis of the primary and secondary variables and other data. The plan should be reviewed and possibly updated as a result of the blind review of the data (see 7.1 for definition) and should be finalised before breaking the blind. Formal records should be kept of when the statistical analysis plan was finalised as well as when the blind was subsequently broken.
If the blind review suggests changes to the principal features stated in the protocol, these should be documented in a protocol amendment. Otherwise, it will suffice to update the statistical analysis plan with the considerations suggested from the blind review. Only results from analyses envisaged in the protocol (including amendments) can be regarded as confirmatory."

This indicated that the ICH principal is followed as long as the statistical analysis plan is finalized or signed off prior to the study unblinding (or database lock if it is open label study). I believe this is the common practice in industry.

There is certaily a trend to push for SAP signoff prior to the study start, especially for late stage trials or for trials with complicated statistical analysis.

For early phase exploratory trials,one of the purpose is to explore the adequate endpoint; control group, study design, sample size, study issues,... for the late confirmatory trials, it is acceptable not to finalize the statistical analysis plan too earlier. If it is phase III, confirmatory trial (or new term A&WC - adequate and well controlled study), it is better to have SAP signoff earlier.

If the study design is complicated or the statistical analysis is complicated (for example using Beyesian approach; using non-inferiority margin; using adaptive design,...), the statistical analysis section in the study protocol may not be sufficient and a detailed statistical analysis plan may have to be sent to FDA at the time of protocol submission.As one of the members from Linkedin commented "The more important a protocol is to the NDA/BLA (i.e., a pivitol trial), the sooner you should get it in front of the FDA for comments."

Another point is that SAP has mainly two parts: the text portion and the mock shells. We may just need to finalize the text portion of the SAP prior to the study start and design the mock up shells after the CRFs, annotations, and sample data are available. In reality, every study protocol contains a section for statistical analysis. The key elements for statistical analysis should be included in this section. If the statistical analysis section is not detailed enough, the expanded statistical analysis section (the text portion of SAP) should detail the things like: prespecified analysis method/statistical model; missing data handling and imputation; prespecified interim analysis plan/method; multiplicity adjustment to p value; justification for non-inferiority margin; detail adaptation method, detail bayesian method, protection of blinding; inclusion of subjects in study population,...).

SAP could become a very lengthy document. here is an example of a SAP with bayesian analysis component from FDA's website.

Safety and Effectiveness of the Alair® System for the Treatment of Asthma: A Multi-center Randomized Clinical Trial (Asthma Intervention Research (AIR2) Trial)

Several weeks ago, I attended DIA/FDA's workshop on "Adaptive design clinical trials - discussion on FDA's draft guidance". FDA has expressed the great concern about the operational biases and the study integrity if the adaptive designs (especially those not not well accepted) are used. FDA's draft guidance on adaptive design has a specific section discussing "Role of the Prospective Statistical Analysis Plan in Adaptive Design Studies"

"The importance of prospective specification of study design and analysis is well recognized for conventional study designs, but it is of even greater importance for many of the types of adaptive designs discussed in sections V and VI, particularly where unblinded interim analyses are planned. As a general practice, it is best that adaptive design studies have a SAP that is developed by the time the protocol is finalized. The SAP should specify all the changes prospectively planned and included in the protocol, describe the statistical methods to implement the adaptations, describe how the analysis of the data from each adaptive stage will be incorporated into the overall study results, and include the justification for the method of control of the Type I error rate and the approach to appropriately estimating treatment effects. The SAP for an adaptive trial is likely to be more detailed and complex than for a non-adaptive trial. Any design or analysis modification proposed after any unblinded interim analysis raises a concern that access to the unblinded data used in the adaptations may have influenced the decision to implement the specific change selected and thereby raises questions about the study integrity. Therefore, such modifications are generally discouraged. Nonetheless, circumstances can occur that call for the SAP to be updated or for some other flexibility for an unanticipated adaptation. The later in the study these changes or updates are made, the more a concern will arise about the revision’s impact. Generally, the justifiable reasons to do so are related to failure of the data to satisfy the statistical assumptions regarding the data (e.g., distribution, proportionality, fit of data to a model). In general, it is best that any SAP updates occur before any unblinded analyses are performed, and that there is unequivocal assurance that the blinding of the personnel determining the modification has not been compromised. A blinded steering committee can make such protocol and SAP changes, as suggested in the ICH E9 guidance and in the DMC guidance, but adaptive designs open the possibility of unintended sharing of unblinded data after the first interim analysis. Any design or analysis modifications made after an unblinded analysis, especially late in the study, may be problematic and should be accompanied by a clear, detailed description of the data firewall between the personnel with access to the unblinded analyses and those personnel making the SAP changes, along with documentation of adherence to these plans. Formal amendments to the protocol and SAP need to be made at the time of such changes (see 1377 21 CFR 312.30)"

Hockey stick phenomenon

2010-04-04T17:09:00.000-04:00

Hockey stick phenomenon or hockey stick curves has been used mostly in describing the climate change. it says that the tempeature variation over centuries are relatively unchanged until after 1900. The temperature rose sharply due to the human activities. Since 1998 Nature article by Mann, Bradley, and Hughes, the hockey stick curve (phenomenon) has stirred quite some debates / contraversies in climate research fields.

Hockey stick curves have also been used in described any change with a normal trend (trajectory), then with a different change or a interruption in the trend. For example, the hockey stick curve may be used to describe the disease progression with gradual progression, then sudden deterioation. In clinical trials, one could observe that patients have initial rebound in the measured parameters (endpoints), then gradually decrease. In clinical trials for Alzheimer disease, the purpose is to prevent the paitent from further deterioation, rather than improvement or cure. If a rebound during the initial phase of the trial, it could be described as 'hockey stick".

During my PhD study, I analyzed the EPA whole effluent toxicity testing data and noticed the non-linear dose response and the phenomenon of 'hormesis' which says that exposure to low or very low dose of toxicants could have benefit effects. The hormesis or low dose response could be described as J-shaped or Hockey stick.

Way before the hockey stick model was used to describe the temperature data, the method was proposed to analyze the data in environmental health data. In 1979, Yanagimoto and Yamamoto published their paper in Environmental Health Perspectives titled "Estimation of Safety Doses: Critical Review of Hockey Stick Regression Method".

From data analysis standpoint, if the data presents with hockey stick phenomenon, the typical linear regression can not be used. Hockey stick regression can be considered as segmented linear regression with just one knot. In a paper by Simpson et al "excess risk thresholds in ultrasound safety studies: statistical method for data on occurrence an dsize of lesions", they used a piecewise linear model. A link from UGA had some discussions about using SAS procedures to model the data with hockey stick:

One thing for sure is that hockey stick could always be contraversial. Additional data may be needed to verify if the hockey stick phenomenon is true or is the data issue or is the data collection issue.

Health Care Globalization and Patients Without Borders

2010-03-30T21:58:00.000-04:00

The term 'globalization' is nothing new. According to Wikipedia, "globalization describes an ongoing process by which regional economies, societies, and cultures have become integrated through a globe-spanning network of communication and trade. The term is sometimes used to refer specifically to economic globalization: the integration of national economies into the international economy through trade, foreign direct investment, capital flows, migration, and the spread of technology." Globalization certainly has its benefits, but it has its victims too, and the results can be deadly. As the global economy knits countries closer together, it becomes easier for diseases to spread through states, over borders and across oceans.

Globalization has impact on the medicines we take (many of them are manufactured outside of a specific country) and the conduct of the clinical trials (the clinical trial data are cross borders from multiple nations). Last year, when I attended the FDA/Industry Statistical Workshop, the theme of the workshop is 'global harmonization' - another way to say 'globalization'.

Recently I attended a conference in Duke, the focus again was 'globalization' with emphasis on Asia. One session discussed the tourism medicine and 'patients without borders'. It will be trend that with globalization, patients can cross border to choose the health care that will better service them (with cost and quality of care in mind). One day, we could share the health care resources much like the sharing of the technologies.

I also understand that the sharing of the health care resource will not be an easy task. Several days ago, one of my American colleagues asked me if it is possible for foreigners to have renal (kidney) transplantation in China (for obvious reason of the shortage in kidney donors). When I posted the question to my alumni email list, I immediately got some response such as the one below "I believe that all Chinese with renal failure have the absolute right for having kidney transplant in China. As a Chinese, I strongly against any give away of basic human right..." I sort of agree with this. The world is not ready to share the health care resource (at least the organs for transplant).

Within the country, there may or may not be any policy or procedure to ensure the fairness between the rich and the poor, not to mention the fairness across countries.

China actually has its policies on organ transplantation including renal
(kidney) transplantation.The policies basically prohibit the tourism medical treatment in China for organ transplantation.

卫生部办公厅关于境外人员申请人体器官移植有关问题的通知 (General Office of the Ministry of Health personnel for human organ transplants outside the Issues)

人体器官移植条例 (Human Organ Transplant Ordinance)

Dealer's choice

2010-03-18T17:57:00.000-04:00

According to dictionary.com, the term "dealer's choice" is defined as "

a card game, as poker, in which the dealer decides what particular game is to be played, often depending on the number of players, and designates any special variations or unusual rules, including setting the stakes."

In contrary to the randomized clinical trial, an approach called "dealer's choice" has now been used in clinical trials or in clinical setting (more often in oncology clinical trials) where the treating physician decides which treatment regimen will be given to the patients/subjects. Typically, this happens when there are several treatment regimens, but there is no sufficient information to decide one is better than others.

While this is ok for treating the patients in clinical setting, it is not ideal when this approach is used in clinical trial. # of subjects who receive on specific treatment regimen will depend on the dealer's choice and may be imbalanced in terms of the sample size and the subject characteristics across different regimens. At the end of the study, if there is any difference among treatment regimens, there may be due to the 'unknown' confounding factor rather than the treatment itself.

Here are some of the examples using "dealer's approach":

An intergroup rectal cancer trial supported by NCI employed the dealer's choice approach in which treatment would be chosen by the physician and patient together.

Interventional Management of Stroke (IMS) III Trial (IMSIII) did not explicitly use the term "dealer's approach". However, in the investigational treatment arm, after initial tPA treatment fails, the doctor will choose--based on the location and extent of the blood clot--one of 4 possible IA treatments given directly in the brain artery that will be most effective in reopening the blocked artery. There is no randomization. It is dealer's choice in choosing one of the four treatment options.

There may be legitimate reason in using "dealer's choice" approach, it certainly compromise the quality of the trial and make the interpretation of the results more difficult.

Ghost writer, ghost surgeon,...

2010-03-13T17:50:00.001-05:00

"Ghost writing" practice has been criticized recently due to the reveal of details targeting several big pharmaceutical companies. With medical ghostwriting, pharmaceutical companies pay both professional writers to produce papers and then pay other scientists or physicans to attach their names to these papers before they are published in a medical or scientific journals."Ghost writer" may never be acknowledged in the publication. Wyeth was actually sued over the ghost writing practice (see reports from NPR and New York Times). Other companies had similar practices: for example, Merck's case and GSK. New York Times recently reported that Drug Maker Wrote Entire Book Under 2 Doctors’ Names.

“To ghostwrite an entire textbook is a new level of chutzpah,” said Dr. David A. Kessler, former commissioner of the Food and Drug Administration, after reviewing the documents. “I’ve never heard of that before. It takes your breath away.”

Actually, "ghost writing" practice has been out there for many years and it is not just in pharmaceutical industry. Celebrities, executives, and political leaders often hire ghostwriters to draft or edit autobiographies, magazine articles, or other written material.No wonder they can publish nice books and articles.

Perhaps, there is nothing wrong in terms of the business model. Medical writer and freelance writer get pays for their services and whether or not being acknowledged is not important to them.

Recently, I heard a new term "ghost surgery": a practice of performing surgery on another phycian's patient by arrangement with the physician but unknown to the patient.A famous surgeon could make an arrangement to have a substitute (perhaps a resident) to perform the surgery without patient's knowledge (patient could be unconscious). What can you do about this practice? can you sue? Not necessarily.

Ghost writer and ghost surgeon are certainly not all 'ghost' out there. There are far more 'ghost' in our daily life and in business practice.

Non-inferiority clinical trials - now comes FDA's draft guidance

2010-03-06T17:08:00.000-05:00

In contrary to adaptive design clinical trials, non-inferiority (NI) design has been there for many years (15-20 years) and many studies have been conducted using non-inferiority design. Many products have been approved by regulatory agencies with based on the pivotal studies in non-inferiority design.

Even though there have been tons of presentations, workshops, and text books about the non-inferiority clinical trials, there is no formal guidance from FDA regarding the inferiority clinical trials (until this week).

This week, FDA issues its draft guidance for industry: non-inferiority clinical trials. The guidance gives advice on when NI studies can be interpretable, on how to choose the NI margin, and how to analyze the results.

NI is considered a special case of equivalence trial with NI comparing to only lower or upper bound of the confidence interval (but not both). While there are many other issues to be considered when design a NI trial, the selection of the NI margin continue to be a key issue in NI design. For a confirmatory trial for licensure, NI margin must be discussed with regulatory agencies and must be agreed by them. Generally, the wider the margin, the easier the NI trial to be successful. the smaller the margin, the larger the sample size.

It is good to see that the guidance includes a question/answer on "in the situation where a placebo-controlled trial would be considered unethical, but a non-inferiority study cannot be performed, what are the options?" Unfortunately, the answer to this question is somehow not clear.

Further readings:

Slide presentation by Bob Temple "FDA experience and perspective on non-inferiority trials"
EMEA "Guideline on the choice of the non-inferiority margin"
EMEA "Points to consider on switching between superiority and non-inferiority"
Slide presentation by Bob Temple "Active control non-inferiority studies theory, assay sensitivity, and choice of margin"
Mary Foulkes "non-inferiority specification of delta"
Lin et al "Statistical issues in specification of delta"

Adaptive design clinical trials - now comes the FDA's draft guidance

2010-02-25T23:16:00.004-05:00

For last several years, 'adaptive design' and 'adaptive clinical trials' are hot topics in biostatistics and clinical trial fields. The industry seems to think that 'adaptive design' is the solution for lengthy and costly drug development program. There are workshops, symposiums, many publications and books about the adaptive design in last several years. However, in reality, many examples, case studies are post hoc and based on the simulation from the historical non-adaptive clinical trial data ("had we implemented the adaptive design, we would have saved xxx time/cost/sample size..."). Recently we do see some implementation of adaptive designs in real clinical trials, but

these are mainly in early stage studies.

This month, FDA issues its guidance for industry on "adaptive design clinical trials for drugs and biologicals". This lengthy guidance contains plenty of references (perhaps with most references for a FDA guidance). Overall tone of this guidance seems to advise the sponsors be cautious in adopting the adaptive design especially those not well established designs. If you don't want to read the entire guidance, slides by two of the FDA working group members summarize the key points in the guidance.

Ahead of FDA, EU has already issued its guidance on adaptive design (or flexible design) . Their opinions have been laid out in EMEA’s REFLECTION PAPER ON METHODOLOGICAL ISSUES IN CONFIRMATORY CLINICAL TRIALS WITH FLEXIBLE DESIGN AND ANALYSIS PLAN issued in 2006.

There was also a well publicized workshop on adaptive design in 2006 in US. However, we have to wait for four years to see FDA's draft guidance. For documentation, all materials from 2006 workshop are now on the web.

Some of the topics are on technical sides, but there are also talks about the perspectives from FDA regarding the adaptive designs.

Dr Robert Temple from FDA regarding “Myth Busting – Clinical”
Dr Robert O’Neill from FDA regarding “Myth Busting – Statistical”

SAS procedures for group sequential design

2010-02-19T22:45:00.001-05:00

Group sequential design has been proposed for a while perhaps since Pocock's paper in 1977. At least for last two decades, the group sequential design and interim analysis have been used in may major clinical trials.

Several years ago, I tried to purchase Cytel's EAST program. But I gave up due to the price and the infrequent use in day-to-day practices.

Now I am pretty happy to know that SAS version 9.2 includes two new procedures for group sequential design. Proc SEQDESIGN allows to calculate the boundaries and Proc SEQTEST allows to perform the tests during the interim analysis whether or not the boundaries have been reached. Proc SEQTEST can also be used to calculate the conditional power (Probability of observing a significant result at full information, given the current data and the specified alternative under the statistical design ) and predictive power.

Reference readings:

Yang Yuan "Group Seuqential Analysis Using the New SEQDESIGN and SEQTEST procedures"

Schulz and Grimes (2005) Multiplicity in randomized trials II: subgroup and interim analyses. Lancet

Cholesterol Drug Approved for People Without High Cholesterol

2010-02-14T14:36:00.001-05:00

Are you willing to take Statin medication if you have normal or mild elevation of LDL-Cholesterol level, but with elevated CRP?

Last week, FDA approved new indication for Crestor - one of the Statin class cholesterol lowering drug. This new indication has nothing to do with cholesterol level, but it is based on the CRP (c-reaction protein). Wall Street Jounal Health Blog had an article titled "Cholesterol drug approved for people without high cholesterol". It seems to me that we have now invented another disease - perhpas hyperCRP (instead of hypercholesterolemia). American are now perhaps inventing more diseases/indications than ever before. In the end, both cholesterol level and CRP level are surrogate endpoint. While the relationship between high cholesterol level and the major cardiovascular events (mortality, MI, stroke,...) has been generally recognized, the relationship between high CRP and the major cardiovascular events mainly relies on one study - JUPITER study - a study stopped early for efficacy.When the conflict of interest issue is considered, the purpose of the study is questioned and skeptical. According to the web blog, there are two conflict of interest issues in this study: "first, the study was funded by AstraZeneca, maker of the study drug, rosuvastatin (Crestor); second, the first author, Paul Ridker of Harvard, owns a patent on the high-sensitivity test for C-reactive protein, the test that would be widely used if the study results are accepted."

It will be inevitable that the next step for pharmaceutical companies like AstraZeneca is to push for routine testing of CRP (CRP screening) in clinical practice to identify the patients with normal or mild elevated LDL-Cholesterol level, but with elevated CRP. A cut point (or normal range) for CRP will be established. This is just how it works in capitalism world.

To find out how convincing of the evidences from JUPITER trial, you can read the publications by yourself. While the evidences seem to be convincing, I just don't want to take medications just for the elevated CRP. Perhaps, instead of the benefit from taking long-term treatment of Crestor, the treatment effect is really due to the detrimental effect of long-term treatment of Placebo. Perhaps, the treatment effect will be gone if we compare the Crestor with no-treatment (instead of Placebo).

Now there is even a study claiming that the Statin prevention in patients with elevated c-reaction protein deems to be cost-effective.

Cost-benefit analysis: put a dollar value on human life?

2010-02-01T15:40:00.000-05:00

Putting a Price Tag on Life: Today, companies and governments often use Jeremy Bentham’s utilitarian logic under the name of “cost-benefit analysis.” In Michael Sandel's lecture at Harvard, he presents some contemporary cases in which cost-benefit analysis was used to put a dollar value on human life. The cases give rise to several objections to the utilitarian logic of seeking “the greatest good for the greatest number.” Should we always give more weight to the happiness of a majority, even if the majority is cruel or ignoble? Is it possible to sum up and compare all values using a common measure like money?

Ford Pinto Case
One of the examples he used is the cost-benefit analysis in Ford Pinto case. According to Wikipedia,the Ford Pinto model became a focus of a major scandal when it was alleged that the car's design allowed its fuel tank to be easily damaged in the event of a rear-end collision which sometimes resulted in deadly fires and explosions. Critics argued that the vehicle's lack of a true rear bumper as well as any reinforcing structure between the rear panel and the tank meant that in certain collisions, the tank would be thrust forward into the differential, which had a number of protruding bolts that could puncture the tank. This, and the fact that the doors could potentially jam during an accident (due to poor reinforcement)allegedly made the car less safe than its contemporaries.
Ford allegedly was aware of this design flaw but refused to pay for a redesign. Instead, it was argued, Ford decided it would be cheaper to pay off possible lawsuits for resulting deaths. Mother Jones Magazine obtained the cost-benefit analysis that it said Ford had used to compare the cost of an $11 repair against the monetary value of a human life, in what became known as the Ford Pinto memo. The characterization of Ford's design decision as gross disregard for human lives in favor of profits led to significant lawsuits. While Ford was acquitted of criminal charges, it lost several million dollars and gained a reputation for manufacturing "the barbecue that seats four."

Repairing the Ford Pinto
Cost of Repairing                           Cost of Not Repairing
$ 11 per part 180 deaths x $200,000
x 12.5 million cars                          + 180 injuries x $67,000
                                                   + 2000 vehicles x $700
=====================================================================
$137 million                                   = $49.5 million
(to improve safety)                          (to let it go)

Philip Morris Czech Republic Cost-Benefit Analysis of Smoking

This is rather more recent case. The detail report can be found at this website.

Cost                                             Benefits
Increased Health Care costs          Tax revenue from cigarette sales
(due to lung cancer)                       Health care savings
                                                        (from early deaths)
                                                     Pension savings
                                                     Savings in housing costs

Net gain if citizens smoke is $147 million
Saving from premature deaths is $1227.00 per person

The fundamental issue in both cases is whether or not we can put a dollar value on human life (in the first example, a $200,000 tag for each death; in the second example, early death from smoking to benefit the government or other people).

We can easily see what is wrong and what is right in both of these examples. However, in our real life, it is not easy to distinguish the right and the wrong.

In UK, a famous government agency is called The National Institute for Health and Clinical Excellence (NICE). NICE's main function is to provide the appraisals that are based primarily on evaluations of efficacy and cost-effectiveness. They actually put a monetory tag on human life (not for each death, but for a good-quality year). The cost-effectiveness limit (or threshold) for NICE is £30,000 per good-quality year of life gained. In many occasions, the novice drug could be denied if the drug is too expensive (over the limit of the price tag). NICE is not nice to the pharmaceutical companies.

If we can not put a price tag on human life, can we put a price tag on 3 months or 6 months of human life? If we can not put a price tag on 3 months or 6 months of human life (saved) and curb the use of extremely expensive drug, how the medical cost will be controlled? I don't think there is an easy solution.

Further readings:

Significant level of 0.00125

2010-01-23T15:29:00.003-05:00

Recently, I found some materials related to a lawsuit against the Nuvelo Pharmaceuticals – a company who used to develop a clot-busting product in the indication of occluded central venous catheter. It is very interesting to see quite some arguments about the study design and the significance of a p-value. Since Nuvelo planned to conduct only one pivotal trial (instead of two), they had agreed with FDA to use an extraordinarily high threshold (0.00125). This stringent p-value required for a single pivotal trial, less than 0.00125, was not met and the development program was eventually terminated. The issue is that such a stringent p-value was not communicated to the investment community upfront – one of the reasons for the lawsuit.

There are scientific basis for this extraordinarily high significant level (0.00125). Here are the contexts extracted from one of the FDA’s NDA Statistical Review for United Therapeutics Corporation Drug: UniprostTM (treprostinol sodium) for pulmonary arterial hypertension.

“...A more important issue is the overall Type I error rate for the proposed analysis in this submission. First, consider the traditional standard for approval at the FDA based on two confirmatory trials. Even if the efficacy of a treatment is shown convincingly in one study, the agency likes to see replication in a second study because we will then be in a better position to infer that the results generalize to the entire population of patients with the disease. The overall Type I error rate (or false positive rate) is the chance that both studies will have a p-value less than 0.05 and the results of both studies are in the same direction. If the treatment effects in the two studies are identically 0, then the chance that both p-values will be less than 0.05 and both treatment effects are in the same direction is 0.001251. For this reason, the Division of Cardio-Renal Drugs has often advised sponsors that one study with a p-value less than 0.00125 may be sufficient for approval…”

While not very common, we do see quite some drug development programs with one-single pivotal trial. One such example is the trial called PROTECT where the sample size and power for primary composite endpoint was based on “90% power at two-sided 0.00125 significance level todetect a difference between a distribution of 33% failure, 35% unchanged and 32% success (placebo group) and 25 failure, 34% unchanged, and 42% success (rolofylline group), using the van Elteren extension of the Wilcoxon test”

Designing one single pivotal trial with a significant level of 0.00125 may not be a good strategy in comparing to the conventional two pivotal trials with a significant level of 0.05. A significant level of 0.00125 is forty times more stringent than a significant level of 0.05. Employing such a small significant level will typically require a large sample size and may be difficult to be successful.

FDA’s perspectives for clinical development of tropical microbicides indicated the followings for a single trial:

No single site provides unusually large fraction of participants
No single investigator or site provides a disproportionate favorable effect
Consistency across study subset
Statistically persuasive
Single Multi-Center Trial Level of Evidence (p value, 2-sided)
· P < 0.001 : persuasive, robust 2*[0.025^2]=0.00125
· 0.05 > P > 0.01: inadequate
· 0.01> p > 0.001: acceptable, if:

- good internal consistency
- low drop-out rates
- Other supportive data

In the end, the evidence of efficacy should not purely rely on the p-values. There are other considerations in assessing the evidence of efficacy. This has been spelled out in FDA’s guidance for Industry: Providing Clinical Evidence of Effectiveness for Human Drug and Biological Products:
Tthe evidence of effectiveness could come from one single study with the following:

Large multicenter study
Consistency across study subsets
Multiple studies in a single study
Multiple endpoints involving different events
Statistically very persuasive finding

Here "statistically very persuasive finding" means a very small p-value even though the guidance does not specifically specify how small the p-value should be. It may depend on the negotiation with the corresponding branches in FDA.

Additional reading:

Lloyd Fisher (1999) ONE LARGE, WELL-DESIGNED, MULTICENTER STUDY AS AN
ALTERNATIVE TO THE USUAL FDA PARADIGM. Drug information journal Vol. 33, pp. 265–271
Boguang Zhen (2007) Consideration of Operational á Level With Different Approval Strategies. Drug Information Journal, Vol. 41, pp. 23–29, 2007 • 0092-8615
FDA guidance
Guidance for Industry: Evidence-Based Review System for the Scientific Evaluation of Health Claims – Final, 2009
Shun Z, Chi E, Durrleman S, Fisher L. (2005) Statistical consideration of the strategy for demonstrating clinical evidence of effectiveness—one larger versus two smaller pivotal studies. Stat Med. 24:1619–1637.

Logistic regression: complete or quasi-complete separation of data points

2010-01-14T11:20:00.003-05:00

When we perform the logistic regression, sometimes, we may run into an issue so called ‘complete or quasi-complete separation of data points’. In this situation, the maximum likelihood estimate does not exist. If we use SAS Proc Logistic, SAS log will give a warning message "WARNING: There is possibly a quasi-complete separation of data points. The maximum likelihood estimate may not exist. WARNING: The LOGISTIC procedure continues in spite of the above warning. Results shown are based on the last maximum likelihood iteration. Validity of the model fit is questionable." SAS will continue to report the Wald test results and odds ratios, however, these tests are no longer valid and results are not reliable (actually not accurate at all).

Complete separation data is something like below:
Y X
0 1
0 2
0 4
1 5
1 6
1 9

There is complete separation because all of the cases in which Y is 0 have X values equal to or less than 4, and the cases in which Y is 1 have X values equal to or greater than 5. In other words, Maximal value in one group is less than the minimal value in another group. When maximal value in one group is equal to the minimal value in another group, quasi-complete separation data may occur.

If the explanatory variable is categorical, complete separation of data points could be something like this:
Response Failure Success
Predictor
0 25 0

1 0 21

Where There are no successes when the value of the predictor variable is 0, and there are no failures when the value of the predictor variable is 1.

For maximum likelihood estimates to exist, there must be some overlaps in the two distributions. Since logistic regression models uses maximum likelihood estimates, when there is no overlaps of data points between two groups, the results from logistic regression models are unreliable and should not be credited.

Starting from SAS version 9.2, Proc Logistic provides Firth estimation for dealing with the issue of quasi or complete separation of data points.

proc logistic;
model y = x /firth;
run;

However, even after Firth estimation, the results should still be interpreted with extreme caution. Complete separation and quasi-complete separation of the data points may occur when the sample size is small and number of data points is not large or in the situation the samples are determined by the outcome (i.e., response) rather than explanatory variables – we see many publications where the analysis is based on the responders vs. non-responders.

When complete separation or quasi-complete separation occurs, for multivariate regression, the explanatory variable causing this situation should be identified and preferably excluded from the model. For univariate regression, other alternative statistical tests (for example group t-test) should be used.

Further reading:

Computation of the Odds Ratio with Small or Zero Cell Counts by Dr Robin High
Convergence Failures in Logistic Regression by Paul Allison
A tutorial on logistic regression by Ying So
What is new in SAS 9.2?

Rasch Analysis

2010-01-03T12:41:00.002-05:00

I recently noticed a new approach so called 'Rasch Analysis' when I worked on a paper in dealing with the MCID (minimal clinically important difference). I have not got chance to do any Rasch analysis on my own, but I have collected some information here for the future use.

Rasch analysis start to be used in education, survey area. In clinical trial, it is mostly used in psycometric, neurology areas where the outcome assessment relies on the instrument which typically contains certain number of items. These instruments are frequently used in CNS and neurology disease such as stroke, alzheimer, dementia. Traditionally, a scale or instrument will need to be validated through the reliability and validity tests. Recently, in addition to the reliability and validity tests, the Rasch measurement model has set new quality standards for outcome measures by appraising a broad range of measurement properties. You will not be surprised to see many papers if you user the search keyword "Rasch analysis" or "Rasch model" in Pubmed.gov.

There is no existing procedure within SAS to perform the Rasch analysis. However, there are some SAS macros for Rasch analysis on the internet developed by Karl Bang Christensen. The most popular software for Rasch analysis is Winsteps which provide a free download of a Ministep with capability of performing Rasch analysis for less items and less records.

Winning the holiday gift

2010-01-02T08:56:00.000-05:00

During the holiday season, it is very typical for a corporate to hold a party for its employees. During the party, one activity is to win the prizes with the raffle tickets.

Say each employee is distributed with 20 tickets in a raffle with 80 prizes. Which gives you a better chance of winning: putting all of your tickets in one of 80 baskets (your favorite item) or spreading them among 20 baskets with each ticket in one basket?

This seems to be a probability issue. There is an answer from AskMarilyn for the similar issue:

If you can see the baskets and tickets, you should wait until the last minute and then put all of your tickets in the basket that appears to contain the fewest tickets. If you can't see the tickets, put all your tickets in the basket for the least-desirable prize. But if you can't see the tickets and the prizes are equal, it doesn't matter what you do.

In previous year, I won nothing because I put all my raffles in a couple of hot items (there are thousands tickets in the boxes for these hot items). This year, I changed the strategy and put my tickets in the basket with the fewest tickets. I won a digital photo cube. Digital photo cube is not my favorite item, but I demonstrated how changing the strategy could increase the possibility of winning.

SAS SGPLOT for creating statistical graphs

2009-12-12T14:56:00.002-05:00

For a long time, we have been using SAS GPLOT for creating graphs. Beginning from SAS version 9.2, there is a new procedure called SAS SGLPOT, which could be a good tool for statisticians.

The detail about this procedure is described in SAS onlind document. There are several white papers about using this procedure.