Odds ratio and risk ratio in clinical trials #2

In my previous article, I discussed the odds ratio and risk ratio (or relative risk ratio). In clinical trials with binary outcome, both odds ratio and relative risk ratio are used. Since the clinical trials are similar to the cohort studies in epidemiology field, it seems to be more reasonable to use relative risk ratio in clinical trials. However, the odds ratio may be more commonly used in practice. This may be due to the fact that the odds ratio can be easily modeled using logistic regression. This could also be due to the fact that the odds ratio is typically larger than relative risk ratio that may be desired by the researcher.

For a non-inferiority or equivalence trials with binary outcome, one may desire to have a smaller standard error, therefore a narrower confidence interval – in this case, the relative risk ratio may be better than odds ratio.

Wikipedia gives an excellent comparison between relative risk ratio and odds ratio.

In an article “How can I estimate relative risk in SAS using proc genmod for common outcomes in cohort studies?”, the calculation of odds ratio, relative risk ratio, and their confidence intervals are illustrated.

Using SAS Proc Genmod, both odds ratio, relative risk ratio, and their confidence intervals can be easily calculated:

For odds ratio:

Proc genmod data = xxx descending;
    class treatment;
    model  outcomevariable = treatment
                                                  / dist = binomial link = logit;
    estimate 'Beta' treatment 1 -1/ exp;
run;

Here, the “link=logit” can be omitted since the logit link function is default when distribution is binomial.

For relative risk ratio, 
proc genmod data = xxx descending;
    class treatment;
    model outcome = treatment
                              / dist = binomial link = log;
    estimate 'Beta' treatment 1 -1/ exp;
run;

Here, the “link=log” can NOT be omitted since the log link function is NOT default when distribution is binomial.

Relative risk ratio can also be estimated using poisson regression especially when the event ratio is small.
 
Proc genmod data = eyestudy;
    class id;
    model outcome = treatment
                          / dist = poisson link = log;
    repeated subject = id/ type = unstr;
    estimate 'Beta' treatment 1 -1/ exp;
run;

Here, the “link=log” can be omitted since the log link function is default when distribution is poisson.

There are several advantages of using Proc Genmod to calculate the odds ratio and risk ratio. Adjusted odds ratio and adjusted relative risk ratio can be easily calculated when there are continuous or categorical covariates. The model can be easily modified to fit the longitudinal data.

Proc Logistic can be used for calculating the odds ratio (and the confidence interval) and can adjust for continuous or categorical covariates. However, Proc Logistic can not be used for calculating the relative risk ratio.

proc logistic;
     model outcome = treatment;
run;

Proc FREQ can be used for calculating the odds ratio and relative risk ratio (and asymptotic confidence interval) using /cmh option. For adjusted odds ratio or risk ratio, only the categorical covariate can be used. 

proc freq order=data;
    tables covariate*treatment*response / CMH;
run;

In the output, the odds ratio will be explicitly indicated while relative risk ratio will be labeled as “col1 risk” or “col2 risk”.

There is no regulatory guidance forcing the use of odds ratio or relative risk ratio. However, in clinical trials, if we compare the ratio of two proportions (eg the proportion of success in treated group vs. the proportion of success in control group), relative risk ratio seems to be better. Relative risk ratio resemble the hazard ratio in may aspects.

In FDA’s guidance “Diabetes Mellitus — Evaluating Cardiovascular Risk in New Antidiabetic Therapies to
Treat Type 2 Diabetes” The calculation of risk ratio is suggested. The guidance indicated
"Sponsors should compare the incidence of important cardiovascular events occurring with the investigational agent to the incidence of the same types of events occurring with the control group to show that the upper bound of the two-sided 95 percent confidence interval for the estimated risk ratio is less than 1.8. This can be accomplished in several ways. The integrated analysis (meta-analysis) of the phase 2 and phase 3 clinical trials described above can be used. Or, if the data from all the studies that are part of the meta-analysis will not by itself be able to show that the upper bound of the two-sided 95 percent confidence interval for the estimated risk ratio is less than 1.8, then an additional single, large safety trial should be conducted that alone, or added to other trials, would be able to satisfy this upper bound before NDA/BLA submission. Regardless of the method used, sponsors should consider the entire range of possible increased risk consistent with the confidence interval and the point estimate of the risk increase. For example, it would not be reassuring to find a point estimate of 1.5 (a nominally significant increase) even if the 95 percent upper bound was less than 1.8.”

In a presentation by Dr Bob O’Neill “Non-Inferiority Clinical Trials Some key statistical issues and concepts” he suggested that Log (Hazard ratio) or Log(relative risk) is preferred when determining the non-inferiority margin

In statistical review for Maxipime (cefepime hydrochloride) NDA, the risk ratio and 95% confidence interval are used. In an article “Relative risks of reported serious injury and death associated with hemostasis devices by gender”, the risk ratio were reported.

However, there are also many cases of using odds ratio instead of risk ratio in clinical trials. Some examples are:

• Antiepileptic drugs and suicidality NDA review
• Tenecteplase, recombinant (TNK-tPA) for Treatment of acute myocardial infarction (AMI) BLA
• In vitro polymerase chain reaction (PCR) based assay for HBV viral load detection PMA review

In summary, while both odds ratio and risk ratio can be used in clinical trials, risk ratio should be given the adequate emphasis in comparing the ratio of two proportions (between two treatment groups). In non-inferiority clinical trials, the risk ratio and its confidence interval are preferred.

FDA's new guidance on device approval process and device clinical trial

For a long time, device makers have been complaining about the FDA's device approval process. I personally heard a lot of talks that the device approval process is easier than the approval of the drug and the biological product. The requirements for clinical trials in device approval has lower standard comparing to  clinical trials in drugs and biological products. FDA device division (CDRH) has the loose criteria for the product approval. In response to the critics, FDA now releases two new draft guidance on August 15, 2011 and is seeking to educate industry on device approval.

The first guidance “Factors to Consider when Making Benefit-Risk Determinations in Medical Device Premarket Review” explains the agency's approval process for diagnostic and therapeutic devices, specifically:

• How the agency weighs the benefits and risks of a device
• How the agency assess the seriousness of a disease or condition
• How many people would use the device if approved
• The availability of other devices approved to treat the same condition

It is interesting enough that in examples used in this guidance, several clinical trials were mentioned as flawed or with unreliable data, however, FDA would approve the device anyway. This is just another reflection that CDRH indeed has lower data quality standard comparing to CDER for drugs and CBER for biological products.

In the second draft document “Design Considerations for Pivotal Clinical Investigations for Medical Devices”, the FDA laid out its expectations for clinical trials for medical devices. The agency stated that it looks for a study to provide reasonable assurance that the device is safe and effective.

Back in July 2011, FDA issued a guidance on “in Vitro Companion Diagnostic Devices” where the In Vitro Companion diagnostic device (IVD companion diagnostic device) is defined as an in vitro diagnostic device that provides information that is essential for the safe and effective use of a corresponding therapeutic product. The guidance intended to accomplish the following:

• Define in vitro companion diagnostic device
• Explain the need for FDA oversight of IVD companion diagnostic devices
• Clarify that, in most circumstances, if use of an IVD companion diagnostic device is essential for the safe and effective use of a therapeutic product, the IVD companion diagnostic device and therapeutic product should be approved or cleared contemporaneously by FDA for the use indicated in the therapeutic product labeling
• Provide guidance for industry and FDA staff on possible premarket regulatory pathways and FDA’s regulatory enforcement policy
• Describe certain statutory and regulatory approval requirements relevant to therapeutic product labeling that stipulates concomitant use of an IVD companion diagnostic device to ensure safety and effectiveness of the therapeutic product

No matter what the device is, if clinical trials are required, the statistical analyses can be based on the FDA guidance “Statistical Guidance on Reporting Results from Studies Evaluating Diagnostic Tests” that was issued in 2007.

In Device field, not all approval requires clincial trials. If clinical trials are required, they do not have to be interventional. If clinical trials are interventional, they do not have to be randomized, controlled. If clinical trials are randomized, controled, they do not have to be double-blinded. Some clinical trials in device field may be using the samples (for example blood samples) from patients with no intervention performed. The blood samples could be historical retains or prospectively collected. The use of the blood samples for device clinical trial still needs the informed consent from the patient.

Equipoise and Lack of Equipoise in Randomized Clinical Trials

According to Merriam-Webster dictionary, the word "Equipoise" means a state of equilibrium. In clinical trial, the concept of 'clinical equipoise' means that there is genuine uncertainty over whether a treatment will be beneficial. In other words, in randomized controlled clinical trials, there should be substantial uncertainty or there is no clear evidence that one treatment arm is particularlly better or worse. Clinical equipoise provides the ethical basis for medical research involving patients assigned to different treatment arms of a clinical trial - it is unethical to assign a subject to an inferior arm if the lack of equipoise exists and if there is substantial evidence that one treatment is better or worse than another treatment.

In the real word, when we plan a randomized, controlled clinical trial, 'lack of equipoise' may often exist. This is especially true in late stage clinical trials. In late stage clinical trials, there typically be some evidences about the benefit and treatment effect of the experimental drug from early phase I or phase II clinical trials. Our sample size calculation is based on the assumed treatment effect of the experimental drug.


We recently published a paper in the Journal of Neurology "Challenges of clinical trial design when there is lack of clinical equipoise: use of a response-conditional crossover design" where we discussed a situation of 'lack of equipoise' and the use of a response-conditional crossover design to ease the concern about the lack of equipoise in clinical trial design. Several small trials had suggested that IVIg is beneficial in treating the disease CIDP - lack of equipoise. However, in the absence of an approved treatment for this indication,  gaining regulatory approval for the use of IVIg in this indication required the conduct of large-scale, placebo-controlled confirmatory trials. Using the response-conditional crossover design, we eased the concern about subjects exposed to the 'perceived' inferior arm (Placebo in this case). The results indicated that we minimized subject's exposure to the inferior treatment arm.

Interpretation of 'clinical equipoise' may be different among clinicians, investigators, patients, clinical trial sponsors, and regulatory agencies. Evidences of treatment effects from small-scale clinical trials may be thought as real evidence for clinician and patients, but not for regulatory agencies.

When we bring the overall benefit/risk into the picture for assessing the 'clinical equipoise', it may be difficult to determine whether or not a clinical equipoise exist or not. A new treatment may have been demonstrated beneficial in efficacy, but with great uncertainty in safety.

Data Query on Diary and Patient Reported Outcome?

We keep hearing that for patient report outcome (PRO) including patient diary data, there is no query or data clarification required and the diary data is source data and can not be queried. For example, in a paper titled “How to clean up dirty data in Patient reported outcomes”, it said “The investigator is not allowed to query any of the patient's answers which leads in general to a lot of dirty data.”. Is it true that the diary or patient reported outcome can not be queried at any circumstance no matter how horrible the data quality is? It is not true and this popular perception is just not true.

Data clarification or data query is an essential process in clinical data management to ensure that the questionable data are corrected. According to EMA reflection paper ON EXPECTATIONS FOR ELECTRONIC SOURCE DOCUMENTS USED IN CLINICAL TRIALS, “Data clarification is part of the process to ensure complete data and the clarification process is one of the processes underlining the need for the maintenance of the audit trail.”

In a clinical study with paper-based case report form, the data clarification is typically issued by the data managers to the investigation sites. The investigator then reviews the issues to provide the responses to the data query. Data managers or clinical monitors can not directly make the changes to the data without issuing query and getting approval from the investigation site. In a clinical study with electronic data capture (EDC), the data clarification/query process is built into the EDC system. The data is entered at the investigation site. The query is issued by data managers or study monitors within EDC system. The investigator will then provide the responses to the query also within EDC system and make the data corrections.

The process becomes vague for patient reported outcome (PRO) or patient diary (no matter it is on the paper or electronic). The key difference for PRO data is that the data is directly recorded or entered by the subject or patient. There is a perception that no matter how poorly the data is, there is no data clarification or query process for PRO data or diary data.

In many clinical trials, the study endpoints rely on the collection of the information provided by the patient (daily symptoms, daily activities, quality of life,…). For example, for clinical studies on urinary incontinence, the primary efficacy endpoint may be the frequency of urinary incontinence episodes (UIE) per week as determined from patient daily diary. For clinical studies on female sexual dysfunction, the clinical endpoint may be the sexual events or encounters recorded daily by the study subjects using diaries. .

I used to work on a clinical trial with irritable bowel syndrome indication where the study endpoints were collected by a touch-tone telephone system (IVR – interactive voice response system). Efficacy parameters were symptom relief, abdominal discomfort or pain, bloating, stool frequency, stool consistency, straining and urgency recorded through IVR on daily basis by the study subjects. For stool frequency, the subject were asked “how many bowel movements did you have today?” As a statistician, I had to check the outliers before analyzing the data. I found some entries with the number of bowel movements being extremely high (55, 66). When I discussed these impossible numbers with the study manager, I was told that patient diary could not be queried. I pointed out that if these obvious data errors could not be corrected, the study data would be severely compromised. Later, we identified many more entries with duplicate numbers (such as 11, 22, 33, 44, 55, 66). After inquiring to the study subjects, it was found that the subject pressed the telephone number key twice for 1, 2, 3, 4, 5, and 6. This is just an example showing that the diary data could be easily recorded wrongly in the database and the query for accuracy is necessary.

FDA’s guidance on “Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims” touched on ensuring the quality of PRO data. In discussion about the ePRO, FDA guidance indicated the concern about clinical investigator inability to maintain and confirm electronic PRO data accuracy. It said “the data maintained by the clinical investigator should include an audit trail to capture any changes made to the electronic PRO data at any point in time after it leaves the patient’s electronic device.” Furthermore, the guidance indicated the concern about ability of any entity other than the investigator (and/or site staff designated by the investigator) to modify the source data”. These statements inexplicitly imply that the diary data can be queried and modified by the clinical investigator (and/or site staff designated by the investigator) and the clinical investigator has responsibility to ensure the completeness and accuracy of the diary data.

Adaptive Licensing for Drug Approval

From this year’s Drug Information Association (DIA) annual conference in Chicago, I learned a new concept of “Adaptive Licensing”. According to http://www.bnid.org/node/6921, “in the past five years, a wave of proposals for reform of drug licensing has emerged in the EU, US, and Canada under the labels of staggered approval, adaptive licensing, managed entry and progressive authorization. Through iterative phases of information gathering followed by regulatory evaluation and correction, these approaches seek to align licensing decisions on market access of drugs with emerging information on benefits and harms of drugs as actual used. It is hoped that this approach will provide patients with earlier access to innovative drugs to address unmet medical needs, better management of known risks, and improved detection of unanticipated adverse effects that emerge in use. “

The current drug licensing process is called ‘phased approach’ which requires the sponsors to conduct a series of clinical trials from phase I to phase III to establish the safety/tolerability and to confirm the efficacy before the regulatory agencies can consider the approval for marketing authorization of a product. This phased approach and the purpose of each phase of the clinical trial are discussed in this free web article. Recently, with the adaptive design concept, we are trying to break the traditional phased approach. The seamless phase II/III studies or seamless phase I/II studies have been much discussed and debated. Even with adaptive design, before a drug can be approved, a series of clinical trials are still required. Let’s now call “learning and confirming”: from early trial for learning to late stage trial for confirmation of the safety and efficacy with Adequate and well-controlled clinical study(ies) (A&WC).

With the phased drug approval approach, there will be a magic moment during the drug approval process. This magic moment is the regulatory reviewer’s action date (or decision date) - The date tells when a regulatory action, such as an original or supplemental approval, takes place. The regulatory agencies will be based on the review of data from pre-marketing clinical studies to make a decision of approving or not approving a production for market authorization. If the efficacy and safety have been demonstrated, the product is approved; if the efficacy and safety have not been sufficiently demonstrated, additional clinical studies may be requested; if the efficacy and safety are not demonstrated, the application of market authorization will be denied.

The adaptive licensing is trying to remove this magic moment from the drug approval process and instead considers the drug licensing as a continuous process. Whether or not a product should be authorized for marketing depends on the risk-benefit ratio. When the benefit outweighs the risk, the product should be approved; when the risk outweighs the benefit, the product should not be approved. For an approved product, if the benefit/risk ratio becomes unfavorable, the product should be withdrawn from the market. I liken this continuous adaptive drug licensing process to the p-value assessment in statistics. The magic moment is like the magic number of p=0.05 (p-value should be a continuous number and p=0.051 (not significant) may not be different from p=0.049 (significant)).

It looks like EU is pioneering in implementing the adaptive licensing. EMA has started to work on Adaptive Licensing to Reach Roadmap Goals (Roadmap to 2015) . However, in US, there is a similar program called “Accelerated Approval of New Drugs for Serious or Life-Threatening Illnesses”. With this program, “FDA may grant marketing approval for a new drug product on the basis of adequate and well-controlled clinical trials establishing that the drug product has an effect on a surrogate endpoint that is reasonably likely, based on epidemiologic, therapeutic, pathophysiologic, or other evidence, to predict clinical benefit or on the basis of an effect on a clinical endpoint other than survival or irreversible morbidity. Approval under this section will be subject to the requirement that the applicant study the drug further, to verify and describe its clinical benefit, where there is uncertainty as to the relation of the surrogate endpoint to clinical benefit, or of the observed clinical benefit to ultimate outcome. Postmarketing studies would usually be studies already underway. When required to be conducted, such studies must also be adequate and well-controlled. The applicant shall carry out any such studies with due diligence.”

However, the recent debates on whether or not Avastin (Bevicuzmab) should be withdrawn from the market for breast cancer demonstrated how difficult to implement this program. Based on the established rule, if a drug is approved through ‘accelerated approval’, the approval is conditional and can be withdrawn from the market if the drug is later showed to be ineffective or with unfavorable benefit/risk ratio. Avastin in breast cancer case just showed how difficult to withdraw a product due to ineffectiveness. In even worse situation, the drug companies may not fulfill the commitment to finish the follow-up studies. There have been several cases of marketing withdrawal due to safety concerns (for example, Vioxx), but it seems to be more difficult to withdraw a product from the market due to the efficacy concern.

Adaptive licensing may be the future direction for drug approval process; however, many issues need to be considered and resolved before this new process can be implemented.

• What is the impact of adaptive licensing on patent expiration?
• What is the tipping point the conditional approval can be granted?
• How to assess the benefit/risk ratio with sound scientific / statistical approaches (ie avoiding subjective assessment in benefit/risk ratio)?
• What if the committed follow-up studies never completed?
• How to deal with the difficulties to withdrew a product when other factors (for example, emotional effect in Avastin case) are kicked in?
• Adaptive Licensing: Taking the Next Step in the Evolution of Drug Approval
• Commentary by Jane Woodcock

Is blinded study really blinded? - assessment of blinding / unblinding in clinical trials

Randomization and blinding are critical components of the clinical trial from the start (design) to the end. Randomized, controlled, and double-blinded trial (RCT) has been the ideal clinical trial design. Inappropriate randomization and blinding (or potential unblinding) affect the integrity for the clinical trial. If the patient or investigator is aware of the treatment assignment, there will be conscious or unconscious biases in assessing efficacy, safety, or patient-reported outcome. With available software and computer programs, generating a randomization schedule is relatively easy. Ten years ago, I wrote a paper on “Generating randomization schedule using SAS programming” to show how easily randomization can be generated. With the interactive response technologies (IRT) including interactive voice response system (IVRS) and interactive web response system (IWRS), implementation of the randomization can also be easily managed. However, maintaining the blinding during the study may not be as easy as we thought.

I still remember the time when I was one of the randomization team members in PPD. After we generated the randomization schedule, we had to put the randomization schedule into an envelope and sealed with signatures. Then we had to put the envelope into a locked security box in a secured randomization room. In order to get the randomization schedule, at least two statisticians had to be present in order to open the security box.

While the actual randomization schedule is locked and secured, the randomization information or treatment assignment concealment can still be compromised by what happened at the site, how the patient and investigator guess the treatment assignment, and how the unblinded personnel communicate with the blinded team members.

There are many factors that can cause the potential unblinding. Here are some examples:

• Guess treatment assignment by the experience of adverse events and side effects. Suppose an intravenously administered drug can cause more headaches than Placebo, a patient with headache may guess he/she is on treatment group and not on Placebo. While this guess may not be 100% accurate, majority of patients may guess their treatment assignment correctly. In a book by Chow et al, ‘Design and analysis of clinical trials: concepts and methodologies’, an example about challenge in maintaining blinding was described “beta-blocker (e.g., pro-pranolol) have specific pharmacologic effects such as lowering blood pressure and the heart rate and distinct adverse effects such as fatigue, nightmares, and depression. Since blood pressure and heart rate are vital signs routinely evaluated at every visit in clinical trials, if a drug such as propranolol is known to lower blood pressure and the heart rate, then preservation of blindness is a huge challenge and seems almost impossible” In a large scale study (BHAT study), at the conclusion of the trial, patients, investigators, and clinical coordinators were asked to guess the patient’s treatment assignment, 79.9%, 69.6%, and 67% of patients, investigators, and clinical coordinators respectively guessed correctly the patient was on Propranolol and 42.8%, 58.6, and 70.6% of patients, investigators, and clinic coordinators respectively guessed correctly that the patient was on Placebo.

• Guess treatment assignment by improvement or no improvement in efficacy. If there is a prior knowledge that an treatment is effective (lack of equipoise), the investigator or patient can guess which treatment the patient is on based on the lack of effect.

• Guess treatment assignment by knowing the blood concentration of the drug or analytes. If a treatment is for augmentation purpose, a patient could have his/her blood sample tested to know whether or not the concentration for augmented drug is increased or not, then guess which treatment group he/she is on.

• In double-blinded studies, there are always some unblinded groups. These groups could include global drug safety for safety monitoring, laboratories that measure drug concentration or biomarkers, study drug supplies, site unblinded pharmacist… all of these groups could potentially reveal the treatment assignment to other study team unintentionally.

• For a study with DMC that involved a third party to prepare the unblinded data for DMC, treatment concealment could potentially be compromised during the information exchange with the blinded study team. This is critical for studies with adaptive designs where the patient data needs to be constantly reviewed and analyzed. An interesting example was discussed by Janet Witts regarding an awkward situation in an adaptive design where the DMC knew the event rate by treatment assignment and the sponsor didn’t. 

There is a dilemma when we develop the informed consent form. On the one hand, we are required to put into the informed consent form as much information as we can. On the other hand, the more information we put into the informed consent form, the more likely we enable the patients to guess their treatment assignment (based on their experience of side effects or perceived efficacy).

Ideally, in a double-blind trial, it is a good practice to evaluate for both the subjects and investigators whether or not blinding / masking has been preserved. However, in the real world, it is rare in double-blinded clinical trials to include a formal assessment of how well the blinding has been preserved. If the assessment of blinding becomes a routine, I think that many studies will show that subjects/investigators guessed correctly more frequently than they should have done by chance alone. Part of the reason this assessment has not been done often is perhaps the difficulty to explain the study results if the blinding is found to be compromised. It will be extremely difficult to assess the magnitude of the impact on the safety and efficacy evaluation if the blinding/treatment assignment concealment is compromised.

Further readings:

• Assuring that double-blind is blind

• Blinded trials taken to the test: an analysis of randomized clinical trials that report tests for the success of blinding

• Blinding, unblinding, and the placebo effect: An analysis of patient’s guesses of treatment assignment in a double-blind clinical trial

• Can keeping clinical trial participants blind to their study treatment adversely affect subsequent care?

• Assessment of blinding in clinical trials

• Concealing treatment allocation in randomised trials

Bland-Altman Plot for Assessing Agreement

Bland-Altman plot is a scatter plot of variable means plotted on the horizontal axis and the differences plotted on the vertical axis which shows the amount of disagreement between the two measures (via the differences) and lets you see how this disagreement relates to the magnitude of the measurements.

When I was in graduate school, the statistical analysis of microarray data just started to be a hot topic. In collaboration with Dr Rick Song, we looked at the microarray data and wrote a manuscript titled “On Graphical Presentation and Quantitative Analysis of cDNA Microarray Data” and we presented in JSM. In this manuscript, we proposed to use Bland-Altman plot. In clinical trials, I have not got a chance to apply this approach, but I do often see articles using the Bland-Altman plot. For example, an article titled “Using the Bland–Altman method to measure agreement with repeated measures” from British Journal of Anaesthesia.

When data is appropriate, Bland-Altman plot can be a handy tool to use. It is worth relaying the paragraphs from our original paper on graphical presentation of micro-array data using Bland-Altman plot.

“Graphical presentation is usually the first step for data analysis of microarray data. In the case without duplication (this is typical in microarray experiment), scatter plots will be drawn and then a regression line drawn through the data. This helps the eye in gauging the degree of agreement between two measurements and also may help us to identify the "outliers" that represent the differentially expressed genes in microarray experiment.

In clinical medicine, to assess agreement between two methods of clinical measurement, Bland and Altman proposed to plot the difference between the methods (A-B) against the mean (A+B)/2[12,13,14,15]. This approach has been extensively used in medical research for assessing measurement error and comparing different measurements for the same quantity. Bland and Altman’s method can be also applied to the microarray data. We can plot (Rm-Gm) against (Rm+Gm)/2 (figure2 above).

Calculating or plotting a regression line is not our focus as we are not concerned with the estimated prediction of one color intensity by another but with the theoretical relationship of equality and deviations from it.

There are several advantages for presenting the microarray data using Brand and Altman’s approach:

The plot of difference against mean allows us to investigate any possible relationship between the discrepancies and the true value. The plot will also show clearly any extreme or outlying observations. If two different samples are used in the experiment, these extreme or outlying observations could indicate the differentially expressed genes. It is often helpful to use the same scale for both axes when plotting differences against mean values. This feature helps to show the discrepancies in relation to the size of the measurement.

Brand and Altman's method makes it easier for us to estimate the precision of the estimated limits of agreement between two color intensities. We want a measure of the agreement that is easy to estimate and to interpret for a measurement on the color intensity of an individual gene. An obvious starting point is the difference between measurements by the two channels on the same gene. There may be a consistent tendency for one channel to exceed the other. This is called calibration factor and can be estimated by the mean difference. There will also be variation about this mean, which we can estimate by the standard deviation of the differences. These estimates are meaningful only if we can assume that calibration factor and variability are uniform throughout all genes.”

More references on Bland-Altman Plot:

• Bland and Altman (1996) Statistics Notes: Measurement error proportional to the mean
• Bland-Altman Analysis using SAS (Using PROC REG & PROC PLOT)
• Bland-Altman Plot from Wikipedia
• Validating the Bland-Altman method of agreement
• Bland-Altman Plot website

Restructure FDA's Drug Review Process?

Last week, I had a chance to listen to a speech by Dr Scott Gottlieb. While he touched several topics in related to the health care reforms, I was specifically interested in his discussion on restructuring FDA’s drug approval process.

Dr Gottlieb gave a lot of insights within FDA and analyzed the root cause of the very long and inefficient FDA drug review process.

Following his speech, I located his paper “SHOULD FDA RESTRUCTURE ITS DRUG REVIEW PROCESS?” from FDLI’s website. A lot of his analyses are so true and to the point.

For example, he elaborated why FDA adopt a matrix management structure for its review program.

“Prior to FDA’s adoption of a matrix management structure for its drug review program, agency scientists were organized largely around the clinical areas in which they worked (oncology, cardio-renal, antiviral, etc). This therapeutically focused structure had some advantages, but also led to some of its own challenges. FDA’s adoption of a matrix structure was aimed at solving some of these problems.

For one thing, sponsors complained that the advice they received about disciplines like biostatistics or clinical pharmacology varied (sometimes significantly) across different therapeutic divisions. Statisticians in one clinical division would be interpreting certain principles of statistics or evaluating a particular protocol design in a manner different than statisticians inside another therapeutic division.

These discrepancies still occur. But it is believed that the matrix organizational structure cuts down on this sort of conflict.

Grouping all of the statisticians or pharmacologists inside the same office increases opportunities for comparable training and cross-calibration on key principles. CDER management pulled the first group of the review divisions—the chemists—in 1995. The impetus was differences in pharmaceutical quality requirements being maintained among the different clinical divisions. Ultimately, having the chemists organized as a single group fostered the development of consistent standards. It also enabled FDA to negotiate the standards established by the International Conference on Harmonization (ICH).

Another reason for establishing the matrix was to improve morale. FDA remains a very “physician centered” culture, but was much more so prior to adoption of the matrix. Staff who lacked medical degrees or who weren’t the clinical reviewers on an application sometimes complained that they felt marginalized in the review process. As one statistician told me, “we were treated like second-class citizens.” Specialists from non-clinical disciplines like statistics also complained that remaining immersed in a single therapeutic area didn’t give them the breadth of experience that they needed for their own professional development.

Similarly, the organization of scientific personnel by therapeutic area was also seen as an impediment to their continued training in their chosen disciplines. For example, statisticians came together for the equivalent of grand rounds or other kinds of shared learning experiences. But these kinds of cross-training opportunities were challenging because staff were ultimately accountable to their divisions. The shared training opportunities weren’t prioritized. Efforts were also made to rotate non-clinical experts across different therapeutic areas. But the challenges endured.”

He then went on discussing the issues with FDA’s weak matrix management structure.

“But in practical terms, the weak matrix means that FDA project managers have limited dominion over key aspects of the review. Key disciplines involved in the review aren’t accountable to the project manager, or the division director. It is a system where there are few management carrots and no sticks. This weak structure also makes it harder to organize collaborative projects or even team meetings. The project manager doesn’t have strong authority when it comes to managing the collaboration between the different scientists involved in a drug’s review.”

He also mentioned the quality of the FDA review scientist and issues with FDA’s policy to allow very flexible working hours and work-from-home schedule. The current FDA drug review team is loosely organized and inefficient in many aspects. However, it is not easy to make big changes to the current process.

Differences between SOPs and WPs

Working in industry setting (versus academic), we all understand that it is critical for us to be familiar with the Standard Operating Procedures (SOPs) / Working Procedures (WPs) and follow these procedures. This is even more obvious in pharmaceutical and drug development industry. An employee could be fired not because of incompetence, but because of violating these procedures. A couple of days ago, I attended ChineseAssociation for Science and Technology Annual Convention, one session was to discuss about the requirements for starting up a new company. Several local entrepreneurs elaborated the intellectual property (IP), business plan, funding, a loyalty team, and so on. Nobody mentioned these procedures. But I think that it is equally important to have a set of necessary SOPs and WPs in order to start a new company, especially a service-type company in highly regulated drug development field such as contract research organization. Scientists working in the academic setting often find the difficulties when they try to land a position in industry mainly because they lack the ‘industry experience’ – one critical part is that they lack the experience in following up the strict industry SOPs. If you interview a candidate who does not even understand the term ‘SOP’, you know that your candidate will have a long way to adapt to the industry setting. 

While there are definitions for Standard Operating Procedures (SOPs), it is hard to define Working Procedures (WPs), not to mention the differences between SOP and WP.

Standard Operating Procedure is established procedure to be followed in carrying out a given operation or in a given situation. In clinical trial setting, the SOP is defined in ICH E6 as “detailed, written instructions to achieve uniformity of the performance of a specific function. “  SOP is one of the components of a Quality Management System (QMS) implemented in many large organizations - improve quality by providing a standard format, approach, and application of common processes.. 

The issue may be with the wording ‘detailed’ in this definition. The common understanding within the industry is that SOP needs to be written in a little bit high level and the working procedure provides the detailed steps according to SOP. If the strict definition of SOP from ICH E6 is followed, there perhaps should not be any working procedures. All working procedures should be called SOPs. There is really no defined boundary for separate the SOPs and WPs. For SOPs, say what you do... but do what you say. The documentation is always important to the SOP compliance. 

My personal opinion is that SOPs are the procedures we must follow, typically on high level, with GCP compliance while WPs are procedures we should follow, typically in more detail, for internal standardization and efficiency. For example, in biostatistics, a SOP must be in place to require the independent validation for statistical outputs. A WP may be created to detail how independent validation should be implemented.

Here are some observations about SOPs and WPs:

1) Within FDA, the term SOPP (standard operating procedure and policies) is used for SOPs and WPs.

http://www.fda.gov/BiologicsBloodVaccines/GuidanceComplianceRegulatoryInformation/ProceduresSOPPs/default.htm

 2) There are two papers containing good discussions about SOP:

• SOP Writing for Clinical Research
• SOP Writing for Clinical Trials webblog

 3) There seems to be no formal definition ‘working procedure’. Different companies may call this differently and different people may have different understanding for ‘working procedure’.

There are some discussions on the web: 

• http://elsmar.com/Forums/showthread.php?t=16890
• http://elsmar.com/Forums/showthread.php?t=14292

Having SOPs and WPs in place is one thing; having everybody trained on SOPs and WPs is another thing. If SOPs and WPs are written in great detail like step-by-step instructions, the procedures may become a burden to be followed. This could in turn cause issues in SOP compliance when there is an audit. 

Last week, we had a group meeting during the lunch time and we ordered some food from Jason’s Deli. It was good to see that the delivery guy brought a Bagset Checklist which listed the items and quantities for each item. Unfortunately, the guy clearly did not follow their procedure and did not check the check list. When he delivered, he realized that he forgot to bring ‘bulk chips’. He had to go back to pick up this missed item. Had he follow the procedure and check his Bagset Checklist, he would have not missed any item. In this case, the consequence of the non-compliance is minor, but in clinical trials, the consequence of the non-compliance could sometimes be very significant.

I-Spy 2 Trial - a study with fancy name and fancy design

If you see the term “I-spy”, you would typically think something related to movie, kids game or TV series. Recently, I-spy has been linked to a novice clinical trial design in breast cancer.

For a short description about this study, clinicaltrials.gov is a good place. In clinicaltrials.gov, I-SPY 2 TRIAL is listed as “Neoadjuvant and Personalized Adaptive Novel Agents to Treat Breast Cancer”. For more detail descriptions about this study, you should check out the following websites:

• Barker et all (2009) I-SPY 2: an adaptive breast cancer trial design in the setting of neoadjuvant chemotherapy, Nature Translational Medicine
• Ispy2.org – I-Spy 2 clinical trial website
• I-Spy 2 Advocate Website – contains many presentations 
• An IT bundle to support adaptive trials with novel agents and emerging biomarkers

This study got a lot of publicities. For example, in FDA’s ‘advancing Regulatory Science for Public Health’, it specifically mentioned the I-Spy 2 trial,

“Personalized treatment for cancer

The “I-SPY 2 TRIAL,” launched in March 2010, represents a groundbreaking new clinical trial model that will help scientists quickly and efficiently test the most promising drugs in development for women with higher risk, rapidly growing breast cancers. During the trial, drugs in development are individually targeted to the biology of each woman’s tumor using specific genetic or biologi­cal markers, known as “biomarkers.” By applying an innovative trial design, researchers will use data from one set of patients’ treatments to treat other patients – more quickly eliminating inef­fective treatments and drugs. The I-SPY 2 trial was developed under the Biomarkers Consortium, a unique public-private partnership that includes the FDA, the National Institutes of Health, and major pharmaceutical companies, led by the Foundation for the National Institutes of Health. “

The claimed advantages for this trial design are:

• Utilized Personalized Medicine
• Uses genetic or biological marker (“biomarkers”) from individual patients’ tumors to screen promising new treatments, identifying which treatments are most effective in specific types of patients
• Improved Efficiency (fewer patients, less time, and fewer resources)

Enable researchers to use early data from one set of patients to guide decisions about which treatments might be more useful for patients later in the trial, and eliminate ineffective treatments more quickly Enable the development of more informed, smaller phase III trials

So what is this trial design? Can this trial really achieve its purpose? Here is what I would say:

• I-Spy 2 is a phase II study, anything generated from this study will need to be confirmed in Phase III confirmatory studies.
• I-Spy 2 is a government-sponsored study and supported by Foundation of NIH, Biomarkers Consortium, NCI, and others. It may never happen if it is an industry-sponsored study.
• I-Spy 2 is academic study and not a study for product licensure. If it is study for product licensure, there may be difficulties for study design to be accepted by regulatory agencies.

• I-Spy 2 design is complicated and it employs Bayesian Adaptive, Response Adaptive Randomization (play-the-winner approach), Biomarker Adaptive, Stopping and adding treatment arms. The algorithm and stopping rules are complicated and the assumptions / basis for generating these algorithms / stopping rules may eventually be demonstrated incorrect. 
• I-Spy 2 is a randomized, open label study. It would be more difficult to implement if this is a double-blinded study
• I-Spy 2 got it fame due to its use of ‘personalized medicine’, ‘adaptive design’ – all fit into the hot areas
• I-Spy 2 is a multi-center, NOT multi-national study. It would be much more challenging if this is a multi-national study

Although I-Spy 2 got a lot of attentions, not everyone is convinced with this design. The study is closed watched – let’s see the fate once it is completed.

Mobile phone text message used in clinical trial?

Recent discussions with my friends in China surprised me a little bit. They are way ahead in applying the advanced technology in public health area. They used the mobile phone text message to promote the smoking cessation. They recently applied research grant to study the effect of mobile phone text message in improving the maternal and child care. For pregnant women or new mothers, the mobile phone text message is used to send reminders, maternal care tips, child care tips,...

There have been some publications demonstrating the effectiveness of text message in these public health promotion areas. For example, a study in New Zealand demonstrated that smoking cessation using mobile phone text messaging is as effective in Maori as non-Maor. The newscientist.com reported that text messages double young smokers' quit rates.There are

It is natural to think that such text message approach can be used in clinical trials. When I search the clinicaltrials.gov, I can find many clinical trials using text message mostly for improving the adherence of clinical visits or adherence of drug taking.

In my opinion, effectiveness of using text message in clinical trials depends on the study population and the country the study is conducted. In China, if the study is conducted in urban areas, mobile phone text message could be very effective because 1) almost everyone has mobile phone; 2) people use text message more often than actual calling. In the United States, for general population, text message may not be a good approach because some family may still rely on residence-line phone instead of cell phone. Even though they have cell phone, they rarely use text message on daily basis. If a study is conducted in high school students or college students, text message could be an effective tool since all students like to use text messages.

Emerging Statistical Issues in the Conduct and Monitoring of Clinical Trials

This Wednesday, I had a chance to attend “University of Pennsylvania Annual Conference on Statistical Issues in Clinical Trials”. The topic for this year is “Emerging Statistical Issues in the Conduct and Monitoring of Clinical Trials”. The number of participants was just right in size and the conference was organized pretty well.

In terms of the topics, there are some of them I like and some of them I don’t like. The presentation slides will eventually be posted on conference’s website, however, I would like to give one or two sentences commenting on each topic.

“Sample size estimation incorporating disease progression” – the key issue is the adequacy of the study endpoint. A good endpoint will incorporate the impact of the disease progression.

“Hurdles and future work in adaptive designs” – it is good to hear the discussion about the hurdles, caveats of the adaptive designs. Still very often, a lot of people only talk about the advantages of adaptive designs – too good to be true.  Similarly, a recent article "a once-rare type of clinical trial that violates one of the sacred tenets of trial design is taking off, but is it worth the risk? " from The-Scientist magazine gave some objective assessments on implementing the adaptive designs.

“Predicting accrual in ongoing trials” – utilizing the complicated statistical model to predict the accrual is a waste of time. Accrual in ongoing clinical trials is 95% clinical operations issues, 5% related to statistics. Is it worth to modeling the accrual?

“New incentive approaches for adherence” – money incentives including lottery is a sensitive topic and ethic issue could follow no matter it is incentive for adherence or for study visit compliance. Money incentives are different depending on participants’ social economic status (family income). $100 lottery may be very incentive to some, but not to others.  

“Efficient source dada verifications in cancer trials” – I always thought that all data fields had to be 100% source data verified. It is not entirely true in large scale trials in oncology or in studies with cardiovascular endpoint. In industry, we are rather conservative.

“Estimation of effect size in trials stopped early” – trials stopped early due to efficacy is not very common and should not be encouraged. Difficulty in estimating the effect size still exists for trials stopped early.

“Accounting in analysis for data errors discovered through sampling” – Unreliable data or large % of missing data is always a concern, even for observational studies. Statistical approach may not be a good option. When data is garbage, the results we draw from the data will also be garbage – so called ‘garbage in, garbage out’ no matter which statistical model is utilized to address the data issues.

“Some practical issues in the evaluation of multiple endpoints” – It is so correct that we should play down the importance of differentiating ‘primary endpoint’, ‘secondary endpoint’, ‘tertiary endpoint’,… Multiple comparison has been expanded so much and is everywhere now (co-primary, primary and secondary, co-secondary, secondary superiority test after non-inferiority test, interim analysis, meta analysis, ISE,…). Are we overdoing this?

Sparse sample and population pharmacokinetics

In drug development, it is necessary to understand the pharmacokinetics profiles (or time concentration profiles) of the experimental drug and calculate the pharmacokinetic (PK) parameters (Area Under the Curve – AUC, Clearance – CL, or Volume of distribution –Vd). These PK parameters can provide the estimate of the dose exposure and assist in the decision on dose timing and dose interval. In order to calculate the PK parameters, we typically need a serial of blood samples at multiple time points (usually more than 6) after the drug administration. In some situations, it is not feasible or not practical to obtain these many blood samples. The obvious example is in pediatric studies where it is not feasible to obtain multiple blood samples due to the blood volume restriction. The specimen may not just be blood samples. If the PK is conducted using other specimens, it is usually difficult to obtain multiple PK samples. For example, we could obtain middle ear fluid (MEF) sample to determine the antibiotic drug concentration in the ear and bronchoalveolar lavage (BAL) to determine the drug exposure in the lung. It is not practical to obtain multiple samples for these special specimens due to the safety concern.

When very few samples are available for each patient, we call it ‘sparse sampling’. With sparse data, we would need to employ a

Population PK

approach to estimate the PK parameters, describe the PK profile, or do PK/PD modeling. The use of population PK during the drug development has been steadily increasing. Regulatory agencies have issued several guidance on the use of population pharmacokinetics.

à        FDA guidance on Population Pharmacokinetics

à        FDA guidance General Considerations on Pediatric PK

à        FDA guidance Pharmacokinetics in Pregnancy

à        EMEA Guidance on the Role of Pharmacokinetics in the Development of Medicinal Products in the Paediatric Population

à        EMEA Guidance on Reporting the Results of Population Pharmacokinetic Analyses

There are different sparse sample designs. Below are some of the sparse sample designs I have seen.  

An example of sparse sampling at fixed time points is described in a paper by Vogelmeier et al. They used BAL fluid sample to study the intrapulmonary half-life of aerosolized product in Normal Volunteers”.

For BAL fluid samples, it is not feasible to obtain serial samples at all six time points (at screening, 0.5, 6, 12, 24, and 36 h). Therefore, in this study, “each volunteer underwent two BALs. The first lavage was done in the screening phase with an interval of between 3 and 7 d before inhalation of the drug. The volunteers were randomly assigned to one of five groups with the second lavage following 0.5, 6, 12, 24, or 36 h after aerosol administration. Each of the groups consisted of six individuals…”

Subjects in group 1 contributed two BAL samples at Screening and at 0.5 hours after inhalation.

Subjects in group 2 contributed two BAL samples at Screening and at 6 hours after inhalation.

Subjects in group 3 contributed two BAL samples at Screening and at 12 hours after inhalation.

Subjects in group 4 contributed two BAL samples at Screening and at 24 hours after inhalation.

Subjects in group 5 contributed two BAL samples at Screening and at 36 hours after inhalation.

With subjects from all five groups combined, a overall picture of the PK profiles over 24 hours after inhalation could be described. Original paper provided only the summary analysis. Nowadays, the data could be further analyzed using nonlinear mixed model from population PK model with software such as NONMEM.

In FDA guidance on Population Pharmacokinetics, an example was provided for estimating the AUC using sparse data (1-2 middle ear fluid samples per subject) in pediatric subjects.

“The penetration of drug X into middle ear fluid (MEF) was investigated using population PK analysis with sparse data (1-2 samples per subject) obtained from 36 pediatric patients (2 months to 2.0 years of age) who underwent clinical therapy with drug X. The estimated area under the concentration-time curve (AUC) that was above the minimum inhibitory concentration (MIC) (AUCMIC) and the half-life of drug X are 12.5 ug.hr/ml and 6.1 hours in MEF, respectively, vs. 23.7 ug.hr/ml and 3.2 hours in plasma, respectively….”

With this short description, we don’t know if MEF samples are taken from subjects at various times or fixed times. However, non-linear mixed model must have been used for analyzing the data.  

FDA’s guidance on population pharmacokinetics states, “the full population PK sampling design is sometimes called experimental population pharmacokinetic design or full pharmacokinetic screen. When using this design, blood samples should be drawn from subjects at various times (typically 1 to 6 time points) following drug administration. The objective is to obtain, where feasible, multiple drug levels per patient at different times to describe the population PK profile. This approach permits an estimation of pharmacokinetic parameters of the drug in the study population and an explanation of variability using the nonlinear mixed-effects modeling approach. “

If a full population PK sampling design is used, the sampling scheme will be something like below. The different subject could contribute different number of samples at various times.  

Subject number	Blood sampling time (t)	concentration at time t  Ct
001	Predose	xxx
001	24 hours post dose	xxx
002	Predose	xxx
002	8 hours post dose	xxx
002	12 hour post dose	xxx
003	Immediately postdose	xxx
003	5 hour post dose	xxx
004	4 hour post dose	xxx
…

Then when non-linear mixed model such as NONMEM is used to fit the data to characterize the PK profile with PK parameter (such as AUC) = function of concentration (Ct) at time t.

In multiple dose studies, if the purpose is to characterize the PK profile at steady state, one could implement a strategy of splitting the number of samples into different dose intervals.

Suppose we need 8 serial blood samples (t1 to t8) to calculate AUC and the dose interval is weekly, we can have these 8 samples split into 4 dose cycles. For each subject, we would only take two samples for each dose cycle. At steady state, for each subject, we expect PK profile after each repeat dose is not much different; the concentration at day 1 after repeat dose #1 would be similar to the concentration at day 1 after repeat dose #4, and so on. In this case, we would be able to calculate AUC for each subject with 8 samples from four dose intervals (instead of 8 samples from one dose interval over 7 days). The drawback is that the study period would be longer.