For multi-center, open label clinical trials, can individual investigation sites disclose results from their sites while study is ongoing?

The golden standard for clinical trials is RCT (randomized, control trials) including three key components: randomization, including a control group (preferably concurrent control), and blinding (or concealment of the treatment assignment). The blinding can minimize the biases (conscious or unconscious) introduced into the study by investigators, patients, and the study team.

However, in some situations, blinding is not possible or is very difficult to implement. A randomized, open-label study may be more appropriate. In the randomized, open-label study, while patients are randomized assigned to receive the experimental new drug or control, the investigator, patients, (in some cases the sponsor study team) are aware of the treatment assignment. 

In an early posting "Some Blinding Techniques in Clinical Trial", some blinding techniques can be applicable even for the randomized, open-label study. On the conservative side, the sponsor can follow the same procedures in handling the randomization, open-label study as the double-blind study - i.e., establishing a firewall to keep the sponsor study team or anyone with the capability to do the aggregate analyses and decision making from accessing the data during the study. 

With a randomized open-label study, both site investigators and patients participating in the study know which treatment group the patient is receiving - knowing the treatment assignment itself may cause biases in efficacy and safety assessments, in concomitant medication selection, in subsequent therapy selection,...

In the last several days, we see an extreme case (presumably due to the special situation in fighting the Covid-19 pandemic). In multi-center, randomized, open-label studies for remdesivir, while the studies are ongoing, the individual sites start to disclose the results / experiences from patients in their perspective sites. 

Statnews.com (04/16/2020) Early peek at data on Gilead coronavirus drug suggests patients are responding to treatment  

"The University of Chicago Medicine recruited 125 people with Covid-19 into Gilead’s two Phase 3 clinical trials. Of those people, 113 had severe disease. All the patients have been treated with daily infusions of remdesivir."

"The best news is that most of our patients have already been discharged, which is great. We’ve only had two patients perish,” said Kathleen Mullane, the University of Chicago infectious disease specialist overseeing the remdesivir studies for the hospital.

 Komo News (04/21/2020) Experimental anti-viral drug Remdesivir used to treat COVID-19 patients

Dr. Vinay Malhotra said at his hospital, 40 patients have been treated with the anti-viral drug.

According to Dr. Malhotra, 32 patients were in severe condition. 8 were in moderate condition. Of the 40 patents treated, Malhotra said 17 of them have already been discharged.

“Everyone is talking about the discharge of these patients. Everyone is talking about how good the effect is in three days,” Malhotra said. "We were fortunate to be chosen for this trial. And that has helped our population tremendously."

Both the University of Chicago Hospital in Illinois and the Multicare Institute for Research and Innovation in Tacoma, Washington are investigation sites participating in two pivotal clinical trials sponsored by Gilead:

• A Phase 3 Randomized Study to Evaluate the Safety and Antiviral Activity of Remdesivir (GS-5734™) in Participants With Severe COVID-19
• A Phase 3 Randomized Study to Evaluate the Safety and Antiviral Activity of Remdesivir (GS-5734™) in Participants With Moderate COVID-19 Compared to Standard of Care Treatment

Both studies (one for severe Covid-19 and one for moderate Covid-19 patients) were designed as randomized, open-label studies with one group receiving Remdesivir (on top of standard of care) and one group receiving standard of care only. According to clinicaltrials.gov, there are 179 and 178 investigation sites for these two studies.

If all sites do the same as the University of Chicago hospital and Multicare Institute for Research and Innovation to disclose their experiences, we could see many reports: some may be good, some may not be very good. The sites with good results may be eager to get their information out; the sites with results not very good may keep silent.

In both of these reports, there was no mention of how many patients in the control group (receiving standard of care) - maybe these patients in the control group are doing pretty well too (unless there are selection biases in determining which patients are allocated to receiving remdesivir or not receiving remdesivir (standard of care)). 

It is not clear if there is an agreement between the sponsor (Gilead) and the investigation sites about disclosing the data while the study is ongoing. Disclosing the data for individual sites while the study is still ongoing should certainly be discouraged or prohibited even in the public health emergency situation. The better way will be that the sponsor designs the study as group sequential design and perform the interim analyses using the data from all sites - if the results are indeed great, the pre-specified boundaries may be crossed and the sponsor may be able to stop the study for efficacy. In this way, the study results may be disclosed earlier while the study integrity is maintained. 

There are already reports that the motive and wrongdoing should be investigated: 

• Congressman calls on SEC to look into leaked remdesivir data: report
• Congressman calls for SEC investigation into leak about possible coronavirus drug

There are also reports that criticize the study design (no adequate control group and no blinding) for Gilead's designed studies: 

The world wants answers on Gilead’s Covid-19 drug. Experts worry next studies may increase uncertainty

 

Gilead's Remdesivir Clinical Trials - What Do Drastic Design Changes Mean?

In the midst of the Covid-19 pandemic, people are rushing to find effective treatments for Covid-19 infected patients. Among the drugs being investigated is remdesivir, an experimental antiviral made by the US drug company Gilead Sciences. It has been characterized as one of the most promising by health authorities, including WHO officials—though that optimism is inspired only by anecdotal information.

There are four randomized, controlled trials with remdesivir that may have the results available in very near term. Two studies conducted in China should have the results available very imminent (before the end of April). Two studies conducted by Gilead itself should have results sometime in May, 2020. 

We noticed that for two pivotal studies run by Gilead, the study design was significantly modified. According to the clinicaltrials.gov, the major changes are listed below: sample sizes are increased; the primary efficacy endpoint is revised, the treatment arms are tweaked.

See the article "Gilead supersizes remdesivir trials, changes primary endpoint".

Protocol Title	Previous Version (before April 6, 2020)	Revised Version (after April 6, 2020)
A Phase 3 Randomized Study to Evaluate the Safety and Antiviral Activity of Remdesivir (GS-5734™) in Participants With Severe COVID-19	400 Subjects Two Strata and Three arms: Remdesivir for 5 days versus Standard of care; Remdesivir for 10 days versus Standard of care	2400 Subjects Four Strata and Five arms: Remdesivir for 5 days (Not Mechanically Ventilated) versus Standard of care; Remdesivir for 10 days (Not Mechanically Ventilated) versus Standard of care Remdesivir, 5 or 10 Days (Extension) versus Standard care Remdesivir 10 days (Mechanically Ventilated) versus Standard of care
	Proportion of Participants With Normalization of Fever and Oxygen Saturation Through Day 14 This is a composite outcome measure. Criteria for fever normalization: Temperature < 36.6 °C armpit, < 37.2 °C oral, or < 37.8 °C rectal sustained for at least 24 hours and criteria for oxygen normalization: peripheral capillary oxygen saturation (Sp02) > 94% sustained for at least 24 hours.	The Odds of Ratio for Improvement on a 7-point Ordinal Scale on Day 14. The odds ratio represents the odds of improvement in the ordinal scale between the treatment groups. The ordinal scale is an assessment of the clinical status at a given day. Each day, the worst score from the previous day will be recorded. The scale is as follows: 1. Death 2. Hospitalized, on invasive mechanical ventilation or Extracorporeal Membrane Oxygenation (ECMO) 3. Hospitalized, on non-invasive ventilation or high flow oxygen devices 4. Hospitalized, requiring low flow supplemental oxygen 5. Hospitalized, not requiring supplemental oxygen - requiring ongoing medical care (coronavirus (COVID-19) related or otherwise) 6. Hospitalized, not requiring supplemental oxygen - no longer required ongoing medical care (other than per protocol Remdesivir administration 7. Not hospitalized.
A Phase 3 Randomized Study to Evaluate the Safety and Antiviral Activity of Remdesivir (GS-5734™) in Participants With Moderate COVID-19 Compared to Standard of Care Treatment	600 Subjects Three arms: Remdesivir for 5 days Remdesivir for 10 days Standard of care	1600 Subjects Added a fourth arm: Experimental: Part B: Extension Treatment, Remdesivir 5 or 10 days Participants will receive continued standard of care therapy together with RDV 200 mg on Day 1 followed by RDV 100 mg on Days 2, 3, 4, 5, 6, 7, 8, 9, and 10.
	Proportion of Participants Discharged by Day 14	The Odds of Ratio for Improvement on a 7-point Ordinal Scale on Day 11. The odds ratio represents the odds of improvement in the ordinal scale between the treatment groups. The ordinal scale is an assessment of the clinical status at a given day. Each day, the worst score from the previous day will be recorded. The scale is as follows: 1. Death 2. Hospitalized, on invasive mechanical ventilation or Extracorporeal Membrane Oxygenation (ECMO) 3. Hospitalized, on non-invasive ventilation or high flow oxygen devices 4. Hospitalized, requiring low flow supplemental oxygen 5. Hospitalized, not requiring supplemental oxygen - requiring ongoing medical care (coronavirus (COVID-19) related or otherwise) 6. Hospitalized, not requiring supplemental oxygen - no longer required ongoing medical care (other than per protocol Remdesivir administration 7. Not hospitalized.

It is not clear what these protocol changes are based on: the external data from other studies or the cumulative data to date from these two ongoing studies? The fact is that these two Gilead-run studies are randomized, open-label – unless they treated them as the blinded studies, the Sponsor would be able to see the cumulative data and monitor the treatment effects along the way. 

A dramatic increase in sample size seems to suggest that the treatment effect is much smaller and the previous study design may be underpowered.

It may also indicate that two randomized, double-blinded Remdesivir studies in China will most likely be underpowered as well (308 subjects for study in mild-moderate patients and 453 subjects for study in severe patients). Unless there is a magic or luck, these two studies from China may not be able to reach the statistical significance because of the smaller sample size. We hope that there will be at least some trends - at a time of no effective treatment for Covid-19 infection, a study with a trend will be encouraging news. 

For the primary efficacy endpoint, odds ratio should be more sensitive to detect the treatment difference for the ordinal data - ordinal logistic regression will need to be used to calculate the odds ratios. 

Gliead asked FDA to rescind the orphan drug designation for coronavirus treatment remdesivir it just received 48 hours earlier - Why?

Last Monday, Gilead received the orphan drug designation from US FDA for remdesivir to treat Covid-19. Two days later on Wednesday, we saw the news that Gilead asked FDA to rescind its orphan drug designation - it was really an unusual step taken by a company in 37-year history since Orphan Drug Act was enacted in 1983.

A drug with orphan designation will give many benefits to the drug developer.

• Exemption from PDUFA (Prescription Drug User Fee Act) marketing application fee (FY 2020: $2.94+ million)
• 7 years of exclusivity in the U.S.
• Tax credits related to clinical trial expenses
• Orphan products grants program
• Submission of a pediatric assessment per PREA is not required
• FDA assistance in clinical trial design

After Gilead received the orphan drug designation, it immediately drew a lot of critics. The company was criticized by lawmakers and patient advocates after receiving the orphan designation for the experimental antiviral drug, saying it was taking advantage of the rapidly accelerating health crisis.

"It's embarrassing to take something that's potentially the most widespread disease in the history of the pharmaceutical industry and claim it's a rare disease."

“This is an unconscionable abuse of a programme designed to incentivise research and development of treatments for rare diseases.”

“Calling COVID-19 a rare disease mocks people’s suffering and exploits a loophole in the law to profiteer off a deadly pandemic,”

Gilead had maintained that at the time that it had filed for the orphan designation, the coronavirus outbreak was nowhere near the level of severity it is at now. It also contended that it had sought the orphan status to expedite approval for the drug, in particular for a required paediatric study plan.

According to the Orphan Drug Act approved in 1983, the orphan drug designation was for drug development in rare disease or condition where the rare disease or condition means: 

(A) affects less than 200,000 persons in the United States, or
(B) affects more than 200,000 in the United States and for which there is no reasonable expectation that the cost of developing and making available in the United States a drug for such disease or condition will recovered from sales in the United States of such drug. Determinations under the preceding sentence with respect to any drug shall be made on the basis of the facts and circumstances as of the date the request for designation of the drug under this subsection is made.

Determining orphan drug designation based on the # of patients is usually good for ultra-rare diseases and for diseases with genetic disorders. However, it can be problematic when it comes to diseases with a dynamic feature in the number count (the Covid-19 pandemic is a great example):

• Defining and counting rare diseases is not straightforward.
• Difficulties in obtaining definitive diagnoses contribute
• Limitations in systems for reporting and tracking orphan disease diagnoses.
• Countries have adopted different definitions of a rare disease
• Researchers are continuously identifying new diseases or disease variants.

The epidemiology of rare diseases—including the determination of prevalence (the number of people affected at any one time), incidence (the number of new cases in a given year), and patterns of disease (e.g., age distribution) in the population—is inexact.

A rare disease may become a prevalent disease

• Some conditions that initially are classified as rare eventually outgrow that categorization. Covid-19 infection is a good example of this. Even though the # of confirmed cases is still below the 200,000 mark, very soon, the threshold of 200,000 cased will be crossed. Another example is HIV/AIDS. When AIDS emerged in the United States, it fit the legislative definition of a rare disease—affecting fewer than 200,000 individuals. As the infection spread, as diagnostic capabilities and data collection systems improved, and as researchers developed effective treatments that reduced mortality without curing the disease, the total number of individuals with AIDS grew to nearly 470,000 by 2007 and the number of individuals with HIV infection exceeded 1.1 million (CDC, 2009c). 
• If effective but not curative treatment can turn a rare disease into a common one,

A prevalent disease may become a rare disease

Effective prevention can, conversely, turn a common condition into a rare disease. This is the case with many once common childhood infections such as mumps and measles. Public health officials are concerned, however, that factors such as the development of drug-resistant infectious agents and the opposition of some parents to childhood vaccinations could reverse the situation for some now rare diseases. The former concern—drug resistance—is partly a significant scientific challenge (i.e., developing new anti-infectives) and partly a public health and clinical practice challenge (i.e., discouraging overuse of antibiotics). Preventing negative health consequences from anti-vaccination sentiment involves public health expertise, social science research, clinician communication skills, and public policy responses.

• Post-polio syndrome
• Precision Medicine – genetic biomarkers further divide the disease into subgroups

For the Covid-19 situation, if we purely base the orphan designation on the # of patients, it will meet the criteria in the beginning, then not meet the criteria, and meet the criteria again. In any way, obtaining an orphan drug designation for remdesivir in treating Covid-19 patients is indeed unethical.  

Six Pivotal Studies Investigating the Efficacy of Remdesivir: Does Calling it Phase II or III Matter?

During the Covid-19 pandemic, people are desperate to find effective treatments to combat the coronavirus. One promising drug is Gilead's Remdesivir. Although Gilead developed remdesivir for Ebola (and did not get FDA's approval though), which belongs to a different family of viruses than SARS-CoV-2, the viral machinery has elements in common. The remdesivir has been repurposed to be used to treat the Covid-19 infected patients under compassionate use or expanded access program and to be used in the clinical trials.

While there are individual cases and anecdotal experience, the efficacy and safety still need to be demonstrated through the pivotal clinical trials - the adequate, well-controlled clinical trials.

There are currently six pivotal clinical trials involving Remdesivir (summarized below). If the information provided in clinicaltrials.gov is accurate, we should be able to get a readout about Remdesivir's efficacy in the coming weeks. The first study to have the efficacy readout is the phase III study in China for severe Covid-19 patients (estimated primary completion date: Apri 3, 2020). Given that the Covid-19 outbreak in China has been under control and very few new cases are reported in China, two studies in China should be wrapped up even though there may be short of the required sample sizes.

The largest trial in Covid-19 is the French study with 3100 patients to be enrolled. In this study, Remdesivir will be compared with standard of care, also head-to-head compared with three other treatments: Lopinavir/ritonavir, interferon Beta-1A, Hydroxychloroquine. The projected study completion date is March 2023 - by that time, the Covid-19 should be long gone.  

Protocol Title	Study Features	Primary Efficacy Endpoints
A Phase 3 Randomized Study to Evaluate the Safety and Antiviral Activity of Remdesivir (GS-5734™) in Participants With Severe COVID-19	Gilead Sciences Phase III, 400 Subjects Three arms: Remdesivir for 5 days, Remdesivir for 10 days, Standard of care Multi-national： US, Hong Kong, Italy, South Korea, Singapore, Spain, Taiwan Estimated Primary Completion Date: May 2020	Proportion of Participants With Normalization of Fever and Oxygen Saturation Through Day 14 This is a composite outcome measure. Criteria for fever normalization: Temperature < 36.6 °C armpit, < 37.2 °C oral, or < 37.8 °C rectal sustained for at least 24 hours and criteria for oxygen normalization: peripheral capillary oxygen saturation (Sp02) > 94% sustained for at least 24 hours.
A Phase 3 Randomized Study to Evaluate the Safety and Antiviral Activity of Remdesivir (GS-5734™) in Participants With Moderate COVID-19 Compared to Standard of Care Treatment	Gilead Sciences Phase III, 600 Subjects Three arms: Remdesivir for 5 days, Remdesivir for 10 days, Standard of care Multi-national： US, Hong Kong, Italy, South Korea, Singapore, Spain, Taiwan Estimated Primary Completion Date: May 2020	Proportion of Participants Discharged by Day 14
A Phase 3 Randomized, Double-blind, Placebo-controlled Multicenter Study to Evaluate the Efficacy and Safety of Remdesivir in Hospitalized Adult Patients With Mild and Moderate 2019-nCoV Respiratory Disease.	Capital Medical University/Chinese Academy of Medical Sciences Phase III, 308 Subjects Two arms: Remdesivir, placebo Mainland China only Estimated Primary Completion Date: April 10, 2020	Time to Clinical recovery (TTCR)  TTCR is defined as the time (in hours) from initiation of study treatment (active or placebo) until normalisation of fever, respiratory rate, and oxygen saturation, and alleviation of cough, sustained for at least 72 hours. Normalisation and alleviation criteria: Fever - ≤36.9°C or -axilla, ≤37.2 °C oral, Respiratory rate - ≤24/minute on room air, Oxygen saturation - >94% on room air, Cough - mild or absent on a patient reported scale of severe, moderate, mild, absent.
A Phase 3 Randomized, Double-blind, Placebo-controlled, Multicenter Study to Evaluate the Efficacy and Safety of Remdesivir in Hospitalized Adult Patients With Severe 2019-nCoV Respiratory Disease.	Capital Medical University Phase III, 453 Subjects Two arms: Remdesivir, placebo Mainland China only Estimated Primary Completion Date: April 3, 2020	Time to Clinical Improvement (TTCI) [Censored at Day 28] TTCI is defined as the time (in days) from initiation of study treatment (active or placebo) until a decline of two categories from status at randomisation on a six-category ordinal scale of clinical status which ranges from 1 (discharged) to 6 (death). Six-category ordinal scale: 6. Death; 5. ICU, requiring ECMO and/or IMV; 4. ICU/hospitalization, requiring NIV/ HFNC therapy; 3. Hospitalization, requiring supplemental oxygen (but not NIV/ HFNC); 2. Hospitalization, not requiring supplemental oxygen; 1. Hospital discharge or meet discharge criteria (discharge criteria are defined as clinical recovery, i.e. fever, respiratory rate, oxygen saturation return to normal, and cough relief).
A Multicenter, Adaptive, Randomized Blinded Controlled Trial of the Safety and Efficacy of Investigational Therapeutics for the Treatment of COVID-19 in Hospitalized Adults	National Institute of Allergy and Infectious Diseases (NIAID) Phase II, 440 Subjects Two arms: Placebo, Remdesivir with additional arms to be added Multi-National: US, Japan, South Korea, Singapore Estimated Primary Completion Date: April 3, 2020	Percentage of subjects reporting each severity rating on an 8-point ordinal scale. The ordinal scale is an assessment of the clinical status at the first assessment of a given study day. The scale is as follows: 1) Death; 2) Hospitalized, on invasive mechanical ventilation or ECMO; 3) Hospitalized, on non-invasive ventilation or high flow oxygen devices; 4) Hospitalized, requiring supplemental oxygen; 5) Hospitalized, not requiring medical care (COVID-19 related or otherwise); 6) Hospitalized, not requiring supplemental oxygen - no longer requires ongoing medical care; 7) Not hospitalized, limitation on activities and/or requiring home oxygen; 8) Not hospitalized, no limitation on activities.
Multi-centre, Adaptive, Randomized Trial of the Safety and Efficacy of Treatments of COVID-19 in Hospitalized Adults	Institut National de la Santé Et de la Recherche Médicale, France Phase III, 3100 Subjects Four arms: Remdesivir, Lopinavir/ritonavir, Interferon Beta-1A, Hydroxychloroquine, Standard of care France Only Estimated Primary Completion Date: March, 2023	Percentage of subjects reporting each severity rating on a 7-point ordinal scale a. Not hospitalized, no limitations on activities b. Not hospitalized, limitation on activities; c. Hospitalized, not requiring supplemental oxygen; d. Hospitalized, requiring supplemental oxygen; e. Hospitalized, on non-invasive ventilation or high flow oxygen devices; f. Hospitalized, on invasive mechanical ventilation or ECMO; g. Death.

While five of these pivotal trials are labeled as Phase III study, only the NIH study (with 440 subjects) is labeled as Phase II study. Several weeks ago, I attended a seminar and three statistical professors discussed the Remdesivir clinical trials and the difficulties in patient recruitments. One professor spent quite some time discussing why two trials in China were labeled as Phase III while the NIH trial was labeled as Phase II - he mistakenly thought that only studies labeled as phase III could be used to support the product approval/registration.

Traditionally, clinical trials are phased: pre-marketing: phase I, phase II, and phase III studies; post-marketing: phase IV study. However, nowadays, these phases are blurred. A clinical trial labeled as phase II can still be sufficient to support the product approval/registration as long as the study is adequate and well-controlled. With adaptive design, some of the different phases of clinical trials are combined - seamless design. In the oncology area, a phase I study with cohort expansion may be sufficient for product approval (see FDA's guidance "Expansion Cohorts: Use inFirst-In-Human Clinical Trialsto Expedite Development ofOncology Drugs and Biologics" and Prowell et al "Seamless Oncology-Drug Development".

21CFR part 314 section 126 defined Adequate and Well-Controlled Studies to provide substantial evidence to demonstrate the efficacy of an investigational drug - there is no mention of the phase of the study.

Sec. 314.126 Adequate and well-controlled studies.

(a) The purpose of conducting clinical investigations of a drug is to distinguish the effect of a drug from other influences, such as spontaneous change in the course of the disease, placebo effect, or biased observation. The characteristics described in paragraph (b) of this section have been developed over a period of years and are recognized by the scientific community as the essentials of an adequate and well-controlled clinical investigation. The Food and Drug Administration considers these characteristics in determining whether an investigation is adequate and well-controlled for purposes of section 505 of the act. Reports of adequate and well-controlled investigations provide the primary basis for determining whether there is "substantial evidence" to support the claims of effectiveness for new drugs. Therefore, the study report should provide sufficient details of study design, conduct, and analysis to allow critical evaluation and a determination of whether the characteristics of an adequate and well-controlled study are present.

Similarly, in FDA's guidance (very important one) "Demonstrating Substantial Evidence of Effectiveness for Human Drug and Biological Products Guidance for Industry", there is no mention of the phases of clinical trials.

If a clinical trial is qualified for 'adequate and well-controlled', it can be used to demonstrate the substantial evidence of effectiveness regardless of the study phase.

PICO, PICOTS, PICOTT Framework for Clinical Questions as a Way to Design Clinical Trials

The PICO process (or framework) is a mnemonic used in evidence-based medicine to frame and answer a clinical question. The PICO framework is also used to develop literature search strategies, for instance in systematic reviews. The PICO acronym stands for:
        P – Patient, Problem or Population
        I – Intervention
       C – Comparison, control or comparator
       O – Outcome(s) (eg. pain, fatigue, nausea, infections, death)

The PICO framework can also be used in designing and planning a clinical trial. In doing so, people  suggest adding T and S to form PICOTS or PICOTT:
        T - Timing, duration or date of publication (eg. measured at 1-month of follow-up)
        S - Study type (eg. randomized controlled trial), sometimes S can be used to stand for Setting or 

              Sample Size

In a book by Matchar DB. Introduction to the Methods Guide for Medical Test Reviews, the PICO and PICOTS frameworks were discussed:

When well built, clinical questions usually have four components:
P: The patient situation, population, or problem of interest.
I: The main intervention, defined very broadly, including an exposure, a diagnostic test, a prognostic factor, a treatment, a patient perception and so forth.
C: A comparison intervention or exposure (also defined very broadly), if relevant.
O: The clinical outcome(s) of interest, including a time horizon, if relevant.

In addition to the standard PICO components, the broader PICOTS framework is extremely useful and important for defining key clinical questions and assessing whether a given study is applicable or not. T refers to Timing and S refers to Setting or Study Design.
T: Timing, i.e. the time it takes to demonstrate an outcome OR the period in which patients are observed.
S: Setting (e.g. ambulatory settings including primary, specialty care and inpatient settings), or sometimes Study Design (such as a randomized controlled trial). 

The PICOTS framework is also discussed in a book by Samson and Schoelles "Chapter 2: Medical Tests Guidance (2) Developing the Topic and Structuring Systematic Reviews of Medical Tests: Utility of PICOTS, Analytic Frameworks, Decision Trees, and Other Frameworks"

The content below is from FDA's website "Using the PICOTS Framework to Strengthen Evidence Gathered in Clinical Trials—Guidance from the AHRQ’s Evidence-based Practice Centers Program"

P: Patient population

Define the patient population that will be studied in the trial and consider how it compares to the general affected population. Consider patient baseline sociodemographic (e.g., age, race, socioeconomic status) and clinical characteristics (e.g., severity of condition, comorbidities) that may contribute to differences in treatment outcomes or treatment preferences. Define the selection criteria and consider how patients in the study may be diagnosed or treated differently in usual clinical care. Consider biases that may be introduced by the selection of patients or attrition of patients.

I: Intervention

Define the intervention, including all of its components. Consider contextual factors, such as prior, concurrent, posttreatments, or specialized training of the provider, which may affect the safety and/or effectiveness of the intervention.

C: Comparator

Define whether there is a placebo or active control comparator. Consider blinding. For placebo-controlled studies, consider the risk and benefit of using sham comparators. An active comparator should be relevant to current practice. If the comparator is “usual care,” define the components of the “usual care” clearly. Do not select an active comparator that has known poor effectiveness in specific subgroup populations.

O: Outcome

Define the safety and effectiveness outcomes that matter to patients and which predict long-term successful results. If surrogate outcomes, such as biochemical or physiological measures, are used, they should be clinically relevant. Consider the validity and reliability of outcome measures, including composite measures. Define the planned outcome measures and analyses in the protocol. Pre-specify subgroup analyses. Report all findings as defined in the protocol. Note any post hoc analyses.

T: Timing

Define the duration of treatment and the follow-up schedule that matter to patients. Consider both long- and short-term outcomes.

S: Setting

Define the setting (primary, specialty, inpatient, nursing homes, or other long-term care setting) where the study is implemented and the relevance of the study setting to real world use.

In summary, trials that provide high strength of evidence:

• Study patients who are likely to be offered the intervention in everyday practice.
• Examine clinical strategies and complexities that are more likely to be replicated in practice.
• Measure the most relevant set of benefits and harms.
• Have low risk of bias.
• Have adequate power to address subgroups.
• Directly compare interventions.
• Include all important intended and unintended effects including adherence and tolerability.

In Duke's website for Evidence-Based Practice, the PICOTT framework was defined as the following:

PATIENT OR PROBLEMHow would you describe a group of patients similar to yours? What are the most important characteristics of the patient?INTERVENTION, EXPOSURE, PROGNOSTIC FACTORWhat main intervention are you considering? What do you want to do with this patient?COMPARISONWhat is the main alternative being considered, if any?OUTCOMEWhat are you trying to accomplish, measure, improve or affect?Type of QuestionTherapy / Diagnosis / Harm / Prognosis / PreventionType of StudySystematic review / RCT / cohort study / case control

The other day, Dr. Sheng Lou from Duke University reviewed the clinical trial design for Remdesivir in treating Coronavirus infection and he used the PICOTS framework. There are two ongoing pivotal studies in China: one for mild/moderate patients and one for severe patients.

Pre-specifications of the statistical analysis plan - story of Ampligen in Chronic Fatigue Syndrome

In the previous post, 'Pre-specification and SAP', various regulatory guidelines and guidance were cited to demonstrate the importance of the pre-specification of the statistical analysis plan in clinical trials - especially the pivotal clinical trials. In the eye of the FDA reviewers, the most critical issue is not to choose or switch the statistical analysis method after the study unblinding or after the sponsor has the knowledge of the unblinded results. If the cherry-picking is allowed, a negative study might be turned into a positive study. 

One example is a drug Ampligen developed by Hemispherx Biopharma Inc. Hemispherx developed Ampligen for the treatment of chronic fatigue syndrome (Myalgic encephalomyelitis). Prior to the PDUFA date, an advisory committee meeting was organized. The discussion of whether or not Ampligen is efficacious was focused on the statistical analysis using untransformed data (result not statistically significant)or log-transformed data (result statistically significant). FDA accused the sponsor of changing to the statistical analysis using untransformed data after the planned analysis using log-transformed data was not statistically significant. 

• FDA Advisory Meeting on Ampligen and Chronic Fatigue Syndrome (ME/CFS) Pt I: Dueling Presentations Set the Stage For Critical Vote
• Help Simmaron AMPlify therapies for M.E.!
• FDA Advisory Committee Makes Recommendations on Ampligen for Chronic Fatigue Syndrome
• 2012 Meeting Materials, Arthritis Advisory Committee
• Webcast of FDA Advisory Committee Meeting for Ampligen in Chronic Fatigue Syndrome

To “log transform” or not to was one of the major issues the FDA raised in their background report; the fact that log untransformed data had shown that Ampligen had a significantly positive benefit while log-transformed data indicated the drug (almost did but) didn’t have that effect.

This technical question would dog Hemispherx throughout the hearing; and they would ultimately answer it but one had the feel that it was too late…that the damage had been done. Hemispherx’s statistician, with years of FDA experience under his belt, showed instructions from the FDA stating that log transformation should not be used unless necessary because it could skew the data, and then

The FDA officials were focused on something else, though, when and why Hemispherx decided to log transform or not the data. The big question was whether Hemispherx saw the data before it decided whether or not to transform the data. The statistician appeared to argue that Hemispherx had to check the variance to determine if the transformation was warranted but in the end stated the biostatistician who prepared the data was no longer with the company and they didn’t know. That was a huge blow…

Never mind that Hemispherx had demonstrated that the log transformation data was not warranted or that non transformed data was appropriate …the FDA was mostly interested in whether the biostatistician had ‘followed the rules’. It was a bizarre thing as a patient to watch a drug that could help ill people be held up on procedural issues but there it was.

This is how the FDA approves drugs? More impromptu debate than rigorous analysis the discussion session kind of flowed along from topic to topic with a pro-FDA moderator calling the shots. In fact, an actual debate, with each side arguing pro’s and con’s of each issue would have been much better.

Instead of issues being drawn up, presented on the screens and then discussed in an organized manner with each side given equal time, the conversation lurched from topic to topic with the Hemispherx reps being frustrated spectators too many times.

After lunch, Hemispherx had appeared confident even after the morning pummeling they’d taken from the FDA team. They felt they had answers to the FDA’s concerns but after the meeting, several members of the team felt they simply were not provided the opportunity to produce them. The moderator did call on Hemispherx several times but, for the most part, the FDA personnel to held the floor.

The short early discussion period clearly left many questions hanging at a time when the issues were fresh in the reviewers minds but the later discussion period felt hurried as well.

Given the ad hoc nature of the discussion period drugs, companies must shake in their boots and investors must tremble when they approach these meetings. Then again, this is an FDA that gives companies 220 pages of background materials and a list of questions to be answered two days before the meeting. The FDA team clearly has the upper hand in these hearings and that’s apparently how they want it.

The consequence is that Ampligen was voted down by the advisory committee and NDA was subsequently rejected by FDA. The sponsor gave up the further pursuit of the Ampligen for the treatment of CFS. Instead, the sponsor changed its name to AIM ImmunoTech Inc. and retool the Ampligen for oncology indications.

On the question: "Based on the information included in the briefing materials and presentations, has the applicant provided sufficient efficacy and safety data to support marketing of Ampligen for the treatment of CFS?," the AAC voted 8 no, 5 yes and 1 non-vote.

Pre-specification and statistical analysis plan

For industry-sponsored clinical trials especially those trials for the regulatory submission purpose (i.e., adequate and well-controlled studies), a prespecified analysis plan is essential. In a previous post, When to Finalize the Statistical Analysis Plan (SAP)? the timing of the SAP development/finalization was discussed. 

According to the current guidelines (such as ICH E9 STATISTICAL PRINCIPLES FOR CLINICAL TRIALS", the SAP may be written as a separate document (as we usually do) and should be finalized before breaking the blind.

"5.1 Prespecification of the Analysis When designing a clinical trial the principal features of the eventual statistical analysis of the data should be described in the statistical section of the protocol. This section should include all the principal features of the proposed confirmatory analysis of the primary variable(s) and the way in which anticipated analysis problems will be handled. In case of exploratory trials this section could describe more general principles and directions. The statistical analysis plan may be written as a separate document to be completed after finalising the protocol. In this document, a more technical and detailed elaboration of the principal features stated in the protocol may be included. The plan may include detailed procedures for executing the statistical analysis of the primary and secondary variables and other data. The plan should be reviewed and possibly updated as a result of the blind review of the data and should be finalised before breaking the blind. Formal records should be kept of when the statistical analysis plan was finalised as well as when the blind was subsequently broken. If the blind review suggests changes to the principal features stated in the protocol, these should be documented in a protocol amendment. Otherwise, it will suffice to update the statistical analysis plan with the considerations suggested from the blind review. Only results from analyses envisaged in the protocol (including amendments) can be regarded as confirmatory.

In the statistical section of the clinical study report the statistical methodology should be clearly described including when in the clinical trial process methodology decisions were made (see ICH E3). "

Similar languages are in ICH E8 "GENERAL CONSIDERATIONS FOR CLINICAL TRIALS":

3.2.4 Analysis

The study protocol should have a specified analysis plan that is appropriate for the objectives and design of the study, taking into account the method of subject allocation, the measurement methods of response variables, specific hypotheses to be tested, and analytical approaches to common problems including early study withdrawal and protocol violations. A description of the statistical methods to be employed, including timing of any planned interim analysis(es) should be included in the protocol (see ICH E3, ICH E6 and ICH E9).  The results of a clinical trial should be analysed in accordance with the plan prospectively stated in the protocol and all deviations from the plan should be indicated in the study report. Detailed guidance is available in other ICH guidelines on planning of the protocol (ICH E6), on the analysis plan and statistical analysis of results (ICH E9) and on study reports (ICH E3).

However, there are randomized controlled studies being single-blind or no blinding (open-label study). According to the ICH E14 revision 1 "GENERAL CONSIDERATIONS FOR CLINICAL STUDIESE8(R1)", the statistical analysis plan should be finalized before the unblinding of study data (for blinded studies) and before the conduct of the study (for open-label studies):

5.1.6 Statistical Analysis

The statistical analysis of a study encompasses important elements necessary to achieving the study objectives. The study protocol should include a statistical methods section that is appropriate for the objectives and study design (ICH E6 and E9). A separate statistical analysis plan may be used to provide the necessary details for implementation. The protocol should be finalised before the conduct of the study, and the statistical analysis plan should be finalised before the unblinding of study data, or in the case of an open-label study, before the conduct of the study. These steps will increase confidence that important aspects of analysis planning were not based on accumulating data in the study or inappropriate use of external data, both of which can negatively impact the reliability of study results. For example, the choice of analysis methods in a randomised clinical trial should not change after examining unblinded study data, and external control subjects should not be selected based on outcomes to be used in comparative analyses with treated study subjects.

It is commonly accepted that the pre-specified analysis plan needs to be included in the protocol for primary efficacy endpoint and perhaps also the secondary efficacy endpoints. A separate statistical analysis plan will be prepared (usually after the study protocol has been implemented and some patient data (in a blinded fashion) is available).

FDA has several guidelines for FDA reviewers for their review of the SAP.  We can see what the FDA's expectations are for the SAP.

Good Review Practice: Statistical Review Template

Data and Analysis Quality

Review the quality and integrity of the submitted data. Examples of relevant issues include the following: 

• Whether it is possible to reproduce the primary analysis dataset, and in particular the primary endpoint, from the original data source 

• Whether it is possible to verify the randomized treatment assignments 

• Findings from the Division of Scientific Investigation or other source(s) that question the usability of the data 

• Whether the applicant submitted documentation of data quality control/assurance procedures (see ICH E3,1 section 9.6; also ICH E6,2 section 5.1) 

• Whether the blinding/unblinding procedures were well documented (see ICH E3, section 9.4.6)

• Whether a final statistical analysis plan (SAP) was submitted and relevant analysis decisions (e.g., pooling of sites, analysis population membership, etc.) were made prior to unblinding.

Applicants are expected to submit data of high quality and make it possible for the FDA to reproduce their results. In turn, FDA reviewers should provide adequate documentation so that the applicant or another data user could reproduce their independent findings. The level of documentation needed will depend on the complexity and novelty of the analysis. If an ordinary ANOVA or ANCOVA is used, for example, it would suffice to identify the dependent and independent variables. If a more unusual analysis is performed, then it may be necessary to provide code. The code should be either included in the report or put in an appropriate digital archive. 

Good Review Practice: Clinical Review of Investigational New DrugApplications

8. STATISTICAL ANALYSIS PLANS
Sponsors should be encouraged to include the SAP as part of the protocol, rather than providing it in a separate document, even if the SAP has not been finalized. If the SAP is changed late in the trial, particularly after the data may be available, it is critical for the sponsor to assure the FDA that anyone making such changes has been unaware of the results. Sponsors should be encouraged to describe the methods used to ensure compliance. Additional information on the principles of statistical analyses of clinical trials is available in ICH E9. 75 The review of the SAP requires close collaboration with the biostatistical reviewer. 

8.1 Planned Analyses Analyses intended to support a marketing application (generally analyses for the phase 3 efficacy trials) should be prospectively identified in the protocol and described in adequate detail. An incomplete description of the proposed analyses in the protocol can leave ambiguity after trial completion in how the trial will be analyzed. 

Nonprospectively defined analyses pose problems because they leave the possibility that various statistical methods were tried and only the most favorable analysis was reported. In such cases, the estimates of drug effect may be biased by the selection of the analysis, and the proper correction for such bias can be impossible to determine. Preplanning of analyses reduces the potential for bias and often reduces disputes between sponsors and the FDA on the interpretation of results. The same principles apply to supportive and/or sensitivity analyses. These analyses should be prospectively specified, despite the fact that the results of such analyses cannot be used as a substitute for the primary analysis. If the protocol pertains to a multinational trial, it is important that an analysis of the regional  differences be prespecified. Clinical reviewers should review these considerations for planned analyses in collaboration with statistical reviewers. 

Although detailed prespecification is essential for the primary efficacy analysis, the ability to interpret findings on other outcomes, such as important secondary efficacy endpoints for which a claim might be sought, is also dependent on the presence of a prospectively described analysis plan. Observations of potential interest, termed descriptive endpoints because the trial will almost always be underpowered in their respect, may be considered in a trial that is successful on its primary endpoint to further explore consistency in demographic subgroups (e.g., sex, age, and race) or evaluate regional differences in multinational trials. Safety outcomes are also important and should be specified prospectively. They will often not be part of the primary analysis unless the trial was designed to assess such an endpoint. Analyses not prospectively defined will in most cases be considered exploratory; see section 8.2.2.1, Descriptive Analysis, for potential use of such descriptive analyses. 

Interim analyses may play an important role in trial design. They present complex issues, including preservation of overall Type I error (alpha spending function), re-estimation of sample size, and stopping guidelines. Plans for interim analyses should be prospectively determined and reviewers should discuss these plans with the statistician. See section 8.1.3, Interim Analysis Plans, for further discussion of these plans. 

8.1.1 Adequacy of the Statistical Analysis Plan
When reviewing the SAP, it is critical to consider whether there is ambiguity about the planned analyses. Particular attention should be paid to the primary endpoint and how it will be analyzed. If there are multiple primary endpoints or analyses, the Type 1 error rate should be controlled appropriately. If there is a single primary endpoint, details of the analysis are important. For example, an SAP that defines the primary analysis as a comparison of the time to event between treatment arms leaves open many possibilities, such as the specific analytical approach (e.g., Cox regression, log rank test), whether the analyses will be adjusted for covariates (and which covariates would be included), and the method for this adjustment. Censoring for subjects who drop out of the trial or who are lost to follow-up should be discussed, particularly since dropout may not be random. Post dropout follow-up may have different implications for superiority and noninferiority trials. 

Consideration also should be paid to other preplanned analyses, such as secondary endpoint analysis, population subset analysis, regional analysis, and interim analysis. Both clinical and statistical reviewers should collaborate in order to make appropriate recommendations. 

When there are possible secondary efficacy endpoints (e.g., different time points, population subsets, different statistical tests, different outcome measures), it is critical to determine how they will be analyzed and their role in the efficacy assessment. In general, secondary analyses are not considered in regulatory decision-making unless there is an effect on the primary endpoint, so that no Type 1 error adjustment is needed for the primary endpoint. A secondary endpoint intended to represent a trial finding (and thus a possible claim) after success on the primary endpoint should be considered as part of the overall SAP and, if there is more than one of these, a multiplicity adjustment or gatekeeper approach may be necessary to protect the Type 1 error rate at a desired level (alpha = 0.05) for such analyses. Positive results in a secondary analysis when the primary endpoint did not demonstrate a statistically significant difference generally will not be considered evidence of effectiveness. 

Protection of the overall (family-wide) Type 1 error rate at a desired level (alpha = 0.05) is essential when the protocol has designated multiple hypotheses testing. Examples include efficacy comparisons among multiple doses with respect to primary and secondary endpoints, subpopulation analysis, and regional analysis. Various commonly used statistical procedures can be used for this multiplicity adjustment (e.g., Bonferroni, Dunnett, Hochberg, Holm, Hommel, and gatekeeping procedures), and these procedures will be considered in a multiplicity guidance under development. The proper use of each procedure depends on the priority of the hypotheses to be tested and the definition of a successful trial outcome. The following two examples are illustrative:

Example 1. A placebo-controlled trial with one primary endpoint and three treatment doses (low, medium, and high) is planned. To assess the efficacy of the three doses as compared to placebo, a commonly used hierarchical procedure tests sequentially from high dose to low against placebo, each at alpha = 0.05, until a pvalue ≤ 0.05 is not attained for a dose. Significance is then declared for all doses that achieved a p-value ≤ 0.05. 

The Bonferroni correction approach also can be used to share alpha = 0.05 among the three doses and test each one at alpha = 0.05/3 = 0.017. This method will be less efficient than the sequential method, if the effect is likely to be positively associated with dose. The primary analysis could also evaluate all three doses pooled versus placebo (less efficient if the low doses are not effective) or of the two highest doses versus placebo. 

Example 2. A placebo-controlled trial of two endpoints, A and B, and three treatment doses (low, medium, and high) is planned. Suppose endpoint A is thought to be more indicative of the true effect than B and so is placed higher in the hierarchy than B. Also, suppose the medium and high doses are hypothesized to be equally effective while the low dose is considered less likely to exhibit significance. Multiple clinical decision rules are designated in hierarchical order to demonstrate the efficacy: 

• Show benefit for each of two higher doses individually compared to placebo with respect to endpoint A and endpoint B 

• Show benefit for the two higher doses pooled compared to placebo with respect to endpoint A or B 

• Show benefit for the low dose with respect to endpoint A 

• Show benefit for the low dose with respect to endpoint B 

Although the Bonferroni, Holm, or Hommel procedure can be applied to test these four hypotheses, a gatekeeper procedure that sequentially tests the four sets of hypotheses in hierarchical order, each at alpha = 0.05, is likely to be more efficient. 

How missing data (particularly from dropouts) are handled can profoundly affect trial outcomes. Sponsors should be encouraged to detail how they plan to minimize dropouts and to specify particular methods for assessing data from dropouts. It is not credible to design these analyses once unblinded data are available. Reviewers should consider the best approach for the particular situation, recognizing that such classic methods as last observation carried forward (known as LOCF) can bias a trial for or against a drug, depending on the reasons for attrition, the time course of the disease, and response to treatment. Dropout may not be random, as subjects may drop out of either the new drug or the control therapy for toxicity or for lack of efficacy. Depending on the cause of the dropout, the use of modeling approaches might have advantages. 

8.1.2 Reviewing Changes to the Statistical Analysis Plan

Changes to critical elements of the analysis (e.g., the primary endpoint, handling of dropouts) during a trial can raise concerns regarding bias, specifically whether the changes could reflect knowledge of unblinded data. Concerns are inevitably greatest when the change is made late and has an important effect on outcome. In theory, if such changes are unequivocally made blindly (e.g., because of data from other trials or careful reconsideration) they should not pose problems, but the assurance of blinding can be hard to provide. For obvious reasons, changes made with data in hand (but purportedly still blinded) pose the greatest difficulties and are hard to support. 

When changes to the original SAP are proposed during the course of conducting the trial, it is critical to determine exactly what information, if any, regarding trial outcomes was available to those involved in proposing the change. Changes made with knowledge of results can introduce bias that can be substantial and impossible to measure. Note that such biases can occur subtly (e.g., the likelihood of adoption of a proposal made by an individual with no knowledge of data can be influenced by the comments or nonverbal communication of an individual who does have such knowledge). Therefore, major protocol changes are not credible if knowledge of interim outcome data is available to any individual who is involved with those planning the change. If there is any potential for such changes, sponsors should be encouraged to describe fully who has had access to data and how the firewalls were maintained, among other information. 

After trial data collection is completed, and before unblinding, there is often a blinded data cleanup phase. During that phase, previously unaddressed specific concerns about the data may be identified (e.g., types and amounts of missing data, concomitant therapies), and decisions are often made by the sponsor as to how to address those concerns. Typically, any changes made during this data cleanup phase should be minor clarifications of the SAP. If more than minor clarifications are made to the SAP, sponsors should be encouraged to submit these changes to the FDA for review as protocol amendments.

Good Review Practice: Good Review Management Principles and Practices for Effective IND Development and Review

Statistical analysis plan. Submission of a detailed statistical analysis plan (SAP) in the initial protocol submission for phase 3 protocols is not required by CDER regulations. However, review staff should strongly encourage sponsors to include the SAP in the initial protocol submission, because phase 3 protocols generally include a detailed section devoted to statistical methods that are closely linked to trial design.

Good Review Practice: Clinical Review Template

"Reviewers are expected to use the study/clinical trial protocol for discussions on study/clinical trial design and planned efficacy analyses and not the final report itself, because documentation of the study/clinical trial design and the statistical analysis plan within the final report are occasionally incomplete or inaccurate."
"With respect to adequate and well-controlled clinical trials, the reviewer should consider: • Minimization of bias (adequacy of blinding, randomization, endpoint committees, prospective statistical analysis plan, and identification of endpoints) • Choice of control group and the limitations of various choices, especially for historical controls or noninferiority clinical trials, including adequacy of documented effect size for the control drug"

"6.1.5 Analysis of Secondary Endpoint(s) Reviewers should describe the secondary endpoints and their potential supportive role. Was an analysis plan prespecified? Were the secondary endpoints considered for analysis as a hierarchical structure? Should any secondary endpoint be assessed if the primary endpoint fails to achieve statistical significance? "

Pre-specified analyses plan should also describe how the safety data should be analyzed even though the pre-specification for safety analyses are not as critical as the efficacy. Unlike the efficacy analyses where statistical analysis methods vary depending on the study design, the endpoint measure, and other issues, the analysis of safety data is relatively standardized. 

• Reviewer GuidanceConducting a Clinical Safety Review of a New Product Application and Preparing a Report on the Review
• Reviewer Guidance Conducting a Clinical Safety Review of a New Product Application and Preparing a Report on the Review