Virtual Clinical Trials

Traditionally the patient enrollment in clinical trials is always through the clinical sites or investigational sites (medical center, hospital, clinic,...). With the new technologies and the trend of patient centered approach in clinical trials, the concept of virtual clinical trial has been formed. In 2011, Pfizer announced their first virtual clinical trial. Today's NPR reported another virtual clinical trial in ALS "Simplified Study Aims To Quickly Test A Long-Shot ALS Treatment". If the virtual trial can be accepted by the medical community and the regulatory authorities, it will significantly lower the cost for conducting clinical trials.

Pragmatic Clinical Trials (PCT) or Large Simple Trials (LST)

Pragmatic trials or large simple trials are becoming a hot topic discussed in recent scientific meetings. These are not really new clinical trial designs, but the trials conducted more close to the real world. The concept of pragmatic trials is not new. I saw an article in British Medical Journal dated back to 1998 “Understanding controlled trials: What are pragmatic trials?” where it said “Trials of healthcare interventions are often described as either explanatory or pragmatic. Explanatory trials generally measure efficacy-the benefit a treatment produces under ideal conditions, often using carefully defined subjects in a research clinic. Pragmatic trials measure effectiveness-the benefit the treatment produces in routine clinical practice.”

FDA commissioner, Dr Califf, has long been a proponent of the pragmatic trials. In his speech at 8th Annual Conference on Statistical Issues in Clinical Trials, he called the traditional clinical trials as Mechanistic trials with intention to evaluate a biological or mechanistic hypothesis and Pragmatic trials are those with Intention to inform decision makers about health and healthcare.

The practical adaptation of pragmatic clinical trials is:
(1) an intent to inform decision-makers (patients,         clinicians, administrators, and policymakers), as opposed to elucidating a biological or social mechanism;
(2) an intent to enroll a population relevant to the decision in practice and representative of the patients/populations and clinical settings for whom the decision is relevant; and
(3) either an intent to
–        (a) streamline procedures and data collection so that the trial can focus on adequate power for informing the clinical and policy decisions targeted by the trial
–        (b) measure a broad range of outcomesPragmatic clinical trials fit for the purpose of informing decision-makers regarding the comparative balance of benefit and risk of a biomedical or behavioral health intervention at the individual or population level

New England Journal of Medicine recently published a paper “Pragmatic Trials” by Dr Ford and Norrie. They proposed nine dimensions for pragmatic trials.

Here are some useful information for pragmatic clinical trials.

• The Patient-Centered Outcomes Research Institute (PCORI), an independent nonprofit, nongovernmental organization located in Washington, DC was authorized by Congress in 2010. PCORI seeks to fund pragmatic clinical trials, large simple trials, or large-scale observational studies that compare two or more alternatives for addressing prevention, diagnosis, treatment, or management of a disease or symptom; improving healthcare system-level approaches to managing care; communicating or disseminating research results to patients, caregivers, or clinicians; or eliminating health or healthcare disparities.
•  Duke University has a website for Rethinking Clinical Trials – A Living Textbook of Pragmatic Clinical Trials 
• In a paper by Dr Patsopoulos “A pragmatic view on pragmatic trials”, the following was stated:

Clinical trials have been the main tool used by the health sciences community to test and evaluate interventions. Trials can fall into two broad categories: pragmatic and explanatory. Pragmatic trials are designed to evaluate the effectiveness of interventions in real-life routine practice conditions, whereas explanatory trials aim to test whether an intervention works under optimal situations. Pragmatic trials produce results that can be generalized and applied in routine practice settings. Since most results from exploratory trials fail to be broadly generalizable, the “pragmatic design” has gained momentum.

• NIH organized a workshop on “Ethical and Regulatory Issues of Pragmatic Clinical Trials Workshop” in June 2016. The videocast of the workshop is available to the public.
• Special Issue of Clinical Trials in 2015 included a total 12 articles discussing the ethical and regulatory issues in pragmatic clinical trials. These articles can be found from Duke website.
• Dr Sandra Kweder from FDA (2013) Opportunities for Large Simple Trials – An FDA Perspective 

•  FDA has increasing interest in clinical trial simplification 1) To decrease difficulty and cost of conducting trials  2) To encourage larger trials that are simple, pragmatic, and robust  3) To reduce unnecessary data collection (e.g., laboratory and/or safety data unlikely to result in a greater understanding of product), which is often cited as major reason physicians are unwilling to participate in clinical trials

• FDA emphasizes data quality, not quantity, and not proscriptive trial design

• FDA encourage the use of of Streamlined Trial Approaches  1) August 24, 2011: Draft Guidance: Oversight of ClinicalInvestigations: A Risk-Based Approach to Monitoring 2) February 2012: Draft Guidance: Determining the Extent ofSafety Data Collection Needed in Late Stage Premarketand Post approval Clinical Investigations (AE Lite)
• Large simple trials designs are consistent with FDA’s 1) Evolving overall approach to safety assessment which emphasizes a focus on information that is useful and informative rather than information that reflects what is already known 2) Risk-based monitoring approaches 3) Quality-by-Design and quality risk management initiative for clinical trials 

Here is a table I would compare the explanatory trials and the pragmatic trials:

Explanatory trials (or Mechanistic trials)	Pragmatic trials (or Large simple trials)
Trials for demonstrating the efficacy	Trials for demonstrating the effectiveness
Limited sample size	Large sample size
Extensive data collection on safety for example adverse events	Limited or selected data collection on safety for example adverse events
Extensive data collection on laboratory tests and vital signs	No or limited data collection on laboratory tests and vital signs
Many inclusion/exclusion criteria	Less inclusion/exclusion criteria
More strict in terms of the protocol deviations, and study drug compliance	More lenient in terms of the protocol deviations and study drug adherence
Homogeneous patient population	More heterogeneous patient population
Conducted in ideal setting or optimal situation	Conducted in real world setting
Results may not be generalizable	Results are more generalizable
Usually not for comparative effectiveness	Usually for comparative effectiveness
The trial results are usually used for drug approval	The trial results are usually used for decision making
Usually with Placebo as control	Usually with comparison of different active drugs
Usually for pre-market studies	Usually for post-market studies
Usually single sponsor	May be involved in multiple sponsors
Less likely to use electronic medical records (EMRs)	More likely to use electronic medical records (EMRs)

The pragmatic trials and the precision medicine seem to be two trends in the clinical trial fields. While the pragmatic trials encourage the large simple trial to mimic the real world situation, the precision medicine encourage the clinical trials with smaller and targeted population, enrichment design, and homogeneous population. 

Type III or Type 3 Error in Clinical Trials

Anybody who learns the statistics will be familiar with the concept of type I and type II error. In hypothesis testing, Type I error, also known as a “false positive”: the error of rejecting a null hypothesis when it is actually true. In other words, this is the error of accepting an alternative hypothesis (the real hypothesis of interest) when the results can be attributed to chance. Plainly speaking, it occurs when we are observing a difference when in truth there is none (or more specifically - no statistically significant difference).

Type II error, also known as a "false negative": the error of not rejecting a null hypothesis when the alternative hypothesis is the true state of nature. In other words, this is the error of failing to accept an alternative hypothesis when you don't have adequate power. Plainly speaking, it occurs when we are failing to observe a difference when in truth there is one. In practice, the statistical power (equals to 1 - type II error) is commonly used. Power is the probability of rejecting the null hypothesis when the null hypothesis is indeed not true (i.e., the alternative hypothesis is true).

Now, there are also a concept of type III error. Fundamentally, Type III errors occur when researchers provide the right answer to the wrong question. While the term ‘type III error’ has been used in literature and presentations, the true meaning of ‘type III error’ is not clearly or consistently defined. People may use the term “type III error” to refer to different things.

In one of presentations, the type III error was used to describe those clinical trials that would have been successful but were not performed due to resource constraint

We make type III error when conclusion is not supported by the data

Type III error referring to an error by rejecting a null hypothesis but inferring the incorrect alternative hypothesis.

A type III error is where you correctly reject the null hypothesis, but it’s rejected for the wrong reason. This compares to a Type I error (incorrectly rejecting the null hypothesis) and a Type II error (not rejecting the null when you should). Type III errors are not considered serious, as they do mean you arrive at the correct decision. They usually happen because of random chance and are a rare occurrence. You can also think of a Type III error as giving the right answer (i.e. correctly rejecting the null) to the wrong question. Either way, you’re still arriving at the correct conclusion for the wrong reason. When we say the “wrong question”, that normally means you’ve formulated your hypotheses incorrectly. In other words, both your null and alternate hypotheses may be poorly worded or completely incorrect.

In a presentation slides titled “Type III and Type IV Errors: Statistical Decision-Making Considerations in addition to Rejecting and Retaining the Null Hypothesis”, type III error was used to refer to the wrong model, right answer and common influences on type III error would be:

• ·         Incorrect operationalization of variables
• ·         Poor theory (e.g., ad hoc explanations of findings)
• ·         Mis-identifying causal architecture (Schwartz & Carpenter, 1999)

Schwartz & Carpenter (1999) The Right Answer for the Wrong Question: Consequences of Type III Error for Public Health Research

In an editorial article of Arch Surg, the type III error was described as “the type III error occurs whenever the conclusions drawn are not supported by the data presented”. The author presented 5 examples using the published articles.

Type III error is solving the wrong problem precisely  – from Raiffa – 1968

Solving the wrong problem is defined as a Type III error by Howard Raiffa (1968, p. 264) and Ian Mitroff (1974). Type III errors are different from Type I and Type II errors, which involve setting the significance level too high or too low in testing the null hypothesis.

USING THE METHOD OF CONTEXT VALIDATION TO MITIGATE TYPE III ERRORS IN ENVIRONMENTAL POLICY ANALYSIS

in an article by Twardowski and Misra (2013) Con: Randomized controlled trials (RCT) have failed in the study of dialysis methods:

“The second trial (the HEMO study) committed a Type III statistical error asking the wrong question and did not bring any valuable results, but at least it did not lead to deterioration of dialysis outcomes in the USA”

Type III error — asking the wrong question and achieving the correct answer:

Kimball [7] postulated a Type III error, an error that gives the right answer to the wrong problem. A Type IV error was subsequently postulated as a type of error that solved the right problem too late [8].

in Flick, U. (2006). An introduction to qualitative research (3rd ed.). Thousand Oaks, CA: Sage.

Flick (2006), for example, discusses qualitative validity in terms of “whether researchers see what they think they see” (p. 371). Moreover, he and others (Kirk & Miller, 1986) argue that three types of error may occur as regards qualitative validity: seeing a relationship, a principle, and so on when they are not correct (Type I error); to reject them when they are correct (Type II error); and asking the wrong questions (Type III error). 

Green and Tones (1999) “For debate: Towards a secure evidence base for health promotion”,

Discussion so far has essentially been concerned with assessing the outcome of interventions and has ignored the nature of the intervention itself. Type III error refers to rejection of the effectiveness of a programme when the programme was inadequate in terms of design or delivery. This is neatly encapsulated in the acronym GIGO – garbage in, garbage out!

Tuck et al (1986) A defence of the small clinical trial: evaluation of three gastroenterological studies

The real worry clinically is the type III error, in which a clinically significantly inferior treatment is preferred to a superior one on the basis of insufficient dataTraditional Type I and II error describe the false positive and negative rates, Type III error describes the opportunity cost of not investigating valid hypotheses due to budgetary limitations

In an article by Counsell (2002) Predicting Outcome After Acute and Subacute Stroke Development and Validation of New Prognostic Models

Predictor variables must be easy to collect (to minimize missing data), clinically relevant, and reliable.The number of variables in multiple regression analyses must also be carefully controlled. Too few variables means that important predictors may be omitted, while too many variables can result in overfitting (a type I error in which false-positive predictors are erroneously included in the model); underfitting (a type II error in which important variables are omitted from the final model); and paradoxical fitting (a type III error in which a variable that, in truth, has a positive association with the outcome is found to have a negative association).The risk of these problems increases as the ratio of outcome events to the number of predictor variables becomes smaller (the events per variable [EPV] ratio, in which the number of events is the lower figure for binary outcomes). The risk of error is especially high with EPVs <10 .="" span="">

In an article by Robin et al (1990) “Type 3 and type 4 errors in the statistical evaluation of clinical trials”, the type III error was referred to 

“Type 3 errors, then, are errors in which the risks of a given medical or public health approach is underestimated, undetected or not specifically sought, leading to an underestimate of the risk-benefit balance.”. The type 3 error was further classified as three categories: type 3A errors arise from a failure to obtain sufficient data to determine the statistical significance of a given risk in an experimental versus a control group.  Type 3-B errors involve failures to look for or detect specific risks in an experimental versus a control group. Type 3-C errors involve risks in which the harm to subjects occurs months to years after the initial use of the modality As a result, the risk-benefit ratio of the modality is seriously underestimated.

Obviously, type III error from clinical trials has greater impact on health policy and medical practice because it involves in making the right decision. While the impact of type I and type II errors are the issues within a clinical trial, the impact of type III error goes beyond a clinical trial - if a type III error is committed, we could potentially adopted a wrong practice due to insufficient information .