Hodges-Lehmann estimator of location shift: Median of Differences versus Difference in Medians or Median Difference

Hodges-Lehmann estimator has been used to compare the treatment effect while the data is non-normal distributed. See my previous posts:

• Hodges-Lehmann Estimator
• Splitting p-value and estimate of the treatment difference

Many of the journal articles used Hodges-Lehmann estimator to the difference in two medians
In a study by Perkins et al "A Randomized Trial of Epinephrine in Out-of-Hospital Cardiac Arrest",

"The Hodges–Lehmann method was used to estimate median differences with 95% confidence intervals for length-of-stay outcomes"

In a study by Devinsky et al "Trial of Cannabidiol for Drug-Resistant Seizures in the Dravet Syndrome"

"Analysis of the primary end point was performed with the use of a Wilcoxon rank-sum test. An estimate of the median difference between cannabidiol and placebo, together with the 95% confidence interval, was calculated with the use of the Hodges–Lehmann approach. Sensitivity analyses of this primary end point were prespecified in the trial protocol and statistical analysis plan"

Similarly, Hodges-Lehmann estimator was used to estimating the treatment effect in licensure trials:

FDA Clinical/Statistical Review for Vascepa (icosapent ethyl) for reduction of triglycerides in patients with very high triglycerides

The median differences between the treatment groups and 95% CIs were estimated with the Hodges-Lehmann method. P-value is from the Wilcoxon rank-sum test.

FDA Statistical review for RLY5016 for Oral Suspension (Veltassa) for Hyperkalemia

To compare Veltassa with placebo, the difference between the mean ranks was tested using a two-sided t-test. The difference and 95% CI between the treatment groups in median change from baseline was estimated using a Hodges-Lehmann estimator.

FDA Medical Review of Oral Treprostinil for Pulmonary Arterial Hypertension

The magnitude of the treatment effects was defined by the Hodges-Lehmann method to estimate the median difference between treatment groups for the change from baseline in 6MWD.

It sounds like we have found a solution to estimate the difference in medians when the data is not normally distributed. However, if we look at how the Hodges-Lehmann is calculated, we will see that it is not accurate to say the Hodges-Lehmann estimator is to compare the difference in medians, it is actually the estimator of the location shift (the term originally used by the authors) or the estimator of the median of differences (further explained below).

• Hodges-Lehmann (1963) Estimates of Location Based on Rank Tests
• SAS Hodges-Lehmann Estimation of Location Shift
• R Tools "Hodges Lehmann: Hodges-Lehmann Estimator of Location"
• How to interpret the estimate from Wilcoxon signed rank paired test?
• Lingling Han (2008) Calculating the point estimate and confidence interval of HodgesLehmann's median using SAS® software
• Elise Coudin and Jean-Marie Dufour, McGill University (2009): Hodges-Lehmann sign-based estimators and generalized confidence distributions in linear median regressions with heterogenous dependent errors

Let's check how medians are calculated using a very simple example: 

Median and the difference in Medians:

	Group A	Group B
Original Measures	4, 7, 5, 3, 6	3, 2, 5, 1, 4
Rank the original measures in order	3, 4, 5, 6, 7	1, 2, 3, 4, 5
Median	5	3
The difference in Medians (A-B)	2

Hodges-Lehmann Estimator of Location Shif (median of differences)

	Group A	Group B
Original Measures	4, 7, 5, 3, 6	3, 2, 5, 1, 4
Rank the original measures in order	3, 4, 5, 6, 7	1, 2, 3, 4, 5
Each number in Group A is compared to each number in Group B	3 is compared to numbers in Group B:    2, 1, 0, -1, -2 4 is compared to numbers in Group B:    3, 2, 1, 0, -1 5 is compared to numbers in Group B:    4, 3, 2, 1, 0 6 is compared to numbers in Group B:    5, 4, 3, 2, 1 7 is compared to numbers in Group B:    6, 5, 4, 3, 2
Rank the differences from these pair comparisons in order	-2, -1, -1, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2, 2, 3, 3, 3, 3, 4, 4, 4, 5, 5, 6
Hodges-Lehmann estimator of location shift	Median of all these differences, in this case, the Hodges-Lehmann estimator is 2

The calculations of the medians can be implemented in the following SAS codes: 

data HodgesLehmann;

  input group $ number @@;

  datalines;

  A 3 A 4 A 5 A 6 A 7

  B 1 B 2 B 3 B 4 B 5

;

proc means data=hodgeslehmann median maxdec=0;

   class group;

   var number;

run;

proc npar1way data=hodgeslehmann hl;

   class group;

   var number;

run;

The Hodges-Lehmann estimation of the location shift is confirmed to be 2. In this example, the Hodges-Lehmann estimation of the location shift (2) is exactly the same as the differences in two medians (5-3 = 2). 

However, in many situations, the Hodges-Lehmann estimation of the location shift will be different from the differences between the two medians. the Hodges-Lehmann should really be called the median of differences between the two groups or the location shift (as the original authors used). 

The example below shows that the Hodges-Lehmann estimation of the location shift can be very different than the differences between the two medians. 

	Group A	Group B
Original Measures	50.6, 39.2, 35.2, 17.0, 11.2, 14.2, 24.2, 37.4, 35.2	38.0, 18.6, 23.2, 19.0, 6.6, 16.4, 14.4, 37.6, 24.4
Rank the original measures in order	11.2 14.2 17.0 24.2 35.2 35.2 37.4 39.2 50.6	6.6 14.4 16.4 18.6 19.0 23.2 24.4 37.6 38.0
Median	35.2	19.0
The difference in Medians (A-B)	16.2

data HodgesLehmann2;                   

   input Group $ number@@;

   datalines;

A 50.6

A 39.2

A 35.2

A 17.0

A 11.2

A 14.2 

A 24.2 

A 37.4 

A 35.2 

B 38.0 

B 18.6 

B 23.2 

B 19.0 

B 6.6 

B 16.4 

B 14.4 

B 37.6 

B 24.4 

; 

proc means data=hodgeslehmann2 median maxdec=1;

  class group;

  var number;

run;

proc npar1way data=hodgeslehmann2 hl;

  class group;

  var number;

run;

As illustrated above, the Hodges-Lehmann estimation of the location shift is 7.8, however, the difference between two medians is 35.2 - 19.0 = 16.2 (the median for groups A is 35.2 and the median for Group B is 19.0).

While the Hodges-Lehmann estimator is often used to measure the treatment difference when the data is not normally distributed, we need to understand how the Hodges-Lehmann is calculated and how Hodges-Lehmann estimator can be very different than the simple difference between two medians. 

The Use of Restricted Mean Survival Time (RMST) Method When Proportional Hazards Assumption is in Doubt

In a recent article from AJRCCM (American Journal of Respiratory and Critical Care Medicine), Harhay et al discussed "An Alternative Approach for the Analysis of Time-to-Event and Survival Outcomes in Pulmonary Medicine'. The alternative approach discussed in the paper is called 'restricted mean survival time' or RMST in short.

In analyzing the time to event data, the most common approach is to draw Kaplan-Meier plots and then use non-parametric method (log-rank test or Wilcoxon test) to compare two different survival curve or use semi-parametric method (proportional hazard model) to perform the regression-type analyses to estimate the magnitude of the treatment difference. A key assumption is that the proportional hazards as the name of the method suggest. What it essentially means is that the ratio of the hazards for any two individuals or for any two groups is constant over time. However, in a lot of situations, the proportional hazard assumption may not hold - we call it non-proportional hazards. If we look at the Kaplan-Meier plots and see two curves crossover, it is likely there exist non-proportional hazards.

In a presentation by FDA statisticians (John Lawrence, Junshan Qiu, Steven Bai, and Jim Hung) "Comparison of Hazard Ratio and Restricted Mean Survival Analysis for Cardiorenal Drug Trials", several examples of the survival data with non-proportional hazards were presented. In the situation of the non-proportional hazards, the common approach such as the Cox proportional hazard model will give a biased estimate.

There are various methods to test the proportional hazard assumption. Please see the link below for details "testing the proportional hazard assumption in Cox models"

In the situation that the proportional hazard assumption is violated, the alternative approach should be explored. One approach coming in handy is the Restricted Mean Survival Time (RMST) method.

The RMST represents the area under the survival curve from time 0 to a specific follow-up time point; it is called restricted mean survival time because given X as the time until any event, the expectation of X (mean survival time) will be the area under the survival function (from 0 to infinity). RMST can be interpreted as the average time until an event occurs during a defined time period ranging from time 0 to a specific follow-up time point.

In the FDA's presentation above, there were final remarks about the RMST method: 

• RMST which is directly related to patient’s survival/event-free time, is viable for quantifying treatment effect. • RMST can give better clinical interpretation of treatment effect.
• The results came from a R function. Yesterday, I found that someone in Lily actually developed SAS macro for RMST.

In the article by Harhay et al, there were also the final comments:

"As shown in these examples, the RMST offers several inferential advantages over other
time-to-event statistics. Though we examined survival, any time-to-event endpoint can be assessed using the RMST approach. Statistical inference (i.e., estimation and hypothesis testing) using the RMST, including p-values, confidence intervals, and covariate-adjustment, can be performed in most popular statistical software packages, such as R and STATA. Study group comparisons using the RMST estimate also confer comparable statistical power to the log-rank test and test for the HR in many situations, thereby providing an alternative and clinically meaningful measure of time gained or lost to inform research and patient care."

Programs have been developed to calculate the RMST.

There is an R package developed by Uno H:

• https://cran.r-project.org/web/packages/survRM2/vignettes/survRM2-vignette3-2.html

In SAS, there was a SAS macro available:

• https://pdfs.semanticscholar.org/5411/7b8fb137211bcab04534017abf12cc581eb6.pdf
• http://bcb.dfci.harvard.edu/~huno/computer-program/rmst2_ver003.sas
• Calculating Restricted Mean Survival Time

Other References:

• An application of restricted mean survival time in a competing risks setting: comparing time to ART initiation by injection drug use
• Comparison of the restricted mean survival time with the hazard ratio in superiority trials with a time-to-event end point: Comparison of the RMST with the HR
• FDA NDA Medical Review for 207997Orig1s000 and 207997Orig2s000
• Tai-Tsang Chen "Alternative Survival Endpoints" from Immuno-Oncology Drug Development Workshop
• Royston P, Parmar M. The use of restricted mean survival time to estimate the treatment effect in randomized clinical trials when the proportional hazards assumption is in doubt. Stat Med. 2011;30:2409-2421.

Adjudication Committee (AC), Endpoint Adjudication Committee (EAC), Clinical Endpoint Committee (CEC)

The clinical trials are getting bigger and more complicated these days. For typical multi-national, multi-center pivotal clinical trials, there will be a lot of committees formed: steering committee (SC), data monitoring committee (DMC), central reader, and adjudication committee, each with specific responsibilities.

Adjudication committee (AC) may also be called endpoint adjudication committee (EAC) or clinical endpoint committee (CEC) and are usually needed when the study endpoints are subjective measures.

What is the Adjudication Committee?

Adjudication committee is an independent group of experts that reviews clinical trial data in order to give expert opinions about clinical safety or efficacy events of interest.

According to the FDA’s Guidance for Industry “Establishment and Operation of Clinical Trial Data Monitoring Committees “, the adjudication committee is mentioned the following:

3.3. Endpoint Assessment/Adjudication Committees Sponsors may also choose to establish an endpoint assessment/adjudication committee (these may also be known as clinical events committees) in certain trials to review important endpoints reported by trial investigators to determine whether the endpoints meet protocol-specified criteria. Information reviewed on each presumptive endpoint may include laboratory, pathology and/or imaging data, autopsy reports, physical descriptions, and any other data deemed relevant. These committees are typically masked to the assigned study arm when performing their assessments regardless of whether the trial itself is conducted in a blinded manner. Such committees are particularly valuable when endpoints are subjective and/or require the application of a complex definition, and when the intervention is not delivered in a blinded fashion. Although such committees do not share responsibility with DMCs for evaluating interim comparisons, their assessments (if performed at frequent intervals throughout the trial with results incorporated into the database in a timely manner) help to ensure that the data reviewed by DMCs are as accurate and free of bias as possible.

Which Clinical Trials Need a Clinical Endpoint Adjudication Committee?

Increasingly, regulatory authorities are placing significant focus on clinical trial processes that ensure consistent, standardized, objective and unbiased reporting of safety and efficacy results; given that the definitions for many endpoint events include subjective components, and investigator-to-investigator subjective assessments may differ. Moreover, an increasing number of trials are now conducted in multiple geographies, and clinical practices across these settings can vary substantially. The likelihood of discrepant interpretations of safety and efficacy endpoints by investigators is thus increased.

Throughout a clinical trial, therefore, it is expected by regulatory agencies that certain events that form safety or efficacy endpoints for the study undergo centralized adjudication by a clinical endpoint adjudication committee (CEC).

A CEC consists of a panel of independent experts who have the relevant therapeutic area expertise, are experienced in clinical trials and have been trained on the specific study protocol. The CEC centrally reviews subject/event data and classifies efficacy and/or safety endpoints in a blinded and unbiased manner. The centralized adjudication process should be designed to both preserve the independence of the CEC and prevent any undue bias that could impact its decision-making processes.

A CEC can be used in any therapeutic area where there is a need for an independent, accurate, consistent and standardized assessment of important study events. CECs are most commonly used in cardiovascular outcome / safety studies; however, they are also frequently used in peripheral vascular disease, neurovascular, respiratory and oncology studies.

In some disease areas, the adjudication of the clinical endpoint is expected or even required by the regulatory agencies.

The study endpoint adjudicated by a central committee will be more reliable and is viewed as more credible.

Examples of Clinical Endpoints that Requires Adjudication

In Cardiovascular outcomes studies, a Major Adverse Cardiac Events (MACE) composite endpoint is often used as the primary endpoint for evaluating the efficacy and/ or safety. MACE is comprised of non-fatal myocardial infarction, non-fatal stroke and cardiovascular death. Once events are confirmed through centralised adjudication to meet protocol endpoint criteria, endpoint data is analysed for the number of occurrences of the composite endpoint in the respective treatment groups.

In the Cardio-Pulmonary field, the clinical trials in pulmonary arterial hypertension with composite morbidity/mortality endpoint will require the adjudication. European Medicines Agency (EMA) has a specific guideline “GUIDELINE ON THE CLINICAL INVESTIGATIONS OF MEDICINAL PRODUCTS FOR THE TREATMENT OF PULMONARY ARTERIAL HYPERTENSION” where a composite endpoint of clinical worsening can be used as a valid primary efficacy endpoint for establishing the efficacy.

The investigation of a composite primary endpoint that reflects, in addition to mortality, time to clinical worsening is encouraged. The composition of this composite endpoint may vary depending on the severity and the aetiology of the disease. The following components are suggested:
1. All-cause death.
2. Time to non-planned PAH-related hospitalization.
3. Time to PAH-related deterioration identified by at least one of the following parameters:
i. increase in WHO FC;
ii. deterioration in exercise testing
iii. signs or symptoms of right-sided heart failure

Any chosen parameter should be clinically relevant, adequately defined, well validated and centrally adjudicated by a blinded adjudication committee.

In oncology studies, the RECIST criteria are used to evaluate if the solid tumor has responded to the treatment. If the response rate is the primary efficacy endpoint, it is usually expected that a central reader facility will be used to review the images centrally to assess the treatment response. The central reader process is like the adjudication process. Here are some discussions about the adjudication in oncology trials.

• Adjudication in Oncology Trials: A Concept Whose Time Has Come
• Adjudication Rates between Readers in Blinded Independent Central Review of Oncology Studies
• Discrepancies of assessments in a RECIST 1.1 phase II clinical trial – the association between adjudication rate and variability in images and tumors selection
• Online Endpoint Adjudication in Oncology

Who are the Adjudicators in the Adjudication Committee?

Adjudication committee is an independent group of experts that reviews clinical trial data in order to give expert opinions about clinical safety or efficacy events of interest.

They are usually the expert in the academic setting. As an independent adjudicator, they cannot be the investigator for the study and cannot serve in other committees (such steering committee and data monitoring committee).

Will the Adjudication Committee be Blinded to the Treatment Assignment?

For double-blinded studies, the adjudication committee members are always blinded to the treatment assignment so that the adjudication is unbiased.

For open-label studies, the adjudication process can still be kept blinded.

What are the Regulatory Requirements for Adjudication Process?

For a study with endpoint adjudication committee, the source data for the primary efficacy endpoint analysis will be based on the adjudicated results – therefore, it is critical to ensure that the adjudication process is adequate, valid, and unbiased.

FDA emphases the importance of the adequate adjudication process in its document “The Use of Clinical Source Data in the Review of Marketing Applications”. In many cases, the adjudication process may be audited by the FDA.

Although many types of clinical source data require minimal or no interpretation after collection (e.g., blood pressure, cholesterol, or other discrete laboratory values), other clinical source data types require detailed interpretation by expert clinicians to assign endpoint values (i.e., endpoint adjudication (e.g., examination of radiographic images to measure tumor size, or examination of hospital records or accumulated data to determine whether a myocardial infarction has occurred)). How the applicant evaluates these source data can critically affect the reported results of the trial. In most cases, it would be expected that such interpretations are made blindly, whether conducted by investigators or special assessment groups (e.g., endpoint assessment committees (EACs)). It is equally critical that there be well-described, prospectively defined, evaluation criteria. In some cases, inspection of the clinical source data by clinical review staff may be necessary to establish the reliability of the data in the CRFs and CRTs for FDA review.
However, the processes used to inspect these source data can themselves pose challenges.

Evaluation of clinical source data is often subjective and, depending on the procedures used, susceptible to bias that could affect both the values of clinical endpoints and the results of efficacy and safety analyses. An FDA audit that reveals deficiencies in endpoint adjudication may trigger the need for additional evaluation of the clinical source data, but the audit itself could be biased. Therefore, just as an applicant’s methods of adjudicating endpoints should be well-defined a priori and free of bias, FDA inspection of such data also should use well-specified audit procedures, generally blinded as to treatment assignment, agreed to before the audit to minimize bias.

On-site inspections related to endpoint adjudication may be warranted under certain circumstances. For example, review of some NDAs may raise questions as to whether proper procedures were followed on endpoint adjudication. Other examples include when re-adjudication requires special equipment only available at the clinical site to access the clinical source data, or when on-site visits are necessary to retrieve clinical source data for re-adjudication. 

If the audit determines that the data in the CRFs and CRTs are not reliable enough for review because of deficiencies in the applicant’s endpoint adjudication process, or in the quality of the actual source data itself, clinical review staff may conclude that a re-adjudication of the endpoints is necessary. Clinical review staff should establish acceptable re-adjudication procedures with the applicant, and the applicant is, in most instances, expected to conduct the re-adjudication and the appropriate reanalysis. Instances in which clinical review staff conduct the re-adjudication itself, excluding the applicant, should be rare and well-justified

Adjudication Committee Charter

A critical document for the adjudication process is the adjudication committee charter. The charter will define the composition of the committee, responsibilities of the adjudication committee, the adjudication process/flowchart, the size of the adjudication committee, …

• Endpoint Adjudication of Events Charter Template: What Should Be Included?
• Endpoint Adjudication of Events Charter Template. What Should Be Included? An Overview
• Development of a Charter for an Endpoint Assessment and Adjudication Committee

What is the Typical Adjudication Process?

A good whitepaper by Quintiles (now IQVIA) described the best practice for adjudication committee.

Establishment and Operation of Clinical Endpoint Committees: Best Practice for Implementation Across the Biopharmaceutical Industry

In the studies with the adjudication process that I was involved, a consensus adjudication process was employed. With a consensus adjudication process, the adjudicators must reach a consensus regarding the endpoint. Suppose we have three adjudicators; the case will be assigned to two independent adjudicators. If two independent adjudicators give the same assessment, the event is considered adjudicated. If two independent adjudicators disagree with the assessment, the event will be sent to the third adjudicator. The third adjudicator must agree with one of the initial adjudicators to close the case.

In some cases, the third adjudicator don’t agree with either of two adjudicators, a meeting may be needed to discuss the case and reach the consensus.

Will a Separate Electronic System be Needed for Adjudication Process?

Yes, usually a separate electronic system will be needed for the adjudication process. The adjudication system will be independent of the electronic data capture (EDC) system that is for collecting the clinical data.

Further Readings:

• Adjudication Strategies in Cardiovascular Outcomes Research
• Centralized adjudication of cardiovascular end points in cardiovascular and noncardiovascular pharmacologic trials: A report from the Cardiac Safety Research Consortium
• Clinical Events Committee ADJUDICATION EFFICIENCIES
• Endpoint Adjudication Data for Submission to FDA and EMA - What Should Be Included?

Placebo Effect, Honest Placebo, Open-label Placebo

In a previous post, I discussed the Placebo Effect and The Choice of Placebo. When a placebo-control trial failed, we would try to understand if it was because the experimental drug had no significant effect or because the placebo treatment had an effect. There could be multiple reasons for a placebo-controlled trial to fail even though the experimental drug was actually effective.

Recently, I read some articles about the placebo effect and people claimed that placebo did work. 

• Placebos DO work and even cause side effects such as headaches, thirst, and insomnia, leading scientist claims
• People Are Now Taking Placebo Pills to Deal With Their Health Problems—And It's Working
• The “honest” placebo: When drugs still work even though patients know they're fake

I believe that the placebo may have some effects, but only in diseases in the CNS and psychological area or in diseases with subjective symptom measures. I also think that the placebo will have an effect only if the patients who receive the placebo treatment do not know which treatment they are receiving (i.e., blinded).

A concept of ‘honest placebo’ blows my mind. The term ‘honest placebo’ is used to describe the open-label placebo – the patient knowingly taking the placebo. When we do placebo-controlled studies, we have always tried to conceal the placebo (through blinding) to avoid the potential biases. In studies with open-label placebo, the blinding is no longer necessary.

The prominent researcher in the area of the placebo effect is Dr. Ted Kaptchuk in Havard University, See Ted Kaptchuk’s TEDMED and his talk Placebo effects make good medicine better. It seems to be boring if someone spends all of his / her efforts to study the placebo effect. You would think that it will be difficult to get the funding to study the placebo effect. However, Dr. Ted Kaptchuk did receive the NIH grants for studying the placebo effect (for example his NIH grant Enhancing the Placebo Effect in Irritable Bowel Syndrome).

In assessing the placebo effect, the following needs to be considered:

• Quantifying the percentage of subjects with the placebo effect
• Don’t expect the placebo effect in diseases beyond certain disease areas (such as CNS, psychological,…). Don’t expect the placebo effect in cancers.
• Using the objective measure to determine if the placebo effects by subjective measures are real
• Considering the composition of the placebo treatment. See a previous post Placebo Effect and The Choice of Placebo
• Considering the treatment compliance (in both the experimental group and the placebo group)
• Considering the course of the disease (some disease symptoms have a pattern of fluctuation, relapse-remission pattern. 
• checking if the placebo effect is triggered by the concomitant medications or concurrent treatments (that the researchers may not be aware of).  

Visit Window for Longitudinal Studies

In the previous post, the time window was discussed. The time window is an issue in studies with relatively shorter durations (such as the clinical pharmacology studies and clinical trials in analgesic drugs). The measures outside the time window can still be used in analysis by using the actual time in calculations. The impact is relatively neglectable.   

In clinical trials with a longitudinal design where the study subjects are followed up at pre-specified schedules for a long period of time, the visit window is an issue to be handled. The visit schedules may not be followed due to the reasons of 1) patient side (for example, travel arrangement); 2) investigator side (for example, investigator may have other responsibilities and not able to see the patient on a specific day); 3) the study procedure cannot be performed on the specified visit day (for example, the CT, MRI, … may not be arranged at the exact visit date).

It is common that the visit window is allowed and specified in the protocol, in this way, not every out of window visit will be recorded as the protocol deviation. The visit window needs to be protocol specific and needs to specified according to the length of the study and interval of the study visits. A study with every four-week visit schedule may have a visit window +/- 1 week; a study with every six-month visit schedule may have a visit window +/- a month.

It is inevitable to have always some subjects with visit outside the visit window. The study will also allow having unscheduled visits and early termination visit (the early termination visit can occur at any time during the study). These visits and measures outside the visit window will need to be used in the analysis so that the missing data can be minimized, and the more representative measures are selected to be used in the analysis. This needs to be handled through programming (usually in analysis dataset programming or CDISC ADaM dataset programming). Generally, two steps are needed:

Specify the Visit Slotting Algorithm

Based on the actual visit date, allocate the actual measure to the specific visit. An example below is for a six-month (24 weeks exactly) study. The visit slotting algorithm can be specified as the following:

Nominal Visit (Scheduled Visit)	Scheduled Visit Day (Study Day)	Slotting Intervals
Baseline	1	-3 to 1 day
Week 1	8	2 to 11 days
Week 2	15	12 to 22 days
Week 4	29	23 to 43 days
Week 8	57	44 to 71 days
Week 12	85	72 to 99 days
Week 16	113	100 to 127 days
Week 20	141	128 to 155 days
Week 24	169	156 to 14 days after the last treatment dose

Selecting the Value for Analysis 

After all the observations have been slotted based on the algorithm above, it is very possible that there are multiple valid observations for an assessment within an assigned analysis visit. Only one of these observations will be used for summary statistics and analyses. In the ADaM program, this is to determine the ANL01FL (‘Y’ if the observation is selected for analysis)

The observation to be used is determined using the following hierarchy (in decreasing order):

The observation closest to the target study day

The latter observation, if 2 observations are equally close to the target study day

Impact on the Missing Data Imputation

If the missing value needs to be imputed, the imputation should be implemented after the above two steps. For missing values where the last observation carried forward (LOCF) algorithm is applied, it is always the last valid observation on treatment carried forward, even though this might not be the observation obtained by the above hierarchy and used in the summaries by visit window.

Impact on analyses using a mixed model such as MMRM (mixed model repeat measure)

After the slotting and determining the analysis flag (ANL01FL), the mixed model will be based on the data with ANL01FL =’Y’ (i.e., some observations within a visit window may be excluded from the analysis).  Can we use all observations and use the actual visit day in the mixed model analysis? If you try, you may run into the issue that the model does not converge. 

Time Windows for PK Blood Sampling

In the drug development process, especially the early non-clinical or clinical studies, it is critical to understand the DMPK. DMPK, or Drug Metabolism and Pharmacokinetics, is an important part of studies often referred to as ADME (Absorption, Distribution, Metabolism, and Elimination).   

• Absorption (how much and how fast, often referred to as the absorbed fraction or bioavailability)
• Distribution (where the drug is distributed, how fast and how extensive)
• Metabolism (how fast, what mechanism/route, what metabolite is formed, and whether they are active or toxic)
• Elimination (how fast, which route)

In clinical pharmacology studies, the purpose of the study is to characterize the pharmacokinetics profile to assess the bioavailability and bioequivalence. In clinical pharmacology studies, usually, the series of blood samples will be collected for measuring the concentration of the study drug and its metabolites. Based on the measured concentrations from the series blood samples, pharmacokinetics parameters can be calculated. The common pharmacokinetics parameters are AUC (area under the time-concentration curve), Cmax (maximum concentration), Tmax (time to maximum concentration), T1/2 (half life),…

When designing a clinical pharmacology study, it is critical to decide and select adequate time points for blood sampling. The ideal blood sampling scheme will include the time points close to the Tmax (so that we can get a good estimate for Cmax and Tmax) and have good spaced time points to characterize the elimination phase (so that we can get a good estimate for T1/2).

From the practicality standpoint, it is not easy to draw the blood samples at the exact time according to the sampling scheme specified in the study protocol. For the phase I study using healthy volunteers at confined clinical research unit, the time windows can be easily controlled. However, for PK studies in patients across many investigational sites, it was very difficult to keep all blood draws within the narrow windows.

It is very common that time windows are allowed for these blood sampling time points. For example, for sampling time at 1 hour (60 minutes) after the study drug administration, we may add a time window to allow the 1-hour sample to be drawn any time between 55 – 65 minutes after the study drug administration, denotes as 60 +/- 5 minutes. A time window of 5 minutes is allowed for this time point. If the blood drawn is outside the time window (for example outside 55 - 65 minutes), a protocol deviation will be recorded. If the data entry is through an EDC (electronic data capture) system, the edit checks are usually built in to track the deviations for sampling time outside windows.

There are some articles arguing the necessity of the time windows.

• Using Sampling “Windows” for PK Blood Samples
• Should you Include a +/- Time Window Around Collection of PK/PD Samples in a Clinical Trial?

The arguments are 1) when we calculate the pharmacokinetic parameters, the actual sampling times are used. The deviations from the nominal time or pre-specified time point have little impact on the calculation of the PK parameters. 2) if the time windows are set too tight and any outside time windows are recorded as protocol deviation, there will be a lot of unnecessary protocol deviations being recorded. If any outside time window is flagged/alerted during the data entry in EDC, it can be very annoying to the data entry person.

My experience is that we can keep the time windows in the study protocol, but not set the time windows too tight. The sites usually need to have some instructions about the time windows, not purely for the PK parameter calculations, but more for operations. The time windows can be set up as a suggestion and we can use the word ‘should’, not ‘must’ for following the time windows. In this way, not any out of window blood drawn will be automatically recorded as a protocol deviation. 

FDA launches new digital tool to help capture real world data from patients to help inform regulatory decision-making

FDA In Brief: FDA launches new digital tool to help capture real world data from patients to help inform regulatory decision-making

FDA’s MyStudies Application (App)

“There are a lot of new ways that we can use real world evidence to help inform regulatory decisions around medical products as the collection of this data gets more widespread and reliable. Better capture of real world data, collected from a variety of sources, has the potential to make our new drug development process more efficient, improve safety and help lower the cost of product development. If done right, it can also help make sure doctors and patients are better informed about the clinical use of new products, enabling them to make more effective, efficient medical choices. This will ultimately help us achieve better outcomes, and safer and more efficient use of expensive technology,” said FDA Commissioner Scott Gottlieb, M.D. “Today we’re unveiling the new MyStudies app for the collection of real world data. This digital platform enables developers to adapt our technology to advance new ways to access and use data collected directly from patients—with the necessary controls in place to ensure patient privacy. Our hope is that the collection of more real world data directly from patients, using a secure app, will lead to more efficient product development and assist with safety monitoring.”

Today the U.S. Food and Drug Administration is announcing the MyStudies app, a new mobile technology to foster the collection of real world evidence via patients’ mobile devices. Real world data can be collected from a variety of sources, such as electronic health records, claims and billing activities, and product and disease registries, as well as from patient-generated data including in home-use settings, or from data gathered from other sources, such as mobile devices.

As part of the agency’s work to foster greater opportunities around real world evidence, the FDA partnered with Kaiser Permanente on a pilot study to measure the functionality and engagement of the MyStudies app. Based on the successful outcome of the study, the FDA is now releasing the open source code and technical documents that will allow researchers and developers to customize and use the FDA’s newly created MyStudies app to expand the diversity of health information available for clinical trials and studies, while directly capturing the perspective of patients. By providing the open source code, the agency is providing a tool that sponsors and developers can adapt to advance their specific clinical trial and real world evidence needs, while also remaining compliant with the FDA’s regulations and guidance for data authenticity, integrity and confidentiality.

For example, patients may be able to securely enroll in and contribute data to traditional clinical trials, pragmatic trials, observational studies and registries. Sponsors may be able to customize their apps to administer questionnaires assessing patient-reported outcomes, symptom scales or patient reports of prescription and over-the-counter medication use.

The overall effort was led by David Martin, M.D., associate director for real world evidence in the Office of Medical Policy in the FDA’s Center for Drug Evaluation and Research, with a grant from the U.S. Department of Health and Human Services’ Patient Centered Outcomes Research Trust Fund. The open source code that serves as the foundation of the MyStudies app, as well as specifications for a secure patient data storage environment, were developed through a collaboration with Harvard Pilgrim Health Care Institute, LabKey and Boston Technology Corporation

Fitting Compartment Models Using PROC NLMIXED

There is a new article on SAS's website "Fitting Compartment Models Using PROC NLMIXED". It is nice to know that SAS Proc NLMIXED can be used for fitting the compartment model in pharmacokinetic (PK) analyses.

Usually, for PK analyses, the non-compartment model will be used - i.e., Cmax, Tmax,... are based on the observed values and AUC is calculated using the trapezoid rule. I would say that for more than 90% of times, the non-compartment model will be good enough for the PK analyses.

However, in some situations, the non-compartment model or non-compartment analysis (NCA) will not be adequate. One-compartment model with first-order elimination and two-compartment model with first-order distribution and elimination processes may be more appropriate.

For PK analyses using NCA, the most commonly used software is Phoenix WinNonLin that was developed by Pharsight and now owned by Certara. For PK analyses based on compartment models, the most commonly used software is NONMEM that was developed by UCSF and now managed by ICON plc.

Potential Unblinding due to Imbalance in Adverse Event Profiles

In a previous post “Is blinded study really blinded? - assessment of blinding/unblinding in clinical trials”, we discussed the potential unblinding if the subjects on different treatment groups experienced a different type of adverse events. For the ethic reasons, the informed consent form needs to list all the potential side effects of the experimental drug. As required by GCP, the investigator’s brochure needs to discuss the details of the potential side effects caused by the experimental drug. If there is an obvious imbalance in adverse event profiles between the two treatment groups (experiment drug versus control), it is possible that the study blinding is not well maintained and the patient or the investigator can guess or predict based on the AE profile which treatment group the patient is on.

This potential unblinding issue can be even more evident if the crossover design is employed where the same subject will have an chance to experience two different drugs. If there are differences in side effects, the subjects will be likely to feel the difference.

In FDA’s recent guidance for industry “Hematologic Malignancy and Oncologic Disease: Considerations for Use of Placebos and Blinding in Randomized Controlled Clinical Trials for Drug Product Development”, the potential unblinding issue was brought up.

“In many cases, because of the toxicity profile of the active treatment, patients and investigators may infer which treatment patients are receiving and thus use of a placebo control may not, in fact, blind the treatment. …”

The guidance also cited the ethic issue for maintaining blinding when the unblinding is necessary to give the best care for the subjects (patients).

“Continued blinding of patients and investigators at the time of disease progression or occurrence of serious adverse events presents additional challenges. For example, in a blinded immunotherapy trial, a patient who develops adverse events on the control arm may receive unnecessary treatments (e.g., immunosuppressive drug products including a high dose of glucocorticoids, cyclophosphamide, interleukin-6 antagonist, or infliximab) for management of adverse events incorrectly attributed to the investigational drug product. Maintaining the blind after disease progression could also affect a patient’s subsequent therapy, potentially preventing a patient who had been on a placebo arm from receiving an approved therapy, or delaying or preventing the patient’s entry into other clinical trials (for those trials of similar drug products that may have specific exclusion criteria based on prior treatment with an active drug or class of drugs). Unblinding would allow informed decision-making with respect to additional treatment 61 options (see below). “

In an analysis by Shah et al Adverse events appear to unblind clinical trials in irritable bowel syndrome, the result “suggests that higher AE incidence on active therapy is associated with more beneficial patient‐reported outcomes in IBS clinical trials. This raises the issue of spontaneous unblinding”

Maintaining the blinding is one of the cornerstones of the RCT (randomized, controlled trial). As mentioned in an article by Stefan Dürr Avoid Unintentional Unblinding In Clinical Trials

“The knowledge of the patient’s treatment could potentially lead to conscious or unconscious bias in the way the site staff recruits the patients (selection bias), how they are treated (performance bias), the assessment of endpoints (detection bias), the handling of withdrawals, and how data is excluded from analysis. The patient attitude to the treatment if known can lead to a difference in reporting symptoms (response bias) or withdrawal from the study (attrition bias). To reduce the chance of bias to the data, clinical trials should be conducted in a double-blind design whenever possible.”

However, in some situation especially when there is an imbalance in the side effects of two treatment groups, blinding may either be difficult to maintain or not be ethical to implement, and alternative approaches are needed. As suggested in FDA's guidance "

“Hematologic Malignancy and Oncologic Disease: Considerations for Use of Placebos and Blinding in Randomized Controlled Clinical Trials for Drug Product Development”, an open-label study or an add-on trial design may be more appropriate. In an early post, I also discussed some blinding techniques. 

Latin Squares for Constructing "Williams Designs" - Balanced for First-order Carry-over (Residual) Effects

Courtesy from the website http://statpages.info/latinsq.html, the list below contains the Williams design with 2 to 26 treatment groups. It is unlikely to have a clinical trial with 26 treatment groups. The most number of treatment groups I see is a study with 6 treatment groups. 

In one of my previous posts, I described the Williams design and its usefulness in phase I or clinical pharmacology studies. In a SAS SUGI paper "Generating Randomization Schedules Using SAS Programming", I described how to use SAS program to construct the randomization schedule for a study with Williams design. 

With the limited sample size, the Williams design is the most efficient design and minimizes the potential impact of the carryover effect. In Williams design with an even number of n treatment groups, there will be n number of sequences or an "n by n" square. Each treatment only occurs one time in one sequence, in one period. Furthermore, each treatment only follows another treatment one time. 

If the number of treatments (n) is an odd number, there will be 2 x n number of sequences or two n by n squares. Each treatment only occurs once in one sequence and in two periods. Each treatment only follows another treatment twice. 

With the Latin Square listed below, we can easily construct the crossover design with treatments, periods, and sequences. For example, a study design with 3 treatment groups will have the following assignments: three treatment groups (A, B, C), three periods (period 1, period 2, and period 3), and six sequences (ABC, BCA, CAB, CBA, ACB, and BAC). 

Treatment Sequence	Period 1	Period 2	Period 3
1	A	B	C
2	B	C	A
3	C	A	B
4	C	B	A
5	A	C	B
6	B	A	C

• Since it is the crossover design, there should be a washout period between each period. It is very common that the washout period is 5 times the half-life. 
• Randomization is for the sequence, not the treatment group. 
• Once we know the number of subjects is needed for each sequence, we will know the total number of subjects for the entire study. 
• Usually, the mixed model is needed for statistical analysis. The statistical model will need to consider the treatment, period, and sequence. An example of the SAS codes for analysis was provided in one of the previous posts "Cookbook SAS Codes for Bioequivalence Test in 2x2x2 Crossover Design"

Williams Design by the Number of Treatment Groups:

Size = 2

A B 
B A 
 

Size = 3

A B C 
B C A 
C A B 
C B A 
A C B 
B A C 
 

Size = 4

A B D C 
B C A D 
C D B A 
D A C B 
 

Size = 5

A B E C D 
B C A D E 
C D B E A 
D E C A B 
E A D B C 
D C E B A 
E D A C B 
A E B D C 
B A C E D 
C B D A E 
 

Size = 6

A B F C E D 
B C A D F E 
C D B E A F 
D E C F B A 
E F D A C B 
F A E B D C 
 

Size = 7

A B G C F D E 
B C A D G E F 
C D B E A F G 
D E C F B G A 
E F D G C A B 
F G E A D B C 
G A F B E C D 
E D F C G B A 
F E G D A C B 
G F A E B D C 
A G B F C E D 
B A C G D F E 
C B D A E G F 
D C E B F A G 
 

Size = 8

A B H C G D F E 
B C A D H E G F 
C D B E A F H G 
D E C F B G A H 
E F D G C H B A 
F G E H D A C B 
G H F A E B D C 
H A G B F C E D 
 

Size = 9

A B I C H D G E F 
B C A D I E H F G 
C D B E A F I G H 
D E C F B G A H I 
E F D G C H B I A 
F G E H D I C A B 
G H F I E A D B C 
H I G A F B E C D 
I A H B G C F D E 
F E G D H C I B A 
G F H E I D A C B 
H G I F A E B D C 
I H A G B F C E D 
A I B H C G D F E 
B A C I D H E G F 
C B D A E I F H G 
D C E B F A G I H 
E D F C G B H A I 
 

Size = 10

A B J C I D H E G F 
B C A D J E I F H G 
C D B E A F J G I H 
D E C F B G A H J I 
E F D G C H B I A J 
F G E H D I C J B A 
G H F I E J D A C B 
H I G J F A E B D C 
I J H A G B F C E D 
J A I B H C G D F E 
 
 
Size = 11

A B K C J D I E H F G 
B C A D K E J F I G H 
C D B E A F K G J H I 
D E C F B G A H K I J 
E F D G C H B I A J K 
F G E H D I C J B K A 
G H F I E J D K C A B 
H I G J F K E A D B C 
I J H K G A F B E C D 
J K I A H B G C F D E 
K A J B I C H D G E F 
G F H E I D J C K B A 
H G I F J E K D A C B 
I H J G K F A E B D C 
J I K H A G B F C E D 
K J A I B H C G D F E 
A K B J C I D H E G F 
B A C K D J E I F H G 
C B D A E K F J G I H 
D C E B F A G K H J I 
E D F C G B H A I K J 
F E G D H C I B J A K 
 

Size = 12

A B L C K D J E I F H G 
B C A D L E K F J G I H 
C D B E A F L G K H J I 
D E C F B G A H L I K J 
E F D G C H B I A J L K 
F G E H D I C J B K A L 
G H F I E J D K C L B A 
H I G J F K E L D A C B 
I J H K G L F A E B D C 
J K I L H A G B F C E D 
K L J A I B H C G D F E 
L A K B J C I D H E G F 
 
 
Size = 13

A B M C L D K E J F I G H 
B C A D M E L F K G J H I 
C D B E A F M G L H K I J 
D E C F B G A H M I L J K 
E F D G C H B I A J M K L 
F G E H D I C J B K A L M 
G H F I E J D K C L B M A 
H I G J F K E L D M C A B 
I J H K G L F M E A D B C 
J K I L H M G A F B E C D 
K L J M I A H B G C F D E 
L M K A J B I C H D G E F 
M A L B K C J D I E H F G 
H G I F J E K D L C M B A 
I H J G K F L E M D A C B 
J I K H L G M F A E B D C 
K J L I M H A G B F C E D 
L K M J A I B H C G D F E 
M L A K B J C I D H E G F 
A M B L C K D J E I F H G 
B A C M D L E K F J G I H 
C B D A E M F L G K H J I 
D C E B F A G M H L I K J 
E D F C G B H A I M J L K 
F E G D H C I B J A K M L 
G F H E I D J C K B L A M 
 
 
Size = 14

A B N C M D L E K F J G I H 
B C A D N E M F L G K H J I 
C D B E A F N G M H L I K J 
D E C F B G A H N I M J L K 
E F D G C H B I A J N K M L 
F G E H D I C J B K A L N M 
G H F I E J D K C L B M A N 
H I G J F K E L D M C N B A 
I J H K G L F M E N D A C B 
J K I L H M G N F A E B D C 
K L J M I N H A G B F C E D 
L M K N J A I B H C G D F E 
M N L A K B J C I D H E G F 
N A M B L C K D J E I F H G 
 
 
Size = 15

A B O C N D M E L F K G J H I 
B C A D O E N F M G L H K I J 
C D B E A F O G N H M I L J K 
D E C F B G A H O I N J M K L 
E F D G C H B I A J O K N L M 
F G E H D I C J B K A L O M N 
G H F I E J D K C L B M A N O 
H I G J F K E L D M C N B O A 
I J H K G L F M E N D O C A B 
J K I L H M G N F O E A D B C 
K L J M I N H O G A F B E C D 
L M K N J O I A H B G C F D E 
M N L O K A J B I C H D G E F 
N O M A L B K C J D I E H F G 
O A N B M C L D K E J F I G H 
I H J G K F L E M D N C O B A 
J I K H L G M F N E O D A C B 
K J L I M H N G O F A E B D C 
L K M J N I O H A G B F C E D 
M L N K O J A I B H C G D F E 
N M O L A K B J C I D H E G F 
O N A M B L C K D J E I F H G 
A O B N C M D L E K F J G I H 
B A C O D N E M F L G K H J I 
C B D A E O F N G M H L I K J 
D C E B F A G O H N I M J L K 
E D F C G B H A I O J N K M L 
F E G D H C I B J A K O L N M 
G F H E I D J C K B L A M O N 
H G I F J E K D L C M B N A O 
 

Size = 16

A B P C O D N E M F L G K H J I 
B C A D P E O F N G M H L I K J 
C D B E A F P G O H N I M J L K 
D E C F B G A H P I O J N K M L 
E F D G C H B I A J P K O L N M 
F G E H D I C J B K A L P M O N 
G H F I E J D K C L B M A N P O 
H I G J F K E L D M C N B O A P 
I J H K G L F M E N D O C P B A 
J K I L H M G N F O E P D A C B 
K L J M I N H O G P F A E B D C 
L M K N J O I P H A G B F C E D 
M N L O K P J A I B H C G D F E 
N O M P L A K B J C I D H E G F 
O P N A M B L C K D J E I F H G 
P A O B N C M D L E K F J G I H 
 

Size = 17

A B Q C P D O E N F M G L H K I J 
B C A D Q E P F O G N H M I L J K 
C D B E A F Q G P H O I N J M K L 
D E C F B G A H Q I P J O K N L M 
E F D G C H B I A J Q K P L O M N 
F G E H D I C J B K A L Q M P N O 
G H F I E J D K C L B M A N Q O P 
H I G J F K E L D M C N B O A P Q 
I J H K G L F M E N D O C P B Q A 
J K I L H M G N F O E P D Q C A B 
K L J M I N H O G P F Q E A D B C 
L M K N J O I P H Q G A F B E C D 
M N L O K P J Q I A H B G C F D E 
N O M P L Q K A J B I C H D G E F 
O P N Q M A L B K C J D I E H F G 
P Q O A N B M C L D K E J F I G H 
Q A P B O C N D M E L F K G J H I 
J I K H L G M F N E O D P C Q B A 
K J L I M H N G O F P E Q D A C B 
L K M J N I O H P G Q F A E B D C 
M L N K O J P I Q H A G B F C E D 
N M O L P K Q J A I B H C G D F E 
O N P M Q L A K B J C I D H E G F 
P O Q N A M B L C K D J E I F H G 
Q P A O B N C M D L E K F J G I H 
A Q B P C O D N E M F L G K H J I 
B A C Q D P E O F N G M H L I K J 
C B D A E Q F P G O H N I M J L K 
D C E B F A G Q H P I O J N K M L 
E D F C G B H A I Q J P K O L N M 
F E G D H C I B J A K Q L P M O N 
G F H E I D J C K B L A M Q N P O 
H G I F J E K D L C M B N A O Q P 
I H J G K F L E M D N C O B P A Q 
 
 

Size = 18

A B R C Q D P E O F N G M H L I K J 
B C A D R E Q F P G O H N I M J L K 
C D B E A F R G Q H P I O J N K M L 
D E C F B G A H R I Q J P K O L N M 
E F D G C H B I A J R K Q L P M O N 
F G E H D I C J B K A L R M Q N P O 
G H F I E J D K C L B M A N R O Q P 
H I G J F K E L D M C N B O A P R Q 
I J H K G L F M E N D O C P B Q A R 
J K I L H M G N F O E P D Q C R B A 
K L J M I N H O G P F Q E R D A C B 
L M K N J O I P H Q G R F A E B D C 
M N L O K P J Q I R H A G B F C E D 
N O M P L Q K R J A I B H C G D F E 
O P N Q M R L A K B J C I D H E G F 
P Q O R N A M B L C K D J E I F H G 
Q R P A O B N C M D L E K F J G I H 
R A Q B P C O D N E M F L G K H J I 
 

Size = 19

A B S C R D Q E P F O G N H M I L J K 
B C A D S E R F Q G P H O I N J M K L 
C D B E A F S G R H Q I P J O K N L M 
D E C F B G A H S I R J Q K P L O M N 
E F D G C H B I A J S K R L Q M P N O 
F G E H D I C J B K A L S M R N Q O P 
G H F I E J D K C L B M A N S O R P Q 
H I G J F K E L D M C N B O A P S Q R 
I J H K G L F M E N D O C P B Q A R S 
J K I L H M G N F O E P D Q C R B S A 
K L J M I N H O G P F Q E R D S C A B 
L M K N J O I P H Q G R F S E A D B C 
M N L O K P J Q I R H S G A F B E C D 
N O M P L Q K R J S I A H B G C F D E 
O P N Q M R L S K A J B I C H D G E F 
P Q O R N S M A L B K C J D I E H F G 
Q R P S O A N B M C L D K E J F I G H 
R S Q A P B O C N D M E L F K G J H I 
S A R B Q C P D O E N F M G L H K I J 
K J L I M H N G O F P E Q D R C S B A 
L K M J N I O H P G Q F R E S D A C B 
M L N K O J P I Q H R G S F A E B D C 
N M O L P K Q J R I S H A G B F C E D 
O N P M Q L R K S J A I B H C G D F E 
P O Q N R M S L A K B J C I D H E G F 
Q P R O S N A M B L C K D J E I F H G 
R Q S P A O B N C M D L E K F J G I H 
S R A Q B P C O D N E M F L G K H J I 
A S B R C Q D P E O F N G M H L I K J 
B A C S D R E Q F P G O H N I M J L K 
C B D A E S F R G Q H P I O J N K M L 
D C E B F A G S H R I Q J P K O L N M 
E D F C G B H A I S J R K Q L P M O N 
F E G D H C I B J A K S L R M Q N P O 
G F H E I D J C K B L A M S N R O Q P 
H G I F J E K D L C M B N A O S P R Q 
I H J G K F L E M D N C O B P A Q S R 
J I K H L G M F N E O D P C Q B R A S 
 

Size = 20

A B T C S D R E Q F P G O H N I M J L K 
B C A D T E S F R G Q H P I O J N K M L 
C D B E A F T G S H R I Q J P K O L N M 
D E C F B G A H T I S J R K Q L P M O N 
E F D G C H B I A J T K S L R M Q N P O 
F G E H D I C J B K A L T M S N R O Q P 
G H F I E J D K C L B M A N T O S P R Q 
H I G J F K E L D M C N B O A P T Q S R 
I J H K G L F M E N D O C P B Q A R T S 
J K I L H M G N F O E P D Q C R B S A T 
K L J M I N H O G P F Q E R D S C T B A 
L M K N J O I P H Q G R F S E T D A C B 
M N L O K P J Q I R H S G T F A E B D C 
N O M P L Q K R J S I T H A G B F C E D 
O P N Q M R L S K T J A I B H C G D F E 
P Q O R N S M T L A K B J C I D H E G F 
Q R P S O T N A M B L C K D J E I F H G 
R S Q T P A O B N C M D L E K F J G I H 
S T R A Q B P C O D N E M F L G K H J I 
T A S B R C Q D P E O F N G M H L I K J 
 

Size = 21

A B U C T D S E R F Q G P H O I N J M K L 
B C A D U E T F S G R H Q I P J O K N L M 
C D B E A F U G T H S I R J Q K P L O M N 
D E C F B G A H U I T J S K R L Q M P N O 
E F D G C H B I A J U K T L S M R N Q O P 
F G E H D I C J B K A L U M T N S O R P Q 
G H F I E J D K C L B M A N U O T P S Q R 
H I G J F K E L D M C N B O A P U Q T R S 
I J H K G L F M E N D O C P B Q A R U S T 
J K I L H M G N F O E P D Q C R B S A T U 
K L J M I N H O G P F Q E R D S C T B U A 
L M K N J O I P H Q G R F S E T D U C A B 
M N L O K P J Q I R H S G T F U E A D B C 
N O M P L Q K R J S I T H U G A F B E C D 
O P N Q M R L S K T J U I A H B G C F D E 
P Q O R N S M T L U K A J B I C H D G E F 
Q R P S O T N U M A L B K C J D I E H F G 
R S Q T P U O A N B M C L D K E J F I G H 
S T R U Q A P B O C N D M E L F K G J H I 
T U S A R B Q C P D O E N F M G L H K I J 
U A T B S C R D Q E P F O G N H M I L J K 
L K M J N I O H P G Q F R E S D T C U B A 
M L N K O J P I Q H R G S F T E U D A C B 
N M O L P K Q J R I S H T G U F A E B D C 
O N P M Q L R K S J T I U H A G B F C E D 
P O Q N R M S L T K U J A I B H C G D F E 
Q P R O S N T M U L A K B J C I D H E G F 
R Q S P T O U N A M B L C K D J E I F H G 
S R T Q U P A O B N C M D L E K F J G I H 
T S U R A Q B P C O D N E M F L G K H J I 
U T A S B R C Q D P E O F N G M H L I K J 
A U B T C S D R E Q F P G O H N I M J L K 
B A C U D T E S F R G Q H P I O J N K M L 
C B D A E U F T G S H R I Q J P K O L N M 
D C E B F A G U H T I S J R K Q L P M O N 
E D F C G B H A I U J T K S L R M Q N P O 
F E G D H C I B J A K U L T M S N R O Q P 
G F H E I D J C K B L A M U N T O S P R Q 
H G I F J E K D L C M B N A O U P T Q S R 
I H J G K F L E M D N C O B P A Q U R T S 
J I K H L G M F N E O D P C Q B R A S U T 
K J L I M H N G O F P E Q D R C S B T A U 
 


Size = 22

A B V C U D T E S F R G Q H P I O J N K M L 
B C A D V E U F T G S H R I Q J P K O L N M 
C D B E A F V G U H T I S J R K Q L P M O N 
D E C F B G A H V I U J T K S L R M Q N P O 
E F D G C H B I A J V K U L T M S N R O Q P 
F G E H D I C J B K A L V M U N T O S P R Q 
G H F I E J D K C L B M A N V O U P T Q S R 
H I G J F K E L D M C N B O A P V Q U R T S 
I J H K G L F M E N D O C P B Q A R V S U T 
J K I L H M G N F O E P D Q C R B S A T V U 
K L J M I N H O G P F Q E R D S C T B U A V 
L M K N J O I P H Q G R F S E T D U C V B A 
M N L O K P J Q I R H S G T F U E V D A C B 
N O M P L Q K R J S I T H U G V F A E B D C 
O P N Q M R L S K T J U I V H A G B F C E D 
P Q O R N S M T L U K V J A I B H C G D F E 
Q R P S O T N U M V L A K B J C I D H E G F 
R S Q T P U O V N A M B L C K D J E I F H G 
S T R U Q V P A O B N C M D L E K F J G I H 
T U S V R A Q B P C O D N E M F L G K H J I 
U V T A S B R C Q D P E O F N G M H L I K J 
V A U B T C S D R E Q F P G O H N I M J L K 
 


Size = 23

A B W C V D U E T F S G R H Q I P J O K N L M 
B C A D W E V F U G T H S I R J Q K P L O M N 
C D B E A F W G V H U I T J S K R L Q M P N O 
D E C F B G A H W I V J U K T L S M R N Q O P 
E F D G C H B I A J W K V L U M T N S O R P Q 
F G E H D I C J B K A L W M V N U O T P S Q R 
G H F I E J D K C L B M A N W O V P U Q T R S 
H I G J F K E L D M C N B O A P W Q V R U S T 
I J H K G L F M E N D O C P B Q A R W S V T U 
J K I L H M G N F O E P D Q C R B S A T W U V 
K L J M I N H O G P F Q E R D S C T B U A V W 
L M K N J O I P H Q G R F S E T D U C V B W A 
M N L O K P J Q I R H S G T F U E V D W C A B 
N O M P L Q K R J S I T H U G V F W E A D B C 
O P N Q M R L S K T J U I V H W G A F B E C D 
P Q O R N S M T L U K V J W I A H B G C F D E 
Q R P S O T N U M V L W K A J B I C H D G E F 
R S Q T P U O V N W M A L B K C J D I E H F G 
S T R U Q V P W O A N B M C L D K E J F I G H 
T U S V R W Q A P B O C N D M E L F K G J H I 
U V T W S A R B Q C P D O E N F M G L H K I J 
V W U A T B S C R D Q E P F O G N H M I L J K 
W A V B U C T D S E R F Q G P H O I N J M K L 
M L N K O J P I Q H R G S F T E U D V C W B A 
N M O L P K Q J R I S H T G U F V E W D A C B 
O N P M Q L R K S J T I U H V G W F A E B D C 
P O Q N R M S L T K U J V I W H A G B F C E D 
Q P R O S N T M U L V K W J A I B H C G D F E 
R Q S P T O U N V M W L A K B J C I D H E G F 
S R T Q U P V O W N A M B L C K D J E I F H G 
T S U R V Q W P A O B N C M D L E K F J G I H 
U T V S W R A Q B P C O D N E M F L G K H J I 
V U W T A S B R C Q D P E O F N G M H L I K J 
W V A U B T C S D R E Q F P G O H N I M J L K 
A W B V C U D T E S F R G Q H P I O J N K M L 
B A C W D V E U F T G S H R I Q J P K O L N M 
C B D A E W F V G U H T I S J R K Q L P M O N 
D C E B F A G W H V I U J T K S L R M Q N P O 
E D F C G B H A I W J V K U L T M S N R O Q P 
F E G D H C I B J A K W L V M U N T O S P R Q 
G F H E I D J C K B L A M W N V O U P T Q S R 
H G I F J E K D L C M B N A O W P V Q U R T S 
I H J G K F L E M D N C O B P A Q W R V S U T 
J I K H L G M F N E O D P C Q B R A S W T V U 
K J L I M H N G O F P E Q D R C S B T A U W V 
L K M J N I O H P G Q F R E S D T C U B V A W 
 

Size = 24

A B X C W D V E U F T G S H R I Q J P K O L N M 
B C A D X E W F V G U H T I S J R K Q L P M O N 
C D B E A F X G W H V I U J T K S L R M Q N P O 
D E C F B G A H X I W J V K U L T M S N R O Q P 
E F D G C H B I A J X K W L V M U N T O S P R Q 
F G E H D I C J B K A L X M W N V O U P T Q S R 
G H F I E J D K C L B M A N X O W P V Q U R T S 
H I G J F K E L D M C N B O A P X Q W R V S U T 
I J H K G L F M E N D O C P B Q A R X S W T V U 
J K I L H M G N F O E P D Q C R B S A T X U W V 
K L J M I N H O G P F Q E R D S C T B U A V X W 
L M K N J O I P H Q G R F S E T D U C V B W A X 
M N L O K P J Q I R H S G T F U E V D W C X B A 
N O M P L Q K R J S I T H U G V F W E X D A C B 
O P N Q M R L S K T J U I V H W G X F A E B D C 
P Q O R N S M T L U K V J W I X H A G B F C E D 
Q R P S O T N U M V L W K X J A I B H C G D F E 
R S Q T P U O V N W M X L A K B J C I D H E G F 
S T R U Q V P W O X N A M B L C K D J E I F H G 
T U S V R W Q X P A O B N C M D L E K F J G I H 
U V T W S X R A Q B P C O D N E M F L G K H J I 
V W U X T A S B R C Q D P E O F N G M H L I K J 
W X V A U B T C S D R E Q F P G O H N I M J L K 
X A W B V C U D T E S F R G Q H P I O J N K M L 
 
 

Size = 25

A B Y C X D W E V F U G T H S I R J Q K P L O M N 
B C A D Y E X F W G V H U I T J S K R L Q M P N O 
C D B E A F Y G X H W I V J U K T L S M R N Q O P 
D E C F B G A H Y I X J W K V L U M T N S O R P Q 
E F D G C H B I A J Y K X L W M V N U O T P S Q R 
F G E H D I C J B K A L Y M X N W O V P U Q T R S 
G H F I E J D K C L B M A N Y O X P W Q V R U S T 
H I G J F K E L D M C N B O A P Y Q X R W S V T U 
I J H K G L F M E N D O C P B Q A R Y S X T W U V 
J K I L H M G N F O E P D Q C R B S A T Y U X V W 
K L J M I N H O G P F Q E R D S C T B U A V Y W X 
L M K N J O I P H Q G R F S E T D U C V B W A X Y 
M N L O K P J Q I R H S G T F U E V D W C X B Y A 
N O M P L Q K R J S I T H U G V F W E X D Y C A B 
O P N Q M R L S K T J U I V H W G X F Y E A D B C 
P Q O R N S M T L U K V J W I X H Y G A F B E C D 
Q R P S O T N U M V L W K X J Y I A H B G C F D E 
R S Q T P U O V N W M X L Y K A J B I C H D G E F 
S T R U Q V P W O X N Y M A L B K C J D I E H F G 
T U S V R W Q X P Y O A N B M C L D K E J F I G H 
U V T W S X R Y Q A P B O C N D M E L F K G J H I 
V W U X T Y S A R B Q C P D O E N F M G L H K I J 
W X V Y U A T B S C R D Q E P F O G N H M I L J K 
X Y W A V B U C T D S E R F Q G P H O I N J M K L 
Y A X B W C V D U E T F S G R H Q I P J O K N L M 
N M O L P K Q J R I S H T G U F V E W D X C Y B A 
O N P M Q L R K S J T I U H V G W F X E Y D A C B 
P O Q N R M S L T K U J V I W H X G Y F A E B D C 
Q P R O S N T M U L V K W J X I Y H A G B F C E D 
R Q S P T O U N V M W L X K Y J A I B H C G D F E 
S R T Q U P V O W N X M Y L A K B J C I D H E G F 
T S U R V Q W P X O Y N A M B L C K D J E I F H G 
U T V S W R X Q Y P A O B N C M D L E K F J G I H 
V U W T X S Y R A Q B P C O D N E M F L G K H J I 
W V X U Y T A S B R C Q D P E O F N G M H L I K J 
X W Y V A U B T C S D R E Q F P G O H N I M J L K 
Y X A W B V C U D T E S F R G Q H P I O J N K M L 
A Y B X C W D V E U F T G S H R I Q J P K O L N M 
B A C Y D X E W F V G U H T I S J R K Q L P M O N 
C B D A E Y F X G W H V I U J T K S L R M Q N P O 
D C E B F A G Y H X I W J V K U L T M S N R O Q P 
E D F C G B H A I Y J X K W L V M U N T O S P R Q 
F E G D H C I B J A K Y L X M W N V O U P T Q S R 
G F H E I D J C K B L A M Y N X O W P V Q U R T S 
H G I F J E K D L C M B N A O Y P X Q W R V S U T 
I H J G K F L E M D N C O B P A Q Y R X S W T V U 
J I K H L G M F N E O D P C Q B R A S Y T X U W V 
K J L I M H N G O F P E Q D R C S B T A U Y V X W 
L K M J N I O H P G Q F R E S D T C U B V A W Y X 
M L N K O J P I Q H R G S F T E U D V C W B X A Y 
 


Size = 26

A B Z C Y D X E W F V G U H T I S J R K Q L P M O N 
B C A D Z E Y F X G W H V I U J T K S L R M Q N P O 
C D B E A F Z G Y H X I W J V K U L T M S N R O Q P 
D E C F B G A H Z I Y J X K W L V M U N T O S P R Q 
E F D G C H B I A J Z K Y L X M W N V O U P T Q S R 
F G E H D I C J B K A L Z M Y N X O W P V Q U R T S 
G H F I E J D K C L B M A N Z O Y P X Q W R V S U T 
H I G J F K E L D M C N B O A P Z Q Y R X S W T V U 
I J H K G L F M E N D O C P B Q A R Z S Y T X U W V 
J K I L H M G N F O E P D Q C R B S A T Z U Y V X W 
K L J M I N H O G P F Q E R D S C T B U A V Z W Y X 
L M K N J O I P H Q G R F S E T D U C V B W A X Z Y 
M N L O K P J Q I R H S G T F U E V D W C X B Y A Z 
N O M P L Q K R J S I T H U G V F W E X D Y C Z B A 
O P N Q M R L S K T J U I V H W G X F Y E Z D A C B 
P Q O R N S M T L U K V J W I X H Y G Z F A E B D C 
Q R P S O T N U M V L W K X J Y I Z H A G B F C E D 
R S Q T P U O V N W M X L Y K Z J A I B H C G D F E 
S T R U Q V P W O X N Y M Z L A K B J C I D H E G F 
T U S V R W Q X P Y O Z N A M B L C K D J E I F H G 
U V T W S X R Y Q Z P A O B N C M D L E K F J G I H 
V W U X T Y S Z R A Q B P C O D N E M F L G K H J I 
W X V Y U Z T A S B R C Q D P E O F N G M H L I K J 
X Y W Z V A U B T C S D R E Q F P G O H N I M J L K 
Y Z X A W B V C U D T E S F R G Q H P I O J N K M L 
Z A Y B X C W D V E U F T G S H R I Q J P K O L N M