Placebo Effect, Honest Placebo, Open-label Placebo

In a previous post, I discussed the Placebo Effect and The Choice of Placebo. When a placebo-control trial failed, we would try to understand if it was because the experimental drug had no significant effect or because the placebo treatment had an effect. There could be multiple reasons for a placebo-controlled trial to fail even though the experimental drug was actually effective.

Recently, I read some articles about the placebo effect and people claimed that placebo did work. 

• Placebos DO work and even cause side effects such as headaches, thirst, and insomnia, leading scientist claims
• People Are Now Taking Placebo Pills to Deal With Their Health Problems—And It's Working
• The “honest” placebo: When drugs still work even though patients know they're fake

I believe that the placebo may have some effects, but only in diseases in the CNS and psychological area or in diseases with subjective symptom measures. I also think that the placebo will have an effect only if the patients who receive the placebo treatment do not know which treatment they are receiving (i.e., blinded).

A concept of ‘honest placebo’ blows my mind. The term ‘honest placebo’ is used to describe the open-label placebo – the patient knowingly taking the placebo. When we do placebo-controlled studies, we have always tried to conceal the placebo (through blinding) to avoid the potential biases. In studies with open-label placebo, the blinding is no longer necessary.

The prominent researcher in the area of the placebo effect is Dr. Ted Kaptchuk in Havard University, See Ted Kaptchuk’s TEDMED and his talk Placebo effects make good medicine better. It seems to be boring if someone spends all of his / her efforts to study the placebo effect. You would think that it will be difficult to get the funding to study the placebo effect. However, Dr. Ted Kaptchuk did receive the NIH grants for studying the placebo effect (for example his NIH grant Enhancing the Placebo Effect in Irritable Bowel Syndrome).

In assessing the placebo effect, the following needs to be considered:

• Quantifying the percentage of subjects with the placebo effect
• Don’t expect the placebo effect in diseases beyond certain disease areas (such as CNS, psychological,…). Don’t expect the placebo effect in cancers.
• Using the objective measure to determine if the placebo effects by subjective measures are real
• Considering the composition of the placebo treatment. See a previous post Placebo Effect and The Choice of Placebo
• Considering the treatment compliance (in both the experimental group and the placebo group)
• Considering the course of the disease (some disease symptoms have a pattern of fluctuation, relapse-remission pattern. 
• checking if the placebo effect is triggered by the concomitant medications or concurrent treatments (that the researchers may not be aware of).  

Visit Window for Longitudinal Studies

In the previous post, the time window was discussed. The time window is an issue in studies with relatively shorter durations (such as the clinical pharmacology studies and clinical trials in analgesic drugs). The measures outside the time window can still be used in analysis by using the actual time in calculations. The impact is relatively neglectable.   

In clinical trials with a longitudinal design where the study subjects are followed up at pre-specified schedules for a long period of time, the visit window is an issue to be handled. The visit schedules may not be followed due to the reasons of 1) patient side (for example, travel arrangement); 2) investigator side (for example, investigator may have other responsibilities and not able to see the patient on a specific day); 3) the study procedure cannot be performed on the specified visit day (for example, the CT, MRI, … may not be arranged at the exact visit date).

It is common that the visit window is allowed and specified in the protocol, in this way, not every out of window visit will be recorded as the protocol deviation. The visit window needs to be protocol specific and needs to specified according to the length of the study and interval of the study visits. A study with every four-week visit schedule may have a visit window +/- 1 week; a study with every six-month visit schedule may have a visit window +/- a month.

It is inevitable to have always some subjects with visit outside the visit window. The study will also allow having unscheduled visits and early termination visit (the early termination visit can occur at any time during the study). These visits and measures outside the visit window will need to be used in the analysis so that the missing data can be minimized, and the more representative measures are selected to be used in the analysis. This needs to be handled through programming (usually in analysis dataset programming or CDISC ADaM dataset programming). Generally, two steps are needed:

Specify the Visit Slotting Algorithm

Based on the actual visit date, allocate the actual measure to the specific visit. An example below is for a six-month (24 weeks exactly) study. The visit slotting algorithm can be specified as the following:

Nominal Visit (Scheduled Visit)	Scheduled Visit Day (Study Day)	Slotting Intervals
Baseline	1	-3 to 1 day
Week 1	8	2 to 11 days
Week 2	15	12 to 22 days
Week 4	29	23 to 43 days
Week 8	57	44 to 71 days
Week 12	85	72 to 99 days
Week 16	113	100 to 127 days
Week 20	141	128 to 155 days
Week 24	169	156 to 14 days after the last treatment dose

Selecting the Value for Analysis 

After all the observations have been slotted based on the algorithm above, it is very possible that there are multiple valid observations for an assessment within an assigned analysis visit. Only one of these observations will be used for summary statistics and analyses. In the ADaM program, this is to determine the ANL01FL (‘Y’ if the observation is selected for analysis)

The observation to be used is determined using the following hierarchy (in decreasing order):

The observation closest to the target study day

The latter observation, if 2 observations are equally close to the target study day

Impact on the Missing Data Imputation

If the missing value needs to be imputed, the imputation should be implemented after the above two steps. For missing values where the last observation carried forward (LOCF) algorithm is applied, it is always the last valid observation on treatment carried forward, even though this might not be the observation obtained by the above hierarchy and used in the summaries by visit window.

Impact on analyses using a mixed model such as MMRM (mixed model repeat measure)

After the slotting and determining the analysis flag (ANL01FL), the mixed model will be based on the data with ANL01FL =’Y’ (i.e., some observations within a visit window may be excluded from the analysis).  Can we use all observations and use the actual visit day in the mixed model analysis? If you try, you may run into the issue that the model does not converge. 

Time Windows for PK Blood Sampling

In the drug development process, especially the early non-clinical or clinical studies, it is critical to understand the DMPK. DMPK, or Drug Metabolism and Pharmacokinetics, is an important part of studies often referred to as ADME (Absorption, Distribution, Metabolism, and Elimination).   

• Absorption (how much and how fast, often referred to as the absorbed fraction or bioavailability)
• Distribution (where the drug is distributed, how fast and how extensive)
• Metabolism (how fast, what mechanism/route, what metabolite is formed, and whether they are active or toxic)
• Elimination (how fast, which route)

In clinical pharmacology studies, the purpose of the study is to characterize the pharmacokinetics profile to assess the bioavailability and bioequivalence. In clinical pharmacology studies, usually, the series of blood samples will be collected for measuring the concentration of the study drug and its metabolites. Based on the measured concentrations from the series blood samples, pharmacokinetics parameters can be calculated. The common pharmacokinetics parameters are AUC (area under the time-concentration curve), Cmax (maximum concentration), Tmax (time to maximum concentration), T1/2 (half life),…

When designing a clinical pharmacology study, it is critical to decide and select adequate time points for blood sampling. The ideal blood sampling scheme will include the time points close to the Tmax (so that we can get a good estimate for Cmax and Tmax) and have good spaced time points to characterize the elimination phase (so that we can get a good estimate for T1/2).

From the practicality standpoint, it is not easy to draw the blood samples at the exact time according to the sampling scheme specified in the study protocol. For the phase I study using healthy volunteers at confined clinical research unit, the time windows can be easily controlled. However, for PK studies in patients across many investigational sites, it was very difficult to keep all blood draws within the narrow windows.

It is very common that time windows are allowed for these blood sampling time points. For example, for sampling time at 1 hour (60 minutes) after the study drug administration, we may add a time window to allow the 1-hour sample to be drawn any time between 55 – 65 minutes after the study drug administration, denotes as 60 +/- 5 minutes. A time window of 5 minutes is allowed for this time point. If the blood drawn is outside the time window (for example outside 55 - 65 minutes), a protocol deviation will be recorded. If the data entry is through an EDC (electronic data capture) system, the edit checks are usually built in to track the deviations for sampling time outside windows.

There are some articles arguing the necessity of the time windows.

• Using Sampling “Windows” for PK Blood Samples
• Should you Include a +/- Time Window Around Collection of PK/PD Samples in a Clinical Trial?

The arguments are 1) when we calculate the pharmacokinetic parameters, the actual sampling times are used. The deviations from the nominal time or pre-specified time point have little impact on the calculation of the PK parameters. 2) if the time windows are set too tight and any outside time windows are recorded as protocol deviation, there will be a lot of unnecessary protocol deviations being recorded. If any outside time window is flagged/alerted during the data entry in EDC, it can be very annoying to the data entry person.

My experience is that we can keep the time windows in the study protocol, but not set the time windows too tight. The sites usually need to have some instructions about the time windows, not purely for the PK parameter calculations, but more for operations. The time windows can be set up as a suggestion and we can use the word ‘should’, not ‘must’ for following the time windows. In this way, not any out of window blood drawn will be automatically recorded as a protocol deviation. 

FDA launches new digital tool to help capture real world data from patients to help inform regulatory decision-making

FDA In Brief: FDA launches new digital tool to help capture real world data from patients to help inform regulatory decision-making

FDA’s MyStudies Application (App)

“There are a lot of new ways that we can use real world evidence to help inform regulatory decisions around medical products as the collection of this data gets more widespread and reliable. Better capture of real world data, collected from a variety of sources, has the potential to make our new drug development process more efficient, improve safety and help lower the cost of product development. If done right, it can also help make sure doctors and patients are better informed about the clinical use of new products, enabling them to make more effective, efficient medical choices. This will ultimately help us achieve better outcomes, and safer and more efficient use of expensive technology,” said FDA Commissioner Scott Gottlieb, M.D. “Today we’re unveiling the new MyStudies app for the collection of real world data. This digital platform enables developers to adapt our technology to advance new ways to access and use data collected directly from patients—with the necessary controls in place to ensure patient privacy. Our hope is that the collection of more real world data directly from patients, using a secure app, will lead to more efficient product development and assist with safety monitoring.”

Today the U.S. Food and Drug Administration is announcing the MyStudies app, a new mobile technology to foster the collection of real world evidence via patients’ mobile devices. Real world data can be collected from a variety of sources, such as electronic health records, claims and billing activities, and product and disease registries, as well as from patient-generated data including in home-use settings, or from data gathered from other sources, such as mobile devices.

As part of the agency’s work to foster greater opportunities around real world evidence, the FDA partnered with Kaiser Permanente on a pilot study to measure the functionality and engagement of the MyStudies app. Based on the successful outcome of the study, the FDA is now releasing the open source code and technical documents that will allow researchers and developers to customize and use the FDA’s newly created MyStudies app to expand the diversity of health information available for clinical trials and studies, while directly capturing the perspective of patients. By providing the open source code, the agency is providing a tool that sponsors and developers can adapt to advance their specific clinical trial and real world evidence needs, while also remaining compliant with the FDA’s regulations and guidance for data authenticity, integrity and confidentiality.

For example, patients may be able to securely enroll in and contribute data to traditional clinical trials, pragmatic trials, observational studies and registries. Sponsors may be able to customize their apps to administer questionnaires assessing patient-reported outcomes, symptom scales or patient reports of prescription and over-the-counter medication use.

The overall effort was led by David Martin, M.D., associate director for real world evidence in the Office of Medical Policy in the FDA’s Center for Drug Evaluation and Research, with a grant from the U.S. Department of Health and Human Services’ Patient Centered Outcomes Research Trust Fund. The open source code that serves as the foundation of the MyStudies app, as well as specifications for a secure patient data storage environment, were developed through a collaboration with Harvard Pilgrim Health Care Institute, LabKey and Boston Technology Corporation

Fitting Compartment Models Using PROC NLMIXED

There is a new article on SAS's website "Fitting Compartment Models Using PROC NLMIXED". It is nice to know that SAS Proc NLMIXED can be used for fitting the compartment model in pharmacokinetic (PK) analyses.

Usually, for PK analyses, the non-compartment model will be used - i.e., Cmax, Tmax,... are based on the observed values and AUC is calculated using the trapezoid rule. I would say that for more than 90% of times, the non-compartment model will be good enough for the PK analyses.

However, in some situations, the non-compartment model or non-compartment analysis (NCA) will not be adequate. One-compartment model with first-order elimination and two-compartment model with first-order distribution and elimination processes may be more appropriate.

For PK analyses using NCA, the most commonly used software is Phoenix WinNonLin that was developed by Pharsight and now owned by Certara. For PK analyses based on compartment models, the most commonly used software is NONMEM that was developed by UCSF and now managed by ICON plc.

Potential Unblinding due to Imbalance in Adverse Event Profiles

In a previous post “Is blinded study really blinded? - assessment of blinding/unblinding in clinical trials”, we discussed the potential unblinding if the subjects on different treatment groups experienced a different type of adverse events. For the ethic reasons, the informed consent form needs to list all the potential side effects of the experimental drug. As required by GCP, the investigator’s brochure needs to discuss the details of the potential side effects caused by the experimental drug. If there is an obvious imbalance in adverse event profiles between the two treatment groups (experiment drug versus control), it is possible that the study blinding is not well maintained and the patient or the investigator can guess or predict based on the AE profile which treatment group the patient is on.

This potential unblinding issue can be even more evident if the crossover design is employed where the same subject will have an chance to experience two different drugs. If there are differences in side effects, the subjects will be likely to feel the difference.

In FDA’s recent guidance for industry “Hematologic Malignancy and Oncologic Disease: Considerations for Use of Placebos and Blinding in Randomized Controlled Clinical Trials for Drug Product Development”, the potential unblinding issue was brought up.

“In many cases, because of the toxicity profile of the active treatment, patients and investigators may infer which treatment patients are receiving and thus use of a placebo control may not, in fact, blind the treatment. …”

The guidance also cited the ethic issue for maintaining blinding when the unblinding is necessary to give the best care for the subjects (patients).

“Continued blinding of patients and investigators at the time of disease progression or occurrence of serious adverse events presents additional challenges. For example, in a blinded immunotherapy trial, a patient who develops adverse events on the control arm may receive unnecessary treatments (e.g., immunosuppressive drug products including a high dose of glucocorticoids, cyclophosphamide, interleukin-6 antagonist, or infliximab) for management of adverse events incorrectly attributed to the investigational drug product. Maintaining the blind after disease progression could also affect a patient’s subsequent therapy, potentially preventing a patient who had been on a placebo arm from receiving an approved therapy, or delaying or preventing the patient’s entry into other clinical trials (for those trials of similar drug products that may have specific exclusion criteria based on prior treatment with an active drug or class of drugs). Unblinding would allow informed decision-making with respect to additional treatment 61 options (see below). “

In an analysis by Shah et al Adverse events appear to unblind clinical trials in irritable bowel syndrome, the result “suggests that higher AE incidence on active therapy is associated with more beneficial patient‐reported outcomes in IBS clinical trials. This raises the issue of spontaneous unblinding”

Maintaining the blinding is one of the cornerstones of the RCT (randomized, controlled trial). As mentioned in an article by Stefan Dürr Avoid Unintentional Unblinding In Clinical Trials

“The knowledge of the patient’s treatment could potentially lead to conscious or unconscious bias in the way the site staff recruits the patients (selection bias), how they are treated (performance bias), the assessment of endpoints (detection bias), the handling of withdrawals, and how data is excluded from analysis. The patient attitude to the treatment if known can lead to a difference in reporting symptoms (response bias) or withdrawal from the study (attrition bias). To reduce the chance of bias to the data, clinical trials should be conducted in a double-blind design whenever possible.”

However, in some situation especially when there is an imbalance in the side effects of two treatment groups, blinding may either be difficult to maintain or not be ethical to implement, and alternative approaches are needed. As suggested in FDA's guidance "

“Hematologic Malignancy and Oncologic Disease: Considerations for Use of Placebos and Blinding in Randomized Controlled Clinical Trials for Drug Product Development”, an open-label study or an add-on trial design may be more appropriate. In an early post, I also discussed some blinding techniques. 

Latin Squares for Constructing "Williams Designs" - Balanced for First-order Carry-over (Residual) Effects

Courtesy from the website http://statpages.info/latinsq.html, the list below contains the Williams design with 2 to 26 treatment groups. It is unlikely to have a clinical trial with 26 treatment groups. The most number of treatment groups I see is a study with 6 treatment groups. 

In one of my previous posts, I described the Williams design and its usefulness in phase I or clinical pharmacology studies. In a SAS SUGI paper "Generating Randomization Schedules Using SAS Programming", I described how to use SAS program to construct the randomization schedule for a study with Williams design. 

With the limited sample size, the Williams design is the most efficient design and minimizes the potential impact of the carryover effect. In Williams design with an even number of n treatment groups, there will be n number of sequences or an "n by n" square. Each treatment only occurs one time in one sequence, in one period. Furthermore, each treatment only follows another treatment one time. 

If the number of treatments (n) is an odd number, there will be 2 x n number of sequences or two n by n squares. Each treatment only occurs once in one sequence and in two periods. Each treatment only follows another treatment twice. 

With the Latin Square listed below, we can easily construct the crossover design with treatments, periods, and sequences. For example, a study design with 3 treatment groups will have the following assignments: three treatment groups (A, B, C), three periods (period 1, period 2, and period 3), and six sequences (ABC, BCA, CAB, CBA, ACB, and BAC). 

Treatment Sequence	Period 1	Period 2	Period 3
1	A	B	C
2	B	C	A
3	C	A	B
4	C	B	A
5	A	C	B
6	B	A	C

• Since it is the crossover design, there should be a washout period between each period. It is very common that the washout period is 5 times the half-life. 
• Randomization is for the sequence, not the treatment group. 
• Once we know the number of subjects is needed for each sequence, we will know the total number of subjects for the entire study. 
• Usually, the mixed model is needed for statistical analysis. The statistical model will need to consider the treatment, period, and sequence. An example of the SAS codes for analysis was provided in one of the previous posts "Cookbook SAS Codes for Bioequivalence Test in 2x2x2 Crossover Design"

Williams Design by the Number of Treatment Groups:

Size = 2

A B 
B A 
 

Size = 3

A B C 
B C A 
C A B 
C B A 
A C B 
B A C 
 

Size = 4

A B D C 
B C A D 
C D B A 
D A C B 
 

Size = 5

A B E C D 
B C A D E 
C D B E A 
D E C A B 
E A D B C 
D C E B A 
E D A C B 
A E B D C 
B A C E D 
C B D A E 
 

Size = 6

A B F C E D 
B C A D F E 
C D B E A F 
D E C F B A 
E F D A C B 
F A E B D C 
 

Size = 7

A B G C F D E 
B C A D G E F 
C D B E A F G 
D E C F B G A 
E F D G C A B 
F G E A D B C 
G A F B E C D 
E D F C G B A 
F E G D A C B 
G F A E B D C 
A G B F C E D 
B A C G D F E 
C B D A E G F 
D C E B F A G 
 

Size = 8

A B H C G D F E 
B C A D H E G F 
C D B E A F H G 
D E C F B G A H 
E F D G C H B A 
F G E H D A C B 
G H F A E B D C 
H A G B F C E D 
 

Size = 9

A B I C H D G E F 
B C A D I E H F G 
C D B E A F I G H 
D E C F B G A H I 
E F D G C H B I A 
F G E H D I C A B 
G H F I E A D B C 
H I G A F B E C D 
I A H B G C F D E 
F E G D H C I B A 
G F H E I D A C B 
H G I F A E B D C 
I H A G B F C E D 
A I B H C G D F E 
B A C I D H E G F 
C B D A E I F H G 
D C E B F A G I H 
E D F C G B H A I 
 

Size = 10

A B J C I D H E G F 
B C A D J E I F H G 
C D B E A F J G I H 
D E C F B G A H J I 
E F D G C H B I A J 
F G E H D I C J B A 
G H F I E J D A C B 
H I G J F A E B D C 
I J H A G B F C E D 
J A I B H C G D F E 
 
 
Size = 11

A B K C J D I E H F G 
B C A D K E J F I G H 
C D B E A F K G J H I 
D E C F B G A H K I J 
E F D G C H B I A J K 
F G E H D I C J B K A 
G H F I E J D K C A B 
H I G J F K E A D B C 
I J H K G A F B E C D 
J K I A H B G C F D E 
K A J B I C H D G E F 
G F H E I D J C K B A 
H G I F J E K D A C B 
I H J G K F A E B D C 
J I K H A G B F C E D 
K J A I B H C G D F E 
A K B J C I D H E G F 
B A C K D J E I F H G 
C B D A E K F J G I H 
D C E B F A G K H J I 
E D F C G B H A I K J 
F E G D H C I B J A K 
 

Size = 12

A B L C K D J E I F H G 
B C A D L E K F J G I H 
C D B E A F L G K H J I 
D E C F B G A H L I K J 
E F D G C H B I A J L K 
F G E H D I C J B K A L 
G H F I E J D K C L B A 
H I G J F K E L D A C B 
I J H K G L F A E B D C 
J K I L H A G B F C E D 
K L J A I B H C G D F E 
L A K B J C I D H E G F 
 
 
Size = 13

A B M C L D K E J F I G H 
B C A D M E L F K G J H I 
C D B E A F M G L H K I J 
D E C F B G A H M I L J K 
E F D G C H B I A J M K L 
F G E H D I C J B K A L M 
G H F I E J D K C L B M A 
H I G J F K E L D M C A B 
I J H K G L F M E A D B C 
J K I L H M G A F B E C D 
K L J M I A H B G C F D E 
L M K A J B I C H D G E F 
M A L B K C J D I E H F G 
H G I F J E K D L C M B A 
I H J G K F L E M D A C B 
J I K H L G M F A E B D C 
K J L I M H A G B F C E D 
L K M J A I B H C G D F E 
M L A K B J C I D H E G F 
A M B L C K D J E I F H G 
B A C M D L E K F J G I H 
C B D A E M F L G K H J I 
D C E B F A G M H L I K J 
E D F C G B H A I M J L K 
F E G D H C I B J A K M L 
G F H E I D J C K B L A M 
 
 
Size = 14

A B N C M D L E K F J G I H 
B C A D N E M F L G K H J I 
C D B E A F N G M H L I K J 
D E C F B G A H N I M J L K 
E F D G C H B I A J N K M L 
F G E H D I C J B K A L N M 
G H F I E J D K C L B M A N 
H I G J F K E L D M C N B A 
I J H K G L F M E N D A C B 
J K I L H M G N F A E B D C 
K L J M I N H A G B F C E D 
L M K N J A I B H C G D F E 
M N L A K B J C I D H E G F 
N A M B L C K D J E I F H G 
 
 
Size = 15

A B O C N D M E L F K G J H I 
B C A D O E N F M G L H K I J 
C D B E A F O G N H M I L J K 
D E C F B G A H O I N J M K L 
E F D G C H B I A J O K N L M 
F G E H D I C J B K A L O M N 
G H F I E J D K C L B M A N O 
H I G J F K E L D M C N B O A 
I J H K G L F M E N D O C A B 
J K I L H M G N F O E A D B C 
K L J M I N H O G A F B E C D 
L M K N J O I A H B G C F D E 
M N L O K A J B I C H D G E F 
N O M A L B K C J D I E H F G 
O A N B M C L D K E J F I G H 
I H J G K F L E M D N C O B A 
J I K H L G M F N E O D A C B 
K J L I M H N G O F A E B D C 
L K M J N I O H A G B F C E D 
M L N K O J A I B H C G D F E 
N M O L A K B J C I D H E G F 
O N A M B L C K D J E I F H G 
A O B N C M D L E K F J G I H 
B A C O D N E M F L G K H J I 
C B D A E O F N G M H L I K J 
D C E B F A G O H N I M J L K 
E D F C G B H A I O J N K M L 
F E G D H C I B J A K O L N M 
G F H E I D J C K B L A M O N 
H G I F J E K D L C M B N A O 
 

Size = 16

A B P C O D N E M F L G K H J I 
B C A D P E O F N G M H L I K J 
C D B E A F P G O H N I M J L K 
D E C F B G A H P I O J N K M L 
E F D G C H B I A J P K O L N M 
F G E H D I C J B K A L P M O N 
G H F I E J D K C L B M A N P O 
H I G J F K E L D M C N B O A P 
I J H K G L F M E N D O C P B A 
J K I L H M G N F O E P D A C B 
K L J M I N H O G P F A E B D C 
L M K N J O I P H A G B F C E D 
M N L O K P J A I B H C G D F E 
N O M P L A K B J C I D H E G F 
O P N A M B L C K D J E I F H G 
P A O B N C M D L E K F J G I H 
 

Size = 17

A B Q C P D O E N F M G L H K I J 
B C A D Q E P F O G N H M I L J K 
C D B E A F Q G P H O I N J M K L 
D E C F B G A H Q I P J O K N L M 
E F D G C H B I A J Q K P L O M N 
F G E H D I C J B K A L Q M P N O 
G H F I E J D K C L B M A N Q O P 
H I G J F K E L D M C N B O A P Q 
I J H K G L F M E N D O C P B Q A 
J K I L H M G N F O E P D Q C A B 
K L J M I N H O G P F Q E A D B C 
L M K N J O I P H Q G A F B E C D 
M N L O K P J Q I A H B G C F D E 
N O M P L Q K A J B I C H D G E F 
O P N Q M A L B K C J D I E H F G 
P Q O A N B M C L D K E J F I G H 
Q A P B O C N D M E L F K G J H I 
J I K H L G M F N E O D P C Q B A 
K J L I M H N G O F P E Q D A C B 
L K M J N I O H P G Q F A E B D C 
M L N K O J P I Q H A G B F C E D 
N M O L P K Q J A I B H C G D F E 
O N P M Q L A K B J C I D H E G F 
P O Q N A M B L C K D J E I F H G 
Q P A O B N C M D L E K F J G I H 
A Q B P C O D N E M F L G K H J I 
B A C Q D P E O F N G M H L I K J 
C B D A E Q F P G O H N I M J L K 
D C E B F A G Q H P I O J N K M L 
E D F C G B H A I Q J P K O L N M 
F E G D H C I B J A K Q L P M O N 
G F H E I D J C K B L A M Q N P O 
H G I F J E K D L C M B N A O Q P 
I H J G K F L E M D N C O B P A Q 
 
 

Size = 18

A B R C Q D P E O F N G M H L I K J 
B C A D R E Q F P G O H N I M J L K 
C D B E A F R G Q H P I O J N K M L 
D E C F B G A H R I Q J P K O L N M 
E F D G C H B I A J R K Q L P M O N 
F G E H D I C J B K A L R M Q N P O 
G H F I E J D K C L B M A N R O Q P 
H I G J F K E L D M C N B O A P R Q 
I J H K G L F M E N D O C P B Q A R 
J K I L H M G N F O E P D Q C R B A 
K L J M I N H O G P F Q E R D A C B 
L M K N J O I P H Q G R F A E B D C 
M N L O K P J Q I R H A G B F C E D 
N O M P L Q K R J A I B H C G D F E 
O P N Q M R L A K B J C I D H E G F 
P Q O R N A M B L C K D J E I F H G 
Q R P A O B N C M D L E K F J G I H 
R A Q B P C O D N E M F L G K H J I 
 

Size = 19

A B S C R D Q E P F O G N H M I L J K 
B C A D S E R F Q G P H O I N J M K L 
C D B E A F S G R H Q I P J O K N L M 
D E C F B G A H S I R J Q K P L O M N 
E F D G C H B I A J S K R L Q M P N O 
F G E H D I C J B K A L S M R N Q O P 
G H F I E J D K C L B M A N S O R P Q 
H I G J F K E L D M C N B O A P S Q R 
I J H K G L F M E N D O C P B Q A R S 
J K I L H M G N F O E P D Q C R B S A 
K L J M I N H O G P F Q E R D S C A B 
L M K N J O I P H Q G R F S E A D B C 
M N L O K P J Q I R H S G A F B E C D 
N O M P L Q K R J S I A H B G C F D E 
O P N Q M R L S K A J B I C H D G E F 
P Q O R N S M A L B K C J D I E H F G 
Q R P S O A N B M C L D K E J F I G H 
R S Q A P B O C N D M E L F K G J H I 
S A R B Q C P D O E N F M G L H K I J 
K J L I M H N G O F P E Q D R C S B A 
L K M J N I O H P G Q F R E S D A C B 
M L N K O J P I Q H R G S F A E B D C 
N M O L P K Q J R I S H A G B F C E D 
O N P M Q L R K S J A I B H C G D F E 
P O Q N R M S L A K B J C I D H E G F 
Q P R O S N A M B L C K D J E I F H G 
R Q S P A O B N C M D L E K F J G I H 
S R A Q B P C O D N E M F L G K H J I 
A S B R C Q D P E O F N G M H L I K J 
B A C S D R E Q F P G O H N I M J L K 
C B D A E S F R G Q H P I O J N K M L 
D C E B F A G S H R I Q J P K O L N M 
E D F C G B H A I S J R K Q L P M O N 
F E G D H C I B J A K S L R M Q N P O 
G F H E I D J C K B L A M S N R O Q P 
H G I F J E K D L C M B N A O S P R Q 
I H J G K F L E M D N C O B P A Q S R 
J I K H L G M F N E O D P C Q B R A S 
 

Size = 20

A B T C S D R E Q F P G O H N I M J L K 
B C A D T E S F R G Q H P I O J N K M L 
C D B E A F T G S H R I Q J P K O L N M 
D E C F B G A H T I S J R K Q L P M O N 
E F D G C H B I A J T K S L R M Q N P O 
F G E H D I C J B K A L T M S N R O Q P 
G H F I E J D K C L B M A N T O S P R Q 
H I G J F K E L D M C N B O A P T Q S R 
I J H K G L F M E N D O C P B Q A R T S 
J K I L H M G N F O E P D Q C R B S A T 
K L J M I N H O G P F Q E R D S C T B A 
L M K N J O I P H Q G R F S E T D A C B 
M N L O K P J Q I R H S G T F A E B D C 
N O M P L Q K R J S I T H A G B F C E D 
O P N Q M R L S K T J A I B H C G D F E 
P Q O R N S M T L A K B J C I D H E G F 
Q R P S O T N A M B L C K D J E I F H G 
R S Q T P A O B N C M D L E K F J G I H 
S T R A Q B P C O D N E M F L G K H J I 
T A S B R C Q D P E O F N G M H L I K J 
 

Size = 21

A B U C T D S E R F Q G P H O I N J M K L 
B C A D U E T F S G R H Q I P J O K N L M 
C D B E A F U G T H S I R J Q K P L O M N 
D E C F B G A H U I T J S K R L Q M P N O 
E F D G C H B I A J U K T L S M R N Q O P 
F G E H D I C J B K A L U M T N S O R P Q 
G H F I E J D K C L B M A N U O T P S Q R 
H I G J F K E L D M C N B O A P U Q T R S 
I J H K G L F M E N D O C P B Q A R U S T 
J K I L H M G N F O E P D Q C R B S A T U 
K L J M I N H O G P F Q E R D S C T B U A 
L M K N J O I P H Q G R F S E T D U C A B 
M N L O K P J Q I R H S G T F U E A D B C 
N O M P L Q K R J S I T H U G A F B E C D 
O P N Q M R L S K T J U I A H B G C F D E 
P Q O R N S M T L U K A J B I C H D G E F 
Q R P S O T N U M A L B K C J D I E H F G 
R S Q T P U O A N B M C L D K E J F I G H 
S T R U Q A P B O C N D M E L F K G J H I 
T U S A R B Q C P D O E N F M G L H K I J 
U A T B S C R D Q E P F O G N H M I L J K 
L K M J N I O H P G Q F R E S D T C U B A 
M L N K O J P I Q H R G S F T E U D A C B 
N M O L P K Q J R I S H T G U F A E B D C 
O N P M Q L R K S J T I U H A G B F C E D 
P O Q N R M S L T K U J A I B H C G D F E 
Q P R O S N T M U L A K B J C I D H E G F 
R Q S P T O U N A M B L C K D J E I F H G 
S R T Q U P A O B N C M D L E K F J G I H 
T S U R A Q B P C O D N E M F L G K H J I 
U T A S B R C Q D P E O F N G M H L I K J 
A U B T C S D R E Q F P G O H N I M J L K 
B A C U D T E S F R G Q H P I O J N K M L 
C B D A E U F T G S H R I Q J P K O L N M 
D C E B F A G U H T I S J R K Q L P M O N 
E D F C G B H A I U J T K S L R M Q N P O 
F E G D H C I B J A K U L T M S N R O Q P 
G F H E I D J C K B L A M U N T O S P R Q 
H G I F J E K D L C M B N A O U P T Q S R 
I H J G K F L E M D N C O B P A Q U R T S 
J I K H L G M F N E O D P C Q B R A S U T 
K J L I M H N G O F P E Q D R C S B T A U 
 


Size = 22

A B V C U D T E S F R G Q H P I O J N K M L 
B C A D V E U F T G S H R I Q J P K O L N M 
C D B E A F V G U H T I S J R K Q L P M O N 
D E C F B G A H V I U J T K S L R M Q N P O 
E F D G C H B I A J V K U L T M S N R O Q P 
F G E H D I C J B K A L V M U N T O S P R Q 
G H F I E J D K C L B M A N V O U P T Q S R 
H I G J F K E L D M C N B O A P V Q U R T S 
I J H K G L F M E N D O C P B Q A R V S U T 
J K I L H M G N F O E P D Q C R B S A T V U 
K L J M I N H O G P F Q E R D S C T B U A V 
L M K N J O I P H Q G R F S E T D U C V B A 
M N L O K P J Q I R H S G T F U E V D A C B 
N O M P L Q K R J S I T H U G V F A E B D C 
O P N Q M R L S K T J U I V H A G B F C E D 
P Q O R N S M T L U K V J A I B H C G D F E 
Q R P S O T N U M V L A K B J C I D H E G F 
R S Q T P U O V N A M B L C K D J E I F H G 
S T R U Q V P A O B N C M D L E K F J G I H 
T U S V R A Q B P C O D N E M F L G K H J I 
U V T A S B R C Q D P E O F N G M H L I K J 
V A U B T C S D R E Q F P G O H N I M J L K 
 


Size = 23

A B W C V D U E T F S G R H Q I P J O K N L M 
B C A D W E V F U G T H S I R J Q K P L O M N 
C D B E A F W G V H U I T J S K R L Q M P N O 
D E C F B G A H W I V J U K T L S M R N Q O P 
E F D G C H B I A J W K V L U M T N S O R P Q 
F G E H D I C J B K A L W M V N U O T P S Q R 
G H F I E J D K C L B M A N W O V P U Q T R S 
H I G J F K E L D M C N B O A P W Q V R U S T 
I J H K G L F M E N D O C P B Q A R W S V T U 
J K I L H M G N F O E P D Q C R B S A T W U V 
K L J M I N H O G P F Q E R D S C T B U A V W 
L M K N J O I P H Q G R F S E T D U C V B W A 
M N L O K P J Q I R H S G T F U E V D W C A B 
N O M P L Q K R J S I T H U G V F W E A D B C 
O P N Q M R L S K T J U I V H W G A F B E C D 
P Q O R N S M T L U K V J W I A H B G C F D E 
Q R P S O T N U M V L W K A J B I C H D G E F 
R S Q T P U O V N W M A L B K C J D I E H F G 
S T R U Q V P W O A N B M C L D K E J F I G H 
T U S V R W Q A P B O C N D M E L F K G J H I 
U V T W S A R B Q C P D O E N F M G L H K I J 
V W U A T B S C R D Q E P F O G N H M I L J K 
W A V B U C T D S E R F Q G P H O I N J M K L 
M L N K O J P I Q H R G S F T E U D V C W B A 
N M O L P K Q J R I S H T G U F V E W D A C B 
O N P M Q L R K S J T I U H V G W F A E B D C 
P O Q N R M S L T K U J V I W H A G B F C E D 
Q P R O S N T M U L V K W J A I B H C G D F E 
R Q S P T O U N V M W L A K B J C I D H E G F 
S R T Q U P V O W N A M B L C K D J E I F H G 
T S U R V Q W P A O B N C M D L E K F J G I H 
U T V S W R A Q B P C O D N E M F L G K H J I 
V U W T A S B R C Q D P E O F N G M H L I K J 
W V A U B T C S D R E Q F P G O H N I M J L K 
A W B V C U D T E S F R G Q H P I O J N K M L 
B A C W D V E U F T G S H R I Q J P K O L N M 
C B D A E W F V G U H T I S J R K Q L P M O N 
D C E B F A G W H V I U J T K S L R M Q N P O 
E D F C G B H A I W J V K U L T M S N R O Q P 
F E G D H C I B J A K W L V M U N T O S P R Q 
G F H E I D J C K B L A M W N V O U P T Q S R 
H G I F J E K D L C M B N A O W P V Q U R T S 
I H J G K F L E M D N C O B P A Q W R V S U T 
J I K H L G M F N E O D P C Q B R A S W T V U 
K J L I M H N G O F P E Q D R C S B T A U W V 
L K M J N I O H P G Q F R E S D T C U B V A W 
 

Size = 24

A B X C W D V E U F T G S H R I Q J P K O L N M 
B C A D X E W F V G U H T I S J R K Q L P M O N 
C D B E A F X G W H V I U J T K S L R M Q N P O 
D E C F B G A H X I W J V K U L T M S N R O Q P 
E F D G C H B I A J X K W L V M U N T O S P R Q 
F G E H D I C J B K A L X M W N V O U P T Q S R 
G H F I E J D K C L B M A N X O W P V Q U R T S 
H I G J F K E L D M C N B O A P X Q W R V S U T 
I J H K G L F M E N D O C P B Q A R X S W T V U 
J K I L H M G N F O E P D Q C R B S A T X U W V 
K L J M I N H O G P F Q E R D S C T B U A V X W 
L M K N J O I P H Q G R F S E T D U C V B W A X 
M N L O K P J Q I R H S G T F U E V D W C X B A 
N O M P L Q K R J S I T H U G V F W E X D A C B 
O P N Q M R L S K T J U I V H W G X F A E B D C 
P Q O R N S M T L U K V J W I X H A G B F C E D 
Q R P S O T N U M V L W K X J A I B H C G D F E 
R S Q T P U O V N W M X L A K B J C I D H E G F 
S T R U Q V P W O X N A M B L C K D J E I F H G 
T U S V R W Q X P A O B N C M D L E K F J G I H 
U V T W S X R A Q B P C O D N E M F L G K H J I 
V W U X T A S B R C Q D P E O F N G M H L I K J 
W X V A U B T C S D R E Q F P G O H N I M J L K 
X A W B V C U D T E S F R G Q H P I O J N K M L 
 
 

Size = 25

A B Y C X D W E V F U G T H S I R J Q K P L O M N 
B C A D Y E X F W G V H U I T J S K R L Q M P N O 
C D B E A F Y G X H W I V J U K T L S M R N Q O P 
D E C F B G A H Y I X J W K V L U M T N S O R P Q 
E F D G C H B I A J Y K X L W M V N U O T P S Q R 
F G E H D I C J B K A L Y M X N W O V P U Q T R S 
G H F I E J D K C L B M A N Y O X P W Q V R U S T 
H I G J F K E L D M C N B O A P Y Q X R W S V T U 
I J H K G L F M E N D O C P B Q A R Y S X T W U V 
J K I L H M G N F O E P D Q C R B S A T Y U X V W 
K L J M I N H O G P F Q E R D S C T B U A V Y W X 
L M K N J O I P H Q G R F S E T D U C V B W A X Y 
M N L O K P J Q I R H S G T F U E V D W C X B Y A 
N O M P L Q K R J S I T H U G V F W E X D Y C A B 
O P N Q M R L S K T J U I V H W G X F Y E A D B C 
P Q O R N S M T L U K V J W I X H Y G A F B E C D 
Q R P S O T N U M V L W K X J Y I A H B G C F D E 
R S Q T P U O V N W M X L Y K A J B I C H D G E F 
S T R U Q V P W O X N Y M A L B K C J D I E H F G 
T U S V R W Q X P Y O A N B M C L D K E J F I G H 
U V T W S X R Y Q A P B O C N D M E L F K G J H I 
V W U X T Y S A R B Q C P D O E N F M G L H K I J 
W X V Y U A T B S C R D Q E P F O G N H M I L J K 
X Y W A V B U C T D S E R F Q G P H O I N J M K L 
Y A X B W C V D U E T F S G R H Q I P J O K N L M 
N M O L P K Q J R I S H T G U F V E W D X C Y B A 
O N P M Q L R K S J T I U H V G W F X E Y D A C B 
P O Q N R M S L T K U J V I W H X G Y F A E B D C 
Q P R O S N T M U L V K W J X I Y H A G B F C E D 
R Q S P T O U N V M W L X K Y J A I B H C G D F E 
S R T Q U P V O W N X M Y L A K B J C I D H E G F 
T S U R V Q W P X O Y N A M B L C K D J E I F H G 
U T V S W R X Q Y P A O B N C M D L E K F J G I H 
V U W T X S Y R A Q B P C O D N E M F L G K H J I 
W V X U Y T A S B R C Q D P E O F N G M H L I K J 
X W Y V A U B T C S D R E Q F P G O H N I M J L K 
Y X A W B V C U D T E S F R G Q H P I O J N K M L 
A Y B X C W D V E U F T G S H R I Q J P K O L N M 
B A C Y D X E W F V G U H T I S J R K Q L P M O N 
C B D A E Y F X G W H V I U J T K S L R M Q N P O 
D C E B F A G Y H X I W J V K U L T M S N R O Q P 
E D F C G B H A I Y J X K W L V M U N T O S P R Q 
F E G D H C I B J A K Y L X M W N V O U P T Q S R 
G F H E I D J C K B L A M Y N X O W P V Q U R T S 
H G I F J E K D L C M B N A O Y P X Q W R V S U T 
I H J G K F L E M D N C O B P A Q Y R X S W T V U 
J I K H L G M F N E O D P C Q B R A S Y T X U W V 
K J L I M H N G O F P E Q D R C S B T A U Y V X W 
L K M J N I O H P G Q F R E S D T C U B V A W Y X 
M L N K O J P I Q H R G S F T E U D V C W B X A Y 
 


Size = 26

A B Z C Y D X E W F V G U H T I S J R K Q L P M O N 
B C A D Z E Y F X G W H V I U J T K S L R M Q N P O 
C D B E A F Z G Y H X I W J V K U L T M S N R O Q P 
D E C F B G A H Z I Y J X K W L V M U N T O S P R Q 
E F D G C H B I A J Z K Y L X M W N V O U P T Q S R 
F G E H D I C J B K A L Z M Y N X O W P V Q U R T S 
G H F I E J D K C L B M A N Z O Y P X Q W R V S U T 
H I G J F K E L D M C N B O A P Z Q Y R X S W T V U 
I J H K G L F M E N D O C P B Q A R Z S Y T X U W V 
J K I L H M G N F O E P D Q C R B S A T Z U Y V X W 
K L J M I N H O G P F Q E R D S C T B U A V Z W Y X 
L M K N J O I P H Q G R F S E T D U C V B W A X Z Y 
M N L O K P J Q I R H S G T F U E V D W C X B Y A Z 
N O M P L Q K R J S I T H U G V F W E X D Y C Z B A 
O P N Q M R L S K T J U I V H W G X F Y E Z D A C B 
P Q O R N S M T L U K V J W I X H Y G Z F A E B D C 
Q R P S O T N U M V L W K X J Y I Z H A G B F C E D 
R S Q T P U O V N W M X L Y K Z J A I B H C G D F E 
S T R U Q V P W O X N Y M Z L A K B J C I D H E G F 
T U S V R W Q X P Y O Z N A M B L C K D J E I F H G 
U V T W S X R Y Q Z P A O B N C M D L E K F J G I H 
V W U X T Y S Z R A Q B P C O D N E M F L G K H J I 
W X V Y U Z T A S B R C Q D P E O F N G M H L I K J 
X Y W Z V A U B T C S D R E Q F P G O H N I M J L K 
Y Z X A W B V C U D T E S F R G Q H P I O J N K M L 
Z A Y B X C W D V E U F T G S H R I Q J P K O L N M