Ecole Nationale Vétérinaire de Toulouse Veterinary clinical studies

Ecole Nationale Vétérinaire de Toulouse Veterinary clinical studies Key issues for statistical analysis Didier Concordet d. concordet@envt. fr ECVPT Workshop Can be downloaded at http: //www. biostat. envt. fr/spip. php? article 34 July 2009

Vocabulary • • • • • • • • Bias (Statistical & Operational) Blind Review Content Validity Double-Dummy Dropout Equivalence Trial Frequentist Methods Full Analysis Set Generalisability, Generalisation Global Assessment Variable Independent Data Monitoring Committee (IDMC) (Data and Safety Monitoring Board, Monitoring Committee, Data Monitoring Committee) Intention-To-Treat Principle Interaction (Qualitative & Quantitative) Inter-Rater Reliability Intra-Rater Reliability Interim Analysis Meta-Analysis Multicentre Trial Non-Inferiority Trial Preferred and Included Terms Per Protocol Set (Valid Cases, Efficacy Sample, Evaluable Subjects Sample) Safety & Tolerability Statistical Analysis Plan Superiority Trial Surrogate Variable Treatment Effect Treatment Emergent Trial Statistician From ICH Topic E 9

Aim of clinical trials To assess the efficacy of a drug in a (target) population Population : the set of individuals that can receive the drug Practically Design/Sampling Inference Population Sample

ISSUES • When designing the trial • When collecting data • When analysing data • When interpreting results

ISSUES • When designing the trial • Sampling the target population • When collecting data • Different kinds of clinical trials • How to detect bias • When analysing data • When interpreting results

ISSUES • When designing the trial • Sampling the target population • When collecting data • Different kinds of clinical trials • How to detect bias • When analysing data • When interpreting results

Sampling the target population There exist sources of variation that make the judgment criterion vary Example with two breeds Judgment criterion

Sampling the target population The two same breeds with different proportions Judgment criterion

Sampling the target population The sample should be representative of the target population Target population 2<=. <3 years 1<=. <2 years . <1 year Sample Female Male b ed re 1 br d ee 2 br d ee 3 6 4 5 ed eed e br br br The sample has the same structure as the population

Two main ways to sample the population Randomization: leave chance make the job the percentage of the animals in each subgroup should be close to the population's one. Stratification: help the chance to do the job Build a sample of animals that has exactly the same percentage of individuals in each subgroup as the population. This requires to know the repartition of subgroups in the population.

Target population definition An experiment in 2 years old beagles showed that the temperature of dogs treated with the antipyretic drug A decreased by 2 °C. What assumptions do we need for this result to hold for all 3 years old beagles dogs man 11

ISSUES • When designing the trial • Sampling the target population • When collecting data • Different kinds of clinical trials • How to detect bias • When analysing data • When interpreting results

Different kinds of clinical trials • Non inferiority • Superiority • Equivalence

Different kinds of clinical trials Non inferiority New treatment Efficacy low high Reference – D (penalty) Reference D : non inferiority margin Superiority New treatment Efficacy Reference D : superiority margin Reference + D

Different kinds of clinical trials Equivalence trial New treatment Efficacy Reference – D Reference + D

Non inferiority trial New treatment Efficacy Reference – D (penalty) Reference The new treatment can have a smaller efficacy than the reference treatment

Non inferiority trial New treatment Efficacy Reference – D (penalty) Reference • the reference treatment is not efficacious • animals included in the trial are not sick • the judgment criterion is not relevant (e. g. does not vary) • delta is too large

Is there a problem ? Is there a problem organizing a non inferiority trial able to demonstrate Reference treatment New treatment Decrease of rectal temperature of at least 1. 5°C Cure rate = 75 % Decrease of rectal temperature of at least 1. 2°C Cure rate = 65 %

A clinical trial should avoid bias Bias : the difference between the compared drugs at the end of the trial due to other things than the drugs • Confusion bias • Selection bias • Follow-up bias • Attrition bias

Confusion bias Arises when one do not taking into account a confusion factor. To avoid such bias, the trial should be comparative and should have a contemporary control group used as a reference group. Questions • Is there a control group ? • Is the treatment effect determined with respect to this control group ? Warning • Despite a control group the treatment effect is measured with a "before-after" comparison.

Selection bias Arises when the two groups to be compared are different (with respect to the endpoint before the beginning of the trial. To avoid it one uses a randomisation : a random allocation of animals into treatment groups Questions • Is there a randomisation procedure ? • Are the two groups balanced ? Warning • There is a historical control group (no randomisation) • The investigators were able to select the animals for a group

Follow-up bias Arises when the follow-up is not the same for the two drugs to be compared. Destroy initial comparability. To avoid it : double blind Questions • Is the trial double blind ? • Is the rate of concomitant medications the same for the two groups ? • Are the protocol deviations similar ? • Are the drop-out number similar ? Warning • The treatments were discernable • The investigators were able to select the animals for a group • The judgment criterion was subjective (eg : the animal feels better )

Attrition bias Arises when some randomised animals are excluded. To avoid it Analysis of the Intention to Treat dataset Questions • Is the number of analysed animals equal to the number of randomized animals ? • Was an imputation method used for missing data ? • Intention to treat analysis Warning • Per Protocol analysis (only the animals alive and non excluded were analysed) • High rate of concomitant treatments ? • High rate of protocol deviations ? • High rate of drop-out ?

Example Temperature (°C) Efficacy of an antipyretic drug. Inclusion of 30 dogs with at least 39. 5°C of temperature. 41 38 Before treatment After treatment

ISSUES • When designing the trial • When collecting data Missing data • When analysing data • When interpreting results

Missing data should be adequately reported Three kinds of missingness mechanism Ignorable missing data: • data Missing Completely At Random (MCAR) allow Data imputation does to treat such missing data The missingness is independent of data ITT dataset Leads to the • data Missing At Random (MAR) The missingness depends on observed data • data Missing Not At Random (MNAR) missing data: Non-Ignorable The missingness mechanism has The missingness depends on the non observed data be clearly described 26

Missing Completely At Random MCAR Missingness and outcome are independent • the owner of the animal missed a visit to the vet • the investigator forgot to write the results • the owner moves house Unlikely to occur in a clinical trial 27

Missing At Random MAR Missingness depends on data that have been observed but not on the unobserved (missing) data • dropout related to baseline characteristics • the animal health has markedly improved or deteriorated since inclusion Assumes that the future trajectories of animals who dropout are similar to those who share the same measurements whether or not they dropout. Frequent in clinical trials. 28

Missing Not At Random MNAR Missingness depends on data that have been unobserved (missing data) • sudden decline or improve in health that has not been observed in the previous visits Assumes that the future trajectories of animals who dropout are different to those who share the same measurements Occurs in clinical trials. 29

Can you classify these missing data ? • The battery of thermometer is discharged. I cannot measure the temperature. • At the last visit, the dog was well. I called the owner by phone, he did not want to come because he said that the dog was cured. • The owner did not come back. I don't know why. 30

ISSUES • When designing the trial • When collecting data • When analysing data • Statistical tests • When interpreting results • Multiple comparisons • Data drying off • What dataset to analyse ?

ISSUES • When designing the trial • When collecting data • When analysing data • Statistical tests • When interpreting results • Multiple comparisons • Data washing • What dataset to analyse ?

Statistical analysis Objective : To draw conclusions on the target population from observation of a sample Inference Sample Population

Things to know about statistical tests Sample Target population P<5% Observed difference There is a difference in the population. This conclusion is drawn with less than 5% risk. Test P≥ 5% Non significant difference. We would take too much risk by claiming a difference in the population.

Repetition of tests also called multiple comparisons Test 1 Test 2 Test 3 Test 4 Risk to wrongly conclude to a difference = 5% n global risk 1 0. 05 2 0. 10 3 0. 13 5 0. 23 10 0. 40 Globally, the risk to wrongly conclude to a difference for 4 comparisons is 18%. Risk inflation

Multiple comparisons One wants to compare the ADG obtained with 5 different diets in pig 1 2 3 4 5 Mean 700 880 730 790 930 SD 48 50 55 44 60 Ten T-tests 1 3 4 2 5 A risk of 5% for each comparison : the global risk can be very large here 40% 36

Choosing the question to get an answer Occurs frequently in the analysis of clinical trials results The question becomes random : it changes with the sample of animals. The question is chosen with its answer in hands… Think about a flip coin game where you win 1€ when tail or head occurs. You choose the decision rule once you know the result of the flip ! Such an approach increases the number of false discoveries. 37

Data drying off: Analysis in subgroups Dog (eff. NSAID) 1 Age<10 2 Age>=10 3 Male 4 Female 5 Format small 5 Format medium 6 Format Large 7 Food dry 8 Food wet P difference with placebo 0. 92 0. 95 Target population? ! 0. 81 0. 78 0. 63 0. 91 0. 74 0. 01 0. 63

Data drying off: a posteriori choice of the judgment criterion • Death all causes • Death cardiovascular origin • Sudden death Main criterion • Death all causes Secondary criteria • Infarct • Death cardiovascular • Vascular cerebral accident • Sudden death • Surgery • Infarct • Vascular cerebral accident No definition of a main criterion • Surgery A single statistical test 7 statistical test Risk to wrongly conclude to efficacy of the new treatment : 30% a priori definition of a main criterion From Cucherat 2005 Risk to wrongly conclude to efficacy = 5%

What dataset to analyse ? • Intention To Treat dataset is based on the initial treatment intent, not on the treatment eventually administered regardless the drop-out. • Per Protocol dataset contains animals who have not dropped out for any reason regardless of initial randomization.

Example : complete data 41

Example : complete data 42

Drop-out (MAR) Drop-out 43

Intention To Treat dataset with Last Observation Carried Forward (LOCF) 44

Per Protocol dataset Only the animals that did not dropped-out were used 45

ISSUES • When designing the trial • When collecting data • When analysing data • When interpreting results • Standard error and standard deviation • P-Values

Standard error / standard deviation The clairance of the drug was equal to 68 ± 5 m. L/mn Two possible meanings depending on the meaning of 5 If 5 is the standard error of the mean (se) there is 95 % chance that the population mean clearance belongs to [68 - 2 5 ; 68 + 2 5 ] If 5 is the standard deviation (SD) 95 % of animals have their clearance within [68 - 2 5 ; 68 + 2 5 ] 47

P values The difference between the effect of the drugs A and B is not significant (P = 0. 56) therefore drug A can be substituted by drug B. NO The only conclusion that can be drawn from such a P value is that you didn't see any difference between the effect of the drugs A and B. That does not mean that such a difference does not exist. Absence of evidence is not evidence of absence 48

P values The drug A has a higher efficacy than the drug B (P = 0. 001) The drug C has a higher efficacy than the drug B (P = 0. 04) Since 0. 001<0. 04 the drug A has a higher than the drug B. NO The only conclusion that can be drawn from such a P value is that you are sure than A>B and less sure than C>B. This does not presume anything about the amplitude of the differences. Significant does not mean important 49

How to avoid these problems ? • Consult your preferred statistician for help in the design of complicated experiments • Use basic descriptive statistics first (graphics, summary statistics, …) • Use common sense • Consider to learn more statistics 50