EURORDIS website
Lesson 6: Significance
Please go through the lesson, making sure to click the green ‘completed’ button once you have read through the content.
Lesson
Confidence vs Scepticism
The Right Number
Fred is late for Carla’s party. Suddenly, on the bus, he realises that he has forgotten her door code. So, he calls Carla from the bus. Unfortunately, Carla’s flat and the bus are very noisy and Carla is not speaking very loudly. Fred cannot understand her response.
- First, he says speak louder! but she cannot (she sang too much yesterday).
- Then, Fred decides to leave the bus, in order to reduce the noise (at least on his side!). Better, but insufficient to understand the code.
- Louder, say it again louder! he shouts. Carla repeats four times, but due to the music level of the party, the message remains unclear (W or I ?, 6 or X…?)
- Resigned, Fred notes a code, not convinced that it is the right one. Grumbling to himself, Fred says: P.I.3.1.4.1.1.6., it makes no sense, with my bad luck today, at least one figure will be wrong and the code will not work.
If you understand Fred’s difficulties and his reactions, you master the main concepts concerning confidence, statistical power and significance.
The Scepticism/Confidence Balance
How can the level of pessimism in Fred’s conclusion be analysed?
The Statistical Scale
Planning a Clinical Trial
Analysing a Clinical TrialÂ
Possible Error(s) in Conclusion
The Two Types of Error
The most obvious error in conclusion is finding a difference when in reality a difference does not exist, “seeing too much in the data”, which is called Type I error, or α (alpha) error (false positive).
In addition, another kind of error has to be considered:
Not finding a difference when in reality a difference does exist, “not seeing enough in the data”, which is called Type II error, or β (beta) error (false negative).
It is essential to take into account the Type II error in the study design. Underestimating Type II error may result in an underestimation of the number of participants and finally to an inconclusive study.
This situation is not only a failure from a scientific perspective, but may be considered as a breach of trust by the orphan participants, since part of their willingness to take part in the study was progress in knowledge.
Therefore, it is unethical to increase the number of participants in a trial to reduce the possibility of and inconclusive clinical trial due to Type II error
When the feasibility of a study seems questionable, all the aspects of the study should be reassessed: the choice of the endpoint, the approach (pragmatic vs explanatory), the concerned population and the criteria of eligibility, etc, rather than adjusting the hypotheses concerning efficacy and variability to the statistical requirements.
A Questionable Use of Statistics
Basic vocabulary: Words you should know
Bias – p value – Significance level – Statistical significance
Technical Vocabulary: Important words to know
Evidence Based Medicine – External validity – Interaction (qualitative and quantitative)- Internal validity
Advanced Vocabulary: Useful words to know
Significative difference – Type I Error (alpha error) – Type II Error (beta error)
Case study
Comparing Treatments
During Phase II development, Quietepil possible new anti-epileptic agent, is compared to a standard treatment (the control treatment) in children aged from 3 to 16 years with pharmaco-resistant epilepsy.
The main endpoint of the study was the number of seizures per month.
The chosen level of significance was p <0.05.
Following 3 months of treatment the results are:
Control group
(51 patients)
14.0 ± 3.9 seizures per month
Quietepilâ„¢ group
(48 patients)
12.7 ± 3.7 seizures per month
Which of these figures play a role in the statistical testing used to compare the two treatments? And in what way?
A Statistical Test Works as a Scale with 4 ParametersÂ
Trial 1 with a Population of 100 3-16 Year Old Children
Trial with a Population of 100 8-12 Year Old Children
Trial with a Population of 600 3-16 Year Old Children
Trial with a Population of 100 8-12 Year Old Children
Trial with a Population of 600 3-16 Year Old Children
Planned Trial 4 for a Population of 3-16 Year Old Children: How Many Participants?
Three Pitfalls in Study Design
Documents for further reading
VALUE of P-VALUE
The Abuse of Power
The Abuse of Power: The Pervasive Fallacy of Power Calculations for Data Analysis John M. Hoeni and Dennis M. Heisey The American Statistician, February 2001, Vol. 55, No. 1
Statistical Power
Statistical power: An historical introduction Jean Descôteaux Tutorials in Quantitative Methods for Psychology 2007, Vol. 3 (2), p. 28- 34.
What Signifies Significance
Statistical Significance and Clinical Relevance: The Importance of Power in Clinical Trials in Dermatology Sachin S. Bhardwaj, MD; Fabian Camacho, MS; Amy Derrow, MS;Alan B. Fleischer, Jr, MD; Steven R. Feldman, MD, PhD Arch Dermatol. 2004;140:1520-1523