Skip to content

Eurordis - Rare Disease Europe


Lesson 6: Significance

Please go through the lesson, making sure to click the green ‘completed’ button once you have read through the content.


Confidence vs Scepticism

The Right Number

Fred is late for Carla’s party. Suddenly, on the bus, he realises that he has forgotten her door code. So, he calls Carla from the bus. Unfortunately, Carla’s flat and the bus are very noisy and Carla is not speaking very loudly. Fred cannot understand her response.

  • First, he says speak louder! but she cannot (she sang too much yesterday).
  • Then, Fred decides to leave the bus, in order to reduce the noise (at least on his side!). Better, but insufficient to understand the code.
  • Louder, say it again louder! he shouts. Carla repeats four times, but due to the music level of the party, the message remains unclear (W or I ?, 6 or X…?)
  • Resigned, Fred notes a code, not convinced that it is the right one. Grumbling to himself, Fred says: P.I., it makes no sense, with my bad luck today, at least one figure will be wrong and the code will not work.

If you understand Fred’s difficulties and his reactions, you master the main concepts concerning confidence, statistical power and significance.

The Scepticism/Confidence Balance

How can the level of pessimism in Fred’s conclusion be analysed?

The Statistical Scale



Planning a Clinical Trial

Analysing a Clinical Trial 

Possible Error(s) in Conclusion

The Two Types of Error

The most obvious error in conclusion is finding a difference when in reality a difference does not exist, “seeing too much in the data”, which is called Type I error, or α (alpha) error (false positive).

In addition, another kind of error has to be considered:

Not finding a difference when in reality a difference does exist, “not seeing enough in the data”, which is called Type II error, or β (beta) error (false negative).

It is essential to take into account the Type II error in the study design. Underestimating Type II error may result in an underestimation of the number of participants and finally to an inconclusive study.

This situation is not only a failure from a scientific perspective, but may be considered as a breach of trust by the orphan participants, since part of their willingness to take part in the study was progress in knowledge.

Therefore, it is unethical to increase the number of participants in a trial to reduce the possibility of and inconclusive clinical trial due to Type II error

When the feasibility of a study seems questionable, all the aspects of the study should be reassessed: the choice of the endpoint, the approach (pragmatic vs explanatory), the concerned population and the criteria of eligibility, etc, rather than adjusting the hypotheses concerning efficacy and variability to the statistical requirements.

A Questionable Use of Statistics 


Basic vocabulary: Words you should know

Bias – p value – Significance level – Statistical significance

Technical Vocabulary: Important words to know

Evidence Based Medicine – External validity –  Interaction (qualitative and quantitative)- Internal validity 

Advanced Vocabulary: Useful words to know

Significative difference – Type I Error (alpha error) – Type II Error (beta error)


Case study

Comparing Treatments

During Phase II development, Quietepil possible new anti-epileptic agent, is compared to a standard treatment (the control treatment) in children aged from 3 to 16 years with pharmaco-resistant epilepsy.

The main endpoint of the study was the number of seizures per month.

The chosen level of significance was p <0.05.

Following 3 months of treatment the results are:

Control group

(51 patients)

14.0 ± 3.9 seizures per month

Quietepilâ„¢ group

(48 patients)

12.7 ± 3.7 seizures per month

Which of these figures play a role in the statistical testing used to compare the two treatments? And in what way?

A Statistical Test Works as a Scale with 4 Parameters 

Trial 1 with a Population of 100 3-16 Year Old Children


Trial with a Population of 100 8-12 Year Old Children

Trial with a Population of 600 3-16 Year Old Children

Trial with a Population of 100 8-12 Year Old Children

Trial with a Population of 600 3-16 Year Old Children

Planned Trial 4 for a Population of 3-16 Year Old Children: How Many Participants?

Three Pitfalls in Study Design


Documents for further reading


The Abuse of Power

The Abuse of Power: The Pervasive Fallacy of Power Calculations for Data Analysis John M. Hoeni  and Dennis M. Heisey The American Statistician, February 2001, Vol. 55, No. 1

Statistical Power 

Statistical power: An historical introduction Jean Descôteaux Tutorials in Quantitative Methods for Psychology 2007, Vol. 3 (2), p. 28- 34.

What Signifies Significance

Statistical Significance and Clinical Relevance: The Importance of Power in Clinical Trials in Dermatology Sachin S. Bhardwaj, MD; Fabian Camacho, MS; Amy Derrow, MS;Alan B. Fleischer, Jr, MD; Steven R. Feldman, MD, PhD Arch Dermatol. 2004;140:1520-1523