Every time you make a decision based on data, there are exactly two ways to be wrong. You can sound the alarm when there is no danger, or you can stay silent when the building is on fire. In statistics, these are called Type I and Type II errors, and the MCAT absolutely loves testing them. The good news: once you learn the fire alarm analogy, you will never confuse them again.
Type I Error (False Positive)
A Type I error occurs when you reject the null hypothesis even though it is actually true. You concluded there was an effect when there really was not one.
Type I = False Positive = “Seeing something that is not there”
The probability of a Type I error is α, which is the significance level you set before the experiment. If α = 0.05, you accept a 5% risk of a Type I error.
Type II Error (False Negative)
A Type II error occurs when you fail to reject the null hypothesis even though it is actually false. You missed a real effect.
Type II = False Negative = “Missing something that IS there”
The probability of a Type II error is β. Unlike α, β is not directly chosen by the researcher - it depends on sample size, effect size, and variability.
The Error Table
The four possible outcomes of hypothesis testing. A Type I error (false positive) occurs when a true null hypothesis is rejected. A Type II error (false negative) occurs when a false null hypothesis is not rejected. Alpha controls the Type I error rate; power (1 - β) is the probability of avoiding a Type II error. Credit: Wikimedia Commons, CC BY-SA 4.0
H0 is True (No Effect)
H0 is False (Real Effect)
Reject H0
Type I Error (α) - False Positive
Correct Decision - True Positive
Fail to Reject H0
Correct Decision - True Negative
Type II Error (β) - False Negative
Statistical Power
Power is the probability of correctly rejecting a false null hypothesis - in other words, the probability of detecting a real effect when one exists.
Power = 1 - β
If β (probability of a Type II error) is 0.20, then power = 0.80, meaning there is an 80% chance of detecting a real effect. Most well-designed studies aim for power of at least 0.80.
How to Increase Power
There are four main ways to increase the power of a study:
Increase sample size. More participants means more data, which makes it easier to detect real effects. This is the most common and practical approach.
Increase the effect size. A larger true effect is easier to detect. Researchers cannot usually control this, but choosing a more potent treatment helps.
Decrease variability. Tighter controls and more precise measurements reduce noise, making the signal easier to see.
Increase α. Using α = 0.10 instead of 0.05 makes it easier to reject H0, which increases power but also increases the Type I error rate.
The Trade-Off
Reducing one type of error increases the other, all else being equal.
If you lower α (stricter threshold, e.g., from 0.05 to 0.01), you reduce Type I errors but increase Type II errors. You are less likely to sound a false alarm, but more likely to miss a real fire.
If you raise α (more lenient, e.g., from 0.05 to 0.10), you reduce Type II errors but increase Type I errors. You catch more real effects but also get more false alarms.
The only way to reduce both error types simultaneously is to increase sample size or reduce variability.
A clinical trial concludes that a new drug is effective (rejects H0), but the drug actually has no effect. What type of error occurred?
Click to reveal answer
Type I error (false positive). The null hypothesis (no effect) was actually true, but the researchers rejected it. They concluded the drug works when it does not. The probability of this error is α, which is typically set at 0.05 (5%).
A study with only 12 participants finds no significant difference between treatments (p = 0.18). A colleague says, "The treatments must be equally effective." What is wrong with this conclusion?
Click to reveal answer
The study likely lacked sufficient statistical power. With only 12 participants, the study may have committed a Type II error (false negative) - failing to detect a real difference because the sample was too small. A non-significant p-value does not prove the treatments are equal. It only means the study did not find enough evidence to declare them different. Increasing the sample size would increase power and provide a more reliable answer.
