Knowledge / Glossary
Experiment Validity
Whether an experiment's results truly reflect cause and effect and can be trusted and generalized.
1 min readUpdated 2026-11-15By Meridian Academy
Whether an experiment's results truly reflect cause and effect and can be trusted and generalized.
Summary
Context can fake results.
What Is Experiment Validity?
Experiment validity covers internal validity, whether the change caused the result, and external validity, whether results apply elsewhere. Threats include small samples, peeking, seasonality, novelty effects and broken randomization.
People may also refer to this concept as:
- 1.Test validity
Purpose and Context
Why It Matters
Invalid tests lead to wrong decisions.
Key Elements
| Element | What it means |
|---|---|
| Internal validity | true cause |
| External validity | generalizability |
| Sample size | enough data |
| Bias | errors in setup |
| Duration | full cycles |
How It Works
- 1.Plan sample size
- 2.Randomize properly
- 3.Run full cycles
- 4.Check for issues
Plan sample size
Before starting.
Randomize properly
Fair groups.
Run full cycles
Weekly patterns.
Check for issues
Novelty and errors.
Examples
The holiday test
The holiday test
A test during holidays showed a big win.
What they did
- 1.Rerun after holidays showed no effect
What happened
Seasonality caused the false win.
Comparisons and Boundaries
| Related term | Relationship | Why it matters |
|---|---|---|
| A/B Testing | Used with | Validity applies to tests |
| Law of Small Numbers | Used with | Small samples threaten validity |
| Experimentation | Used with | Experiments need validity |
| Incrementality | Used with | Incrementality tests need validity |
Common Misconceptions
| Mistake | What to do instead |
|---|---|
| Peeking | Stopping early. |
| Ignoring seasonality | Results vary. |
Related concepts
Sources
- [1] Ron Kohavi, Diane Tang, and Ya Xu, Trustworthy Online Controlled Experiments
- [2] Rory Sutherland, Alchemy
- [3] Richard Shotton, The Choice Factory