Knowledge / Glossary
A/B Testing
Comparing two versions of something, such as a page or email, with random groups to see which performs better.
1 min readUpdated 2026-11-15By Meridian Academy
Comparing two versions of something, such as a page or email, with random groups to see which performs better.
Summary
Let customers decide arguments.
What Is A/B Testing?
A/B testing splits an audience randomly so one group sees version A and the other sees version B. The difference in results shows the effect of the change. Valid tests need a clear hypothesis, enough sample size and a fixed duration.
People may also refer to this concept as:
- 1.Split testing
- 2.A/B test
Purpose and Context
Why It Matters
Testing replaces opinion with evidence and compounds small improvements over time.
Key Elements
| Element | What it means |
|---|---|
| Hypothesis | what change and why |
| Variants | control and challenger |
| Randomization | fair groups |
| Sample size | enough data |
| Significance | confidence the result is real |
How It Works
- 1.Write a hypothesis
- 2.Build variants
- 3.Run to sample size
- 4.Analyze and act
Write a hypothesis
Define change and expected effect.
Build variants
Change one main element.
Run to sample size
Do not stop early.
Analyze and act
Roll out or learn.
Examples
The pricing page headline
The pricing page headline
A software company debated two pricing page headlines.
What they did
- 1.Ran an A/B test for three weeks with 20,000 visitors
- 2.Measured trial starts
- 3.Waited for the planned sample
What happened
The benefit-led headline lifted trials 11%.
Comparisons and Boundaries
| Related term | Relationship | Why it matters |
|---|---|---|
| Hypothesis | Builds on | Tests start with hypotheses |
| Experiment Validity | Used with | Validity makes results trustworthy |
| Conversion Rate Optimization | Used with | CRO relies on testing |
| Test Prioritization | Used with | Prioritization picks tests |
Common Misconceptions
| Mistake | What to do instead |
|---|---|
| Stopping early | Early leads are often chance. |
| Testing many changes at once | You will not know what worked. |
Related concepts
Sources
- [1] John Caples, Tested Advertising Methods
- [2] Tim Ash, Maura Ginty, and Rich Page, Landing Page Optimization
- [3] Ron Kohavi, Diane Tang, and Ya Xu, Trustworthy Online Controlled Experiments