# What is A/B Testing?

What is A/B Testing? Comparing two versions with equal-sized groups. Version A is the existing one, version B is the change.

## How it works in practice

A/B testing is a scientific experiment. Without it, you're guessing. With it, you know. 40% of A/B tests reveal that the 'improvement' you assumed would help actually reduces conversions.

## Related metrics

- CRO — A/B testing is the central tool
- Statistical Significance — without it, results aren't valid
- Landing Page — a common place to A/B test
- Email Subject Lines — the best candidate for testing

## Common misconception

'A one-week A/B test is enough.' Wrong. You need 100-200 conversions PER VARIANT. CR = 2% with 200 daily visits gives 4 conversions/day — 50 days to a result.

## How to fix it

Not reaching significance? (1) Continue until you hit the required sample size, (2) check for external factors (holidays, unusual promotions), (3) don't change anything else during the test.

## Real-world example

An oils store tested 'Buy Now' vs. 'Add to Cart.' Over 3 weeks and 4,000 visits: 'Add to Cart' had a 3.1% CR, 'Buy Now' had 2.7%. A 14.8% difference at 97% significance.

## Where to start

Highest-ROI order to test: (1) email subject line, (2) product-page CTA, (3) main image, (4) free-shipping threshold, (5) review placement.

---

Category: UX · From the [Jimmy OS KPI Glossary](https://jimmy-os.com/glossary/kpi.md) · Source page: https://jimmy-os.com/glossary/kpi/a-b-testing