Plan a test
A/B test sample size calculator
Visitors each variant needs, and how many days that takes at your traffic.
A/B testing
How long to run an A/B test
How long to run an A/B test depends on one number you can work out before launch: the sample size each version needs. Divide it by your daily traffic, round up to whole weeks, and do not stop early. This guide shows the calculation, a worked example and what to do when your traffic is low.
How long should you run an A/B test?
An A/B test should run until each version has reached the sample size you calculated before launch, rounded up to whole weeks. For most websites that means at least one full week, usually two or more, and never less than one full business cycle. The right length comes from your traffic and the size of change you want to detect, not from the calendar.
There is no single correct number of days. A busy store testing a big change can finish in two weeks. A small lead generation site testing a subtle change might need months, which is a sign to change the test rather than wait. The sections below show how to work out your own answer.
How to calculate A/B test duration in 6 steps
To calculate A/B test duration, find the sample size each variant needs, multiply it by the number of variants, divide by the number of visitors who enter the test each day, and round up to whole weeks. The formula is: days = (sample size per variant x number of variants) / daily visitors in the test.
-
Find your baseline conversion rate
Take the current conversion rate of the exact page or step you will test, for the visitors who will enter the test, over recent whole weeks without unusual promotions. A lower baseline needs a bigger sample.
-
Choose the smallest lift worth detecting
This is the minimum detectable effect: the smallest improvement that would change what you do. Halving it roughly quadruples the sample, so be honest about what matters to the business rather than hoping to catch tiny effects.
-
Set confidence and power
Most teams use 95% confidence and 80% power. Higher settings protect you better against false winners and missed winners, and they make the test longer.
-
Calculate the sample size per variant
Enter those numbers into the free A/B test sample size calculator. It returns the visitors each variant needs, and it shows its formula.
-
Divide by your daily traffic into the test
Multiply the sample per variant by the number of variants, including the control, then divide by the visitors who actually enter the test each day. Count only the traffic that sees the tested page, not your whole site.
-
Round up to whole weeks and write it down
Round the days up to full weeks so every day of the week is counted equally. Note the planned end date, the sample and the success metric before launch, and stick to them.
How long to run an A/B test: a worked example
Say a product page converts at 3% and you want to detect a 20% relative lift, from 3% to 3.6%, at 95% confidence and 80% power, with one variant against the control. 2,000 visitors a day reach that page.
- Sample per variant: 13,914 visitors (from the sample size calculator).
- Total sample: 13,914 x 2 variants = 27,828 visitors.
- Days: 27,828 / 2,000 a day = 13.9, rounded up to 14 days.
- Whole weeks: run the test for 2 weeks, then stop and read the result.
If the same page only got a quarter of that traffic, the test would need four times as long. That is why the duration question is really a traffic question.
How traffic and effect size change test length
Traffic and the size of the change you want to detect set the length of an A/B test more than anything else. The table below shows whole weeks for a page converting at 3%, with one variant against the control, 95% confidence and 80% power. Small lifts on low traffic quickly become impractical.
| Smallest lift to detect | Visitors per variant | 500 visitors a day | 2,000 visitors a day | 10,000 visitors a day |
|---|---|---|---|---|
| 10% (3% to 3.3%) | 53,211 | 31 weeks: change the plan | 8 weeks | 2 weeks |
| 20% (3% to 3.6%) | 13,914 | 8 weeks | 2 weeks | 1 week |
| 30% (3% to 3.9%) | 6,455 | 4 weeks | 1 week | 1 week |
Every figure in the table comes from the same formula as the sample size calculator, with a one-week minimum. Anything over 8 weeks is marked "change the plan": a test that long is exposed to cookie loss, seasonality and site changes, and it ties up the page.
What affects how long an A/B test takes?
Seven things decide how long an A/B test takes: your baseline conversion rate, the smallest lift you want to detect, the traffic that enters the test, the number of variants, your confidence and power settings, whether the test is one-sided or two-sided, and how long your customers take to decide. The first three matter most.
| Factor | Longer test when | Shorter test when |
|---|---|---|
| Baseline conversion rate | The page converts rarely | The page converts often |
| Smallest lift to detect | You want to catch small lifts | You only care about large lifts |
| Traffic into the test | Few visitors reach the tested page | The page is one of your busiest |
| Number of variants | Several variants share the traffic | One variant against the control |
| Confidence and power | Set higher, such as 99% and 90% | Set to the usual 95% and 80% |
| One-sided or two-sided | Two-sided, which also checks for harm | One-sided, planned before launch |
| Buying cycle | Customers take weeks to decide | Customers decide in one visit |
What is the minimum time to run an A/B test?
The minimum time to run an A/B test is one full week, even when the planned sample arrives sooner. Visitors on a Monday morning and a Saturday night are often different people with different intent, so a test that only saw two weekdays has only measured those two weekdays. Two full weeks is a safer floor for most sites.
Cover a full business cycle
If your customers usually take a week or two to decide, or your sales follow paydays, monthly billing or a weekly email, run long enough to include at least one full cycle. Otherwise the test measures the people who decide fastest, who may not behave like everyone else.
Allow for the novelty effect
A change can win early simply because returning visitors notice something new, then fade as the novelty wears off. A test that runs for whole weeks gives that early bump time to settle, which is one more reason not to call a result in the first few days.
Avoid unusual periods
Sales, holidays, a big email campaign or a press mention can bring visitors who behave differently from your normal traffic. If the test overlaps one, either extend it so normal weeks dominate or note the overlap when you read the result.
Great results. easy to work with, and do proper split-testing. Even noticed when I accidentally made a change to a page they were testing, and they stopped the test and started again to make sure I didn't distort results. The most important thing to know: they had a positive result on revenue.
Why you should not stop an A/B test early
Stopping an A/B test early, the moment one version looks like a winner, is one of the most common ways to get a false result. Results swing widely in the first days of a test. If you check repeatedly and stop the first time the numbers cross your significance threshold, the chance that your "winner" is only noise rises well above the level you set.
This is often called peeking. Researchers Ramesh Johari, Leo Pekelis and David Walsh describe it in Always Valid Inference: Bringing Sequential Analysis to A/B Testing (2015): standard p-values are only reliable when the sample size is fixed in advance, not when the people running the test decide when to stop by watching the results. A peer-reviewed version, with Pete Koomen as a coauthor, was published in Operations Research in 2022 (doi.org/10.1287/opre.2021.2135). The p-value calculator page sets out what a p-value does and does not tell you about a test, peeking included.
When can you stop an A/B test early?
- The test is broken. A variant does not load on some browsers, tracking stopped, or someone changed the page mid-test. Stop, fix it and start again with fresh data.
- The variant is clearly causing harm. Agree a guardrail before launch, such as a large drop in orders, and stop if it is crossed. Stopping to protect revenue is not the same as stopping to declare a win.
- You planned for it. Sequential testing methods, which some testing tools offer, are designed to be checked as data arrives and set their own stopping rules. Choose the method before launch, not after you have seen the numbers.
Can an A/B test run too long?
Yes, an A/B test can run too long. Over many weeks visitors clear cookies, switch devices and may see both versions, which blurs the difference you are measuring. Seasonality, new campaigns and other site changes creep in, and every extra week delays rolling out a winner or moving on from a loser.
Set a ceiling before launch. If the calculation says the test needs many months, do not start it and hope: change the plan using the options below. A test that cannot finish in a reasonable time is not a test worth running in that form.
What to do when your A/B test would take too long
When the numbers say an A/B test would take too long, change what you test rather than cutting the test short. Bigger changes, busier pages, a goal closer to the change and fewer variants all shorten the test. Some problems do not need a test at all.
- Test a bigger change. A rewritten offer or a new page layout can produce a larger difference than a button color, and a larger difference needs a smaller sample.
- Test where the traffic is. Your homepage, top landing pages or the first step of checkout see more visitors than a deep product page.
- Measure a nearer goal. Clicks to checkout or form starts happen more often than purchases, so they reach a sample faster. Check that the nearer goal really tracks the final one.
- Use two variants, not five. Every extra variant splits the same traffic further.
- Fix the obvious without testing. Broken forms, slow pages and missing information should be fixed, not tested. Use session recordings and visitor polls to find them.
Not sure which problems on your site are obvious fixes and which need a test? That is where Convertica's free CRO audit can help: it checks your page for people and for AI agents and gives you every finding, with how to fix each one.
How long real A/B tests ran: three Convertica case studies
Convertica's published case studies show how test length follows traffic and the size of the change. Each test below ran until it had enough data, and none was stopped on the first good day.
| Case study | What was tested | How long it ran | Stated result |
|---|---|---|---|
| dScryb | Calls to action and access levels in membership search results, desktop and mobile | 17 days, 20,000 visitors, 98% confidence | +49.4% paid membership sign-ups |
| JustThrive | Product page trust signals and auto-ship offer, then a second variation against the winner | Test 1: 7 December to 15 January. Test 2: 15 February to 24 March | +101.82% revenue, cumulative across both tests |
| Empire Flippers | A rebuilt valuation tool submission process, split-URL tested | First test: two weeks. Result after rollout: 47 days | +51.6% conversions (valuation tool submissions) in 47 days |
Checklist: is your A/B test ready to stop?
Before you stop an A/B test and act on the result, check each line below. If any answer is no, keep the test running or fix the problem and restart it.
- Each variant has reached the sample size you planned before launch
- The test has run for whole weeks, at least one and ideally two
- It covered at least one full business cycle for your customers
- Nothing changed on the tested page, the offer or the tracking during the test
- Traffic was split as planned and no variant was broken on any device or browser
- The result is checked on the metric you chose before launch, not one picked afterwards
- You have checked significance with a statistical significance calculator
- You have looked at desktop and mobile separately before rolling out
Free tools to plan and read your test
Plan the length of your next A/B test before you launch it, and read the result properly when it ends. Each calculator shows its formula and a worked example.
Check a result
Statistical significance calculator
Whether a finished A/B test result is bigger than chance.
Measure
Conversion rate calculator
Your baseline conversion rate from visitors and conversions.
Short answer
An A/B test should run until each version has reached the sample size you calculated before launch, rounded up to whole weeks. For most websites that means at least one full week, usually two or more, and never less than one full business cycle. To get the days, divide the total sample by the daily visitors who enter the test.
- The formula is days = (sample size per variant x number of variants) / daily visitors in the test.
- Halving the smallest lift you want to detect roughly quadruples the sample, so pick a lift that matters to the business.
- Count only the traffic that sees the tested page, not your whole site.
- Write down the end date, sample and success metric before launch, and do not stop early because a variant looks like a winner.
- If a test would take months, change the test rather than wait.
Questions about A/B test duration
How long should you run an A/B test?
Run an A/B test until each variant reaches the sample size you calculated before launch, then round up to whole weeks. In practice that means at least one full week, ideally two, and long enough to cover a full business cycle. Never stop early just because one version is ahead.
How do you calculate how long to run an A/B test?
To work out how long to run an A/B test, calculate the sample size each variant needs from your baseline conversion rate, the smallest lift you care about, your confidence level and your power. Multiply by the number of variants, divide by the visitors who enter the test each day, and round up to whole weeks.
What is the minimum time to run an A/B test?
The minimum is one full week, even if the planned sample arrives sooner, because people behave differently on different days of the week. Two full weeks is a safer floor for most sites, and longer if customers usually take more than a week to decide.
Can an A/B test run too long?
Yes. Over many weeks, visitors clear cookies or switch devices and can see both versions, seasonality and marketing changes creep in, and you delay rolling out a winner. If the plan says the test needs months, change the plan: test a bigger change, a busier page or a nearer goal.
Can I stop an A/B test as soon as it reaches statistical significance?
No, not with a standard fixed-sample test. Checking results repeatedly and stopping the first time the result looks significant makes false winners much more likely. Wait for the planned sample, or use a testing method built for early stopping, such as a sequential test, and set its rules before launch.
What if my site does not have enough traffic to A/B test?
Test bigger changes, test on your busiest pages, measure a step closer to the change, such as clicks to checkout, and limit the test to two variants. If that is still too slow, fix clear problems without a test and use research such as session recordings and visitor polls to guide the changes.
How long should an email A/B test run?
Email A/B tests work differently from website tests: each recipient gets one send, so the test lasts until most opens and clicks have come in, often hours rather than weeks. The rules on sample size still apply, so a small list may not give a reliable answer.
Does a longer A/B test give a more accurate result?
Only up to a point. More visitors narrow the uncertainty around the result, but past the planned sample the gains are small and the risks of a long test grow. Plan the sample you need, run to it in whole weeks, then decide.
