Guide
A/B testing guide
How to plan, run and read a single A/B test, step by step.
Testing method
CRO testing: the types of test and how to plan them
CRO testing checks a change to your website against the current version with real visitors before you keep it. This guide covers the types of CRO test, which one fits your traffic, and how to build a testing plan and roadmap that decides the rules before the test starts.
What is a CRO test?
A CRO test is a controlled experiment that shows real visitors two or more versions of a page or flow and measures which one gets more of them to convert. Visitors are split at random, so the only systematic difference between the groups is the change. That lets you credit a lift to the change instead of to a busy week or a new ad.
CRO testing is one stage of conversion rate optimization, the wider process of finding out why visitors do not convert and fixing it. Research tells you what to change; the test tells you whether the change worked. Skipping the research is the most common reason a testing program produces a long list of flat results.
This page is the map: which type of CRO test to use, what each needs, and how to plan a series of them. The step-by-step mechanics of a single split test are in our A/B testing guide.
When is a CRO test worth running?
A CRO test is worth running when the page gets enough conversions to reach a result in a few weeks, the change is uncertain enough that you could be wrong, and the decision matters enough to wait for. If any of the three is missing, ship the fix, measure before and after, or research further instead.
- Enough conversions. What limits a test is the number of conversions, not visits. A page with plenty of traffic but few sales may still be too thin to test small changes.
- Real uncertainty. Fixing a broken button or a checkout error does not need a test. Testing obvious bugs spends traffic you could use on questions with two plausible answers.
- A decision that matters. A new pricing layout, a shorter form or a rebuilt product page can change revenue for years. A footer tweak rarely justifies weeks of traffic.
What are the types of CRO tests?
There are three families of CRO test. Controlled experiments (A/B, A/B/n, split URL, multivariate and bandit tests) split live traffic at random. Comparisons over time (before-and-after) need no split but give weaker evidence. Qualitative tests (usability tests, painted door tests) explain behavior or measure demand with very little traffic. The table shows what each needs.
| Test type | What changes | Traffic needed | Best for |
|---|---|---|---|
| A/A test | Nothing: two identical versions | Same as an A/B test | Checking that the tool splits traffic and counts conversions correctly before real tests |
| A/B test | One version against the current page | The least of any controlled test | Most questions about one page: a headline, a form, a layout, an offer |
| A/B/n test | Several versions against the current page | More with every extra variant | Comparing a few distinct ideas for the same element when traffic allows |
| Split URL test | Two separate pages on different URLs | Same as an A/B test | Full redesigns, new page templates and rebuilt flows that a visual editor cannot make |
| Multivariate test | Several elements at once, in every combination | The most, growing with each combination | High-traffic pages where you want to know how elements interact |
| Multi-armed bandit | Several versions, with traffic shifted toward the leader | Similar to A/B, with less certainty at the end | Short campaigns and promotions where earning during the test matters more than learning |
| Before-and-after | The whole site or page, compared across two periods | Any, but the evidence is weakest | Low-traffic sites and changes that cannot be split, read with care |
| Usability test | Nothing live: people try the page while you watch | None; a handful of participants | Finding why people struggle, and generating hypotheses to test |
| Painted door test | A link or button to something not built yet | Enough clicks to judge interest | Measuring demand for a feature, plan or offer before building it |
A/B and A/B/n tests
An A/B test sends half the visitors to the current page and half to one changed version, then compares conversion rates. It is the default CRO test because it needs the least traffic for a clear answer. An A/B/n test adds more variants; each one takes a share of the traffic, so the test needs longer to reach the same certainty. The full method, from hypothesis to reading results, is in the A/B testing guide, and real ones are collected in our A/B testing examples.
Split URL tests
A split URL test is an A/B test between two whole pages hosted at different addresses, with traffic split by a redirect. Use it when the new version is a rebuild rather than an edit, such as a new checkout or a step-by-step form in place of a long one. Set a canonical tag on the variant so search engines treat the original as the main page.
Multivariate tests
A multivariate test changes several elements at once, such as a headline, an image and a button, and shows every combination. It tells you which combination wins and whether elements affect each other, but every added element multiplies the number of versions, so it suits only pages with heavy traffic. On most sites, a sequence of A/B tests answers the same questions sooner.
Multi-armed bandits
A bandit test moves traffic toward whichever version is ahead while the test runs, so fewer visitors see the weaker version. The cost is a less certain answer, because the losing version gets too few visitors to measure well. Bandits suit short promotions; for lasting decisions a fixed split is the safer read. Our guide to AI A/B testing covers how tools automate this.
Before-and-after comparisons
A before-and-after comparison measures conversion for a period before a change and a period after it. It needs no traffic split, so it works on any site, but seasonality, campaigns and changes in traffic mix all land in the result. Keep other changes on hold, compare equal lengths of time, and treat the result as a strong hint rather than proof.
Usability and painted door tests
A usability test watches a handful of people try to complete a task on your page and asks them to think aloud. It does not prove a lift; it shows why people get stuck, which is where good hypotheses come from. A painted door test shows a link to an offer or feature that does not exist yet and counts clicks; tell the visitors who click that it is coming, so the test stays honest.
Which test types did Convertica's case studies use?
Convertica's published case studies used four of the test types above: single A/B tests, split URL tests for rebuilt flows, sequential split tests where each round tested against the last winner, and a before-and-after comparison. The table lists each result with the metric, timeframe and source exactly as the case study states them. Results from one site do not predict another's.
| Test type | Case study | Result and metric | Timeframe | Source |
|---|---|---|---|---|
| Split URL test, then rollout | Empire Flippers | +51.6% conversions (valuation tool submissions) | 47 days | Empire Flippers, valuation tool submissions, 47 days, split-URL test then rollout |
| Two sequential split tests | JustThrive | +101.82% revenue, cumulative across two sequential split tests | 7 December to 24 March | JustThrive, revenue, two sequential split tests, Dec to Mar |
| A/B test | dScryb | +49.4% paid membership sign-ups | 17-day A/B test | dScryb, paid membership sign-ups, 17-day A/B test, 98% confidence |
| Split test across several pages | 10Beasts | +66.3% ad fees, winning pages only, test 2 | 20 March to 18 April | 10Beasts, Amazon Associates report, split test on 6 money pages, 4 winners rolled out |
| A/B test, mobile | Sticky CTA | +252.9% rate of order completion on mobile | 20-day A/B test | Online store, landing page A/B test, mobile, 98% confidence |
| Before-and-after comparison | Robert's affiliate site, part 1 | +$15,048 monthly revenue, affiliate site | 34 days, August 2017 | Robert's affiliate site, monthly revenue before the work vs August (a before and after comparison, not a single split test) |
Two patterns stand out. The rebuilt flows (Empire Flippers' step-by-step submission process) were split URL tests, because the new version was a different page, not an edit. And the biggest cumulative result came from a roadmap, not a single test: JustThrive's second test ran against the first test's winner. The second part of Robert's story, about 50 split tests in three months, shows the same approach at volume. Every write-up is in the CRO case studies.
How to build a CRO testing plan in 7 steps
To build a CRO testing plan, start from research rather than ideas, turn each problem into a hypothesis, rank the hypotheses, and for each one fix the metric, sample size, length and decision rule before launch. Then build and check the variant, run it untouched, and record the result. The plan's job is to settle every judgment call while nobody has seen data yet. The research that feeds the plan is covered step by step in our guide to CRO analysis.
Start from evidence, not opinions
List the problems your analytics, session recordings, surveys, usability tests and an audit of the page have found. Each test idea should trace back to something you observed, such as visitors abandoning a form at the phone number field, not to a competitor's design or a hunch.
Write a hypothesis for each problem
Turn every problem into a testable statement of what you will change, for whom, and which metric should move as a result. The format is in the next section. A hypothesis you cannot prove wrong is not ready to test.
Prioritize the list
Score each hypothesis with a simple framework such as PIE or ICE, described below, and sort by score. The score is a way to argue less and start sooner, not a forecast.
Choose one primary metric and your guardrails
Pick the single metric that decides the test, as close to revenue as your traffic allows: orders, sign-ups, qualified leads. Add guardrail metrics that must not get worse, such as average order value, refund requests or page load time.
Work out sample size and length before you start
Enter your baseline conversion rate and the smallest lift worth detecting into an A/B test sample size calculator, then turn the sample into whole weeks with our guide on how long to run an A/B test. If the answer is months, test a bolder change or a higher-traffic page.
Build the variant and check it
Build the change, then check it on desktop and phone, in the main browsers, logged in and out. Confirm the conversion event fires once per action in both versions. Run an A/A test first if the tool is new to the site.
Run it untouched, then decide by the rule you wrote
Do not edit the variant, change traffic sources or stop early while the test runs. When it reaches its planned sample, read it with a statistical significance calculator and apply the decision rule from your plan.
How do you write a CRO test hypothesis?
A CRO test hypothesis names the evidence, the change, the audience and the expected effect on one metric. A common format is: because we saw [evidence], we believe that [change] for [audience] will [increase or decrease] [metric]. We will know when [metric] changes by at least [smallest lift worth detecting] at the planned sample size.
An illustrative example, not taken from a real test: because session recordings show mobile visitors scrolling past the shipping cost and leaving at checkout, we believe that showing the shipping cost beside the price on product pages for mobile visitors will increase completed orders. The evidence, the change and the metric are all specific, so a flat result still teaches you something.
How do you prioritize CRO tests?
Prioritize CRO tests by scoring each hypothesis on a few factors, usually how much it could move the metric, how strong the evidence is, and how easy it is to build, then running the highest scores first. PIE and ICE are the two most widely used scoring frameworks. Both are rough by design; their value is that the whole team scores the same way.
| Framework | Factors, each scored on the same scale | Strength | Weakness |
|---|---|---|---|
| PIE | Potential (how much room the page has to improve), Importance (how valuable its traffic is), Ease (how simple the test is to build and run) | Ranks pages as well as ideas, so it shows where to test first | Potential is a judgment call, and scores drift between people |
| ICE | Impact (how much the change could move the metric), Confidence (how strong the evidence is), Ease | Quick to apply, and Confidence rewards ideas backed by research | Says nothing about traffic, so a high score can still be untestable |
Whichever you use, add a traffic check before a test enters the queue: if the page cannot reach its sample size in a reasonable number of weeks, the idea moves to a before-and-after list or waits for a bolder version. Some teams replace the subjective scores with yes-or-no questions (is it above the fold, is it backed by user research) to keep scoring consistent.
A CRO test plan template
A CRO test plan template is one page per test that records the problem, the hypothesis, the setup and the decision rule before launch, then the result after. Copy the fields below into a document or spreadsheet. The example column is illustrative, made up to show how each field reads, and not from a real test.
| Field | What to write | Illustrative example |
|---|---|---|
| Problem and evidence | What you observed and where | Mobile visitors leave checkout at the shipping step; recordings show surprise at the cost |
| Hypothesis | Because, we believe, for, will, measured by | Showing shipping cost on product pages for mobile visitors will increase completed orders |
| Test type | A/B, A/B/n, split URL, multivariate or other | A/B test |
| Page and audience | URL, devices, traffic sources, new or returning | All product pages, mobile only, all sources |
| Variants | What each version shows, with screenshots | Control: price only. Variant: price plus shipping line |
| Primary metric | The one metric that decides the test | Completed orders per visitor |
| Guardrail metrics | Metrics that must not get worse | Average order value, add to cart rate |
| Smallest lift worth detecting | The minimum change that would justify keeping it | A relative lift you choose from your margins |
| Sample size and length | From the calculator, in whole weeks | The calculator's figure per variant, rounded up to full weeks |
| Decision rule | What counts as a win, a loss and inconclusive | Ship if significant at the planned sample and guardrails hold; otherwise keep control |
| Result and learning | Filled in after: numbers, segments, what it taught | Written after the test ends, win or lose |
What is a CRO roadmap?
A CRO roadmap is the ordered list of tests you plan to run over the coming months, page by page, with what each depends on. It turns a scored backlog into a schedule: which page is tested when, which tests cannot overlap because they share traffic, and which follow-up test is waiting on a result. Review it after every finished test.
- One test per page and audience at a time. Two tests on the same visitors at once blur each other's results. Run tests in parallel only on separate pages or separate audiences.
- Build on winners. A winning variant becomes the new control, and the next test challenges it, as JustThrive's second split test did.
- Mix sizes. Alternate bold tests, which teach a lot and need less traffic to detect, with smaller refinements on pages that have plenty of conversions.
- Keep a learning log. Record every result, including losses and inconclusive tests, so the next round of hypotheses starts from what you now know.
What to do with winners, losers and inconclusive tests
After a CRO test, act on the decision rule you wrote before it started. Roll out a clear winner and make it the new control. Keep the current version after a clear loss and record why the idea failed. For an inconclusive result, ship whichever version is simpler to maintain or test a bolder change. Every outcome goes into the learning log.
- Winner. Roll it out to everyone, then keep watching the primary metric for a few weeks; a lift measured in a test can shrink once novelty wears off. Check segments only to plan the next test, not to rescue a loser.
- Loser. A clear loss tells you something visitors value. If removing an element hurt, that element was doing work; the next hypothesis might strengthen it instead.
- Inconclusive. The change was too small to detect at this traffic, or it does not matter to visitors. Either answer frees you to move on. Do not extend the test indefinitely hoping it tips over.
- Broken. If tracking failed, the split was uneven or the variant had a bug, the result does not count. Fix it and run the test again from zero.
Audit and rebuilds
Where Convertica fits
Good tests start from good evidence, and the free CRO audit is a quick source of it: run it on the page you plan to test and it scores the page on eight checks, for people and for AI agents, and gives you every finding with how to fix it. Each finding is a ready-made problem for the first step of your testing plan. Run it from the free CRO audit page.
If you would rather have the fixes built, Convertica's conversion rate optimization services offer two fixed-price rebuilds: the one page rebuild ($497), where Convertica's team builds every fix the audit found on one page, and the full site rebuild ($1,997) for your whole site. Both are on the pricing page. Past results, with metric, timeframe and source, are in the CRO case studies.
Keep reading
Related guides
Guide
How long to run an A/B test
Turn a sample size into a test length in whole weeks.
Guide
A/B testing examples
Real tests with what changed and what moved.
Short answer
CRO testing means checking a website change against the current version with real visitors before you keep it. The main types are A/B, A/B/n, split URL and multivariate tests, with before-and-after comparisons and usability tests for low traffic. A testing plan fixes the hypothesis, metric, sample size and test length before the test starts.
- Pick the test type by your traffic: A/B and split URL need the least, multivariate the most, and usability tests need almost none.
- Write every test as a hypothesis tied to evidence: what you saw, what you will change, for whom, and which metric should move.
- Decide the primary metric, sample size and test length before launch, and do not stop a test early because one version is ahead.
- Rank ideas with a simple score such as PIE or ICE, then run them as a roadmap where each test builds on the last result.
- An inconclusive test is a result too: record it, and either ship the simpler version or test a bolder change.
Questions about CRO testing
What is CRO testing?
CRO testing is the part of conversion rate optimization where you check a proposed change against the current page with real visitors before keeping it. The most common form is an A/B test, but split URL tests, multivariate tests, before-and-after comparisons and usability tests are all CRO tests. The choice depends on your traffic and the question.
What is CRO in testing?
In testing, CRO stands for conversion rate optimization: improving the share of visitors who take an action such as buying, signing up or requesting a quote. A CRO test is the experiment that shows whether a specific change actually moved that share, rather than assuming it did.
What is CRO an acronym for?
In marketing and web work, CRO is short for conversion rate optimization. In pharmaceuticals and clinical research the same letters mean contract research organization, which is unrelated. This guide uses the marketing meaning throughout.
What are the types of CRO tests?
The main types of CRO tests are A/A tests to check the setup, A/B tests, A/B/n tests with several variants, split URL tests between two separate pages, multivariate tests of several elements at once, and multi-armed bandits. Before-and-after comparisons, usability tests and painted door tests fill the gaps when traffic is low or the question is about demand.
What should a CRO testing plan include?
A CRO testing plan should include the problem and its evidence, a hypothesis, the page and audience, the variants, one primary metric and any guardrail metrics, the minimum detectable effect, the sample size and test length worked out before launch, and the decision rule for a win, a loss or an inconclusive result.
What is the difference between CRO and SEO?
SEO brings more of the right visitors to a site through search; CRO gets more of the visitors you already have to take the action you want. They work on different numbers, traffic and conversion rate, and they help each other: a page that answers the visitor's question clearly tends to serve both.
What is the difference between CRO and A/B testing?
CRO is the whole process of finding why visitors do not convert and fixing it. A/B testing is one method inside that process, used to prove that a fix worked. You can do CRO without testing every change, and you can run A/B tests that have nothing to do with research, which is how many tests end up teaching nothing.
How long should a CRO test run?
A CRO test should run until it reaches the sample size you calculated before launch, in whole weeks so every weekday is counted, and for at least one full buying cycle. Decide the length in advance and do not stop early because one version looks ahead: early leads often fade as more visitors arrive.
What if my site does not have enough traffic for CRO testing?
With low traffic, test fewer and bolder changes, measure a step closer to the top of the funnel such as add to cart, and use usability tests and before-and-after comparisons for the rest. A before-and-after comparison is weaker evidence, so keep everything else steady and read it alongside the same weeks of the previous year where you can.
