Guide
The complete A/B testing guide
The method behind every conversion test, step by step.
A/B testing
Multivariate testing: what it is and when it is worth running
Multivariate testing changes several elements of one page at the same time and tests every combination, so you learn which mix works best and whether the elements affect each other. It answers a richer question than an A/B test, and it costs far more traffic. This guide shows how MVT testing works, the arithmetic behind it, and when a plain A/B test is the better choice.
What is multivariate testing?
Multivariate testing is a method of testing several elements of one web page at once. You create versions of each element, such as two headlines and two hero images, and the testing tool shows each visitor one combination. Comparing the combinations shows which mix converts best and how much each element contributes on its own.
The name comes from statistics: "multivariate" means many variables. In testing, each element you change is a variable (sometimes called a factor), and each version of it is a level. MVT testing is the short form you will see in testing tools and job ads. It is one member of a family of controlled tests; the complete A/B testing guide covers the method they all share.
How does a multivariate test work?
A multivariate test works by multiplying versions. The number of combinations is the number of versions of each element multiplied together, the current version included. Visitors are split evenly across every combination, each visitor stays in the same one, and the tool records conversions for each combination and for each version of each element.
Take a product page where you want to test three things. The table shows how quickly the combinations add up.
| Element (factor) | Versions (levels) | Running total of combinations |
|---|---|---|
| Headline | 3: current, benefit-led, question | 3 |
| Hero image | 2: product alone, product in use | 3 x 2 = 6 |
| Button text | 2: "Add to cart", "Buy now" | 6 x 2 = 12 |
| Delivery note under the price | 2: absent, present | 12 x 2 = 24 |
Three elements make 12 versions of the page; one more yes-or-no element makes 24. Each of those is a page some visitors will see, so each needs enough visitors and conversions to be judged. That multiplication is the whole story of multivariate testing: it is what makes the method informative, and it is what makes it expensive.
What a multivariate test measures
- The best combination. Which of the 12 (or 24) pages converted best.
- Main effects. How much each element moves conversions on average, across all the versions of everything else. For example, whether the benefit-led headline did better whichever image it sat beside.
- Interaction effects. Whether two elements change each other's effect. A headline that promises speed may work with the "Buy now" button and fall flat with "Add to cart". An A/B test that changes one thing at a time cannot see this.
Multivariate testing vs A/B testing
The difference between multivariate testing and A/B testing is what changes and how traffic is shared. An A/B test compares whole versions of a page and tells you which won. A multivariate test varies several elements in every combination and tells you which element mattered and how they interact, at the cost of splitting traffic across many more versions.
| Test | What changes | What it tells you | Traffic needed | Best for |
|---|---|---|---|---|
| A/B test | One version against the current page | Which version won | Lowest | Most tests, bold changes, modest traffic |
| A/B/n test | Several whole versions against the current page | Which of several versions won | Grows with each version | Comparing a few distinct ideas |
| Split URL test | Two different pages on different addresses | Which page won | Same as an A/B test | Redesigns and new templates |
| Multivariate test | Several elements, every combination | Best combination, each element's effect, interactions | Highest: grows with every combination | Refining a settled design on a high-traffic page |
A practical way to choose: an A/B test answers "is this new direction better?" and a multivariate test answers "which details of this direction matter?". If you do not yet know the direction, an MVT spreads your traffic thin to optimize details of a page that may be the wrong page. For worked examples of the A/B side, see these A/B test examples, which include a well-known multivariate test from a political campaign.
How much traffic does a multivariate test need?
A multivariate test needs roughly the sample one A/B variant would need, multiplied by the number of combinations. Each combination is compared with the others, so each has to collect enough conversions on its own. Work out the per-version sample with a calculator first, multiply by the combinations, then divide by your daily traffic to see how long it would take.
A worked example with round, hypothetical figures: suppose a sample size calculator says each version of your page needs 10,000 visitors to detect the lift you care about. An A/B test needs 20,000 visitors (control plus one variant). The 12-combination test above needs about 120,000. At 2,000 visitors a day, that is 10 days for the A/B test and 60 days for the MVT, before you round up to whole weeks.
Why the bill is often even higher
Comparing many combinations means many chances of a false positive: with enough versions, one will look like a winner by luck alone. Correcting for this, as a careful analysis should, raises the sample each combination needs. The A/B test sample size calculator shows how the total grows as you add variants, and why each extra version costs more than the last.
Main effects need less than the best combination
There is one piece of good news. A main effect, such as "the benefit-led headline does better", is estimated from every visitor who saw that headline, across all images and buttons. So you can learn which version of each element helps with less traffic than it takes to crown a single winning combination with confidence. Plan which question you need answered before you size the test.
Full factorial vs fractional factorial multivariate tests
A full factorial multivariate test shows every possible combination, so it can measure every interaction, but it needs the most traffic. A fractional factorial test shows a planned subset of combinations and estimates the rest from them. It needs less traffic, but it assumes some interactions are too small to matter and cannot measure the ones it leaves out.
| Design | Combinations shown | Measures interactions? | Trade-off |
|---|---|---|---|
| Full factorial | All of them | Yes, all of them | Highest traffic; the honest default when you can afford it |
| Fractional factorial (including the Taguchi method) | A planned subset | Only some; the rest are assumed small | Less traffic, but a wrong assumption gives a confidently wrong answer |
Most website testing tools that offer MVT run full factorial tests by default, and some offer a fractional option. If your tool proposes a fractional design, ask which interactions it assumes away. If the reason you wanted a multivariate test was to find an interaction, a fractional design may skip exactly the one you were looking for.
When should you use multivariate testing?
Use multivariate testing when a page already gets many conversions, its overall design is settled, and you suspect two or three specific elements affect each other. Use an A/B test when you are trying a new direction, testing a single idea, or working with modest traffic. Most sites run far more A/B tests than multivariate tests.
MVT is a good fit when
- The page has high traffic and many conversions a week: a home page, a top landing page, a busy product template.
- The layout is settled and you are refining details, not choosing a direction.
- You have a reason to expect an interaction, such as a headline promise that only works if the button repeats it.
- You can limit the test to two or three elements with two or three versions each.
An A/B test is the better choice when
- Your planned MVT would need more than a couple of months to reach its sample.
- You are testing a redesign, a new offer or a new page structure.
- You only care which version wins, not why.
- The change is bold enough that a single test will show a clear answer.
How to run a multivariate test in 7 steps
To run a multivariate test, start from evidence about the page, pick two or three elements, limit their versions, count the combinations, size the test before launch, run it for whole weeks without stopping early, and read the main effects and interactions before you trust any single winning combination.
-
Start from evidence about the page
Use analytics, session recordings, surveys and support questions to find what holds visitors back on this page. Elements chosen because they are easy to change waste the most expensive kind of test.
-
Pick two or three elements
Choose elements you believe matter and might interact: the headline and the button, the hero image and the headline, the price display and the delivery note. Write down why you expect each to matter.
-
Limit the versions of each
Two or three versions per element, current version included. Every extra version multiplies the combinations. Make each version a real alternative, not a near copy, so it has a chance of showing a difference.
-
Count the combinations and remove nonsense
Multiply the versions to get the number of pages. Check that every combination makes sense together; a headline that refers to an image that is not shown is a broken page, not a test.
-
Size the test before you launch
Find the per-version sample for the smallest lift worth detecting, multiply by the combinations, and divide by daily traffic. If the answer is longer than you can wait, cut elements or versions, or run an A/B test instead.
-
Run whole weeks and do not stop early
Choose one main metric and a stopping rule in advance. Weekday and weekend visitors behave differently, so run whole weeks. With many combinations, early leaders change often, and stopping on one is the fastest way to a false win.
-
Read main effects and interactions, then confirm
Look at which version of each element helped on average and whether any pair interacted. Check the result in a statistical significance calculator that accounts for the number of comparisons. If one combination won narrowly, confirm it with a follow-up A/B test against the current page before you build it.
Common multivariate testing mistakes
The most common multivariate testing mistakes are testing too many elements for the traffic, stopping as soon as one combination leads, ignoring the extra false positives that many comparisons create, choosing elements without evidence, and building a narrow winning combination without confirming it first.
- Too many elements. Four elements with three versions each is 81 pages. Few sites can fill that.
- Calling it early. With many combinations, one is always ahead by chance at some point. Stick to the planned sample.
- Ignoring multiple comparisons. Twelve combinations are many comparisons, and some will look significant by luck.
- Testing what is easy, not what matters. Button colors are simple to vary and rarely the problem.
- Near-identical versions. Tiny differences need huge samples to detect, multiplied by every combination.
- Broken combinations. Check every page the test will create, on phones as well as desktops.
- Building an unconfirmed winner. A narrow lead among many combinations deserves a follow-up A/B test.
- Leaving the winner in the testing tool. Build it into the page itself, so it loads fast and AI agents that read the HTML see it too.
What to look for in a multivariate testing tool
A multivariate testing tool should let you define elements and versions separately, generate and preview every combination, keep each visitor in one combination, report main effects as well as combination results, and correct for multiple comparisons. Many general A/B testing tools include MVT, but some only on higher plans, so check before you buy.
- Element-level setup. You define the headline versions and image versions; the tool builds the combinations.
- Preview of every combination. So you can catch broken pages before visitors do.
- Main effect reporting. Not just a leaderboard of combinations.
- Full or fractional choice, explained. If the tool offers fractional designs, it should say which interactions are assumed away.
- Server-side or fast client-side delivery. Many combinations means many page changes; flicker and slow loading can bias the result.
Is multivariate testing worth it for a small site?
Multivariate testing is rarely worth it for a small site. Splitting modest traffic across a dozen combinations leaves each with too few conversions to judge within a few weeks. A small site learns more from one bold A/B test at a time, run in sequence, or from fixing clear problems directly and watching the numbers afterwards.
Sequential A/B tests lose the interaction effects, but on most pages the biggest gains come from fixing obvious problems: an unclear headline, a hidden price, a form that asks too much. Those do not need a multivariate test, or any test, to find. A CRO audit is a quick way to see which ones your page has.
Audit and rebuilds
Where Convertica fits
Convertica has run conversion research and A/B tests for clients since 2017, and its CRO case studies show the results with metric, timeframe and source. Today it offers a free CRO audit, then a one page rebuild or a full site rebuild. The free CRO audit checks your page for people and for AI agents and gives you every finding, with why each one matters. If you want the fixes built for you, the one page rebuild has Convertica's team build every fix the audit found on that page, and the full site rebuild covers your whole site.
Related guides and tools
More on testing
Free tool
A/B test sample size calculator
How many visitors each version needs before you start.
Free tool
Statistical significance calculator
Check whether a finished result is real before you act on it.
A/B testing services
Want your A/B tests run for you?
Convertica runs A/B tests for clients, quoted one by one as a custom proposal. Monthly traffic comes first, because it decides whether a test on your site can reach a clear result, and how long it takes.
The split tests in Convertica's case studies were run by its team for past clients, and each result is published with its metric, timeframe and source.
Short answer
Multivariate testing (MVT) tests several elements of one page at the same time, such as a headline, an image and a button, by showing visitors every combination of their versions. It reveals which combination converts best and how the elements interact, but every combination needs its own sample, so it suits pages with high traffic and many conversions.
- A multivariate test multiplies versions: three headlines, two images and two buttons make 12 combinations.
- Each combination needs about the sample one A/B variant would, so the traffic bill grows with every element you add.
- Its unique answer is the interaction effect: whether two changes work better, or worse, together.
- Use MVT to refine a settled design on a high-traffic page; use an A/B test for bold changes and modest traffic.
- Plan the combinations, sample size and stopping rule before launch, and treat any single combination's lead with suspicion.
Questions about multivariate testing
What is multivariate testing?
Multivariate testing is a way of testing several elements of one page at once, such as the headline, the image and the button, by showing visitors every combination of their versions. It shows which combination converts best and how much each element contributes, including whether two elements work better or worse together.
What is the difference between A/B testing and multivariate testing?
An A/B test compares whole versions of a page, usually the current page against one alternative. A multivariate test changes several elements and tests every combination of them. The A/B test tells you which version won; the multivariate test also tells you which element mattered, but it splits your traffic across many more versions.
What does MVT stand for in testing?
MVT stands for multivariate testing. MVT testing, multivariate testing and multivariate experiments all mean the same method: several page elements varied at once, with every combination shown to a share of visitors.
How much traffic do you need for a multivariate test?
Each combination needs roughly the sample a single variant would need in an A/B test, so the total is about that sample multiplied by the number of combinations. Three headlines, two images and two buttons make 12 combinations, which means about 12 times the visitors of a one-variant test. Work the per-version figure out with a sample size calculator first.
What is the difference between full factorial and fractional factorial testing?
A full factorial test shows every possible combination, which measures every interaction but needs the most traffic. A fractional factorial test, sometimes called the Taguchi method, shows a planned subset of combinations and estimates the rest. It needs less traffic, but it assumes some interactions do not matter, and it cannot measure the ones it skipped.
When should you use multivariate testing instead of A/B testing?
Use multivariate testing when a page already gets a lot of conversions, the overall design is settled, and you want to know which of a few elements to refine and whether they interact. Use an A/B test for a bold redesign, a new offer or any page with modest traffic.
Can a small website run multivariate tests?
Usually not well. Splitting modest traffic across a dozen combinations means each one collects too few conversions to give a reliable answer within a few weeks. A small site learns more from one bold A/B test at a time, or from fixing clear problems found in an audit without testing them.
