Plan a test
A/B test sample size calculator
Visitors each version needs, and how many days that takes at your traffic.
A/B testing
Shopify A/B testing
Shopify A/B testing shows two versions of your store to different shoppers at the same time and measures which one sells more. This guide covers Shopify's own Rollouts Experiments, testing apps and script tools, what each plan lets you test, and the checkout and price testing limits, from Shopify's own help pages.
What is Shopify A/B testing?
Shopify A/B testing is a controlled experiment on a Shopify store: shoppers are split at random between the current version of a page, theme or offer and a changed version, at the same time, and the store compares how many in each group add to cart, reach checkout or buy. Because both groups shop in the same period, the difference can be put down to the change.
The method is the same as on any website, and the A/B testing guide covers it in full. What is different on Shopify is how you build the variant and what the platform lets you change: the theme, the checkout, prices and discounts each have their own rules, and some depend on your plan. This page is not affiliated with Shopify; every statement about Shopify's features comes from Shopify's own help pages.
Does Shopify have built-in A/B testing?
Yes. Shopify's Rollouts, found in Markets > Rollouts in the Shopify admin, schedule changes to your store, and one rollout type, the Experiment, compares a treatment against a control so you can measure a change before it goes to everyone. Rollouts are on the Basic plan or higher; Experiments need the Grow plan or higher.
Shopify's Rollouts help pages describe three types:
- Launch: merges your changes into the store permanently.
- Event: publishes changes for a set period, then rolls them back.
- Experiment: shows the changes to part of your eligible visitors and compares them with the rest. The split is 50% control and 50% treatment by default, and the default end date is 90 days after the start. You can change both before saving.
What a Rollouts Experiment can test
A rollout can include changes to your main theme (edit the current theme or replace it with another one), checkout and accounts configurations, discounts and product catalogs. For a theme Experiment, Shopify reports conversion rate over time, bounce rate, reached checkout rate and add to cart rate. A checkout and accounts Experiment reports checkout conversion rate.
What a Rollouts Experiment cannot do
- Experiments apply only to your online store, not headless or custom storefronts or Shopify POS.
- You cannot change Liquid templates as part of a rollout, and rollouts do not work on vintage themes.
- You cannot target a rollout to a customer segment, a customer tag or a device.
- You cannot filter the results, for example by device or by country.
- Only visitors who consent are counted, and a visitor who sees both versions in the same browser is left out.
Shopify's analytics page for rollouts does not say how a winner is decided or mention statistical significance, so check the result yourself with a statistical significance calculator before you apply it.
Three ways to A/B test on Shopify
There are three ways to A/B test on Shopify: Shopify's own Rollouts Experiments, an A/B testing app from the Shopify App Store, or a general testing tool that adds a script to your theme or runs in your own code. They differ mainly in what they can change, how you target shoppers and how the result is reported.
| Approach | Good for | Limits to check |
|---|---|---|
| Rollouts Experiment (Shopify) | A whole theme, an edited copy of your theme, checkout and accounts configurations, discounts, catalog changes | Grow plan or higher, online store only, no Liquid template changes, no device or segment targeting, no result filters |
| A/B testing app (Shopify App Store) | Listings in the App Store's A/B testing collection describe testing prices, shipping rates, content, themes, checkout and post-purchase pages | What each app can change on your plan, how it reads results, and its effect on page speed. Check the app's own listing |
| General testing tool (script or server side) | Single elements with a visual editor, targeting by device or audience, custom goals, tests outside Shopify such as a headless storefront | A script that changes the page after it loads can make it flicker; the checkout is limited by your Shopify plan |
Whichever you choose, test where your traffic is and where a change can make a visible difference. The A/B testing tools guide compares general tools in detail; the Shopify App Store's A/B testing collection lists the apps built for Shopify. Convertica does not rank or recommend a single app, and has no affiliate links on this page.
How to run an A/B test on Shopify in 8 steps
To run an A/B test on Shopify, start from a problem in your store data, write a hypothesis, choose one metric and a sample size, build the variant in Rollouts, an app or a testing tool, split traffic at random, wait for the planned sample, then check the result before you roll anything out.
-
Find the problem first
Read Shopify's analytics funnel by device: sessions, add to cart, reached checkout and orders. Watch session recordings on the step where most shoppers drop out. A Shopify CRO audit turns that into a ranked list of problems worth testing.
-
Write a hypothesis
State what you will change, for which shoppers, and why you expect it to help: "Showing shipping costs on the product page will lift orders on mobile, because shoppers leave the cart when they first see the shipping fee."
-
Choose one primary metric
Pick the metric that decides the test before launch: orders per visitor or revenue per visitor for most store tests, add to cart rate only when the change sits on the product page and you have checked that it tracks orders.
-
Work out the sample size and duration
Enter your baseline conversion rate and the smallest lift worth detecting into the A/B test sample size calculator, then divide by the daily visitors who will see the tested page. Round up to whole weeks, as set out in how long to run an A/B test.
-
Pick the tool that can change what you want to test
A whole theme or checkout configuration fits a Rollouts Experiment. A price or shipping rate needs Rollouts catalog changes or an app built for it. A single section on one template with device targeting fits an app or a general testing tool.
-
Build and check the variant
Preview both versions on a phone and a desktop, add a product to the cart, apply a discount code and place a test order. Check that orders, revenue and the test's goal are recorded for both versions.
-
Launch and leave it alone
Split traffic at random, usually 50/50. Do not edit the theme, prices or the tested section while the test runs; Shopify notes that editing the treatment or control copy during an Experiment can affect its results.
-
Read the result, then roll out or roll back
When the planned sample is reached, check significance, look at mobile and desktop separately (outside Rollouts, which cannot filter by device), then apply the winner permanently or keep the original. Record what you learned either way.
What can you A/B test on Shopify?
On Shopify you can A/B test the homepage, collection pages, product pages, the cart, a whole theme, discounts and prices on every plan that supports your chosen tool. The checkout is the exception: how much of it you can change depends on your plan, and its information, shipping and payment steps are only open to apps on Shopify Plus.
| Part of the store | Typical test | How on Shopify |
|---|---|---|
| Homepage | Hero message, featured collections, trust and shipping information | Edited theme in Rollouts, an app or a testing tool |
| Collection pages | Filters, product card details, sort order | Edited theme in Rollouts, an app or a testing tool |
| Product pages | Image gallery, variant selectors, reviews position, delivery information, Add to Cart behavior | Edited theme in Rollouts, an app or a testing tool |
| Cart | Cart drawer or cart page, free shipping message, upsells | Edited theme in Rollouts, an app or a testing tool |
| Whole theme | A new or redesigned theme against the live one | Replacement theme in a Rollouts Experiment (not vintage themes), or a theme testing app |
| Checkout | Checkout and accounts configuration, branding, extra content | Rollouts checkout and accounts Experiment; apps on the information, shipping and payment steps need Shopify Plus |
| Prices and shipping | A higher or lower price, a shipping threshold or rate | Catalog or discount changes in Rollouts, or a price testing app |
For ideas by page type, with real tests behind them, see ecommerce A/B testing and the A/B testing examples.
Can you A/B test the Shopify checkout?
You can A/B test the Shopify checkout only within what your plan allows. Shopify's checkout customization page says standard customizations are on the Basic plan or higher and advanced customization only on Shopify Plus. Its pricing page lists checkout customization as "Limited" on Basic, Grow and Advanced and "Full" on Plus.
- On every plan: eligible apps can customize the thank you and order status pages, and branding such as the logo, colors and font can be set.
- Shopify Plus only: eligible apps on the information, shipping and payment steps, and access to the Checkout Branding API.
- Rollouts: checkout and accounts configurations can be put in an Experiment, which reports checkout conversion rate. Headless and custom storefront checkouts are not supported for these changes.
On plans below Plus, most of the testing that moves orders happens before the checkout: product pages, delivery information and the cart. The checkout optimization guide covers what to fix there.
Shopify price testing: what to know first
Shopify price testing shows different prices to different shoppers and measures which earns more. It is possible through catalog changes in a Rollouts Experiment or through price testing apps, but Shopify itself warns that while a catalog change is in an Experiment, your online store can show different prices or products than Google or other sales channels.
- Ads, feeds and apps can disagree. Shopify notes that product ads, listings, feeds and third-party apps might not display the same offer as the store during the test.
- Rollbacks are not instant everywhere. When a rollout is paused or an Event ends, Shopify POS and product feeds return to current prices on their next sync and can show the tested prices until then.
- Judge on profit, not conversion rate. A lower price can win on orders and lose on margin. Use revenue or gross profit per visitor as the metric.
- Shoppers compare. A returning visitor who sees a different price on another device may lose trust. Keep the test short, and check the rules on pricing where you sell.
The price testing guide covers how to plan and read a price test on any platform.
How to A/B test a Shopify theme
To A/B test a Shopify theme, create a Rollouts Experiment and add a theme change: either edit a copy of your current theme or replace it with another theme. Shopify then splits eligible visitors between the live theme and the new one. A theme replacement always applies to all markets, and vintage themes cannot be used.
A whole theme test answers "is the new theme better overall?", not "which change made the difference". If the new theme changes ten things and wins, you will not know which of them helped. When you need that answer, test the biggest changes one at a time on your busiest templates.
Landing page A/B testing on Shopify
Landing page A/B testing on Shopify usually means testing a page built for one campaign, such as a paid social offer. You can test two versions of the page in an app or testing tool, or send the campaign's traffic to two separate page addresses and compare them, which is split URL testing. The landing page A/B testing guide covers both.
How much traffic does a Shopify A/B test need?
A Shopify A/B test needs enough visitors on the tested page for each version to reach the sample size you calculate from your conversion rate and the smallest lift worth detecting. There is no single minimum: a store with few orders a day can still test a big change, but a small change on a quiet product page may take months.
If the sample size calculator says your test would take too long, test a bigger change, move it to a busier template such as the product page that most shoppers see, measure a nearer step such as reached checkout, or fix the clear problems without a test. Rollouts' 90-day default end date is a ceiling, not a reason to run that long.
A real Shopify A/B test: Add to Cart straight to the cart
A UK homewares store on Shopify found that most shoppers bought one item per order, yet clicking Add to Cart did not take them to the cart. Convertica A/B tested sending shoppers straight to the cart after Add to Cart, on mobile and desktop, for 19 days, until the test reached 98% statistical significance.
On mobile, the conversion rate rose 47.7% and revenue per visitor rose 71.4%. Desktop showed no increase in conversions, so the winner was rolled out to mobile users only. The lesson for Shopify stores: read results by device, because the same change can win on one and do nothing on the other. The full story is in the homewares case study.
+26.9%
Conversions, overall, with the winner rolled out to mobile users, 19-day A/B test
Source: UK homewares store on Shopify, winner rolled out to mobile only, as stated in the original case study
Common Shopify A/B testing mistakes
The most common Shopify A/B testing mistakes are testing without enough traffic, stopping as soon as one version looks ahead, editing the theme mid-test, judging a price test on conversion rate alone, and reading results for all devices together. Each one can turn a real result into noise or a false winner.
- Stopping early. Results swing in the first days. Wait for the planned sample in whole weeks.
- Changing the store mid-test. A theme edit, a new app, a sale or a price change during the test muddies both versions.
- Ignoring devices. Most store tests behave differently on mobile and desktop. Rollouts cannot filter by device, so plan another way to read them if device matters.
- Forgetting the apps. Reviews, upsell and popup apps load on both versions. Check that they work the same on each and do not break the variant.
- Counting the wrong shoppers. Rollouts count only visitors who consent; tools that count everyone will not match it. Compare like with like.
- Testing what should just be fixed. A broken cart drawer or a slow product page needs a fix, not a test.
Great results. easy to work with, and do proper split-testing. Even noticed when I accidentally made a change to a page they were testing, and they stopped the test and started again to make sure I didn't distort results. The most important thing to know: they had a positive result on revenue.
Where to start with Shopify A/B testing
The best place to start Shopify A/B testing is a list of real problems on your store, ranked by the sales at stake, so your traffic goes on tests worth running. Convertica's free CRO audit checks a store page for shoppers and for AI agents and gives you every finding, with why each one matters, which is a ready list of test ideas.
If you would rather have a team plan and run the tests, Convertica quotes A/B testing one by one as a custom proposal: see A/B testing services to ask for an A/B testing proposal.
Free tools to plan and read a Shopify test
Plan the length of your next Shopify A/B test before you launch it, and check the result properly when it ends. Each calculator shows its formula and a worked example.
Check a result
Statistical significance calculator
Whether a finished test result is bigger than chance.
Find test ideas
Shopify CRO audit
Find the problems on your store worth testing first.
A/B testing services
Want your A/B tests run for you?
Convertica runs A/B tests for clients, quoted one by one as a custom proposal. Monthly traffic comes first, because it decides whether a test on your site can reach a clear result, and how long it takes.
The split tests in Convertica's case studies were run by its team for past clients, and each result is published with its metric, timeframe and source.
Short answer
Shopify A/B testing means showing two versions of a store page, theme or offer to random shoppers at the same time and comparing sales. Shopify's Rollouts include Experiments for theme, checkout and discount changes on the Grow plan or higher. Apps and script based testing tools cover more, such as prices, shipping and single elements.
- Shopify's own option is an Experiment in Markets > Rollouts: a 50/50 split by default, available on the Grow plan or higher, for the online store only.
- Rollouts cannot change Liquid templates, run on vintage themes, target a device or segment, or filter results by device.
- Apps in the Shopify App Store and general testing tools cover more: prices, shipping rates, single page elements and, for some, checkout and post-purchase pages.
- Changes to the checkout's information, shipping and payment steps through apps need Shopify Plus.
- Work out the sample size before launch: low traffic makes a Shopify test slow or inconclusive.
Questions about Shopify A/B testing
Does Shopify have built-in A/B testing?
Yes. Shopify's Rollouts, in Markets > Rollouts in the admin, include a rollout type called an Experiment that compares a treatment against a control. Rollouts are on the Basic plan or higher, but Experiments need the Grow plan or higher, and they apply only to your online store.
How do you do Shopify A/B testing?
Find a problem in your store data, write a hypothesis, pick one metric, work out the sample size, then build the variant in Rollouts, an app or a testing tool. Split traffic at random, wait for the planned sample in whole weeks, check significance, then roll out the winner or keep the original.
What are the best A/B testing tools for Shopify?
There is no single best tool: the right one depends on what you want to test. Rollouts Experiments suit whole theme, checkout and discount changes. Apps from the Shopify App Store's A/B testing collection cover prices, shipping and content. General testing tools add a script to your theme. Compare them by what they can test on your plan.
Can you A/B test the checkout on Shopify?
Partly. Rollouts Experiments can compare checkout and accounts configurations, measured by checkout conversion rate. Apps that change the information, shipping and payment steps need Shopify Plus; on Basic, eligible apps can only customize the thank you and order status pages. Most tests on other plans happen on product pages and the cart.
Can you A/B test prices on Shopify?
Yes, with care. Rollouts can put a catalog change, which sets prices, in an Experiment, and several apps describe price testing. Shopify warns that during such a test your online store can show different prices than Google and other channels, and ads, feeds and third-party apps might not show the same offer.
How long should a Shopify A/B test run?
Run a Shopify A/B test until each version reaches the sample size you calculated before launch, rounded up to whole weeks, and never less than one full week. Rollouts set a default end date 90 days after an Experiment starts, which is a ceiling, not a target. Do not stop early because one version looks ahead.
Can you A/B test two Shopify themes against each other?
Yes. When you add theme changes to a Rollouts Experiment, you can edit your current theme or replace it with another theme, so one Experiment can test a whole new theme against the live one. You cannot change Liquid templates in a rollout, and rollouts do not work on vintage themes.
Is Shopify split testing the same as Shopify A/B testing?
Usually, yes. Split testing is another name for A/B testing. Some tools use split URL testing for the case where each version lives at its own address, such as two landing pages, and shoppers are sent to one or the other.
