Shopify A/B Testing: How Rollouts Work and When Your Store Has Enough Traffic
Written by Saif Ur Rehman, Lead CRO Auditor and founder of Cartvane. Updated .
The short answer
Shopify now has A/B testing built in. In your admin it lives under Markets > Rollouts, the test type is called an Experiment, and your store must be on the Grow plan or higher to run one.
Whether a test can tell you anything is a different question, and it's about traffic. At a 2% conversion rate, spotting a 20% relative change (2.0% to 2.4%) takes roughly 20,000 sessions per version. A store with 20,000 sessions a month would need about two months to finish that one test. Below that level, you get more from fixing problems that research already explains and comparing equal before and after periods.
This guide covers how Rollouts Experiments work, the traffic maths with every input shown, the rules that keep a test honest, and what to do instead when the numbers say you can't test yet.
Not enough traffic to test yet? See what an audit finds first. Free scan, no card.
How Shopify Rollouts Experiments work
Everything in this section comes from Shopify's Help Center pages on Rollouts (linked in Sources), read on October 26, 2026. Shopify updates this feature often, so check those pages before you rely on a detail.
Three rollout types
A rollout is a scheduled set of changes. Shopify offers three types:
| Type | What it does |
|---|---|
| Launch | Merges your changes into the store permanently. It's the default type. |
| Event | Publishes changes for a set period, then rolls them back. It needs an end date. |
| Experiment | Shows a treatment to some visitors and a control to others so you can measure the change before a full Launch. |
What an Experiment can change
A rollout can hold one change to your main theme (edit it or replace it), one change to your checkout and accounts configuration, one or more catalog changes (set a catalog to Active or Draft), and one discount change (activate or deactivate discounts you pick).
For most stores the useful one is the theme change: test a new product page layout or a different cart against the current one. Testing a discount is possible too, but read Shopify's caveat first. Analytics for a discount rollout don't include discount revenue or orders, and a discount code activated in an Experiment can be shared with people outside the test group.
Defaults you can change
A new Experiment splits traffic 50% control, 50% treatment and ends 90 days after it starts. You can change both before saving.
There's also a separate traffic allocation setting. If you set allocation to 50% and the split to 50/50, only a quarter of eligible visitors see the treatment. That keeps risk low, but it also makes the test take longer. The traffic maths below assumes 100% allocation.
How visitors are assigned
This part matters for whether results can be trusted, and Shopify handles it well:
- Each visitor is put in a group before the page loads, so the page doesn't flicker between versions.
- The group is saved in an essential cookie, so returning visitors see the same version until the cookie expires or is cleared. A new browser or device can get a different group.
- If you change the split mid-test, only new visitors are affected.
- Visitors get a group even without consent to tracking, but only visitors who consent are counted in results. If many of your shoppers decline tracking (common with EU traffic), your counted sample is smaller than your traffic.
- A visitor who sees both versions in the same browser is dropped from the results.
- Bot traffic isn't counted. Results update a few minutes after each visit.
What you can measure
For a theme Experiment, Shopify shows conversion rate over time, bounce rate, reached checkout rate and add to cart rate. For a checkout and accounts Experiment, it shows checkout conversion rate. You can't customize the metrics, and you can't filter results by device or country.
Shopify's analytics page lists these metrics but doesn't explain how, or whether, it judges a winner statistically. So decide your own sample size and stopping rule before you start (next two sections), and don't treat a line that looks higher on the chart as a result.
One more rule from Shopify: editing the treatment or control while an Experiment is active can affect its results. Finish the change before you start the test.
The traffic maths: can your store finish a test?
Most A/B testing advice skips this step. It's the one that decides whether a test is worth starting.
The inputs
A sample-size calculation needs four numbers:
- Baseline conversion rate: your current rate. Use Shopify's conversion rate for the pages in the test.
- Minimum detectable effect (MDE): the smallest change you care about spotting. A 20% relative MDE on 2% means detecting 2.0% vs 2.4%.
- Significance level: how often you accept a false alarm. The usual choice is 5%.
- Statistical power: how often you want to catch a real change of the MDE's size. The usual choice is 80%.
The numbers below use 5% significance (two-sided) and 80% power, the defaults in Evan Miller's sample size calculator, computed with the same formula the calculator uses. Your results may differ by one or two from rounding. These are arithmetic, not Cartvane results or a promise of any lift.
Sessions needed per version
| Baseline conversion rate | Detect a 10% relative change | Detect a 20% relative change | Detect a 30% relative change |
|---|---|---|---|
| --- | --: | --: | --: |
| 1% | 157,697 | 39,986 | 18,017 |
| 2% | 78,040 | 19,785 | 8,913 |
| 3% | 51,487 | 13,051 | 5,878 |
Double each number for the total across control and treatment.
What that means in weeks
Same inputs, 50/50 split, 100% allocation, every session counted. Weeks are rounded up.
| Test (baseline, change to detect) | 5,000 sessions/month | 20,000 sessions/month | 100,000 sessions/month |
|---|---|---|---|
| --- | --: | --: | --: |
| 2%, 10% change | 136 weeks | 34 weeks | 7 weeks |
| 2%, 20% change | 35 weeks | 9 weeks | 2 weeks (run 3 or 4, see below) |
| 3%, 10% change | 90 weeks | 23 weeks | 5 weeks |
| 3%, 20% change | 23 weeks | 6 weeks | 2 weeks (run 3 or 4) |
| 1%, 20% change | 70 weeks | 18 weeks | 4 weeks |
Shopify's default Experiment runs 90 days, about 13 weeks. At 5,000 sessions a month, none of the tests in this table can finish inside that window. At 20,000 a month, only changes of 20% or more fit, and only on stores converting at 2% or better.
Two things make real tests take longer than this table:
- Consent. Only consenting visitors are counted, so if 30% of visitors decline tracking, you need roughly 40% more traffic (1 ÷ 0.7).
- Repeat visits. Shopify assigns visitors but reports conversion per session. For planning, treating the figures above as sessions per version is a reasonable shortcut; stores with many repeat visits should leave extra room.
A useful cross-check
At a 2% baseline and a 20% MDE, each version needs 19,785 sessions, which comes to about 400 orders in the control alone (19,785 × 2%). That lines up with the stopping rule Peep Laja describes in CXL's article on significance and validity: he doesn't believe tests with fewer than 250 to 400 conversions per variation. If your store won't get a few hundred orders per version in a few weeks, plan for the "not enough traffic" path below.
Rules that keep a Shopify test honest
- Write the hypothesis first. "Because [what you saw], we believe [the change] will [effect] for [shoppers]. We'll know when [metric] moves." If you can't fill in the first blank, the idea isn't ready to test.
- Fix the sample size and end date before you start. Then don't stop early because the chart looks good. Evan Miller's How Not To Run an A/B Test explains why: significance figures assume the sample size was fixed in advance, and checking repeatedly and stopping at the first "significant" moment makes those figures meaningless. CXL's article above shows a test that looked 100% significant after two days and reversed ten days later.
- Run whole weeks, at least three. The same CXL article gives Laja's rule as at least 3 weeks (better 4), and André Morys's rule of thumb as 3 to 4 weeks, so every weekday and weekend and every traffic source is included. That's why the 2-week rows above say to run 3 or 4.
- Pick one decision metric. For a theme test, use conversion rate. Treat add to cart rate and reached checkout rate as supporting signs. A rise in add to cart with flat orders usually means the problem moved down the funnel.
- One change set per Experiment. Shopify allows one theme change per rollout. Keep it to one idea, like a new buy box, so you know what caused the result.
- Don't edit while it runs. Shopify warns this can affect results. Also avoid starting a big ad campaign or sale mid-test, because that changes who your visitors are.
- Don't slice small results. You can't filter Shopify's Experiment results by device or country. Even if you segment elsewhere, the CXL article's rule applies to each segment: you need the full sample size inside each one.
- Write down the result either way. Record the hypothesis, dates, sessions, orders per version and decision. A clean "no difference" result still tells you not to spend more time there.
What to test first
Test changes close to the sale on pages with the most traffic. A product page used by every buyer will finish a test far sooner than a page few people visit.
Good first candidates are things a store audit can flag. Some examples, named after public Cartvane checks:
- Add to cart visible on the first mobile screen: move the buy box up on product pages.
- Free shipping bar is honest: does a bar that matches checkout totals change reached checkout rate?
- Express wallets shown first in checkout: test it with a checkout and accounts Experiment, measured by checkout conversion rate.
Some issues aren't worth testing. If a check finds something broken, like a filter that doesn't work on mobile or a free shipping bar that shows the wrong amount, fix it. A test would only confirm that broken things perform worse.
For scoring which ideas to test first, see how to prioritize Shopify CRO fixes. That guide covers prioritization; this one covers whether and how to run the test.
When you don't have the traffic
Most small Shopify stores can't finish a valid test, and that's fine. A test answers "is B better than A?" but if you already know A is broken, there's no need to ask. Here's what to do instead:
- Fix what research already explains. Hidden shipping costs, forced account creation and slow mobile pages are well documented in public research (see the Shopify CRO audit guide for sources). Fix them rather than test them.
- Watch real sessions. Recordings and heatmaps show where shoppers get stuck, even at low traffic. The comparison of Cartvane vs heatmap and analyzer apps explains how to use them together.
- Compare equal before and after periods, carefully. Ship the change, then compare the same number of whole weeks before and after in Shopify's reports (for example, Conversion rate breakdown). Write down anything else that changed: ads, email sends, sales, seasonality. This won't prove cause and effect, but it catches big drops. Note that Shopify changed how sessions are measured in a rollout on September 21–23, 2026, so a comparison across those dates may shift without shoppers behaving differently. The homepage CRO guide covers that caveat.
- Use a Launch for safety, not proof. A Launch with partial traffic allocation lets you release a change to part of your visitors first and watch for problems before going to 100%. Shopify shows sales, orders, conversion rate and the Conversion rate breakdown for a Launch, but that's monitoring, not a controlled test.
- Test bigger changes when you do test. If you'll only ever detect a 30% change, don't test a button colour. Test a rebuilt product page or a new cart.
If you're weighing whether paid help is worth it at your traffic level, see is a Shopify CRO audit worth it.
What Cartvane checks
This section is about Cartvane's product. Everything above is Shopify's documentation, public research and arithmetic.
Cartvane runs 91 checks across your homepage, collection pages, product pages, cart, checkout, speed and trust, and ranks what it finds by priority. That ranking is the starting point for a testing plan: broken things to fix now, then bigger changes worth an Experiment if you have the traffic.
The free preview gives your health score, your 3 biggest leaks marked on your pages, and the full fix for one of them. No card, no account.
The full report is $19.99, one time, with no subscription. It includes every issue ranked by priority, phone and desktop screenshots, fix steps, how to test each change with the research behind it, and a 90-day plan. If the report finds nothing worth fixing, you get your money back within 14 days. See pricing.
Cartvane doesn't promise a conversion lift, and neither should anyone else before a test or a fix is measured.
Run the free scan · Read a sample report
FAQ
Does Shopify have built-in A/B testing?
Yes. Rollouts include an Experiment type that splits visitors between a control and a treatment. You'll find it under Markets > Rollouts in your admin.
Which Shopify plan do I need for experiments?
Shopify's Help Center says your store must be on the Grow plan or higher to run experiments.
How long should a Shopify A/B test run?
Until it reaches the sample size you calculated before starting, and at least three whole weeks. Shopify's default end date is 90 days after the start, but that's a default, not a sign the test has enough data.
Can I A/B test checkout on Shopify?
Yes. A rollout can include a checkout and accounts configuration change, and an Experiment of that type reports checkout conversion rate.
Does Shopify tell me which version won?
Shopify shows metrics for each version, such as conversion rate, add to cart rate and reached checkout rate. Its analytics documentation doesn't describe a significance test, so set your own sample size and stopping rule.
Is a before and after comparison as good as an A/B test?
No. It can't separate your change from everything else that changed at the same time. It's still useful for catching large drops when you don't have the traffic for a test.
Do I need a third-party testing app?
Not for theme, checkout, catalog or discount tests that Rollouts supports. Third-party tools may offer things Rollouts doesn't, like filtering results by device. Check what you need before paying for one.
About the author
Saif Ur Rehman is the founder and Lead CRO Auditor of Cartvane, based in Lahore, Pakistan. He has spent 5+ years focused on Shopify conversion rate optimization and has worked on 110+ Shopify CRO projects.
Related reading
- Shopify CRO audit: complete guide and checklist
- How to prioritize Shopify CRO fixes
- Cartvane vs heatmap and CRO analyzer apps
- Shopify homepage CRO
- Sample report · Pricing
Sources
- Shopify Help Center, Rollouts
- Shopify Help Center, Types of rollouts and changes
- Shopify Help Center, Rollout analytics
- Shopify Help Center, Managing rollouts
- Shopify Help Center, Behaviour reports
- Evan Miller, Sample Size Calculator
- Evan Miller, How Not To Run an A/B Test (April 18, 2010)
- Peep Laja, CXL, Statistical Significance Does Not Equal Validity (last updated Dec 20, 2022)
Get your free Shopify CRO audit
Paste your store URL and see your biggest conversion leaks in minutes. 91 checks, no card required.