Collection version test
A collection version test splits the shoppers who open one collection between two of its saved versions. Each shopper sees the version they are assigned: its product order, content cards, product duplicates and image settings. Depict reports how each group clicked, added to cart and ordered, names a winner once there is enough data, and lets you promote the winner to the live version.Before you start
- Plan. The store’s plan must include Merchandising at Custom (including the earlier Custom plan). See Your plan.
- Switched on for the store. Depict enables collection version tests per store, once the theme integration below is in place. Until then the Compare with A/B test button is disabled and reads Not yet available; ask your Depict contact.
- Theme integration. The Depict app embed serves the test and reorders the collection grid in the shopper’s browser, so the theme must carry Depict’s grid markup and render the collection from Depict’s metaobjects. Your theme developer follows Theme integration for market versions and A/B tests.
- Two saved versions of the collection. With one version the button reads Create a second version to compare. Versions in a running test are locked until the test ends.
Starting a test
1
Open the collection
Open the collection in the editor from Collections, click the
version badge to open the versions menu, and click Compare with A/B
test.
2
Pick the versions to compare
Variant A is pre-selected as the main version and is usually the version
live today; Variant B is the one you want to try. Each preview shows the
version’s first 48 products with its content cards in place.
3
Choose the markets
All markets splits every shopper who opens the collection, in every
market Depict serves. Specific market group splits only shoppers in
that group; shoppers in other markets keep seeing the collection as
today, including their own market version. While a test runs on a
market, no other test can start on that market of this collection.
4
Set the traffic split
Shown as Variant A / Variant B, from 90/10 to 10/90 in steps of 10.
50/50 gives the fastest answer. Weighting toward A, say 80/20, keeps most
shoppers on the version you trust, but the test needs to run longer.
5
Set the duration
7, 14, 21 or 30 days, 14 by default. Seven days covers one full week of
shopping patterns; longer tests detect smaller differences. Depict reads
the collection’s traffic from the last four weeks and estimates how long
the test needs.
6
Name it and start
The name is optional and shows on the test’s page and in the versions
menu badge. Click Start test. The split starts at once.
How the split works
Every shopper who opens the collection is assigned to a group once, in their browser, and keeps that group for the whole test on that device. The same person on another device or browser can land in the other group. Shoppers in markets outside the test’s scope are not assigned and see the collection as today. Only shoppers who accepted cookies are measured; shoppers who did not still see their variant. Each version’s product order is read when the test starts. Refresh on the test page re-reads both versions, for instance after products joined or left the collection. Depict reorders the products on each page the theme renders. On a collection that spans several pages, which page a product is on follows the live version’s order; its position within the page follows the variant.Reading the results
Every number is a count of people and every rate is per visitor, so a rate never exceeds 100% and an uneven split does not tilt the outcome.
Each row shows the rate per variant, the likely lift, and Chance B is
better: the probability that Variant B’s true rate is above Variant A’s,
given what was observed.
The verdict rests on the click rate. A variant is called better once
the chance reaches 95%, after at least 7 days and 1,000 visitors per
variant. Orders and revenue per visitor act as guardrails once each variant
has 25 buyers: if the variant with more clicks is losing on orders or
revenue at a 90% chance, the headline says so, for example Variant B gets
more clicks, but fewer of its visitors order, and advises against
promoting on clicks alone. Below 25 buyers the page says there are not
enough orders yet to say anything about orders or revenue.
Until the minimums are met the page says Still collecting data and
shows the date a verdict is expected, based on how fast the slower variant
gathers visitors. If that date falls after the planned end, the page offers
to extend the test to it, up to 30 days after the start. A collection too
quiet to reach a verdict in 30 days is flagged; test a busier collection or
two versions that differ more.
Below the verdict the page shows the results by device (mobile, tablet,
desktop) and by new versus returning shoppers, daily charts, and revenue in
the shop currency with a toggle to the currencies shoppers paid in. A segment
is compared only once both variants hold 100 visitors; below that it shows
counts only.
Can these numbers be trusted? checks that visitors arrived in the
configured proportion and reports shoppers seen in both groups, who are left
out of the numbers. A split that is far off means tracking or assignment is
broken in one variant; the page then says The traffic split looks off
and the results cannot be trusted.
Ending and promoting
- End early (End once the planned end has passed) stops the split. Shoppers see the live version again. A test ended before 7 days and 1,000 visitors per variant has no verdict.
- Promote winner, available once the test has ended, publishes the winning version the same way as pressing Publish on it. Pick the market groups, or All other markets for the main version. A group the winning version does not belong to gets a copy of it, which you can edit separately afterwards. Versions unlock when the test ends.
- Delete, on an ended or promoted test, removes the test and its results page.
Troubleshooting
- The button says “Not yet available”. Collection version tests are switched on per store by Depict.
- The button says “Upgrade to a Custom plan”. The store needs Merchandising at Custom.
- Both groups see the same products in the same order. Check, in this order: the Depict Brand Features app embed is on for the live theme; the theme carries Depict’s grid markup and the theme integration; you are not looking at a filtered or re-sorted view, a market outside the test’s scope, or the Depict portal’s preview.
- No visitors recorded. Check that the Depict app embed is on and the tracking pixel is active.
- The traffic split looks off. One variant loads or tracks differently. Check that both versions render on the storefront, then start a new test.
- No verdict after the planned end. The test did not reach 7 days and 1,000 visitors per variant. Run it again for longer, or on a busier collection.
Merchandising test
A merchandising test runs on a single collection and measures the effect of that collection’s visual merchandising: content cards, product duplicates and per-collection image settings. Shoppers in the test group see the collection as merchandised; shoppers in the control group see it without those elements.Limits and behavior
- Tests run on one collection at a time.
- The traffic split is 50/50 and cannot be changed.
- Group assignment is per browser and persists for the whole test.
- The product order is identical in both groups: Depict writes the order into the Shopify collection itself, so ordering is not part of what this test varies. To test two product orders, run a collection version test.
- Add-to-carts and purchases only count products that are in the collection.
- Merchandising tests are available on plans with Merchandising at Large or Custom (including the earlier Pro and Custom plans), and during the Unlimited Trial. See Your plan.
Before you start
- The store’s plan must include Merchandising at Large or Custom, or the store must be in the Unlimited Trial.
- The collection must contain at least one content card, product duplicate or image setting; without them, the two groups would be identical.
Starting a test
1
Open the collection
Open the collection in the editor from Collections.
2
Start the A/B test setup
Start the A/B test setup from the collection editor.
3
Complete the Todos step
The Todos step lists anything the collection still needs before a
test can run.
4
Configure the test
In Settings, enter a name, a start date, an end date, and optionally
an email address to notify about the test.
5
Launch
In Review, click Launch.
How the split works
Every visitor is randomly assigned to one of the two groups, 50/50. The assignment is stored in the visitor’s browser and persists for the whole test, so a returning shopper stays in the same group on the same device. The same person on a different device or browser can land in the other group.Reading the results
The Insights view shows, per group and per day:
Statistical significance is computed with a two-tailed z-test for each rate
metric, over the whole test window so far. The p-value shows as a dash until
both groups have page views, and it is recomputed as data accumulates, so it
moves; a single early dip below a threshold is not a conclusion. Low-traffic
collections need longer test windows before the p-values are meaningful.
When the test concludes, an AI-written summary states which group won, with
what confidence, and suggests next steps.
Troubleshooting
- The A/B test option asks me to upgrade. Merchandising tests require Merchandising at Large or Custom; they are also available during the Unlimited Trial.
- The setup won’t launch. The collection must contain at least one content card, product duplicate or image setting.
- Both groups look the same. The control group hides content cards, duplicates and image settings; the product order is the same for everyone. Check which group your own browser was assigned to.
- No p-value yet. The p-value appears once both groups have page views.