Setting up an Extensions A/B Test

Compare two checkout extensions head-to-head, or test an extension against showing nothing

Date: May 7, 2026

Table of Contents

• Step 1: Create the test
• Step 2: Fill in the details
• Step 3: Set rules (optional)
• Step 4: Allocate traffic
• Step 5: Connect to Shopify and launch
• Managing your test

An A/B test compares two experiences side by side to see which performs better. Instead of guessing, you split real customer traffic between versions and let the data tell you which drives more conversions, revenue, or whatever metric matters most. PDQ runs the test automatically — set it up, launch it, and check the results.

An Extensions A/B Test compares two checkout extensions against each other. For example: testing whether a Checkout Upsell with free shipping messaging performs better than one with a discount badge. One extension is the control, the other is the test variant.

You can also test nothing vs. an extension — one group sees no extension at all, the other sees your chosen extension. Useful for measuring the true incremental impact of an extension on your checkout metrics.

Note: Any display rules already configured on each extension still apply during the test. The test doesn't override those settings — it respects them.

Step 1: Create the test

In the left sidebar, go to Optimize > A/B Tests. Click New Test and select Extensions Test from the dropdown.

Step 2: Fill in the details

• Test Name — short and clear (e.g. "Upsell with FST vs Upsell with Discount").
• Hypothesis — what you expect to learn (min. 100 characters). Be specific: "Showing a free shipping threshold message will increase AOV more than a discount badge by motivating customers to add one more item."
• End Test After X Checkouts — optional. Leave blank to run until you stop it manually.
• Goal Metric — Checkout Conversion, Gross Profit, AOV, Shipping Revenue, or Net Revenue.

Click Next.

Step 3: Set rules (optional)

Add conditions to limit which shoppers enter the test (cart value, product type, location, etc.). Leave empty to run for everyone. Click Next.

Step 4: Allocate traffic

Two ways to run the test:

• Extension vs. Extension — pick a Control extension and a Test extension. Set the traffic split (typically 50/50).
• Nothing vs. Extension — set Control to NULL (no extension), Test to your chosen extension. Measures the true impact of the extension vs showing nothing.

Click Create draft test.

Step 5: Connect to Shopify and launch

Once the draft is created, a Token for Shopify tab appears inside the test. How you use the token depends on test type:

• Nothing vs. Extension — copy the token and add it to the desired Shopify checkout block (Settings > Checkout > select your checkout > Add block > paste the token). PDQ serves the extension to the Test group and nothing to Control.
• Extension vs. Extension — paste the same test token into both extensions' Shopify checkout blocks. PDQ uses the token to determine which extension to serve to each shopper.

Once Shopify is connected, find your test in the Drafts tab in PDQ, click ⋯, and select Start Test to go live.

Managing your test

• Editing a draft — Find it in the Drafts tab, click ⋯, select Edit.
• Launching — Click ⋯ on the draft, select Start Test. Goes live immediately.
• Monitoring — A Running tab appears between Drafts and Concluded once at least one test is active. Performance data lives in Optimize > Analytics Dashboard, available the day after launch.
• Ending — In the Running tab, click ⋯, select End Test. View final results in Optimize > Analytics Dashboard, or click ⋯ in Concluded and pick Get Analytics.

Happy testing! Reach out to support@prettydamnquick.com if you need help.