How do I set up an A/B test?

  1. Turn on A/B testingOn the Pro plan, turn on the test in the A/B test card below the redisplay settings on the edit page.
  2. Add version BSet an image for B (required), alt text, and, if needed, a link URL and how the link opens.
  3. Choose the split ratioChoose an A:B ratio of 50:50, 70:30, or 90:10.
  4. Save and deliverThe position, display conditions, and redisplay settings are shared by A and B. The same customer (the same browser) always sees the same version.

What does each metric mean?

Impressions
The number of times the popup was actually shown.
Clicks
The number of times the link was clicked. After a click, the popup is also treated as closed, but the click is not counted in closes.
Closes
The number of times the popup was closed.
CTR
Clicks as a percentage of impressions.

You can see impressions and CTR on the list page, and every metric in the performance card on the edit page. The numbers are totals since tracking began; there is no daily breakdown. Viewing tracking data is available on Basic and above. Tracking continues on the Free plan, so when you move to Basic or above, you can see the data collected so far. Deleting a popup also deletes its tracking data, and this cannot be undone. During an A/B test, the performance card shows the CTR of A and B and which version is leading.

Things to keep in mind when judging results

  • Don't draw conclusions while impressions are still low
  • The "leading" label is a simple CTR comparison; statistical significance is not tested
  • Look beyond CTR and also check results on the destination page

Does a reset affect the test?

Resetting the hidden state also clears each visitor's A/B assignment. The same browser may then be assigned the other version, so avoid resetting in the middle of a test.