A/B Testing for B2C Commerce Known Issues

We've compiled a list of known issues and troubleshooting to help you configure your A/B tests in B2C Commerce.

"Visitor" Metric Incremented Twice Upon Logout

Sometimes B2C Commerce counts a visitor metric twice. For example, an A/B test has Everyone as the assigned customer group. If a registered customer logs out after placing an order, and then places another order as a guest, B2C Commerce counts the visitor metric twice. B2C Commerce counts this customer twice even though it's the same customer using the same PC in the same session. When a customer performs a guest checkout only, B2C Commerce counts the customer once, no matter how many orders the customer makes. B2C Commerce counts them only once because when a registered customer logs out, B2C Commerce increments the visits and visitors metrics as if a new, anonymous customer has come to the site. B2C Commerce considers this anonymous customer another visitor and evaluates them again for A/B test participation. If the customer qualifies for an A/B test, B2C Commerce generates relevant data as part of the customer's participation.

This can cause a problem with the metrics because B2C Commerce counts the registered customer twice and a guest checkout once. When collecting A/B testing metrics, B2C Commerce doesn’t know how many customers are registered or how many users can result in a guest checkout.

Download of A/B Test Results Displays Security Warning in IE8

If you download A/B tests into a .csv file for the first time using an IE8 browser, a security message appears. The message informs you that the download has been blocked. After you confirm the message, the A/B test list page appears and the file isn't downloaded. If you try a second time, the download begins. If you leave the A/B test module and return, the security warning reappears.

This error only occurs with Microsoft® IE8.

A/B Test Limitations Enforced by Quotas

Quotas enforce the following limitations:

• B2C Commerce allows a flexible distribution of test segments across a maximum of 12 A/B tests.
• Maximum of five test segments within a test, including the control group segment, so four user-defined segments total.

Anonymous Customer Participates in A/B Test for Registered Customers

An anonymous customer can participate in an A/B test configured for registered customers only in the following scenario. A registered customer participates in an A/B test that has expiration set to Never and is configured for registered customers. The registered customer leaves or logs out of the site and the personalization cookie still exists in the respective system. Now, an anonymous customer can participate in the test.

This is to help produce a consistent storefront experience for the A/B test at the expense of potential false positives. When the cookie was set in a previous visit for someone using the same browser, we assume it was also set for the current browser user.

Participation in A/B Test for Registered Customers Can Happen Rarely

B2C Commerce uses a round-robin assignment of participants to tests. For example, an A/B test is configured for Everyone. A second A/B test configured for Registered Customers only has participants if those customers are remembered and the round-robin mechanism has chosen that test.

For example:

A/B test	Triggered	Customer group
A	Immediately	Everyone
B	Immediately	Registered

Potential scenarios:

• A customer starts a new session. The customer typically starts as Anonymous. so A/B test B doesn't apply. B2C Commerce selects A/B test A every time because it loops through all allowed A/B tests until one matches, and B never matches.
• A customer logs in later. Participation is always in progress. Therefore, A/B test B isn't assigned.
• A customer logs in with the Remember Me flag set to true. They return later to the store and are remembered. A/B test B is next in the round-robin sequence. Only then is the customer selected for participation in A/B test B.

We recommend that you run one test at a time for most participants. Running one test at a time can shorten the time to statistical significance; and no overlap, to prevent skewing the results.

Cannot Add a Campaign or A/B Test Schedule to a Content Slot

Users occasionally get confused about why they can’t configure a campaign within the Business Manager Content Slot module. This is by design because users who have permission to access the Content Slot module don’t always have permission to access the Campaigns module. You can only create campaigns and A/B tests if you have permission to access the Business Manager Campaigns module.

A/B Test Experiences Not Applied to Page Designer Category Pages

A/B test experiences are not applied to category pages rendered by Page Designer. Categories must use the standard controller-based rendering path for A/B test experiences to take effect. To work around this limitation, remove the Page Designer page assignment from the affected category so it uses the standard rendering path.