You are here:
A/B Testing for B2C Commerce FAQ
Frequently asked questions for A/B testing for B2C Commerce.
Q: How is A/B testing sorting rule inheritance related to category sorting rule inheritance?
A: They are directly related: always configure your A/B tests to take advantage of that inheritance. When configuring A/B tests, only select the top-level categories and rely on inheritance. Don’t select all subcategories in the category tree (explicitly assigning a sorting rule to each category).
We don't recommend A/B testing your site search sorting rule because you can't get a clear participation trigger. The search-show pipeline is also called when you enter a category. So, customers are brought into a search test with either search or a category page view.
Q: Is it true that you must add <varyby="price_promotion"> to each
template that uses iscache for A/B testing to work properly?
For example, the SiteGenesis product-related templates already include this, but the slot content templates don't. To use a slot content as an A/B test experience, do I need to add this parameter to the slot content-related templates?
A: Caching is a complex topic, and there is no general rule for all templates. If you have a
template that shows static content, cache that template for a reasonably long period, such as
60 minutes. If you have a template that is specific to each storefront customer, such as the
cart contents, never cache that template. If you have a template that is personalized by
price, promotion, search results order, or a custom A/B test experience, cache that template
using <varyby="price_promotion">.
If you have an unpersonalized home page, for example, cache it for 60 minutes. The A/B tests still work for the site. B2C Commerce places storefront customers in A/B test segments as expected despite this cached page, whether they view it right away or never see it.
If your search uses a sorting rule, cache your search results page with
<varyby="price_promotion"> to account for the possibility of
personalized sorting rules. Even if you have no A/B tests or no A/B tests with sorting rule
experiences, do this. There can be campaigns that personalize the sorting rule for a search
because of customer group, source code, or coupon qualifiers.
Using <varyby="price_promotion"> to cache pages isn't really about A/B
tests, but about personalization. A/B tests are one way to personalize. Slot caching is a
different way. Slot content personalization depends only on the customer groups and A/B test
segments that apply for a storefront customer, and not the promotions or price books.
You cache slot content for 60 minutes, for example, even if that content only appears to a
specific customer group or a specific A/B test segment. Salesforce B2C Commerce only renders
that content specifically for that group or segment, and everyone in that group or segment
sees the same content. For example, only a specific customer group or A/B test segment sees
the slot content, and that content includes the storefront customer's name. It can't be cached
for 60 minutes because everyone sees the same name. There are no personalized slot content
templates in SiteGenesis, which is why <varyby="price_promotion">
doesn't appear there.
Q: Why are there discrepancies between A/B Test metrics observed in B2C Commercevs. external analytics tools?
A: Each Analytics tool has its own, usually proprietary, way of tracking the storefront. The logic of external analytics tool is not disclosed to us, so we can’t help with comparing metrics. However, there are some common causes that can explain the discrepancies between analytics tools.
If you have multiple A/B tests running concurrently in B2C, the shopper enters the first test that they qualify for (usually when they land on the homepage or category page). Once assigned, they’re omitted from any other A/B tests. This omission from further B2C A/B tests can cause Site Optimizers to see more visits to their other test pages in external tools. However, only a subset of those shoppers actually receives the test, and Salesforce Commerce Cloud (SFCC) tracks them.
If the participation trigger is set as Per Session or Per Customer, the shopper is reconsidered for an A/B test each time they return to the site.
If you're using plug-ins, you can experience issues with your analytic’s data. For example, you can receive an artificial inflation of Facebook-referring traffic and an increase in pageviews. Some plug-ins redirect when a user interacts with the plug-in, even when the plug-in only appears on the page. The redirect goes to a third site and returns the user to the storefront site. The redirect doesn't pass the referrer, so the traffic sources report show that the visit originated from the third site, overwriting the correct campaign or traffic source.
These plug-ins can be affected:
- Activity Feed
- Recommendations
- Like Button
For the SiteGenesis or Mobile web Storefront applications, the Like button is supposed to be an inactive display element until a user clicks it. However, it actually makes a call to Facebook that issues another call-back to the referenced product detail page. When a customer views a product, the Like button causes a second visit to the product detail page, which B2C Commerce counts as another A/B test visitor. Because this is a consistent factor across all segments, it does not change the ratio for the test results.
Finally, bot traffic can also cause skewed results in A/B testing compared to other analytics tools.
Q: Are any A/B testing parameters enforced by quotas?
A: Quotas enforce one A/B testing limitation: Maximum of five test segments within a test, including the control group segment, so four user-defined segments total.
Q: You can create or import an unlimited number of active A/B tests, yet only a limited number of them can be truly active. How does B2C Commerce select the active tests?
A: The Business Manager selects the active tests by:
- Comparing their end dates and selecting the tests that end first
- If they have the same end dates, comparing their start dates and selecting the ones that started the most recently
This determination is dynamic. If a test is enabled or newly created, it can immediately take precedence over one of the others. B2C Commerce allows a maximum of 12 A/B test segments for all active A/B tests. A single A/B test can have up to four segments plus the control group.
Q: Is Active Data tagging a pre-requisite to using A/B testing?
A: No. Active data isn't a requirement, but it's recommended. For example, active data can be used for sorting rules.
Q: A/B testing setup is done on staging and then replicated to production. Do I also need to reindex?
A: No. Reindexing is not necessary.
Q: Is it true that the more tests you run, the more diluted your participant pool gets?
A: Running tests one after the other has no effect on the participant pool. Running tests simultaneously results in the tests sharing the possible participants, because each customer can participate in only one test at a time. The more tests run on the site, the fewer participants in each test. Fewer participants mean a smaller sample size, which often requires tests to run longer. However, there can be only three simultaneous A/B tests in B2C Commerce.
Q: What experiences does B2C Commerce support?
A: B2C Commerce supports experiences with promotions, content slots, and sorting rules. In addition, we support script customization where you can define other, custom experiences.
Q: Do you recommend running one test for a campaign with multiple experiences per segment, instead of running two separate tests? For example, one test with a content slot and promotion or two tests, one with a content slot and the other has the promotion.
A: The answer depends on what you want to test. To test independently the effect of slots versus the effect of sorting rules, run separate tests. To test the combined effect of a strategy that includes both slots and sorting rules, run a single test. If you run a single test, you can't tell if the results are because of the slots, the sorting rules, or a combination of both. Run individual tests to better understand this level of detail.
Q: Do parallel running tests influence each other?
A: Yes and no. Yes, if experience overlap, because users don't participate in other tests, which can skew the results. No, if experiences don't overlap. However, if you measure against the same metric, for example, conversion rate, it's difficult to argue that either experience caused an increase. Also, it takes longer until the tests are conclusive because there are more test segments into which the traffic is distributed. In general, we recommend that you run one test at a time, to eliminate all doubt of other influences.
Q: If I have an A/B test for slots on the homepage, how do I determine the percentage to allocate for segments versus the control group?
A: The reasons to select one percentage over another depends on the reason for the test and its configuration. For example, a merchant wants to test content slots on the homepage, comparing conservative content with something risky or provocative. The merchant chooses a low allocation for the group seeing the risky content. Alternatively, to test two versions of the same slot with minor differences, such as color or font, a merchant can select equal percentage (50/50 or 33/33/33) allocations. Using a lower allocation results in a smaller sample size of participants in that segment, which on average requires you to run the test for a longer period. Optionally, run the risky content with a 50/50 allocation for a shorter test. However, with a smaller percentage allocation, pull the test faster if it generates a negative reaction, with fewer users having seen the risky content.
Q: What events cause an email notification to be sent, and how frequently are they sent?
A: B2C Commerce sends email notifications when:
- The A/B test ends.
- The results of the test become statistically significant.
- The A/B test is paused.
- The A/B test is resumed.
