Configure a B2C Commerce A/B Test to Compare Product Templates

Determine whether the location of product reviews impacts revenue. Product reviews are important and you want the reviews to have a more prominent placement on product detail pages. You ask a developer for help.

1. The developer copies the product detail template pt_productdetails.isml from Studio. Then, the developer modifies the copy to include the new or the old layout by using the scripting API. The scripting API helps determine the active test group for the customer (ABTestMgr.isParticipant). The new layout differs significantly from the original layout, but the developer only wants to test it on a few selected products.
   • Changes the product detail page pipeline (Product-Show). The pipeline calls product.isml, which calls pt_productdetails.isml. This pipeline dynamically selects the copied product detail page template for products of brand MyBrand and the old one.
   • Uses remote includes for dynamic inclusion of the new or old template for each customer.
2. Clicks App Launcher , selects Merchant Tools, and then goes to Online Marketing | A/B Tests, and configures a test called Product-reviews.
   • Enable the test and enter an optional description that is for the developer's own usage.
   • Include an email address for notifications when there are statistically significant results for the key metric.
   • Select Adds to Baskets as the key metric, because of expecting the product review placement to result in more adds to the basket.
   • Set the test to Never expire for the life of the test. (A slot on the product detail page always appears regardless of the test group that the customer is assigned to.
   • Set the test to run for 1 week.
   • Make the test available to the Everyone group, so any customer that views the product detail page can be a participant.
   • Create two segments, each allocated at 50%, but with no test segment experiences.
   • Click New Test Segment and set the segment to enable Custom Template Scripting.

The developer stages the test to production in concert with the cartridge changes containing the templates and pipeline. After 5 days, the developer checks the results. The old template seems to result in more adds to baskets than the new template, not the results expected. The developer extends the test on production for an extra week, and after five more days concludes that the old template results in more Add to Baskets.

Note Salesforce B2C Commerce doesn't compute statistical significance when the control group outperforms the test segments.

On production, the developer changes the end date of the test to the previous day. The changed templates and pipeline remain in place. However, all traffic now sees the old template because customers are no longer considered participants in the test group that shows the new template. The developer also coordinates a rollback of the template and pipeline changes to cache the old template as it was before.

When the merchant reviews the results, the more prominent location of the product reviews pushed the Add to Cart button too far down the page. The merchant asks the developer to create a test that accounts for the locations of both of these elements.

Note Your customized implementation can use different templates from this example.