Success Insights: Testing for Improved Metrics
Goals of Testing
Improve the overall user experience in ways that can be verified by response data
- Create a learning culture to foster continuous improvement and know what resonates with your customers
- Increase conversions through subscriber engagement with emails (opens, clicks)
- Reduce Attrition (Unsubscribers, complaints)
- Increase subscriber satisfaction
Level of Effort
|Subject Lines: Copy||Low||Low||Medium|
|Subject Lines: Symbols||Low||Low||Low|
|Subject Lines: Personalization||Medium||Medium||Medium|
|Personalization: Dynamic Content||High||High||High|
|Day of Week||Low||Low||Medium|
|Call to Actions (Buttons, Text)||Medium||Medium||Medium|
Testing Best Practices
- Plan for regular testing: develop a calendar but adjust as test results yield new theories
- Understand the lag between getting results and applying them to messages in production
- Determine the statisically significant sample size
- Isolate one variable; hold all others constant. (The variable can be overall treatment) Once the winner is chosen, isolate specific components to test and optimize.
Review and Refine:
- Ensure differences are statistically significant
- To perserve institutional memory, capture and share test results regularly
- Keep all information in a central file
- Create a report with background, results, key discoveries, recommendations and screen captures
- What other theories spring up as a result of the test?
- Should we retest?
- Where factors cannot be held constant (seasonality), will we need to retest to verify results?
- Can we be certain that the lift was from the test factors: does correlation = causation?
- Can we state the conclusion of the test with confidence?
In conclusion - testing is the fastest and easiest way to give your customers the power to shape the communication you send to them. By creating a plan and being intentional about constant analysis - you can ensure you are sending your subscribers the right message and at the right time.