Product Spotlight: Understand the True Impact of Your Campaigns with Holdout Testing

Blueshift

| July 29, 2021

Product Spotlight: Understand the True Impact of Your Campaigns with Holdout Testing

As a marketer, you spend countless hours creating the ideal campaigns and journeys for your customers. You keep a constant eye on engagement, conversion, and revenue generated and continue to test campaigns to optimize performance. But with so many campaigns and competing messages running simultaneously, how do you distinguish the impact of a campaign from all other marketing touchpoints? We want to think that our marketing activity is influencing engagement and buying decisions, but how do you know whether customers would have converted regardless of whether they saw your messages? A big question on the minds of marketers and business leaders looking to optimize spend: what’s the incremental lift driven by my marketing?

Holdout tests provide the definitive answer about the real impact driven by campaigns. Now, with the release of campaign holdout reporting, you can run holdout tests and understand the incremental impact of campaigns within the Blueshift platform.

What Is Holdout Testing?

Holdout testing is a foundational campaign testing best practice that’s been around for decades. It provides a scientific way to measure the incremental increase in purchases, visits, or other business goals driven by your marketing efforts by comparing campaign exposure to a baseline of doing nothing.

Holdout testing is simple. You randomly divide your customers into two groups: a group that receives messages from the campaign (the test group) and a holdout group (also referred to as a control group) who is intentionally withheld from receiving your messages. By randomly assigning customers to groups, the profile of the test group matches the profile of the control group, and the differences between each group’s response will be meaningful. After the campaign wraps up, you measure how the test group performed compared to the baseline control group to see the incremental lift generated by your campaign.

Holdout Report for: Weekly Promotions screenshot

How To Do Holdout Testing Within Blueshift

Blueshift makes holdout testing and reporting easy and supports holdout testing across all types of campaigns and journeys, not simply single messages. You can set up two types of holdout groups:

Global Control Group: A percentage of users will be held out from ALL campaigns and syndications. This allows you to analyze the overall impact of your marketing strategy.
Campaign Control Group: A group of users will be held out from one specific campaign.

Holdout groups can be applied to entire campaigns and customer journeys. You can run them on lifecycle and triggered campaigns in addition to standard campaigns. Plus, you can analyze holdout test results using numerous goal metrics.

Setting up holdout groups for your campaigns can be done in a matter of clicks. Then, you can access the results in holdout reports. Here, you can slice and dice the results by selecting the desired goal metric and time period for your analysis. Goals can be standard goal events, such as visits or purchases, or custom goal events important to your business. You’ll then see the results for the selected metric for each of the control and test groups along with the statistical significance level to understand if the results are reliable.

Testing is an important tool for marketers, allowing you to understand what works, identify new opportunities for growth, and make better business decisions. By being able to simply apply holdout testing across your campaigns, you can understand the real impact drivers and double down on revenue-driving activities.

Ready to get started? Learn more about holdout testing and how to set up holdout testing for a campaign here, then reach out to your customer success manager to enable holdout analysis for your Blueshift account.