November Updates

If you are running an A/B test to select the best template or subject line, you will now have access to a detailed analysis under the Reports -> A/B Test Results tab of the campaign studio. Depending upon the metric you are optimizing for (e.g. clicks, impressions or a custom goal), Blueshift will show which variation had the best performance with respect to the control along with the confidence level of the results (i.e. whether the results are random). Blueshift will also select the winner if the results are statistically significant (i.e. repeatable).

With the new feature, marketers like yourself can compare performance on multiple goal metrics any time, unlike some competitor products where only one goal is allowed each time, and needs to be configured before the test can be set live. You can also change the control variation at any time. This is useful for long running campaigns where the control variation from original A/B test might have been archived and the winning variant is selected as the benchmark/control for a future test. Compared to us, our competitors’ products can only support a single control variation.

In order to make our ‘delivered’ metric at par with the industry standard, we have updated how we compute the ‘delivered’ metric for push messages. Instead of relying on an acknowledgement from the SDK (i.e. the user’s phone), we are now inferring a ‘delivered’ event if we don’t receive a ‘bounce’ from APNS or FCM for a specific ‘send’. This change means that the ‘delivered’ count will approximate equal Sends minus Bounces.

With the change you will also notice higher delivered counts for your push channel. However because this updated definition will be applied for both historical and new data, your reports should still show the correct trend. Please note this change only applies to ‘delivered’ events for push notification and will not impact any other channel or metric unless the metric is a compound metric using “delivered” in the formula.