Behind-the-Scenes: Real-Time Segments with Blueshift

Behind the scenes, Blueshift builds a continually changing graph of users and items in the catalog. The edges in the graph come from user’s behavior (or implied behavior), we call this the “Interaction graph”. The “interaction graph” is further enriched by machine-learning models that add predicted edges and scores to the graph (if you liked item X, you may also like item Y) and also expand user attributes through 3rd party data sources (example: given the firstname “John”, with reasonable confidence we can infer gender is male).

The segment service can run complex queries against the “interaction graph” like: “Female users that viewed ‘Handbags’ over $500 in last 90 days, with lifetime purchases over $1,000 and not using mobile apps recently and having a high churn probability” and return those users within a few seconds to a couple of minutes.

360-degree user profiles

For every user on your site/mobile app, Blueshift creates a user profile that tracks anonymous user behavior and merges it with their logged-in activities across devices. These rich user profiles combine CRM data, aggregate lifetime statistics, catalog-related activity, predictive attributes, campaign & channel activity and website / mobile app activity. The unified user profiles form the basis for segmentation. A segment query matches these 360 degree user profiles against the segment definition to identify the target set of users.

Multiple data stores (no one store to rule them all)
The segmentation engine is powered by several different data stores. A given user action or attribute that hits the event API is replicated across these data stores including: timeseries stores for events, relational database for metadata, in-memory stores for aggregated data & counters, key-value stores for user lookups, as well as a reverse index to search across any event or user attributes quickly. The segmentation engine is tuned for fast retrieval of complex segment definitions compared to a general purpose SQL-style database where joins across tables could take hours to return results. The segmentation engine leverages data across all these data stores to pull the right set of target users that match the segment definition.

Real-time event processing

Website & mobile apps send data to Blueshift’s event APIs via SDKs and tag managers. The events are received by API end-points and written to in-memory queues. The event queues are processed continuously in-order, and updates are made across multiple data stores (as described above). The user profiles and event attributes are updated continuously with respect to the incoming event stream. Campaigns pull the audience data just-in-time for messaging, which result in segments that are continuously updated and always fresh. Marketers do not have to worry about out of date segment definitions and avoid the “list pull hell” with data-warehouse style segmentation.

Dynamic attribute binding

The segmentation engine further simplifies onboarding user or event attributes by removing the need to model (or declare) attribute types ahead of time. The segmentation engine dynamically assesses the type of each new attribute based on sample usage in real-time. For instance, an attribute called “loyalty_points” with a value of “450”, would be interpreted as a number (and show related numeric operators for segmentation), while an attribute like “membership_level” with a value of “gold” would be dynamically interpreted as a string (and show related string comparison operators for segmentation), or an attribute like “redemption_at” with a value like “2016-09-23” will be interpreted as a timestamp (and show relative time operators).

Several Blueshift customers have thousands of CRM & event attributes, and are able to use these attributes without any data modeling or declaring their data upfront, saving them numerous days of implementing data schemas in SQL-based implementations.

The combination of 360-degree user profiles, real-time event processing, multiple specialized data stores and dynamic attribute binding, empowers marketers to create always fresh and continuously updated segments.