DocumentationExperience PlatformIntelligent Services Guide

Discover insights with Customer AI

Last update: Thu Jul 20 2023 00:00:00 GMT+0000 (Coordinated Universal Time)

CREATED FOR:

  • Developer
  • User
  • Admin
  • Leader

Customer AI, as part of Intelligent Services provides marketers with the power to leverage Adobe Sensei to anticipate what your customers next action is going to be. Customer AI is used to generate custom propensity scores such as churn and conversion for individual profiles at-scale. This is accomplished without having to transform the business needs to a machine learning problem, picking an algorithm, training, or deployment.

This document serves as a guide for interacting with service instance insights in the Intelligent Services Customer AI user interface.

Getting started

In order to utilize insights for Customer AI, you need to have a service instance with a successful run status available. To create a new service instance visit Configuring a Customer AI instance. If you recently created a service instance and it is still training and scoring, please allow 24 hours for it to finish running.

Service instance overview

In the Adobe Experience Platform UI, select Services in the left navigation. The Services browser appears and displays available Intelligent Services. In the container for Customer AI, select Open.

Accessing your instance

The Customer AI service page appears. This page lists service instances of Customer AI and displays information about them, including the name of the instance, propensity type, how often the instance is run, and the status of the last update.

NOTE
Only service instances that have completed successful scoring runs have insights.

Create instance

Select a service instance name to begin.

Create instance

Next, the insights page for that service instance appears with the option to select Latest scores or Performance summary. The default tab Latest scores provides visualizations of your data. The visualizations and what you can do with the data are explained in more detail throughout this guide.

The Performance summary tab shows the actual churn or conversion rates for each propensity bucket. To learn more, see the section on performance summary metrics.

setup page

Service instance details

There are two ways to view service instance details: from the dashboard or within the service instance.

Service instance dashboard

To view an overview of the service instance details within the dashboard, select a service instance container, avoiding the hyperlink that is attached to the name. This opens a right rail that provides additional details. The controls contain the following:

  • Edit: Selecting Edit allows you to modify an existing service instance. You can edit the name, description, and scoring frequency of the instance.
  • Clone: Selecting Clone copies the currently selected service instance set up. You can then modify the workflow to make minor tweaks and rename it as a new instance.
  • Delete: You can delete a service instance, including any historical runs.
  • Data source: A link to the dataset used by this instance.
  • Run Frequency: How often a scoring run takes place and when.
  • Score definition: A quick overview of the goal you configured for this instance.

NOTE
In the event that a scoring run fails, an error message is provided. The error message is listed under Last run details in the right rail which is only visible to failed runs.

failed run message

Show more insights dropdown

The second way to view additional details for a service instance is located within the insights page. Select Show more in the top-right to populate a drop down. Details are listed such as the score definition, when it was created, the propensity type, and the datasets used. For more information on any of the properties listed, please visit Configuring a Customer AI instance.

show more

Customer AI dataset preview popover

If more than one dataset is used by Customer AI, a hyperlink labeled **Multiple ** followed by the number of datasets in brackets () is provided.

multiple datasets

Selecting the multiple datasets link opens the Customer AI dataset preview popover. Each color in the preview represents a dataset as shown by the color key to the left of the dataset columns. In this example, you can see that only Dataset 1 contains the PROP1 column.

show more

Edit an instance

To edit an instance, select Edit in the top-right navigation.

click the edit button

The edit dialog box appears, allowing you to edit the name, description, status, and scoring frequency of the instance. To confirm your changes and close the dialog, select Save in the bottom-right corner.

edit popover

More actions

The More actions button is located in the top-right navigation next to Edit. Selecting More actions opens a dropdown that allows you to select one of the following operations:

  • Clone: Selecting Clone copies the service instance set up. You can then modify the workflow to make minor tweaks and rename it as a new instance.
  • Delete: Deletes the instance.
  • Access scores: Selecting Access scores opens a dialog providing a link to the downloading scores for Customer AI tutorial, the dialog also provides the dataset id required for making API calls.
  • View run history: A dialog containing a list of all the scoring runs associated with the service instance appears.

more actions

Scoring summary scoring-summary

Scoring summary displays the total number of profiles scored and categorizes them into buckets containing high, medium, and low propensity. The propensity buckets are determined based on score range, low is less than 24, medium is 25 to 74, and high is above 74. Each bucket has a color corresponding to the legend.

NOTE
If it is a conversion propensity score, the high scores show in green and the low scores in red. If you are predicting churn propensity this is flipped, the high scores are in red and the low scores are green. The medium bucket remains yellow regardless of what propensity type you choose.

scoring summary

You can hover over any color on the ring to view additional information, such as a percentage and total number of profiles belonging to a bucket.

Distribution of Scores

The Distribution of Scores card gives you a visual summary of the population based on the score. The colors that you see in the Distribution of Scores card represent the type of propensity score generated. Hovering over any of the scoring distributions provides the exact count belonging to that distribution.

distribution of scores

Influential factors

For each score bucket, a card is generated that shows the top 10 influential factors for that bucket. The influential factors give you additional details on why your customers belong to various score buckets.

Influential factors

Influential factor drilldowns

Hovering over any of the top influential factors further breaks down the data. You are provided an overview as to why certain profiles belong to a propensity bucket. Depending on the factor, you may be given number, categorical, or boolean values. The example below displays categorical values by region.

drilldown screenshot

Additionally, using drilldowns, you are able to compare a distribution factor if it occurs in two or more propensity buckets and create more specific segments with these values. The following example illustrates the first use case:

You can see that profiles with low propensity to convert are less likely to have made a recent visit to the adobe.com webpages. The “Days since last webVisit” factor has only 8% coverage compared to 26% in medium propensity profiles. Using these numbers, you can compare the distribution within each bucket for the factor. This information can be used to infer that the recency in webvisit is not as influential in the low propensity bucket, as it is in medium propensity bucket.

Create a segment

Selecting the Create Segment button in any of the buckets for low, medium, and high propensity redirects you to the segment builder.

NOTE
The Create Segment button is only available if Real-Time Customer Profile is enabled for the dataset. For more information on how to enable Real-Time Customer Profile, visit the Real-Time Customer Profile overview.

Click create segment

Create a segment

The segment builder is used to define a segment. When selecting Create Segment from the Insights page, Customer AI automatically adds the selected buckets information to the segment. To finish creating your segment, simply fill in the Name and Description containers located in the right rail of the segment builder user interface. After you have given the segment a name and description, select Save in the top-right.

NOTE
Since the propensity scores are written to the individual profile, they are available in the Segment builder like any other profile attributes. When you navigate to the segment builder to create new segments you can see all the various propensity scores under your namespace Customer AI.

Segment fill in

To view your new segment in the Platform UI, select Segments in the left navigation. The Browse page appears and displays all available segments.

All your Segments

Historical performance historical-performance

The Performance summary tab shows the actual churn or conversion rates, separated into each of the propensity buckets scored by Customer AI.

Performance summary tab

Initially only expected rates (dotted lines) are displayed. Expected rates are displayed when a scoring run has not occurred and data is not yet available. However, once an outcome window has passed, the expected rate is replaced with an actual rate (solid line).

Hovering over the lines displays the date and actual/expected rate for that day in that bucket.

Bucket example

You can filter the timeframe for the expected and actual rates being displayed. Select the calendar icon icon then select a new date range. The results in each of the buckets are updated to display within the new date range.

Date selector

Individual scoring run rates

The bottom half of the Performance summary tab displays the results for each individual scoring run. Select the dropdown date in the top-right to display results for a different scoring run.

Depending on if you are predicting churn or conversion, the Distribution of Scores graph displays the distribution of profiles churned/converted and not churned/not converted in each increment.

individual scoring

Model evaluation model-evaluation

In addition to tracking the predicted and actual outcomes over time on the Historical Performance tab, marketers have even more transparency over model quality with the Model Evaluation tab. You can use the Lift and Gains charts to determine the differences in using a predictive model vs random targeting. Additionally, you are able to determine how many positive outcomes would be captured at each score cutoff. This is useful for segmentation and for aligning return on investment with marketing actions.

Lift chart

lift chart

The lift chart measures the improvement of using a predictive model instead of random targeting.

High quality model indicators include:

  • High lift values in the first few deciles. This means that the model is good at identifying the users with the highest propensity to take the action of interest.
  • Descending lift values. This means that Customers with higher scores are more likely to take the action of interest than people with lower scores.

Gains chart

gains chart

The cumulative gains chart measures the percentage of positive outcomes captured by targeting scores above a certain threshold. After sorting the customers by propensity score from high to low, the population is split into deciles - 10 equally sized groups. A perfect model would capture all of the positive outcomes in the highest score deciles. A baseline random targeting method captures positive outcomes proportionally to the size of the group - targeting 30% of the users would capture 30% of the outcomes.

High quality model indicators include:

  • The cumulative gains approach 100% quickly.
  • The cumulative gains curve for the model is closer to the upper left corner of the chart.
  • The cumulative gains chart can be used to determine the score cutoffs for segmentation and targeting. For example, if the model captures 70% of the positive outcomes in the first 2 score deciles, targeting users with PercentileScore > 80 is expected to capture approximately 70% of the positive outcomes.

AUC (Area under the curve)

The AUC reflects the strength of the relationship between the ranking by score and the occurrence of the predicted goal. An AUC of 0.5 means the model is no better than a random guess. An AUC of 1 means the model can perfectly predict who will take the relevant action.

Next steps

This document outlined the insights provided by a Customer AI service instance. You can now continue to the tutorial on downloading scores in Customer AI or browse the other Adobe Intelligent Services guides that are offered.

Additional resources

The following video outlines how to use Customer AI to see the output of the models and influential factors.

https://video.tv.adobe.com/v/32666?learn=on&quality=12

Transcript
Hi, I’m Hithala Trinidade, senior product manager. Let’s look at what kind of insights Customer AI generates, and how those insights can be used. Luma, an athletic apparel retailer, wants to increase sales of watches, and they’ve already created a customer AI instance to generate propensity scores. We will now deep dive into customer AI reports to see how Luma can use the insights and scores in the marketing. Intervention services are available from the Services link in the left navigation. Clicking on the Customer AI service card will show you all the instances created in the account. When a service instance is new, the hyperlink is disabled, as the insights have not been generated. You can click elsewhere on the row to open the right trail, and see additional details of the service such as name, description, scoring frequency, the prediction goal, and eligible population. After training and scoring have successfully done, the predictive scores are written back to Experience platform on each profile. We could look up any individual yield and customer profile, to see their propensity to purchase Luma watch, and we can also see the predictive scores in aggregate. Once you click on the service instance, the same details are available at the top, and are easily accessible via Show Mode. In the middle panel on the left, the scoring summary shows the total number of profiles scored, and categorized in low, medium, and high propensity buckets. The propensity buckets are data mined based on score range alone. That is, low is less than 24, medium is 25 to 74, and high is 75 and above. The score distribution on the right shows a visual summary of the population based on the score. The colors that you see here are based on the type of propensity score chosen in the configuration. Conversion are shown. If it’s a conversion propensity score, high scores are good, and we show them in green. However, if it’s a churn propensity score, high scores are bad, and we will show them in red. In the lower panel, we see the high, medium, and low distribution of scores, across Luma’s customer base. For each score bucket, we show the top 10 influential factors for that bucket. The influential factors give you additional details on why people belong to various score buckets. These score buckets and influential factors could be used for data analysis, customer segmentation, proselytization, experimentation, promotions, customer relations access management, product improvement. With these scores you can power customer segmentation, and targeting. Let’s create a segment from this high propensity users. The logic is already applied with the segment definition, and you can make any additional updates if you want. You can add additional conditions to the segment, like profile attributes, events, et cetera. We will give the segment a name, and description, and save it.
Now that we have this high value audience, we can activate it with both Adobe Application and non-Adobe applications. Since the propensity scores are returned to the individual profile, they are available in the Segment Builder, like any other profile attributes. When you navigate to the segment builder to create new segments. You will see all the various propensity scores under the customer’s name space, under customer AI.
At the individual profile level, Customer AI adds the percentile. This value provides information regarding the performance of a profile, related to other similarly-scored profiles. For example, a profile with a percentile rank of 99 for churn indicates that it is at a higher risk of churning compared to 99% of other profiles that we scored. Use percentile for segment creation when you want to target top X person of your population for marketing campaigns. Probability. This attribute is the true probability that a customer will achieve the predicted goal, within the defined time frame. When comparing outputs across different goals, it recommended that you consider probability over percentile or score. Probability should always be used when determining the average probability across the eligible population, as the probability will tend to be on the lower side for events that do not occur frequently. The score. Score is the relative likelihood a customer will achieve the predicted goal with the defined time frame. The value is not to be treated as a probability percentage, but rather, the likelihood of an individual compared to the overall population. Influential factors. These are predicted reasons why a profile is likely to convert, churn, or take the specified action. Factors are comprised of the following attributes. The code, which is the profile or the behavioral attribute which positively influences profile’s predicted score. The value of the profile, or behavioral attribute. The importance, which indicates the weight of the profile, or behavioral attribute, has on the predicted score. It ranges from low to medium to high. The output of customer AI can be used for personalization, targeting, et cetera, using Adobe Experience Cloud Applications, and services, and third party tools. Customers can create segments, leveraging the propensity scores within the Segment Builder, and these audiences will be available for using Adobe Advertising Cloud, Adobe Audience Manager, Adobe Campaign, and Adobe Target. Customer AI scores can also be uploaded in Adobe Analytics for exploratory data analysis. All real-time customer data platform customers will be able to create segments, leveraging the propensity scores, and activate them via destinations. Of course, the propensity scores can be uploaded into third party tools as well. So, you should now know how you can use customer AI propensity scores, and Insights to convert and retain customers. -
recommendation-more-help
8959a20a-a58f-4057-9f82-870706c576e9