Monitor dataflows for destinations in the UI
Use the various destinations in the Experience Platform catalog to activate your data from Platform to countless external partners. Platform makes the process of tracking the flow of data to your destinations easier by providing transparency with dataflows.
The monitoring dashboard provides you with a visual representation of the journey of a dataflow, including the destination the data is activated to. This tutorial provides instructions on how you can either monitor dataflows directly in the destinations workspace or use the monitoring dashboard to monitor dataflows for your destinations using the Experience Platform user interface.
Getting started getting-started
This guide requires a working understanding of the following components of Adobe Experience Platform:
- Dataflows: Dataflows are a representation of data jobs that move data across Platform. Dataflows are configured across different services, helping move data from source connectors to target datasets, to Identity and Profile, and to Destinations.
- Dataflow runs: Dataflow runs are the recurring scheduled jobs based on the frequency configuration of selected dataflows.
- Destinations: Destinations are pre-built integrations with commonly used applications that allow for the seamless activation of data from Platform for cross-channel marketing campaigns, email campaigns, targeted advertising, and many other use cases.
- Sandboxes: Experience Platform provides virtual sandboxes which partition a single Platform instance into separate virtual environments to help develop and evolve digital experience applications.
Monitor dataflows in the Destinations workspace monitor-dataflows-in-the-destinations-workspace
In the Destinations workspace within the Platform UI, navigate to the Browse tab and select the name of a destination that you want to view.
A list of existing dataflows appears. On this page is a list of viewable dataflows, including information about their destination, username, number of dataflows, and status.
See the following table for more information on statuses:
Enabled status indicates that a dataflow is active and is exporting data according to the schedule it was provided.Disabled status indicates that a dataflow is inactive and is not exporting any data.Processing status indicates that a dataflow is not yet active. This status is often encountered immediately after a new dataflow is created.Error status indicates that the activation process of a dataflow has been disrupted.Dataflow runs for streaming destinations dataflow-runs-for-streaming-destinations
For streaming destinations, the Dataflow runs tab provides an hourly update for metric data on your dataflow runs. The most prominent statistics labelled are for identities.
Identities represent the different facets of a profile. For example, if a profile contains both a phone number and an email address, that profile has two identities.
A list of individual runs and their particular metrics is displayed, along with the following totals for identities:
- Identities activated: The total number of profile identities successfully activated to the selected destination. This metric includes identities that are created, updated, and removed from exported segments.
- Identities excluded: The total number of profile identities that are skipped for activation based on missing attributes and consent violation.
- Identities failed: The total number of profile identities that are not activated to the destination due to errors.
Each individual dataflow run shows the following details:
-
Dataflow run start: The time that the dataflow run started at. For streaming dataflow runs, Experience Platform captures metrics based on the start of the dataflow run, in the form of hourly metrics. For streaming dataflow runs, if a dataflow run started, for example, at 10:30PM, the metric shows the start time as 10:00 PM in the UI.
-
Processing time: The amount of time that it took for the dataflow run to process.
- For completed runs, the processing time metric always shows one hour.
- For dataflow runs which are still in a processing state, the window to capture all the metrics stays open for more than an hour, to process all metrics that correspond to the dataflow run. For example, a dataflow run that started at 9:30 AM might stay in a processing state for one hour and thirty minutes to capture and process all the metrics. Then, once the processing window closes and the status of the dataflow run updates to completed, the displayed processing time is changed to one hour.
-
Profiles received: The total number of profiles received in the dataflow.
-
Identities activated: The total number of profile identities that were successfully activated to the selected destination as part of the dataflow run. This metric includes identities that are created, updated, and removed from exported segments.
-
Identities excluded: The total number of profile identities that are excluded from activation based on missing attributes and consent violation.
-
Identities failed The total number of profile identities that are not activated to the destination due to errors.
-
Activation rate: The percentage of received identities that have either been successfully activated or skipped. The following formula demonstrates how this value is calculated:
-
Status: Represents the state the dataflow is in: either Completed or Processing. Completed means that all the identities for the corresponding dataflow run were exported within the one-hour period. Processing means that the dataflow run has not yet finished.
To view the details of a particular dataflow run, select the run’s start time from the list.
The details page for a dataflow run contains additional information such as the number of profiles received, the number of identities activated, the number of identities failed, and the number of identities excluded.
The details page also displays a list of identities that failed and identities that were excluded. Information for both the failed and excluded identities is displayed, including the error code, identity count, and description. By default, the list displays the failed identities. To show skipped identities, select the Identities excluded toggle.
Dataflow runs for batch destinations dataflow-runs-for-batch-destinations
For batch destinations, the Dataflow runs tab provides metric data on your dataflow runs. A list of individual runs and their particular metrics is displayed, along with the following totals for identities:
- Identities activated: The total number of profile identities successfully activated to the selected destination. This metric includes identities that are created, updated, and removed from exported segments.
- Identities excluded: The count of individual profile identities excluded from activation for the selected destination, based on missing attributes and consent violation.
Each individual dataflow run shows the following details:
- Dataflow run start: The time that the dataflow run started at.
- Segment: The name of the segment associated with each dataflow run.
- Processing time: The amount of time it took for the dataflow run to be processed.
- Profiles received: The total number of profiles received in the dataflow. This value is updated every 60 minutes.
- Identities activated: The total number of profile identities that were successfully activated to the selected destination as part of the dataflow run. This metric includes identities that are created, updated, and removed from exported segments.
- Identities excluded: The total number of profile identities that are excluded from activation based on missing attributes and consent violation.
- Status: Represents the state the dataflow is in. This can be one of three states: Success, Failed, and Processing. Success means that the dataflow is active and is exporting data according to its provided schedule. Failed means that the activation of data has been suspended due to errors. Processing means that the dataflow is not yet active and is generally encountered when a new dataflow is created.
To view details of a specific dataflow run, select the run’s start time from the list.
The details page for a dataflow, in addition to the details shown on the dataflows list, displays more specific information about the dataflow:
- Size of data: The size of the dataflow that is being exported.
- Total files: The total number of files exported in the dataflow.
- Last updated: The time the dataflow run was last updated.
The details page also displays a list of identities that failed and identities that were excluded. Information for both the failed and excluded identities is displayed, including the error code and description. By default, the list displays the failed identities. To show excluded identities, select the Identities excluded toggle.
Monitoring Destinations dashboard monitoring-destinations-dashboard
- Destinations monitoring functionality is currently supported for all destinations in Experience Platform except the Adobe Target and Custom personalization destinations.
- For the Amazon Kinesis, Azure Event Hubs, and HTTP API destinations, the metrics related to identities excluded, failed, and activated are estimated. Higher volumes of activation data lead to higher accuracy of the metrics.
To access the Monitoring dashboard, select Monitoring (
Use the Destinations dashboard to get an overall idea of the health of your activation flows. Start by getting insights on an aggregated level for all batch and streaming destinations and then drill down into detailed views for dataflows, dataflow runs, and activated segments for an in-depth look at your activation data. The screens in the Monitoring dashboard provide actionable insights through metrics and error descriptions to help you troubleshoot any problems that might arise in your activation scenarios.
At the center of the dashboard is the Activation panel, which contains metrics and graphs that display data on the activation rate of the data which is exported to streaming destinations, as well as on the failed batch dataflow runs to batch destinations.
By default, the data displayed contains the activation information from the last 24 hours. Select Last 24 hours to adjust the time frame of records displayed. Available options include Last 24 hours, Last 7 days, and Last 30 days. Alternatively, you can select the dates on the calendar pop-up window that appears. Once you have selected dates, select Apply to adjust the time frame of the information shown.
Use the arrow icon (
- Streaming activation rate: Represents the percentage of received identities that have either been successfully activated or skipped. The formula used to calculate this percentage is described further above on this page, in the Dataflow runs for streaming destinations section.
- Batch failed dataflow runs: Represents the number of failed dataflow runs in the selected time interval.
The Activation graph is displayed by default and you can disable it to expand the list of destinations below. Select the Metrics and graphs toggle to disable the graphs.
The Activation panel displays a list of destinations that contain at least one existing account. This list also includes information on the profiles received, identities activated, identities failed, identities excluded, activation rate, total failed dataflows, and the last updated date for these destinations. Not all metrics are available for all destination types. The table below outlines which metrics are available per destination type, streaming or batch.
You can also filter your list of destinations to only display the selected category of destinations. Select the My destinations dropdown, and select the destination category that you want to filter to.
Additionally, you can enter a destination into the search bar to isolate to a single destination. If you want to see the destination’s dataflows you can select the filter
If you want to view all existing dataflows across all destinations, select Dataflows.
A list of dataflows appears, sorted by the last dataflow run. You can see additional details for a specific dataflow by locating the destination you want to monitor, selecting the filter
Once you select a dataflow for further inspection, the dataflow details page contains a toggle which allows you to see the activated data in the dataflow, broken down by dataflow runs or segments.
Dataflow runs view dataflow-runs-view
When Dataflow runs is selected, you can see a list of dataflow runs for the selected dataflow and further information about each run.
Use the Show failures only toggle to display only the failed runs for a dataflow.
Segment-level view segment-level-view
When Segments is selected, you see a list of the segments which were activated to the selected dataflow, within the selected time range. This screen includes segment-level information about the identities activated, identities excluded, as well as the status and the time of the last dataflow run. By reviewing the metrics for identities excluded and activated, you can verify if a segment has been successfully activated or not.
For example, you are activating a segment called “Loyalty Members in California” to an Amazon S3 destination “Loyalty Members California December”. Let’s assume that there are 100 profiles in the selected segment but only 80 out of 100 profiles contain Loyalty ID attributes and you have defined the export mapping rules as loyalty.id is required. In this case, on a segment level, you will see 80 identities activated, and 20 identities excluded.
- The segment-level view is currently only available for batch destinations.
- Segment level metrics are currently recorded for successful dataflow runs only. They are not recorded for failed dataflow runs and excluded records.
In the segment-level view, the metrics are aggregated across multiple dataflow runs within the selected time range. If there are multiple dataflow runs, you can drill down from the segment level to see the breakdown for each dataflow run, filtered by the selected segment.
Use the filter button
Dataflow runs page dataflow-runs-page
The dataflow runs page displays information on your dataflow runs, including the dataflow run start time, processing time, profiles received, identities activated, identities excluded, identities failed, activation rate, and status.
When you drill down into the dataflow runs page from the segment-level view, you have the option of filtering the dataflow runs by the following options:
- Dataflow runs with failed identities: For the selected segment, this option lists all the dataflow runs that failed for activation. To inspect why identities in a certain dataflow run failed, see the dataflow run details page for that dataflow run.
- Dataflow runs with skipped identities: For the selected segment, this option lists all the dataflow runs where some of the identities were not fully activated and some profiles were skipped. To inspect why identities in a certain dataflow run were skipped, see the dataflow run details page for that dataflow run.
- Dataflow runs with activated identities: For the selected segment, this option lists all the dataflow runs that have identities which were successfully activated.
To see more details about a specific dataflow run, select the filter
Dataflow run details page dataflow-run-details-page
The dataflow run details page, in addition to the details shown on the dataflow runs list, displays more specific information about the dataflow:
- Dataflow run ID: The ID of the dataflow.
- IMS org ID: The organization the dataflow belongs to.
- Last updated: The time the dataflow run was last updated.
The details page also has a toggle to switch between dataflow run errors and segments. This option is only available for dataflow runs in batch destinations.
The dataflow run errors view displays a list of identities that failed and identities that were excluded. Information for both the failed and excluded identities is displayed, including the error code, identity count, and description. By default, the list displays the failed identities. To show skipped identities, select the Identities excluded toggle.
When Segments is selected, you see a list of the segments which were activated in the selected dataflow run. This screen includes segment-level information about the identities activated, identities excluded, as well as the status and the time of the last dataflow run.
Next steps next-steps
By following this guide, you now know how to monitor dataflows for both batch and streaming destinations, including all the relevant information such as processing time, activation rate, and status. To learn more about dataflows in Platform, read the dataflows overview. To learn more about destinations, read the destinations overview.