Data ingestion notifications

The process of ingesting data into Adobe Experience Platform is comprised of multiple steps. Once you identify data files that need to be ingested into Platform, the ingestion process begins and each step occurs consecutively until the data is either successfully ingested or fails. The ingestion process can be initiated using the Adobe Experience Platform Batch Ingestion API or using the Experience Platform user interface.

Data loaded into Platform must go through multiple steps in order to reach its destination, the Data Lake or the Real-Time Customer Profile data store. Each step involves processing the data, validating the data, and then storing the data before passing it on to the next step. Depending on the amount of data being ingested, this can become a time consuming process and there is always a chance of the process failing due to validation, semantics, or processing errors. In the event of a failure, the data issues need to fixed and then the entire ingestion process must be restarted using the corrected data files.

To assist in monitoring the ingestion process, Experience Platform makes it possible to subscribe to a set of events that are published by each step of the process, notifying you to the status of the ingested data and any possible failures.

Register a webhook for data ingestion notifications

In order to receive data ingestion notifications, you must use Adobe Developer Console to register a webhook to your Experience Platform integration.

Follow the tutorial on subscribing to Adobe I/O Event notifications for detailed steps on how to accomplish this.

IMPORTANT
During the subscription process, ensure that you select Platform notifications as the event provider, and select the Data ingestion notification event subscription when prompted.

Receive data ingestion notifications

Once you have successfully registered your webhook and new data has been ingested, you can start receiving event notifications. These events can be viewed using the webhook itself, or by selecting the Debug Tracing tab in your project’s event registration overview in Adobe Developer Console.

The following JSON is an example of a notification payload that would be sent to your webhook in the case of a failed batch ingestion event:

{
  "event_id": "93a5b11a-b0e6-4b29-ad82-81b1499cb4f2",
  "event": {
    "xdm:ingestionId": "01EGK8H8HF9JGFKNDCABHGA24G",
    "xdm:customerIngestionId": "01EGK8H8HF9JGFKNDCABHGA24G",
    "xdm:imsOrg": "{ORG_ID}",
    "xdm:completed": 1598374341560,
    "xdm:datasetId": "5e55b556c2ae4418a8446037",
    "xdm:eventCode": "ing_load_failure",
    "xdm:sandboxName": "prod",
    "sentTime": "1598374341595",
    "processStartTime": 1598374342614,
    "transformedTime": 1598374342621,
    "header": {
      "_adobeio": {
        "imsOrgId": "{ORG_ID}",
        "providerMetadata": "aep_observability_catalog_events",
        "eventCode": "platform_event"
      }
    }
  }
}
Property
Description
event_id
A unique, system-generated ID for the notification.
event
An object which contains the details of the event that triggered the notification.
event.xdm:datasetId
The ID of the dataset to which the ingestion event applies.
event.xdm:eventCode
A status code indicating the type of event that was triggered for the dataset. See the appendix for specific values and their definitions.

To view the full schema for event notifications, refer to the public GitHub repository.

Next steps

Once you have registered Platform notifications to your project, you can view received events from the Project overview. Refer to the guide on tracing Adobe I/O Events for detailed instructions on how to trace your events.

Appendix

The following section contains additional information about interpreting data ingestion notification payloads.

Available status notification events event-codes

The following table lists the available data ingestion status notifications that you can subscribe to.

Event code
Platform Service
Status
Event description
ing_load_success
Data Ingestion
success
A batch was successful ingested into a dataset within the Data Lake.
ing_load_failure
Data Ingestion
failure
A batch failed to be ingested into a dataset within the Data Lake.
ps_load_success
Real-Time Customer Profile
success
A batch was successful ingested into the Profile data store.
ps_load_failure
Real-Time Customer Profile
failure
A batch failed to be ingested into the Profile data store.
ig_load_success
Identity Service
success
Data was successfully loaded into the identity graph.
ig_load_failure
Identity Service
failure
Data failed to be loaded into the identity graph.
NOTE
There is only one event topic provided for all data ingestion notifications. In order to distinguish between different statuses, the event code can be used.
recommendation-more-help
2ee14710-6ba4-4feb-9f79-0aad73102a9a