Profile export behavior for different destination types

There are several destination types in Experience Platform, as shown in the diagram below. These destinations have slightly different export patterns with regards to what triggers a destination export and what is included in an export, as described in the sections further below.

Message aggregation in streaming destinations

Before diving into specific information per destination type, it is important to understand the concept of message aggregation for streaming destinations.

Experience Platform destinations export data to API-based integrations as HTTPS calls. Once the destinations service is notified by other upstream services that profiles have been updated as a result of batch ingestion, streaming ingestion, batch segmentation, streaming segmentation or identity graph changes, data is exported and sent to streaming destinations.

Profiles are aggregated into HTTPS messages before being dispatched to destination API endpoints.

Take the Facebook destination with a configurable aggregation policy as an example - data is sent in an aggregated fashion, where the destinations service takes all the incoming data from the profile service upstream and aggregates it by one of the following, before dispatching it to Facebook:

Whichever of the thresholds above is first met triggers an export to Facebook. So, in the Facebook Custom Audiences dashboard, you might see audiences coming in from Experience Platform in 10,000 record increments. You might be seeing 10,000 records every 2-3 minutes because the data gets processed and aggregated faster than the 300 seconds export interval, and gets sent faster, so about every 2-3 minutes until all records have been processed. If there are insufficient records to make up a 10,000 batch, then the current number of records will be sent as is when the time window threshold is met, so you might see smaller batches sent to Facebook as well.

As another example, consider the HTTP API destination, which has a best effort aggregation policy, with maxUsersPerRequest: 10. This means that a maximum of ten profiles will be aggregated before an HTTP call is fired to this destination, but Experience Platform tries to dispatch profiles to the destination as soon as the destinations service receives updated re-evaluation information from an upstream service.

The aggregation policy is configurable, and destination developers can decide how to configure the aggregation policy to best meet the rate limitations of the API endpoints downstream. Read more about aggregation policy in the Destination SDK documentation.

Streaming profile export (enterprise) destinations streaming-profile-destinations

The enterprise destinations in Experience Platform are Amazon Kinesis, Azure Event Hubs, and HTTP API.

Experience Platform optimizes the profile export behavior to your enterprise destination, to only export data to your API endpoint when relevant updates to a profile have occurred following audience qualification or other significant events. Profiles are exported to your destination in the following situations:

In all the cases described above, only the profiles where relevant updates have occurred are exported to your destination. For example, if an audience mapped to the destination flow has a hundred members, and five new profiles qualify for the segment, the export to your destination is incremental and only includes the five new profiles.

Note that all the mapped attributes are exported for a profile, no matter where the changes lie. So, in the example above all the mapped attributes for those five new profiles will be exported even if the attributes themselves haven’t changed.

What determines a data export and what is included in the export

Regarding the data that is exported for a given profile, it is important to understand the two different concepts of what determines a data export to your enterprise destination and which data is included in the export.

For example, consider this dataflow to an HTTP destination where three audiences are selected in the dataflow, and four attributes are mapped to the destination.

A profile export to the destination can be determined by a profile qualifying for or exiting one of the three mapped segments. However, in the data export, in the segmentMembership object, other unmapped audiences might appear, if that particular profile is a member of them and if these share the same merge policy as the audience that triggered the export. If a profile qualifies for the Customer with DeLorean Cars audience but is also a member of the Watched “Back to the Future” movie and Science fiction fans segments, then these other two audiences will also be present in the segmentMembership object of the data export, even though these are not mapped in the dataflow, if these share the same merge policy with the Customer with DeLorean Cars segment.

From a profile attributes point of view, any changes to the four attributes mapped above will determine a destination export and any of the four mapped attributes present on the profile will be present in the data export.

Streaming API-based destinations streaming-api-based-destinations

The profile export behavior for streaming destinations such as Facebook, Trade Desk, and other API-based integrations is very similar to the behavior described above for enterprise destinations.

Examples of streaming destinations are the destinations belonging to the social and advertising categories in the catalog.

Experience Platform optimizes the profile export behavior to your streaming destination, to only export data to streaming API-based destinations when relevant updates to a profile have occurred following audience qualification or other significant events. Profiles are exported to your destination in the following situations:

In all the cases described above, only the profiles where relevant updates have occurred are exported to your destination. For example, if an audience mapped to the destination flow has a hundred members, and five new profiles qualify for the segment, the export to your destination is incremental and only includes the five new profiles.

Note that all the mapped attributes are exported for a profile, no matter where the changes lie. So, in the example above all the mapped attributes for those five new profiles will be exported even if the attributes themselves haven’t changed.

What determines a data export and what is included in the export

Regarding the data that is exported for a given profile, it is important to understand the two different concepts of what determines a data export to your streaming API destination and which data is included in the export.

For example, consider this dataflow to a streaming destination where three audiences are selected in the dataflow.

A profile export to the destination can be determined by a profile qualifying for or exiting one of the three mapped segments. If a profile qualified for the Customer with DeLorean Cars segment, this will trigger an export. The other audiences (City - Dallas and Basic Site Active) might be exported as well in case the profile has that audience present with one of the possible statuses (realized or exited). Unmapped audiences (like Science fiction fans) will not be exported.

From a profile attributes point of view, any changes to the three attributes mapped above will determine a destination export.

Batch (file-based) destinations file-based-destinations

When exporting profiles to file-based destinations in Experience Platform, there are three types of schedules (listed below) and two file export options (full or incremental files) that you can use. All these settings are set on an audience level, even when multiple audiences are mapped to a single destination dataflow.

In any of the export situations above, the exported files include the profiles that qualified for the export, alongside the columns you selected as XDM attributes for export.

Incremental file exports incremental-file-exports

Not all updates on a profile qualify a profile to be included in incremental file exports. For example, if an attribute was added to or removed from a profile, that does not include the profile in the export. Only profiles for which the segmentMembership attribute has changed will be included in exported files. In other words, only if the profile becomes part of the audience or is removed from the audience is it included in incremental file exports.

Similarly, if a new identity (new email address, phone number, ECID, and so on) is added to a profile in the identity graph, that does not represent a reason to include the profile in a new incremental file export.

If a new audience is added to a destination mapping, that does not affect qualifications and exports for another segment. Export schedules are configured individually per audience and files are exported separately for every segment, even if the audiences have been added to the same destination dataflow.

For example, in the export setting illustrated below, where an audience is exporting incremental file updates, note the following circumstances where a profile is included in an incremental file export or not:

What determines a data export and what is included in the export

Based on the information in the section above, the profile export behavior to file-based destinations can be summarized as described below:

Full file exports

The full active population of the audience is exported every day.

Incremental file exports

In the first file export after setting up the activation workflow, the entire population of the audience is exported. In subsequent exports, only the modified profiles are exported.

Next steps next-steps

After reading this document, you now know what to expect to see in profile exports to streaming, enterprise, and file-based destinations.

Next, you can read about how identities are handled in the activation workflow.