Aggregation policy
To ensure maximum efficiency when exporting data to your API endpoint, you can use various settings to aggregate exported profiles into larger or smaller batches, group them by identity, and other use cases. This also allows you to tailor data exports to any downstream limitations on your API endpoint (rate limiting, number of identities per API call, etc.).
Use configurable aggregation to dive deep into the settings provided by Destination SDK or use best effort aggregation to tell Destination SDK to batch the API calls as best as it can.
When building a real-time (streaming) destination with Destination SDK, you can configure how the exported profiles should be combined in the resulting exports. This behavior is determined by the aggregation policy settings.
To understand where this component fits into an integration created with Destination SDK, see the diagram in the configuration options documentation or see the guide on how to use Destination SDK to configure a streaming destination.
You can configure the aggregation policy settings via the /authoring/destinations
endpoint. See the following API reference pages for detailed API call examples where you can configure the components shown in this page.
This article describes all the supported aggregation policy settings that you can use for your destination.
After reading through this document, refer to the documentation on using templating and the aggregation key examples to understand how to include the aggregation policy in your message transformation template based on your selected aggregation policy.
Supported integration types supported-integration-types
Refer to the table below for details on which types of integrations support the functionality described on this page.
Best effort aggregation best-effort-aggregation
Best effort aggregation works best for destinations which prefer fewer profiles per request and would rather take in more requests with less data than fewer requests with more data.
The example configuration below shows a best effort aggregation configuration. For an example of configurable aggregation, see the configurable aggregation section. The parameters applicable to best effort aggregation are documented in the table below.
"aggregation":{
"aggregationType":"BEST_EFFORT",
"bestEffortAggregation":{
"maxUsersPerRequest":10,
"splitUserById":false
}
}
aggregationType
Indicates the type of aggregation policy that your destination should use. Supported aggregation types:
BEST_EFFORT
CONFIGURABLE_AGGREGATION
bestEffortAggregation.maxUsersPerRequest
This value indicates the maximum number of profiles that your endpoint should receive in a single HTTP call. Note that this is a best effort aggregation. For example, if you specify the value 100, Platform might send any number of profiles smaller than 100 on a call.
If your server does not accept multiple users per request, set this value to
1
.bestEffortAggregation.splitUserById
true
if your server only accepts one identity per call, for a given identity namespace.Configurable aggregation configurable-aggregation
Configurable aggregation works best if you’d rather take in large batches, with thousands of profiles on the same call. This option also allows you to aggregate the exported profiles based on complex aggregation rules.
The example configuration below shows a configurable aggregation configuration. For an example of best effort aggregation, see the best effort aggregation section. The parameters applicable to configurable aggregation are documented in the table below.
"aggregation":{
"aggregationType":"CONFIGURABLE_AGGREGATION",
"configurableAggregation":{
"splitUserById":true,
"maxBatchAgeInSecs":2400,
"maxNumEventsInBatch":5000,
"aggregationKey":{
"includeSegmentId":true,
"includeSegmentStatus":true,
"includeIdentity":true,
"oneIdentityPerGroup":true,
"groups":[
{
"namespaces":[
"IDFA",
"GAID"
]
},
{
"namespaces":[
"EMAIL"
]
}
]
}
}
}
aggregationType
Indicates the type of aggregation policy that your destination should use. Supported aggregation types:
BEST_EFFORT
CONFIGURABLE_AGGREGATION
configurableAggregation.splitUserById
true
if your server only accepts one identity per call, for a given identity namespace.configurableAggregation.maxBatchAgeInSecs
Used in conjuction with maxNumEventsInBatch
, this parameter determines how long Experience Platform should wait until sending an API call to your endpoint.
- Minimum value (seconds): 1800
- Maximum value (seconds): 3600
For example, if you use the maximum value for both parameters, Experience Platform will wait either 3600 seconds OR until there are 10000 qualified profiles before making the API call, whichever happens first.
configurableAggregation.maxNumEventsInBatch
Used in conjunction with maxBatchAgeInSecs
, this parameter determines how many qualified profiles should be aggregated in an API call.
- Minimum value: 1000
- Maximum value: 10000
For example, if you use the maximum value for both parameters, Experience Platform will wait either 3600 seconds OR until there are 10000 qualified profiles before making the API call, whichever happens first.
configurableAggregation.aggregationKey
configurableAggregation.aggregationKey.includeSegmentId
true
if you want to group profiles exported to your destination by audience ID.configurableAggregation.aggregationKey.includeSegmentStatus
includeSegmentId
to true
, if you want to group profiles exported to your destination by audience ID and audience status.configurableAggregation.aggregationKey.includeIdentity
true
if you want to group profiles exported to your destination by identity namespace.configurableAggregation.aggregationKey.oneIdentityPerGroup
true
if you want the exported profiles to be aggregated into groups based on a single identity (GAID, IDFA, phone numbers, email, etc.).configurableAggregation.aggregationKey.groups
Next steps next-steps
After reading this article, you should have a better understanding of how you can configure aggregation policies for your destination.
To learn more about the other destination components, see the following articles: