Amazon Kinesis source
Adobe Experience Platform provides native connectivity for cloud providers like AWS, Google Cloud Platform, and Azure. You can bring your data from these systems into Platform.
Cloud storage sources can bring your own data into Platform without the need to download, format, or upload. Ingested data can be formatted as XDM JSON, XDM Parquet, or delimited. Every step of the process is integrated into the Sources workflow. Platform allows you to bring in data from Amazon Kinesis in real time.
Prerequisites
The following section provides further information on prerequisite set up required before you can create a Kinesis source connection.
Set up access policy
A Kinesis stream requires the following permissions to create a source connection:
GetShardIterator
GetRecords
DescribeStream
ListStreams
These permissions are arranged through the Kinesis console and are checked by Platform once you enter your credentials and select your data stream.
The example below displays the minimum access rights required to create a Kinesis source connection.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"kinesis:GetShardIterator",
"kinesis:GetRecords",
"kinesis:DescribeStream",
"kinesis:ListStreams"
],
"Resource": [
"arn:aws:kinesis:us-east-2:901341027596:stream/*"
]
}
]
}
kinesis:GetShardIterator
kinesis:GetRecords
kinesis:DescribeStream
kinesis:ListStreams
For more information on controlling access for Kinesis data streams, see the following Kinesis document.
Configure iterator type
Kinesis supports the following iterator types to allow you to specify the order of how your data is read:
AT_SEQUENCE_NUMBER
AFTER_SEQUENCE_NUMBER
AT_TIMESTAMP
TRIM_HORIZON
LATEST
A Kinesis UI source currently only supports TRIM_HORIZON
, while the API supports both TRIM_HORIZON
and LATEST
as modes to get data. The default iterator value that Platform uses for the Kinesis source is TRIM_HORIZON
.
For more information on iterator types, see the following Kinesis document.
Connect Amazon Kinesis to Platform
The documentation below provides information on how to connect Amazon Kinesis to Platform using APIs or the user interface: