Create a YOURSOURCE connection using the Flow Service API
As you go through this template, replace or delete all the paragraphs in italics (starting with this one).
Start by updating the metadata (title and description) at the top of the page. Please ignore all instances of DNL on this page. This is a tag that helps our machine translation processes correctly translate the page into the multiple languages that we support. We will add tags to your documentation after you submit it.
Overview
Provide a short overview of your company, including the value it provides to customers. Include a link to your product documentation homepage for further reading.
Prerequisites
Add information in this section about anything that customers need to be aware of before starting to set up the source in the Adobe Experience Platform user interface. This can be about:
- needing to be added to an allow list
- requirements for email hashing
- any account specifics on your side
- how to obtain an API key to connect to your platform
Gather required credentials
In order to connect YOURSOURCE to Experience Platform, you must provide values for the following connection properties:
For more information on these credentials, see the YOURSOURCE authentication documentation. Please add link to your platform’s authentication documentation here.
Connect YOURSOURCE to Platform using the Flow Service API
The following tutorial walks you through the steps to create a YOURSOURCE source connection and create a dataflow to bring YOURSOURCE data to Platform using the Flow Service API.
Create a base connection base-connection
A base connection retains information between your source and Platform, including your source’s authentication credentials, the current state of the connection, and your unique base connection ID. The base connection ID allows you to explore and navigate files from within your source and identify the specific items that you want to ingest, including information regarding their data types and formats.
To create a base connection ID, make a POST request to the /connections
endpoint while providing your YOURSOURCE authentication credentials as part of the request body.
API format
POST /connections
Request
The following request creates a base connection for YOURSOURCE:
curl -X POST \
'https://platform.adobe.io/data/foundation/flowservice/connections' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}' \
-H 'Content-Type: application/json' \
-d '{
"name": "{YOURSOURCE} base connection",
"description": "{YOURSOURCE} base connection to authenticate to Platform",
"connectionSpec": {
"id": "6360f136-5980-4111-8bdf-15d29eab3b5a",
"version": "1.0"
},
"auth": {
"specName": "OAuth generic-rest-connector",
"params": {
"accessToken": "{ACCESS_TOKEN}",
"refreshToken": "{REFRESH_TOKEN}",
"expirationDate": "{EXPIRATION_DATE}"
}
}
}'
name
description
connectionSpec.id
auth.specName
auth.params.
Response
A successful response returns the newly created base connection, including its unique connection identifier (id
). This ID is required to explore your source’s file structure and contents in the next step.
{
"id": "70383d02-2777-4be7-a309-9dd6eea1b46d",
"etag": "\"d64c8298-add4-4667-9a49-28195b2e2a84\""
}
Explore your source explore
Using the base connection ID you generated in the previous step, you can explore files and directories by performing GET requests.
Use the following calls to find the path of the file you wish to bring into Platform:
API format
GET /connections/{BASE_CONNECTION_ID}/explore?objectType=rest&object={OBJECT}&fileType={FILE_TYPE}&preview={PREVIEW}&sourceParams={SOURCE_PARAMS}
When performing GET requests to explore your source’s file structure and contents, you must include the query parameters that are listed in the table below:
{BASE_CONNECTION_ID}
objectType=rest
rest
.{OBJECT}
fileType=json
json
is the only supported file type.{PREVIEW}
{SOURCE_PARAMS}
{SOURCE_PARAMS}
, you must encode the entire list_id
string in base64. In the example below, "list_id": "10c097ca71"
encoded in base64 equates to eyJsaXN0SWQiOiIxMGMwOTdjYTcxIn0=
.Request
curl -X GET \
'https://platform.adobe.io/data/foundation/flowservice/connections/70383d02-2777-4be7-a309-9dd6eea1b46d/explore?objectType=rest&object=json&fileType=json&preview=true&sourceParams=eyJsaXN0SWQiOiIxMGMwOTdjYTcxIn0=' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}'
Response
A successful response returns the structure of the queried file.
{
"data": [
{
"members": [
{
"id": "cff65fb4c5f5828666ad846443720efd",
"email_address": "roykent@gmail.com",
"unique_email_id": "72c758cbf1",
"full_name": "Roy Kent",
"web_id": 547094062,
"email_type": "html",
"status": "subscribed",
"merge_fields": {
"FNAME": "Roy",
"LNAME": "Kent",
"ADDRESS": {
"addr1": "",
"addr2": "",
"city": "Richmond",
"state": "Virginia",
"zip": "",
"country": "US"
},
"PHONE": "",
"BIRTHDAY": ""
},
"stats": {
"avg_open_rate": 0,
"avg_click_rate": 0
},
"ip_signup": "",
"timestamp_signup": "",
"ip_opt": "103.43.112.97",
"timestamp_opt": "2021-06-01T15:31:36+00:00",
"member_rating": 2,
"last_changed": "2021-06-01T15:31:36+00:00",
"language": "",
"vip": false,
"email_client": "",
"location": {
"latitude": 0,
"longitude": 0,
"gmtoff": 0,
"dstoff": 0,
"country_code": "",
"timezone": ""
},
"source": "Admin Add",
"tags_count": 0,
"tags": [
],
"list_id": "10c097ca71"
}
],
"list_id": "10c097ca71",
"total_items": 2,
"_links": [
{
"rel": "self",
"href": "https://us6.api.mailchimp.com/3.0/lists/10c097ca71/members",
"method": "GET",
"targetSchema": "https://us6.api.mailchimp.com/schema/3.0/Definitions/Lists/Members/CollectionResponse.json",
"schema": "https://us6.api.mailchimp.com/schema/3.0/Paths/Lists/Members/Collection.json"
},
{
"rel": "parent",
"href": "https://us6.api.mailchimp.com/3.0/lists/10c097ca71",
"method": "GET",
"targetSchema": "https://us6.api.mailchimp.com/schema/3.0/Definitions/Lists/Members/Response.json"
},
{
"rel": "create",
"href": "https://us6.api.mailchimp.com/3.0/lists/10c097ca71/members",
"method": "POST",
"targetSchema": "https://us6.api.mailchimp.com/schema/3.0/Definitions/Lists/Members/Response.json",
"schema": "https://us6.api.mailchimp.com/schema/3.0/Definitions/Lists/Members/POST.json"
}
]
}
]
}
Create a source connection source-connection
You can create a source connection by making a POST request to the Flow Service API. A source connection consists of a connection ID, a path to the source data file, and a connection spec ID.
API format
POST /sourceConnections
Request
The following request creates a source connection for YOURSOURCE:
curl -X POST \
'https://platform.adobe.io/data/foundation/flowservice/sourceConnections' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}' \
-H 'Content-Type: application/json' \
-d '{
"name": "{YOURSOURCE} Source Connection",
"description": "{YOURSOURCE} Source Connection",
"baseConnectionId": "70383d02-2777-4be7-a309-9dd6eea1b46d",
"connectionSpec": {
"id": "6360f136-5980-4111-8bdf-15d29eab3b5a",
"version": "1.0"
},
"data": {
"format": "json"
},
"params": {
"server": "us6",
"listId": "10c097ca71"
}
}'
name
description
baseConnectionId
connectionSpec.id
data.format
json
.Response
A successful response returns the unique identifier (id
) of the newly created source connection. This ID is required in a later step to create a dataflow.
{
"id": "246d052c-da4a-494a-937f-a0d17b1c6cf5",
"etag": "\"712a8c08-fda7-41c2-984b-187f823293d8\""
}
Create a target XDM schema target-schema
In order for the source data to be used in Platform, a target schema must be created to structure the source data according to your needs. The target schema is then used to create a Platform dataset in which the source data is contained.
A target XDM schema can be created by performing a POST request to the Schema Registry API.
For detailed steps on how to create a target XDM schema, see the tutorial on creating a schema using the API.
Create a target dataset target-dataset
A target dataset can be created by performing a POST request to the Catalog Service API, providing the ID of the target schema within the payload.
For detailed steps on how to create a target dataset, see the tutorial on creating a dataset using the API.
Create a target connection target-connection
A target connection represents the connection to the destination where the ingested data is to be stored. To create a target connection, you must provide the fixed connection specification ID that corresponds to the Data Lake. This ID is: c604ff05-7f1a-43c0-8e18-33bf874cb11c
.
You now have the unique identifiers a target schema a target dataset and the connection spec ID to the Data Lake. Using these identifiers, you can create a target connection using the Flow Service API to specify the dataset that will contain the inbound source data.
API format
POST /targetConnections
Request
The following request creates a target connection for YOURSOURCE:
curl -X POST \
'https://platform.adobe.io/data/foundation/flowservice/targetConnections' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}' \
-H 'Content-Type: application/json' \
-d '{
"name": "{YOURSOURCE} Target Connection",
"description": "{YOURSOURCE} Target Connection",
"connectionSpec": {
"id": "c604ff05-7f1a-43c0-8e18-33bf874cb11c",
"version": "1.0"
},
"data": {
"format": "json"
},
"params": {
"dataSetId": "5ef4551c52e054191a61a99f"
}
}'
name
description
connectionSpec.id
c604ff05-7f1a-43c0-8e18-33bf874cb11c
.data.format
params.dataSetId
Response
A successful response returns the new target connection’s unique identifier (id
). This ID is required in later steps.
{
"id": "7c96c827-3ffd-460c-a573-e9558f72f263",
"etag": "\"a196f685-f5e8-4c4c-bfbd-136141bb0c6d\""
}
Create a mapping mapping
In order for the source data to be ingested into a target dataset, it must first be mapped to the target schema that the target dataset adheres to. This is achieved by performing a POST request to Data Prep API with data mappings defined within the request payload.
API format
POST /conversion/mappingSets
Request
curl -X POST \
'https://platform.adobe.io/data/foundation/conversion/mappingSets' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}' \
-H 'Content-Type: application/json' \
-d '{
"version": 0,
"xdmSchema": "https://ns.adobe.com/{TENANT_ID}/schemas/995dabbea86d58e346ff91bd8aa741a9f36f29b1019138d4",
"xdmVersion": "1.0",
"id": null,
"mappings": [
{
"destinationXdmPath": "_id",
"sourceAttribute": "Id",
"identity": false,
"identityGroup": null,
"namespaceCode": null,
"version": 0
},
{
"destinationXdmPath": "person.name.firstName",
"sourceAttribute": "FirstName",
"identity": false,
"identityGroup": null,
"namespaceCode": null,
"version": 0
},
{
"destinationXdmPath": "person.name.lastName",
"sourceAttribute": "LastName",
"identity": false,
"identityGroup": null,
"namespaceCode": null,
"version": 0
}
]
}'
xdmSchema
mappings.destinationXdmPath
mappings.sourceAttribute
mappings.identity
Response
A successful response returns details of the newly created mapping including its unique identifier (id
). This value is required in a later step to create a dataflow.
{
"id": "bf5286a9c1ad4266baca76ba3adc9366",
"version": 0,
"createdDate": 1597784069368,
"modifiedDate": 1597784069368,
"createdBy": "{CREATED_BY}",
"modifiedBy": "{MODIFIED_BY}"
}
Create a flow flow
The last step towards bringing data from YOURSOURCE to Platform is to create a dataflow. By now, you have the following required values prepared:
A dataflow is responsible for scheduling and collecting data from a source. You can create a dataflow by performing a POST request while providing the previously mentioned values within the payload.
To schedule an ingestion, you must first set the start time value to epoch time in seconds. Then, you must set the frequency value to one of the five options: once
, minute
, hour
, day
, or week
. The interval value designates the period between two consecutive ingestions however, creating a one-time ingestion does not require an interval to be set. For all other frequencies, the interval value must be set to equal or greater than 15
.
API format
POST /flows
Request
curl -X POST \
'https://platform.adobe.io/data/foundation/flowservice/flows' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}' \
-H 'Content-Type: application/json' \
-d '{
"name": "{YOURSOURCE} dataflow",
"description": "{YOURSOURCE} dataflow",
"flowSpec": {
"id": "6499120c-0b15-42dc-936e-847ea3c24d72",
"version": "1.0"
},
"sourceConnectionIds": [
"246d052c-da4a-494a-937f-a0d17b1c6cf5"
],
"targetConnectionIds": [
"7c96c827-3ffd-460c-a573-e9558f72f263"
],
"transformations": [
{
"name": "Mapping",
"params": {
"mappingId": "bf5286a9c1ad4266baca76ba3adc9366",
"mappingVersion": "0"
}
}
],
"scheduleParams": {
"startTime": "1625040887",
"frequency": "minute",
"interval": 15
}
}'
name
description
flowSpec.id
6499120c-0b15-42dc-936e-847ea3c24d72
.flowSpec.version
1.0
.sourceConnectionIds
targetConnectionIds
transformations
transformations.name
transformations.params.mappingId
transformations.params.mappingVersion
0
.scheduleParams.startTime
scheduleParams.frequency
once
, minute
, hour
, day
, or week
.scheduleParams.interval
once
and should be greater than or equal to 15
for other frequency values.Response
A successful response returns the ID (id
) of the newly created dataflow. You can use this ID to monitor, update, or delete your dataflow.
{
"id": "993f908f-3342-4d9c-9f3c-5aa9a189ca1a",
"etag": "\"510bb1d4-8453-4034-b991-ab942e11dd8a\""
}
Appendix
The following section provides information on the steps you can to monitor, update, and delete your dataflow.
Monitor your dataflow
Once your dataflow has been created, you can monitor the data that is being ingested through it to see information on flow runs, completion status, and errors. For complete API examples, read the guide on monitoring your sources dataflows using the API.
Update your dataflow
Update the details of your dataflow, such as its name and description, as well as its run schedule and associated mapping sets by making a PATCH request to the /flows
endpoint of Flow Service API, while providing the ID of your dataflow. When making a PATCH request, you must provide your dataflow’s unique etag
in the If-Match
header. For complete API examples, read the guide on updating sources dataflows using the API
Update your account
Update the name, description, and credentials of your source account by performing a PATCH request to the Flow Service API while providing your base connection ID as a query parameter. When making a PATCH request, you must provide your source account’s unique etag
in the If-Match
header. For complete API examples, read the guide on updating your source account using the API.
Delete your dataflow
Delete your dataflow by performing a DELETE request to the Flow Service API while providing the ID of the dataflow you want to delete as part of the query parameter. For complete API examples, read the guide on deleting your dataflows using the API.
Delete your account
Delete your account by performing a DELETE request to the Flow Service API while providing the base connection ID of the account you want to delete. For complete API examples, read the guide on deleting your source account using the API.