Update dataflows using the Flow Service API
This tutorial covers the steps for updating a dataflow, including its basic information, schedule, and mapping sets using the Flow Service API.
Getting started
This tutorial requires you to have a valid flow ID. If you do not have a valid flow ID, select your connector of choice from the sources overview and follow the steps outlined before attempting this tutorial.
This tutorial also requires you to have a working understanding of the following components of Adobe Experience Platform:
- Sources: Experience Platform allows data to be ingested from various sources while providing you with the ability to structure, label, and enhance incoming data using Platform services.
- Sandboxes: Experience Platform provides virtual sandboxes which partition a single Platform instance into separate virtual environments to help develop and evolve digital experience applications.
Using Platform APIs
For information on how to successfully make calls to Platform APIs, see the guide on getting started with Platform APIs.
Look up dataflow details
The first step in updating your dataflow is to retrieve dataflow details using your flow ID. You can view the current details of an existing dataflow by making a GET request to the /flows
endpoint.
API format
GET /flows/{FLOW_ID}
{FLOW_ID}
id
value for the dataflow you want to retrieve.Request
The following request retrieves updated information regarding your flow ID.
curl -X GET \
'https://platform.adobe.io/data/foundation/flowservice/flows/2edc08ac-4df5-4fe6-936f-81a19ce92f5c' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}'
Response
A successful response returns the current details of your dataflow including its version, schedule, and unique identifier (id
).
{
"items": [
{
"id": "2edc08ac-4df5-4fe6-936f-81a19ce92f5c",
"createdAt": 1612310475905,
"updatedAt": 1614122324830,
"createdBy": "{CREATED_BY}",
"updatedBy": "{UPDATED_BY}",
"createdClient": "{CREATED_CLIENT}",
"updatedClient": "{UPDATED_CLIENT}",
"sandboxId": "{SANDBOX_ID}",
"sandboxName": "{SANDBOX_NAME}",
"imsOrgId": "{ORG_ID}",
"name": "Database dataflow using BigQuery",
"description": "collecting test1.Mytable from Google BigQuery",
"flowSpec": {
"id": "14518937-270c-4525-bdec-c2ba7cce3860",
"version": "1.0"
},
"state": "enabled",
"version": "\"5400d99c-0000-0200-0000-60358d540000\"",
"etag": "\"5400d99c-0000-0200-0000-60358d540000\"",
"sourceConnectionIds": [
"b7581b59-c603-4df1-a689-d23d7ac440f3"
],
"targetConnectionIds": [
"320f119a-5ac1-4ab1-88ea-eb19e674ea2e"
],
"inheritedAttributes": {
"sourceConnections": [
{
"id": "b7581b59-c603-4df1-a689-d23d7ac440f3",
"connectionSpec": {
"id": "3c9b37f8-13a6-43d8-bad3-b863b941fedd",
"version": "1.0"
},
"baseConnection": {
"id": "6990abad-977d-41b9-a85d-17ea8cf1c0e4",
"connectionSpec": {
"id": "3c9b37f8-13a6-43d8-bad3-b863b941fedd",
"version": "1.0"
}
}
}
],
"targetConnections": [
{
"id": "320f119a-5ac1-4ab1-88ea-eb19e674ea2e",
"connectionSpec": {
"id": "c604ff05-7f1a-43c0-8e18-33bf874cb11c",
"version": "1.0"
}
}
]
},
"scheduleParams": {
"startTime": "1612310466",
"frequency": "week",
"interval": "15",
"backfill": "true"
},
"transformations": [
{
"name": "Copy",
"params": {
"deltaColumn": {
"name": "Datefield",
"dateFormat": "YYYY-MM-DD",
"timezone": "UTC"
}
}
},
{
"name": "Mapping",
"params": {
"mappingId": "0b090130b58b4819afc78b6dc98b484d",
"mappingVersion": "0"
}
}
],
"runs": "/flows/2edc08ac-4df5-4fe6-936f-81a19ce92f5c/runs",
"lastOperation": {
"started": 1614122316652,
"updated": 1614122324830,
"percentCompleted": 100.0,
"status": {
"value": "completed",
"errors": []
},
"ops": [
{
"op": "replace",
"path": "/scheduleParams/frequency",
"value": "week"
}
],
"operation": "update"
},
"lastRunDetails": {
"id": "a10cc80b-fbea-4c6b-873e-d7fd32f4d12d",
"state": "success",
"startedAtUTC": 1613079975512,
"completedAtUTC": 1613080027511
}
}
]
}
Update dataflow
To update your dataflow’s run schedule, name, and description, perform a PATCH request to the Flow Service API while providing your flow ID, version, and the new schedule you want to use.
If-Match
header is required when making a PATCH request. The value for this header is the unique version of the connection you want to update. The etag value updates with every successful update of a dataflow.API format
PATCH /flows/{FLOW_ID}
Request
The following request updates your flow run schedule, as well as your dataflow’s name and description.
curl -X PATCH \
'https://platform.adobe.io/data/foundation/flowservice/flows/2edc08ac-4df5-4fe6-936f-81a19ce92f5c' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}'
-H 'If-Match: "1a0037e4-0000-0200-0000-602e06f60000"' \
-d '[
{
"op": "replace",
"path": "/scheduleParams/frequency",
"value": "day"
},
{
"op": "replace",
"path": "/name",
"value": "Database Dataflow Feb2021"
},
{
"op": "replace",
"path": "/description",
"value": "Database dataflow for testing update API"
}
]'
op
add
, replace
, and remove
.path
value
Response
A successful response returns your flow ID and an updated etag. You can verify the update by making a GET request to the Flow Service API, while providing your flow ID.
{
"id": "2edc08ac-4df5-4fe6-936f-81a19ce92f5c",
"etag": "\"50014cc8-0000-0200-0000-6036eb720000\""
}
Update mapping
You can update the mapping set of an existing dataflow by making a PATCH request to the Flow Service API and providing updated values for your mappingId
and mappingVersion
.
API format
PATCH /flows/{FLOW_ID}
Request
The following request updates your dataflow’s mapping set.
curl -X PATCH \
'https://platform.adobe.io/data/foundation/flowservice/flows/2edc08ac-4df5-4fe6-936f-81a19ce92f5c' \
-H 'Authorization: Bearer {ACCESS_TOKEN}' \
-H 'x-api-key: {API_KEY}' \
-H 'x-gw-ims-org-id: {ORG_ID}' \
-H 'x-sandbox-name: {SANDBOX_NAME}'
-H 'If-Match: "50014cc8-0000-0200-0000-6036eb720000"' \
-d '[
{
"op": "replace",
"path": "/transformations/0",
"value": {
"name": "Mapping",
"params": {
"mappingId": "c5f22f04e09f44498e528901546a83b1",
"mappingVersion": 2
}
}
}
]'
op
add
, replace
, and remove
.path
transformations
is being updated.value.name
value.params.mappingId
value.params.mappingVersion
Response
A successful response returns your flow ID and an updated etag. You can verify the update by making a GET request to the Flow Service API, while providing your flow ID.
{
"id": "2edc08ac-4df5-4fe6-936f-81a19ce92f5c",
"etag": "\"2c000802-0000-0200-0000-613976440000\""
}
Next steps
By following this tutorial, you have updated the basic information, schedule, and mapping sets of your dataflow using the Flow Service API. For more information on using source connectors, see the sources overview.