Set up a recurring import workflow setting-up-a-recurring-import

Create a new workflow template from Resources > Templates > Workflow templates.

Add the following activities:

• Data loading (file): Define the expected structure of the file containing the data to import.
• Enrichment: Reconcile the imported data with database data.
• Split: Create filters to process records differently depending on whether they could be reconciled or not.
• Deduplication: Deduplicate the data from the incoming file before it is inserted in the database.
• Update data: Update the database with the imported profiles.

Configure the Data Loading (file) activity:

• Define the expected structure by uploading a sample file. The sample file should contain only a few lines but all the columns necessary for the import. Check and edit the file format to make sure that the type of each column is set correctly: text, date, integer, etc. For example:

  code language-none
  lastname;firstname;birthdate;email;crmID Smith;Hayden;23/05/1989;hayden.smith@mailtest.com;123456

• In the Name of the file to load section, select Upload a file from the local machine and leave the field blank. Each time a new workflow is created from this template, you can specify here the file you want, as long at it corresponds to the defined structure.

  You can use any of the options but you have to modify the template accordingly. For example, if you select Specified in the transition, you can add a File Transfer activity before to retrieve the file to import from a FTP/SFTP server. With S3 or SFTP connection, you can also import segment data to Adobe Campaign with Adobe Real-time Customer Data platform. For more on this, refer to this documentation.

  

Configure the Enrichment activity. The purpose of this activity in this context is to identify the incoming data.

• In the Enrichment tab, select Add data and define a link between the imported data and the recipients targeting dimension. In this example, the CRM ID custom field is used to create the join condition. Use the field or combination of fields you need as long it allows to identify unique records.
• In the Reconciliation tab, leave the Identify the document from the working data option unchecked.

Configure the Split activity to retrieve reconciled recipients in one transition and recipients that could not be reconciled but who have enough data in a second transition.

The transition with reconciled recipients can then be used to update the database. The transition with unknown recipients can then be used to create new recipient entries in the database if a minimum set of information is available in the file.

Recipients that cannot be reconciled and do not have enough data are selected in a complement outbound transition and can be exported in a separate file or simply ignored.

• In the General tab of the activity, select Use the additional data only as filtering setting and make sure that the Targeting dimension is automatically set to Enrichment.

  Check the Generate complement option to be able to see if any record cannot be inserted in the database. If you need, you can then apply further processing to the complementary data: file export, list update, etc.

• In the first subset of the Subsets tab, add a filtering condition on the inbound population to select only records for which the recipient primary key is not equal to 0. This way, data from the file that are reconciled with recipients from the database are selected in that subset.

  

• Add a second subset that selects unreconciled records that have enough data to be inserted in the database. For example: email address, first name and last name.

  Subsets are processed in their creation order, meaning that when this second subset is processed, all records that already exist in the database are already selected in the first subset.

  

• All records that are not selected in the first two subsets are selected in the Complement.

Configure the Update data activity located after the first outbound transition of the Split activity configured previously.

• Select Update as Operation type since the inbound transition only contains recipients already present in the database.

• In the Record identification section, select Using reconciliation keys and define a key between the targeting dimension and the link created in the Enrichment. In this example, the CRM ID custom field is used.

• In the Fields to update section, indicate the fields from the recipients dimension to update with the value of the corresponding column from the file. If the names of the file columns are identical or almost identical to the names of the recipients dimension fields, you can use the magic wand button to automatically match the different fields.

  

Configure the Deduplication activity located after the transition containing unreconciled recipients:

• Select Edit configuration and set the targeting dimension to the temporary schema generated from the Enrichment activity of the workflow.

  

• In this is example, the email field is used to find unique profiles. You can use any field you are sure is filled and part of a unique combination.

• In the Deduplication method screen, select Advanced parameters and check the Disable automatic filtering of 0 ID records option to make sure records that have a primary key equal to 0 (which should be all records of this transition) are not excluded.

Configure the Update data activity located after the Deduplication activity configured previously.

• Select Insert as Operation type since the inbound transition only contains recipients not present in the database.

• In the Record identification section, select Directly using the targeting dimension and choose the Recipients dimension.

• In the Fields to update section, indicate the fields from the recipients dimension to update with the value of the corresponding column from the file. If the names of the file columns are identical or almost identical to the names of the recipients dimension fields, you can use the magic wand button to automatically match the different fields.

  

After the third transition of the Split activity, add a Data extraction (file) activity and a File transfer activity if you want to keep track of data not inserted in the database. Configure those activities to export the column you need and to transfer the file on a FTP or SFTP server where you can retrieve it.

Add an End activity and save the workflow template.