DocumentationExperience PlatformIngest streaming data

Ingest streaming data

Last update: Mon Sep 25 2023 00:00:00 GMT+0000 (Coordinated Universal Time)

CREATED FOR:

  • Beginner
  • Intermediate
  • Developer

In this lesson, you will stream data using the Adobe Experience Platform Web SDK.

There are two main tasks we must complete in the Data Collection interface:

  • We must implement Web SDK on the Luma website to send data about visitor activity from the website to the Adobe Edge network. We will do a simple implementation using tags (formerly Launch)

  • We must configure a datastream, which tells the Edge network where to forward the data. We will configure it to send the data to our Luma Web Events dataset in our Platform sandbox.

Data Engineers will need to ingest streaming data outside of this tutorial. When implementing Adobe Experience Platform’s Web or Mobile SDKs, typically a web or mobile developer is involved in the data layer creation and tag property configuration.

Before you begin the exercises, watch these two short videos to learn more about streaming data ingestion and Web SDK:

https://video.tv.adobe.com/v/28425?learn=on

https://video.tv.adobe.com/v/34141?learn=on

NOTE
While this tutorial focuses on streaming ingestion from websites with Web SDK, you can also stream data using the Adobe Mobile SDK, Apache Kafka Connect, and other mechanisms.

Permissions required

In the Configure Permissions lesson, you set up all the access controls required to complete this lesson.

Configure the datastream

First we will configure the datastream. A datastream tells the Adobe Edge network where to send the data after receiving it from the Web SDK call. For example, do you want to send the data to Experience Platform, Adobe Analytics, or Adobe Target? Datastreams are managed in the Data Collection user interface (formerly Launch) and are critical to data collection with Web SDK.

To create your datastream:

  1. Log into the Experience Platform Data Collection user interface

  2. Select Datastreams in the left navigation

  3. Select the New Datastream button in the upper-right hand corner

    Select datastreams in the left navigation

  4. For the Friendly Name, enter Luma Platform Tutorial (add your name to the end, if multiple people from your company are taking this tutorial)

  5. Select the Save button

    Name the datastram and save

On the next screen, you specify where you want to send data. To send data to Experience Platform:

  1. Toggle on Adobe Experience Platform to expose additional fields
  2. For Sandbox, select Luma Tutorial
  3. For Event Dataset, select Luma Web Events Dataset
  4. If you use other Adobe applications, feel free to explore the other sections to see which information is required in the Edge Configuration of these other solutions. Remember, Web SDK was developed not only to stream data into Experience Platform, but also to replace all previous JavaScript libraries used by other Adobe applications. The Edge Configuration is used to specify the account details of each application to which you want to send the data.
  5. Select Save
    Configure the datastream and save

Once the Edge Configuration has saved, the resulting screen will show three environments have been created for Development, Staging, and Production. Additional Development environments can be added:
Each Edge Configuration can have multiple environments
All three environments contain the Platform details you just entered. However, these details can be configured differently per environment. For example, you could have each environment send data to a different Platform sandbox. In this tutorial, we will not do any additional customization to our datastream.

Install the Web SDK Extension

Add a property

First we must create a tag property (formerly a tag property). A property is a container for all the JavaScript, rules, and other features required to collect details from a web page and send it to various locations.

To create a property:

  1. Go to Properties in the left navigation
  2. Select the New Property button
    Add a new property
  3. As the Name, enter Luma Platform Tutorial (add your name to the end, if multiple people from your company are taking this tutorial)
  4. As the Domains, enter enablementadobe.com (explained later)
  5. Select Save
    Property details

Add the Web SDK extension

Now that you have a property you can add the Web SDK using an extension. An extension is a package of code that extends the Data Collection interface and functionality. To add the extension:

  1. Open your tag property
  2. Go to Extensions in the left navigation
  3. Go to the Catalog tab
  4. There are many extensions available for tags. Filter the catalog with the term Web SDK
  5. In the Adobe Experience Platform Web SDK extension, select the Install button
    Install the Adobe Experience Platform Web SDK extension
  6. There are several configurations available for the Web SDK extension, but there only two we are going to configure for this tutorial. Update the Edge Domain to data.enablementadobe.com. This setting allows you to set first party cookies with your Web SDK implementation, which is encouraged. Later in this lesson, you will map a website on the enablementadobe.com domain to your tag property. The CNAME for the enablementadobe.com domain has already been configured so that data.enablementadobe.com will forward to Adobe servers. When you implement Web SDK on your own website, you will need to create a CNAME for your own data collection purposes, for example, data.YOUR_DOMAIN.com
  7. From the Datastream dropdown, select your Luma Platform Tutorial datastream.
  8. Feel free to look at the other configuration options (but don’t change them!) and then select Save

Create a rule to send data

Now we will create a rule to send data to Platform. A rule is a combination of events, conditions, and actions that tell tags to do something. To create a rule:

  1. Go to Rules in the left navigation
  2. Select the Create New Rule button
    Create a rule
  3. Name the rule All Pages - Library Loaded
  4. Under Events, select the Add button
    Name the rule and add an event
  5. Use the Core Extension and select Library Loaded (Page Top) as the Event Type. This setting means that our rule fires whenever the Launch library loads on a page.
  6. Select Keep Changes to return to the main rule screen
    Add the Library Loaded event
  7. Leave Conditions empty, since we want this rule to fire on all pages, as per the name we gave it
  8. Under Actions, select the Add button
  9. Use the Adobe Experience Platform Web SDK Extension and select Send Event as the Action Type
  10. On the right, select web.webpagedetails.pageViews from the Type dropdown. This is one of the XDM fields in our Luma Web Events Schema
  11. Select Keep Changes to return to the main rule screen
    Add the Send Event action
  12. Select Save to save the rule
    Save the rule

Publish the rule in a library

Next we will publish the rule to our development environment so we can verify that it works.

To create a library:

  1. Go to Publishing Flow in the left navigation
  2. Select Add Library
    Select Add Library
  3. For the Name, enter Luma Platform Tutorial
  4. For the Environment, select Development
  5. Select the Add All Changed Resources button. (In addition to the Adobe Experience Platform Web SDK extension and the All Pages - Library Loaded rule, you will also see the Core extension added which contains the base JavaScript required by all Launch web properties.)
  6. Select the Save & Build for Development button
    Create and build the library

The library may take a few minutes to build and when it is complete it displays a green dot to the left of the library name:
Build complete

As you can see on the Publishing Flow screen, there is a lot more to the publishing process which is beyond the scope of this tutorial. We are just going to use a single library in our Development environment.

Validate the data in the request

Add the Adobe Experience Platform Debugger

The Experience Platform Debugger is an extension available for Chrome and Firefox browsers which helps you see the Adobe technology implemented in your web pages. Download the version for your preferred browser:

If you’ve never used the Debugger before—and this one is different from the older Adobe Experience Cloud Debugger—you might want to watch this five-minute overview video:

https://video.tv.adobe.com/v/32156?learn=on

Open the Luma website

For this tutorial, we use a publicly hosted version of the Luma demo website. Let’s open it and bookmark it:

  1. In a new browser tab, open the Luma website.
  2. Bookmark the page for use throughout the rest of the tutorial

This hosted website is why we used enablementadobe.com in the Domains field of our initial tag property configuration and why we used data.enablementadobe.com as our first-party domain in the Adobe Experience Platform Web SDK extension. See, I had a plan!

Luma homepage

Use the Experience Platform Debugger to map to your tag property

The Experience Platform Debugger has a cool feature that allows you to replace an existing tag property with a different one. This is useful for validation and allows us to skip many implementation steps in this tutorial.

  1. Make sure you have the Luma site open and select the Experience Platform Debugger extension icon

  2. The Debugger will open and show some details of the hardcoded implementation, which is unrelated to this tutorial (you may need to reload the Luma site after opening the Debugger)

  3. Confirm that the Debugger is “Connected to Luma” as pictured below and then select the “lock” icon to lock the Debugger to the Luma site.

  4. Select the Sign In button on the top right to authenticate.

  5. Now go to Launch in the left navigation

  6. Select the Configuration tab

  7. To the right of where it shows you the Page Embed Codes, open the Actions dropdown, and select Replace
    Select Actions > Replace

  8. Since you are authenticated, the Debugger is going to pull in your available Launch properties and environments. Select your Luma Platform Tutorial property

  9. Select your Development environment

  10. Select the Apply button
    Select the alternate tag property

  11. The Luma website will now reload with your tag property. Help, I’ve been hacked! Just kidding.
    tag property replaced

  12. Go to Summary in the left navigation, to see the details of your Launch property
    Summary tab

  13. Now go to AEP Web SDK in the left navigation to see the Network Requests

  14. Open the events row

    Adobe Experience Platform Web SDK request

  15. Note how we can see the web.webpagedetails.pageView event type we specified in our Send Event action, and other, out-of-the-box variables adhering to the AEP Web SDK ExperienceEvent Mixin format
    Event details

  16. These types of request details are also visible in the browser’s web developer tools Network tab. Open it and reload the page. Filter for calls with interact to locate the call, select it, and then look in the Headers tab, Request Payload area.
    Netowrk tab

  17. Go to the Response tab and note how the ECID value is included in the response. Copy this value as you will use it to validate the profile information in the next exercise.
    Netowrk tab

Validate the data in Experience Platform

You can validate that data is landing in Platform by looking at the batches of data arriving in the Luma Web Events Dataset. (I know, it’s called streaming data ingestion but now I’m saying it arrives in batches! It streams in real-time to Profile, so it can be used for real-time segmentation and activation, but is sent in batches every 15 minutes to the data lake.)

To validate the data:

  1. In the Platform user interface, go to Datasets in the left navigation
  2. Open the Luma Web Events Dataset and confirm that a batch has arrived. Remember, they are sent every 15 minutes, so you might need to wait for the batch to show up.
  3. Select the Preview dataset button
    Open the dataset
  4. In the preview modal, note how you can select different fields of the schema on the left to preview those specific data points:
    Preview the fields

You can also confirm that the new profile is showing up:

  1. In the Platform user interface, go to Profiles in the left navigation
  2. Select the ECID namespace and search for your ECID value (copy it from the respon. The profile will have its own id, separate from the ECID.
  3. Select the Profile ID to open the profile
    Find and open the profile
  4. Select the Events tab to see the pages you viewed
    Profile Events

Add custom data to the event

Create a data element for page name

  1. In the Data Collection tags interface, on the top-right corner of your Luma Platform Tutorial property, open the Select a Working Library dropdown and select your Luma Platform Tutorial library. This setting makes it easier to publish additional updates to our library.

  2. Now go to Data Elements in the left navigation

  3. Select the Create New Data Element button

    Create a new data element

  4. As the Name, enter Page Name

  5. As the Data Element Type, select JavaScript Variable

  6. As the JavaScript variable name, enter digitalData.page.pageInfo.pageName

  7. To help standardize the format of the values, check the boxes for Force lowercase value and Clean text

  8. Make sure that Luma Platform Tutorial is selected as the working library

  9. Select Save to Library
    Create a data element for page name

Map the page name to the XDM Object data element

Now we will map our page name to the Web SDK.

IMPORTANT
In order to complete this task, we need to make sure that your user first has access to the Prod sandbox. If you don’t already have access to the Prod sandbox from a different product profile, quickly open your Luma Tutorial Platform profile and add the permission item Sandboxes > Prod. After doing so, do a SHIFT-Reload on the Data Elements page to clear your cache
Add the Prod sandbox

On the Data Elements page:

  1. Create a new data element
  2. As the Name, enter XDM Object
  3. As the Extension, select Adobe Experience Platform Web SDK
  4. As the Data Element Type, select XDM object
  5. As the Sandbox, select your Luma Tutorial sandbox
  6. As the Schema, select your Luma Web Events Schema
  7. Select the web.webPageDetails.name field
  8. As the Value, select the icon to open the data element selection modal and choose your Page Name data element
  9. Select Save to Library
    Map the page name to the XDM Object data element

This same process is used to map additional custom data on your website to XDM fields.

Add the XDM data to your Send Event action

Now that you have data mapped to XDM fields, you can include it in your Send Event action:

  1. Go to the Rules screen
  2. Open your All Pages - Library Loaded rule
  3. Open the Adobe Experience Platform Web SDK - Send Event action
  4. As the XDM data, select the icon to open the data element selection modal and choose your XDM Object data element
  5. Select the Keep Changes button
    Add the XDM data to your Send Event action
  6. Now, since you have had Luma Platform Tutorial selected as your working library for the last several exercises, your recent changes have been saving directly to the library. Instead of having to publish our changes via the Publishing Flow screen, you can just open the dropdown on the blue button and select Save to Library and Build
    Save to Library and Build

This starts building a new tag library with the three changes you just made.

Validate the XDM data

You should now be able to reload the Luma homepage, while mapped to your tag property using the Debugger as you learned earlier, and see that the page name field populates in the request!
Validate the XDM data

You can also validate the page name data was received in Platform, by previewing the dataset and profile.

Send additional identities

Your Web SDK implementation is now sending events with the Experience Cloud ID (ECID) as the primary identifier. The ECID is generated automatically by the Web SDK and is unique per device and browser. A single customer can have multiple ECIDs depending on which device and browser they are using. So how can we get a unified view of this customer and link their online activity to our CRM, Loyalty, and Offline Purchase data? We do that by collecting additional identities during their session and deterministically linking their profile via identity stitching.

If you recall, I mentioned that we would be using the ECID and CRM Id as identities for our web data in the Map Identities lesson. So let’s collect the CRM Id with the Web SDK!

Add Data Element for the CRM Id

First we store the CRM Id in a data element:

  1. In the tags interface, add a data element named CRM Id
  2. As the Data Element Type, select JavaScript Variable
  3. As the JavaScript variable name, enter digitalData.user.0.profile.0.attributes.username
  4. Select the Save to Library button (Luma Platform Tutorial should still be your working library)
    Add Data Element for the CRM Id

Add the CRM Id to the Identity Map data element

Now that we have captured the CRM Id value, we must associate it with a special data element type called the Identity Map data element:

  1. Add a data element named Identities

  2. As the Extension, select Adobe Experience Platform Web SDK

  3. As the Data Element Type, select Identity map

  4. As the Namespace, enter Luma CRM Id, which is the namespace we created in an earlier lesson

    note warning
    WARNING
    The Adobe Experience Platform Web SDK extension version 2.2 allows you to select Namespace from a pre-populated dropdown using the actual values in your Platform account. Unfortunately, this feature is not yet “sandbox aware” and thus the Luma CRM Id value may not appear in the dropdown. This may prevent you from completing this exercise. We will post a workaround once confirmed.

  5. As the ID, select the icon to open the data element selection modal and choose your CRM Id data element

  6. As the Authenticated State, select Authenticated

  7. Leave Primary unchecked. Since the CRM Id is not present for most visitors to the Luma website, you definitely do not want to override the ECID as the primary identifier. It would be a rare use case to use anything other than the ECID as the primary identifier. Usually I don’t mention the default settings in these instructions, but I am calling this one out to help you avoid headaches later on in your own implementation.

  8. Select the Save to Library button (Luma Platform Tutorial should still be your working library)
    Add the CRM Id to the Identity Map data element

NOTE
You can pass multiple identifiers using the Identity map data type.

Add the Identity map data element to the XDM Object

There is one more data element we must update—the XDM Object data element. It may seem weird to have to update three separate data elements to pass this one identity, but this process is designed to scale for multiple identities. Don’t worry, we’re almost done with this lesson!

  1. Open your XDM Object data element
  2. Open the IdentityMap XDM field
  3. As the Data element, select the icon to open the data element selection modal and choose your Identities data element
  4. Now, since you have had Luma Platform Tutorial selected as your working library for the last several exercises, your recent changes have been saving directly to the library. Instead of having to publish our changes via the Publishing Flow screen, you can open the dropdown on the blue button and select Save to Library and Build
    Add the IdentityMap data element to the XDM Object

Validate the identity

To validate that the CRM Id is now being sent by the Web SDK:

  1. Open the Luma website
  2. Map it to your tag property using the Debugger, as per the earlier instructions
  3. Select the Login link on the top right of the Luma website
  4. Log in using the credentials test@adobe.com/test
  5. Once authenticated, inspect the Experience Platform Web SDK call in the Debugger (Adobe Experience Platform Web SDK > Network Requests > events of the most recent request) and you should see the lumaCrmId:
    Validate the identity in the Debugger
  6. Look up the user profile using the ECID namespace and value again. In the profile, you will see the CRM Id and also the Loyalty Id and the profile details like the name and phone number. All identities and data have been stitched together into a single, real-time customer profile!
    Validate the identity in Platform

Additional Resources

Great job! That was a lot of information about Web SDK and Launch. There is a lot more involved in a full-blown implementation, but those are the basics to help you get started and see the results in Platform.

NOTE
Now that you are done with the Streaming Ingestion lesson, you can remove the Prod sandbox from your Luma Tutorial Platform product profile

Data Engineers, if you like you can skip ahead to the run queries lesson.

Data Architects, you can move on to merge policies!

recommendation-more-help
513160b6-bf42-4c58-abdd-4f817b1cccad