Model data in schemas
In this lesson, you will model Luma’s data into schemas. This is one of the longest lessons in the tutorial, so get a glass of water and buckle up!
Standardization and interoperability are key concepts behind Adobe Experience Platform. Experience Data Model (XDM) is an effort to standardize customer experience data and define schemas for customer experience management.
XDM is a publicly documented specification designed to improve the power of digital experiences. It provides common structures and definitions for any application to use to communicate with Platform services. By adhering to XDM standards, all customer experience data can be incorporated into a common representation that can deliver insights in a faster, more integrated way. You can gain valuable insights from customer actions, define customer audiences through segments, and express customer attributes for personalization purposes.
XDM is the foundational framework that allows Adobe Experience Cloud, powered by Experience Platform, to deliver the right message to the right person, on the right channel, at exactly the right moment. The methodology on which Experience Platform is built, XDM System, operationalizes Experience Data Model schemas for use by Platform services.
Data Architects will need to create schemas outside of this tutorial, but Data Engineers will work closely with the schemas created by the Data Architect.
Before you begin the exercises, watch this short video to learn more about schemas and the Experience Data Model (XDM):
Permissions required
In the Configure Permissions lesson, you set up all the access controls required to complete this lesson.
Create Loyalty Schema via UI
In this exercise, we will create a schema for Luma’s loyalty data.
- Go to the Platform user interface and ensure that your sandbox is selected.
- Go to Schemas in the left navigation
- Select the Create Schema button on the top right
- From the dropdown menu, select XDM Individual Profile, since we will be modeling attributes of an individual customer (points, status, and so on).
Add standard field groups
Next you will be prompted to add field groups to the schema. All fields must be added to schemas using groups. You can choose from a large set of industry-standard field groups provided by Adobe or create your own. As you start modeling your own data in Experience Platform, it is good to become familiar with the industry-standard field groups provided by Adobe. Whenever possible, it is a best practice to use them as they sometimes power downstream services, such as Customer AI, Attribution AI, and Adobe Analytics.
When working with your own data, a big step will be to determine which of your own data should be captured in Platform and how it should be modeled. This large topic is discussed in more depth in the course Model Your Customer Experience Data with XDM. In this tutorial, I will just be guiding you through the implementation of some pre-determined schemas.
To add field groups:
-
In the Add Field groups modal, select following field groups:
- Demographic Details for basic customer data like name and birthdate
- Personal Contact Details for basic contact details like email address and phone number
-
You can preview the fields contributed in the field group by selecting the icon on the right side of the row.
-
Check the Industry > Retail box to expose industry-specific field groups.
-
Select Loyalty to add the loyalty program fields.
-
Select Add field group to add all three field groups to the schema.
Now take some time to explore the current state of the schema. The field groups have added standard fields related to a person, their contact details, and loyalty program status. You may find these two field groups useful when you create schemas for your own company’s data. Select a specific field group row or check the box next to the field group name to see how the visualization changes.
To save the schema:
- Select the top node of the schema.
- Enter
Luma Loyalty Schema
as the Display Name. - Select Save.
Add a custom field group
Now let’s create a custom field group.
While the loyalty field group contained a loyaltyID
field, Luma would like to manage all of their system identifiers in a single group to help ensure consistency across their schemas.
Field groups must be created in the schema workflow. To create the field group:
- Select Add under the Schema Field Groups heading
- Select Create new field group
- Use
Luma Identity profile field group
as the Display name - Use
system identifiers for XDM Individual Profile class
as the Description - Select Add field groups
The new, empty field group is added to your schema. The + buttons can be used to add new fields to any location in the hierarchy. In our case we want to add fields at the root level:
-
Select + next to the name of the schema. This adds a new field under your tenant id namespace to manage conflicts between your custom fields and any standard fields.
-
In the Field properties sidebar add the details of the new field:
- Field name:
systemIdentifier
- Display name:
System Identifier
- Type: Object
- Select Apply
- Field name:
Now add two fields under the systemIdentifier
object:
-
First field
- Field name:
loyaltyId
- Display name:
Loyalty Id
- Type: String
- Field name:
-
Second field
- Field Name:
crmId
- Display Name:
CRM Id
- Type: String
- Field Name:
Your new field group should look like this. Select the Save button to save your schema, but leave the schema open for the next exercise.
Create a data type
Field groups, such as your new Luma Identity profile field group
, can be reused in other schemas, allowing you to enforce standard data definitions across multiple systems. But they can only be reused in schemas that share a class, in this case the XDM Individual Profile class.
The data type is another multi-field construct which can be reused in schemas across multiple classes. Let’s convert our new systemIdentifier
object into a data type:
With the Luma Loyalty Schema
still open, select the systemIdentifier
object and select Convert to new data type
If you Cancel out of the schema and navigate to the Data types tab, you will see your newly created data type. We will use this data type later in the lesson.
Create CRM Schema via API
Now we will create a schema using the API.
- Use the XDM Individual Profile class
- Name it
Luma CRM Schema
- Use the following field groups: Demographic Details, Personal Contact Details, and Luma Identity profile field group
First we create the empty schema:
-
Open Postman
-
If you don’t have an access token, open the request OAuth: Request Access Token and select Send to request a new access token.
-
Open your environment variables and change the value of CONTAINER_ID from
global
totenant
. Remember, you must usetenant
whenever you want to interact with your own custom elements in Platform, such as creating a schema. -
Select Save
-
Open the request Schema Registry API > Schemas > Create a new custom schema.
-
Open the Body tab and paste the following code and select Send to make the API call. This call creates a new schema using the same
XDM Individual Profile
base class:code language-json { "type": "object", "title": "Luma CRM Schema", "description": "Schema for CRM data of Luma Retail ", "allOf": [{ "$ref": "https://ns.adobe.com/xdm/context/profile" }] }
note note NOTE The namespace references in this and subsequent code samples (for example https://ns.adobe.com/xdm/context/profile
), can be obtained by using list API calls with the CONTAINER_ID and accept header set to the correct values. Some are also easily accessible in the user interface. -
You should get a
201 Created
response -
Copy
meta:altId
from Response body. We will use it later in another exercise.
-
The new schema should be visible in the user interface but without any field groups
meta:altId
or schema id can also be obtained by making the API request Schema Registry API > Schemas > Retrieve a list of schemas within the specified container. with the CONTAINER_ID set to tenant
and an accept header application/vnd.adobe.xdm+json
.- No auth token: Run the OAuth: Request Access Token request to generate a new token
401: Not Authorized to PUT/POST/PATCH/DELETE for this path : /global/schemas/
: Update the CONTAINER_ID environment variable fromglobal
totenant
403: PALM Access Denied. POST access is denied for this resource from access control
: Verify your user permissions in the Admin Console
Add standard field groups
Now it’s time to add the field groups to the schema:
-
In Postman, Open the request Schema Registry API > Schemas > Update one or more attributes of a custom schema specified by ID.
-
In the Params tab, paste the
meta:altId
value from the previous response as theSCHEMA_ID
-
Open the Body tab and paste the following code and select Send to make the API call. This call adds the standard field groups to your
Luma CRM Schema
:code language-json [{ "op": "add", "path": "/allOf/-", "value": { "$ref": "https://ns.adobe.com/xdm/context/profile-personal-details" } }, { "op": "add", "path": "/allOf/-", "value": { "$ref": "https://ns.adobe.com/xdm/context/profile-person-details" } } ]
-
You should get a 200 OK status for the response and the field groups should be visible as part of your schema in the UI
Add custom field group
Now let’s add our Luma Identity profile field group
to the schema. First, we need to find the id of our new field group, using a list API:
-
Open the request Schema Registry API > Field groups > Retrieve a list of field groups within the specified container.
-
Select the Send button to retrieve a list of all of the custom field groups in your account
-
Grab the
$id
value of theLuma Identity profile field group
(yours will be different from the value in this screenshot)
-
Open the request Schema Registry API > Schemas > Update one or more attributes of a custom schema specified by ID. again
-
The Params tab should still have the
$id
of your schema -
Open the Body tab and paste the following code, replacing the
$ref
value with the$id
of your ownLuma Identity profile field group
:code language-json [{ "op": "add", "path": "/allOf/-", "value": { "$ref": "REPLACE_WITH_YOUR_OWN_FIELD_GROUP_ID" } }]
-
Select Send
Verify that the field group has been added to the schema by checking both the API response and in the interface.
Create Offline Purchase Events Schema
Now let’s create a schema based on the XDM ExperienceEvent class for Luma’s offline purchase data. Since you are now getting familiar with the schema editor user interface, I will reduce the number of screenshots in the instructions:
- Create a schema with the XDM ExperienceEvent class
- Add the standard field group Commerce Details to capture common order details. Spend a few minutes exploring the objects inside.
- Search for
Luma Identity profile field group
. It is not available! Remember that field groups are tied to a class, and since we are using a different class for this schema we can’t use it. We we need to add a new field group for the XDM ExperienceEvent class containing the identity fields. Our data type will make that really easy! - Select the Create new field group radio button
- Enter the Display name as
Luma Identity ExperienceEvent field group
and select the Add field groups button - Make sure the + buttons appear on in the Structure section so you can add new fields
- In Structure section, select + at top level of the schema
- As the Field Name, enter
systemIdentifier
- As the Display Name, enter
System Identifier
- As the Type, select System Identifier which is the custom data type you created earlier
- Select the Apply button
- Name your schema
Luma Offline Purchase Events Schema
- Select the Save button
Note how the data type added all of the fields!
Also, select XDM ExperienceEvent under the Class heading and inspect some of the fields contributed by this class. Note that _id and timestamp fields are required when using the XDM ExperienceEvent class—these fields must be populated for every record you ingest when using this schema:
Create Web Events Schema
Now we are going to create one more schema for Luma’s website data. By this point you should be an expert at creating schemas! Build the following schema with these properties
Select the Consumer Experience Event field group. This field group contains the commerce and productListItems objects that were also in Commerce Details. Indeed Consumer Experience Event is a combination of several other standard field groups that are also available separately. AEP Web SDK ExperienceEvent Mixin field group also contains other field groups, including some of the same ones in Consumer Experience Event. Fortunately, they blend together seamlessly.
Notice that we didn’t add the Luma Identity ExperienceEvent field group
to this schema. This is because the Web SDK has a different way of collecting identities. If you select the XDM ExperienceEvent class in the Composition section of the schema editor, you will notice that one of the fields it adds by default is called IdentityMap. IdentityMap is used by various Adobe applications to link to Platform. You will see how identities are sent to Platform via identityMap in the streaming ingestion lesson.
Create Product Catalog Schema
By using the Commerce Details and Consumer Experience Event field groups, Luma reports some details of product-related events via the standard productListItems data type. But they also have additional product detail fields that they would like to send to Platform. Instead of capturing all of these fields in their point-of-sale and e-commerce systems, Luma would prefer to ingest these fields directly from their product catalog system. A “schema relationship” allows you to define a relationship between two schemas for the purposes of classification or lookups. Luma will use a relationship to classify their product details. We will begin the process now and complete it at the end of the next lesson.
First we must create a schema for Luma’s product catalog using a custom class:
-
Select the Create schema button and select the Browse option from the dropdown
-
Select the Create new class radio button
-
Name it
Luma Product Catalog Class
-
Leave the Behavior as Record
-
Select the Assign Class button
-
Create a new field group called
Luma Product Catalog field group
with the following fields:- productName: Product Name: String
- productCategory: Product Category: String
- productColor: Product Color: String
- productSku: Product SKU: String | Required
- productSize: Product Size: String
- productPrice: Product Price: Double
-
Name the schema
Luma Product Catalog Schema
(be sure to update the correct field and not update the class name) -
Save the schema
Your new schema should look like this. Note how the productSku
field is listed in the Required fields section:
The next step is to define the relationship between the two ExperienceEvent schemas and the Luma Product Catalog Schema
, however there are a few additional steps that we must take in the next lesson before we can do that.
Additional Resources
Now that you have your schemas you can map identities!