Content Search and Indexing indexing

Changes in AEM as a Cloud Service changes-in-aem-as-a-cloud-service

With AEM as a Cloud Service, Adobe is moving away from an AEM instance-centric model to a service-based view with n-x AEM Containers, driven by CI/CD pipelines in the Cloud Manager. Instead of configuring and maintaining Indexes on single AEM instances, the Index configuration has to be specified before a deployment. Configuration changes in production are clearly breaking CI/CD policies. The same holds true for index changes since it can impact system stability and performance if not specified tested and reindexed before bringing them into production.

Below is a list of the main changes compared to AEM 6.5 and earlier versions:

Limitations:

How to Use how-to-use

Index definitions can be categorized into three primary use cases, as follows:

For both points 1 and 2 above, you need to create an index definition as part of your custom code base in the respective Cloud Manager release schedule. For more information, see the Deploying to AEM as a Cloud Service documentation.

Index Names index-names

An index definition can fall into one of the following categories:

Preparing the New Index Definition preparing-the-new-index-definition

For customizations of an OOTB index prepare a new package that contains the actual index definition that follows this naming pattern:

<indexName>-<productVersion>-custom-<customVersion>

For a fully customizsed index, prepare a new index definition package that contains the index definition that follows this naming pattern:

<prefix>.<indexName>-<productVersion>-custom-<customVersion>

Deploying Custom Index Definitions deploying-custom-index-definitions

To illustrate the deployment of a customized version of the out-of-the-box index damAssetLucene-8, we will provide a step-by-step guide. In this example, we will rename it to damAssetLucene-8-custom-1. Then the process is as follows:

Project Configuration

We strongly recommend to use version >= 1.3.2 of the Jackrabbit filevault-package-maven-plugin. The steps to incorporate it into your project are as follows:

After adding the new index definition, deploy the new application using Cloud Manager. This deployment initiates two jobs, responsible for adding (and merging if necessary) the index definitions to MongoDB and Azure Segment Store for author and publish, respectively. Prior to the switch, the underlying repositories undergo reindexing with the updated index definitions.

Index Management using Rolling Deployments index-management-using-rolling-deployments

What is Index Management what-is-index-management

Index management is about adding, removing, and changing indexes. Changing the definition of an index is fast, but applying the change (often called “building an index”, or, for existing indexes, “reindexing”) requires time. It is not instantaneous: the repository must be scanned for data to be indexed.

What are Rolling Deployments what-are-rolling-deployments

A rolling deployment can reduce downtime. It also allows for zero downtime upgrades and provides fast rollbacks. The old version of the application runs at the same time as the new version of the application.

Read-Only and Read-Write Areas read-only-and-read-write-areas

Certain areas of the repository (read-only parts of the repository) can be different in the old and in the new version of the application. The read-only areas of the repository are typically /app and /libs. In the following example, italic is used to mark read-only areas, while bold is used for read-write areas.

The read-write areas of the repository are shared between all versions of the application, while for each version of the application, there is a specific set of /apps and /libs.

Index Management Without Rolling Deployments index-management-without-rolling-deployments

During development, or when using on-premise installations, indexes can be added, removed, or changed at runtime. Indexes are used when they are available. If an index is not yet used in the old version of the application, then the index is typically built during a scheduled downtime. The same occurs when removing an index, or changing an existing index. When removing an index, it becomes unavailable when it is removed.

Index Management With Rolling Deployments index-management-with-rolling-deployments

With rolling deployments, there is no downtime. For some time during an update, both the old version (for example, version 1) of the application, and the new version (version 2), run concurrently against the same repository. If version 1 requires a certain index to be available, then this index must not be removed in version 2. The index should be removed later, for example, in version 3, at which point it is guaranteed that version 1 of the application is no longer running. Also, applications should be written such that version 1 works well, even if version 2 is running, and if indexes of version 2 are available.

After upgrading to the new version is complete, old indexes can be garbage collected by the system. The old indexes might still stay for some time, to speed up rollbacks (if a rollback should be needed).

The following table shows five index definitions: index cqPageLucene is used in both versions while index damAssetLucene-custom-1 is used only in version 2.

The version number is incremented each time the index is changed. To avoid custom index names colliding with index names of the product itself, custom indexes, and changes to out-of-the-box indexes must end with -custom-<number>.

Changes to Out-of-the-Box Indexes changes-to-out-of-the-box-indexes

After Adobe changes an out-of-the-box index like “damAssetLucene” or “cqPageLucene”, a new index named damAssetLucene-2 or cqPageLucene-2 is created. Or, if the index was already customized, the customized index definition is merged with the changes in the out-of-the-box index, as shown below. Merging of changes happens automatically. That means you do not need to do anything if an out-of-the-box index changes. However, it is possible to customize the index again later.

Current Limitations current-limitations

Index management is only supported for indexes of type lucene, with compatVersion set to 2. Internally, other indexes might be configured and used for queries, for example, Elasticsearch indexes. Queries that are written against the damAssetLucene index might, on AEM as a Cloud Service, in fact be run against an Elasticsearch version of this index. This difference is invisible to the application user, however certain tools such as the explain feature reports a different index. For differences between Lucene and Elasticsearch indexes, see the Elasticsearch documentation in Apache Jackrabbit Oak. Customers cannot and do not need to configure Elasticsearch indexes directly.

Only built-in analyzers are supported (that is, those analyzers that are shipped with the product). Custom analyzers are not supported.

For best operational performance, indexes should not be excessively large. The total size of all indexes can be used as a guide. If this size increases by more than 100% after custom indexes have been added, and standard indices have been adjusted on a development environment, custom index definitions should be adjusted. AEM as a Cloud Service can prevent the deployment of indexes that would negatively impact system stability and performance.

Adding an Index adding-an-index

To add a fully custom index named /oak:index/acme.product-custom-1, to be used in a new version of the application and later, the index must be configured as follows:

acme.product-1-custom-1

This configuration works by prepending a custom identifier to the index name, followed by a dot (.). The identifier should be from 2 through 5 characters in length.

As above, this configuration ensures that the index is only used by the new version of the application.

Changing an Index changing-an-index

When an existing index is changed, a new index must be added with the changed index definition. For example, consider the existing index /oak:index/acme.product-custom-1 is changed. The old index is stored under /oak:index/acme.product-custom-1, and the new index is stored under /oak:index/acme.product-custom-2.

The old version of the application uses the following configuration:

/oak:index/acme.product-custom-1

The new version of the application uses the following (changed) configuration:

/oak:index/acme.product-custom-2

Undoing a Change undoing-a-change

At times, it becomes necessary to undo a modification in an index definition. This could occur due to an inadvertent error or that the modification is no longer required. Take, for instance, the index definition damAssetLucene-8-custom-3, which was mistakenly created and has already been deployed. Consequently, you want to revert back to the previous index definition, damAssetLucene-8-custom-2. To accomplish this, you need to introduce a new index named damAssetLucene-8-custom-4 that incorporates the definition from the prior index, damAssetLucene-8-custom-2.

Removing an Index removing-an-index

The following only applies to custom indexes. Product indexes may not be removed as they are used by AEM.

If an index is removed in a later version of the application, you can define an empty index (an empty index that is never used, and does not contain any data), with a new name. For this example, you can name it /oak:index/acme.product-custom-3. This name replaces the index /oak:index/acme.product-custom-2. After /oak:index/acme.product-custom-2 is removed by the system, the empty index /oak:index/acme.product-custom-3 can then be removed. An example of such an empty index is:

<acme.product-custom-3
        jcr:primaryType="oak:QueryIndexDefinition"
        async="async"
        compatVersion="2"
        includedPaths="/dummy"
        queryPaths="/dummy"
        type="lucene">
        <indexRules jcr:primaryType="nt:unstructured">
            <rep:root jcr:primaryType="nt:unstructured">
                <properties jcr:primaryType="nt:unstructured">
                    <dummy
                        jcr:primaryType="nt:unstructured"
                        name="dummy"
                        propertyIndex="{Boolean}true"/>
                </properties>
            </rep:root>
        </indexRules>
</acme.product-custom-3>

If it is no longer needed to have a customization of an out-of-the-box index, then you must copy the out-of-the-box index definition. For example, if you have already deployed damAssetLucene-8-custom-3, but no longer need the customizations and want to switch back to the default damAssetLucene-8 index, then you must add an index damAssetLucene-8-custom-4 that contains the index definition of damAssetLucene-8.

Index and Query Optimizations index-query-optimizations

Apache Jackrabbit Oak enables flexible index configurations to efficiently handle search queries. Indexes are especially important for larger repositories. Ensure that all queries are backed by an appropriate index. Queries without a suitable index may read thousands of nodes, which are then logged as a warning.

See this document for information on how queries and indexes can be optimized.