Debugging AEM as a Cloud Service using logs

Logs act as the frontline for debugging AEM applications in AEM as a Cloud Service, but are dependent on adequate logging in the deployed AEM application.

All log activity for a given environment’s AEM service (Author, Publish/Publish Dispatcher) is consolidated into a single log file, even if different pods within that service generate the log statements.

Pod Ids are provided in each log statement, and allowing filtering or collating of log statements. Pod Id’s are in the format of:

Custom log files

AEM as a Cloud Services does not support custom log files, however it does support custom logging.

For Java logs to be available in AEM as a Cloud Service (via Cloud Manager or Adobe I/O CLI), custom log statements must be written the error.log. Logs written to custom named logs, such as example.log, will not be accessible from AEM as a Cloud Service.

Logs can be written to the error.log using a Sling LogManager OSGi configuration property in the application’s org.apache.sling.commons.log.LogManager.factory.config~example.cfg.json files.

{
   ...
   "org.apache.sling.commons.log.file": "logs/error.log"
   ...
}

AEM Author and Publish service logs

Both AEM Author and Publish services provide AEM runtime server logs:

AEM Publish Dispatcher logs

Only AEM Publish Dispatcher provides Apache web server and Dispatcher logs, as these aspects only exists in the AEM Publish tier, and not on the AEM Author tier.

Cloud Manager cloud-manager

Adobe Cloud Manager allows for the download of logs, by day, via an environment’s Download Logs action.

These logs can be downloaded and inspected via any log analysis tools.

Adobe I/O CLI with Cloud Manager plugin aio

Adobe Cloud Manager supports accessing AEM as a Cloud Service logs via the Adobe I/O CLI with the Cloud Manager plugin for the Adobe I/O CLI.

First, set up the Adobe I/O with Cloud Manager plugin.

Ensure the relevant Program Id and Environment Id have been identified, and use list-available-log-options to list the log options that are used to tail or download logs.

$ aio cloudmanager:list-programs
Program Id Name      Enabled
14304      Program 1 true
11454      Program 2 true
11502      Program 3 true

$ aio config:set cloudmanager_programid <PROGRAM ID>

$ aio cloudmanager:list-environments
Environment Id Name            Type  Description
22295          program-3-dev   dev
22310          program-3-prod  prod
22294          program-3-stage stage

$ aio cloudmanager:list-available-log-options <ENVIRONMENT ID>
Environment Id Service    Name
22295          author     aemaccess
22295          author     aemerror
22295          author     aemrequest
22295          publish    aemaccess
22295          publish    aemerror
22295          publish    aemrequest
22295          dispatcher httpdaccess
22295          dispatcher httpderror
22295          dispatcher aemdispatcher

Tailing logs aio-cli-tail-logs

Adobe I/O CLI provides the ability to tail logs in real-time from AEM as a Cloud Service using the tail-logs command. Tailing is useful to watching real-time log activity as actions are performed on the AEM as a Cloud Service environment.

$ aio config:set cloudmanager_programid <PROGRAM ID>
$ aio cloudmanager:tail-logs <ENVIRONMENT ID> <SERVICE> <NAME>

Other command line tools, such as grep can be used in concert with tail-logs to help isolate log statements of interest, for example:

$ aio cloudmanager:tail-logs 12345 author | grep com.example.MySlingModel

… only displays log statements generated from com.example.MySlingModel or contain that string in them.

Downloading logs aio-cli-download-logs

Adobe I/O CLI provides the ability to download logs from AEM as a Cloud Service using the download-logs) command. This provides the same end result as downloading the logs from the Cloud Manager web UI, with the difference being the download-logs command consolidates logs across days, based on the how many days of logs are requested.

$ aio config:set cloudmanager_programid <PROGRAM ID>
$ aio cloudmanager:download-logs <ENVIRONMENT> <SERVICE> <NAME> <DAYS>

Understanding logs

Logs in AEM as a Cloud Service have multiple pods writing log statements into them. Because multiple AEM instances write to the same log file, it is important to understand how to analyze, and reduce noise while debugging. To explain, the following aemerror log snippet is used:

01.01.2020 12:00:00.000 [cm-p12345-e56789-aem-author-abcdefg-1111] *DEBUG* [qtp2078364989-269] com.example.components.impl.ExampleModelImpl Preparing to collect resources
01.01.2020 12:00:01.002 [cm-p12345-e56789-aem-author-abcdefg-2222] *WARN*  [qtp40782847611-87] com.example.services.impl.ExampleServiceImpl Unable to resolve resource [ /content/example ] to a resource. Aborting.
01.01.2020 12:00:02.003 [cm-p12345-e56789-aem-author-abcdefg-1111] *ERROR* [qtp2078364989-269] com.example.components.impl.ExampleModelImpl Unable to collect any resources

Using the Pod Ids, the data point after the date and time, the logs can be collated by Pod, or AEM instance within the service, making it easier to trace and understand code execution.

Pod cm-p12345-e56789-aem-author-abcdefg-1111

01.01.2020 12:00:00.000 [cm-p12345-e56789-aem-author-abcdefg-1111] *DEBUG* [qtp2078364989-269] com.example.components.impl.ExampleModelImpl Preparing to collect resources
01.01.2020 12:00:02.003 [cm-p12345-e56789-aem-author-abcdefg-1111] *ERROR* [qtp2078364989-269] com.example.components.impl.ExampleModelImpl Unable to collect any resources

Pod cm-p12345-e56789-aem-author-abcdefg-2222

01.01.2020 12:00:01.002 [cm-p12345-e56789-aem-author-abcdefg-2222] *WARN*  [qtp2078364989-269] com.example.services.impl.ExampleServiceImpl Unable to resolve resource [ /content/example ] to a resource. Aborting.

Recommended log levels log-levels

Adobe’s general guidance on log levels per AEM as a Cloud Service environment are:

Setting the most appropriate log level for each environment type is with AEM as a Cloud Service, log levels are maintained in code

…and thus, require a deployment to change.

Environment specific variables to set Java log levels

An alternative to setting static well-known Java log levels for each environment is to use AEM as Cloud Service’s environment specific variables to parameterize log levels, allowing the values to be changed dynamically via the Adobe I/O CLI with Cloud Manager plugin.

This requires updating the logging OSGi configurations to use the environment specific variable placeholders. Default values for log levels should be set as per Adobe recommendations. For example:

/apps/example/config/org.apache.sling.commons.log.LogManager.factory.config~example.cfg.json

{
    ...
    "org.apache.sling.commons.log.names": ["com.example"],
    "org.apache.sling.commons.log.level": "$[env:LOG_LEVEL;default=DEBUG]"
    ...
}

This approach has downsides that must be taken into account:

Environment specific variables do not work for Apache web server or Dispatcher log configurations as these are not configured via OSGi configuration.