Custom Fields

Disclaimer: This page describes a feature, that is designed for the use with SAP Cloud Platform Application Logging. If you use this library outside this environment, it will most likely not be needed.

Introduction to Custom Fields

Within the SAP Cloud Platform log messages are indexed into Elasticsearch. They can be searched by any terms (words) appearing in any field of the emitted JSON object. But only certain predefined fields are included as indexed field, which allows filtering and limiting searches on those fields. These predefined fields are known as Context Fields in the library and exposed via LogContext. getContextFieldsKeys() (c.f. LogContext).

Users can provide so called Custom Fields in a nested object within the JSON message. These key-value-pairs will be indexed as keyword fields in Elasticsearch. This allows users to use these fields in custom queries and visualisations to improve there Elasticsearch experience. The library supports two ways to generate Custom Fields:

• Declaration of MDC entries
• Parameters of log messages

Declaration of MDC entries as Custom Fields

SLF4j defines the Mapped Diagnostic Context (MDC) as a map of key-value-pairs to allow tagging common sets of log messages by custom entries. All MDC entries will be attached to the generated JSON object as top-level entries by the logging library. However, they will not be indexed by the SAP Cloud Platform Logging Service, unless they are Context Fields.

To emit MDC fields as Custom Fields the library supports configuration for both logging backends Log4j and Logback. In the standard configuration, such fields will be moved from top-level to the custom_fields nested object within the JSON object. To reduce the message size, such fields will no longer be available at top-level. But there is an additional configuration, to retain the top-level copy. Note, that Context Fields will never be removed from top-level, but only a copy will be created in the custom_fields object.

Log4j Example

The following configuration shows two MDC fields custom-field and retained-field, that are declared as custom fields with the latter one declared to retain a top-level copy.

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="warn" strict="true"
	packages="com.sap.hcp.cf.log4j2.converter,com.sap.hcp.cf.log4j2.layout">
	<Appenders>
		<Console name="STDOUT-JSON" target="SYSTEM_OUT" follow="true">
			<JsonPatternLayout charset="utf-8">
				<customField mdcKeyName="custom-field" />
				<customField mdcKeyName="retained-field" retainOriginal="true" />
			</JsonPatternLayout>
		</Console>
	</Appenders>
	<Loggers>
		<Root level="${LOG_ROOT_LEVEL:-WARN}">
			<AppenderRef ref="STDOUT-JSON" />
		</Root>
		<Logger name="com.sap.hcp.cf" level="INFO" />
	</Loggers>
</Configuration>      

Logback Example

The following configuration shows two MDC fields custom-field and retained-field, that are declared as custom fields with the latter one declared to retain a top-level copy.

<?xml version="1.0" encoding="UTF-8"?>
<configuration debug="false" scan="false">
    <appender name="STDOUT-JSON" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="com.sap.hcp.cf.logback.encoder.JsonEncoder">
        	<customFieldMdcKeyName>custom-field</customFieldMdcKeyName>
        	<customFieldMdcKeyName>retained-field</customFieldMdcKeyName>
        	<retainFieldMdcKeyName>retained-field</retainFieldMdcKeyName>
        </encoder>
    </appender>
    <root level="${LOG_ROOT_LEVEL:-WARN}">
        <appender-ref ref="STDOUT-JSON" />
    </root>
    <logger name="com.sap.hcp.cf" level="INFO" />
</configuration>

Typed Parameter of Log Message

SLF4j allows arbitrary parameters to a log message, usually used for formatting the log message. The library offers a special parameter type CustomField, that will be emitted as Custom Field in the generated log message. An example of this feature looks as follows:

import static com.sap.hcp.cf.logging.common.customfields.CustomField.customField;

// omitted for brevity 

LOGGER.info("Logging with custom field {}", customField("customKey", "customValue"));

Note: This feature should be considered as deprecated. It will be marked as such in upcoming releases of the library.