Merge pull request #34891 from rolfedh/RHDEVDOCS-2677

RHDEVDOCS-2677 Document "Allow storing and querying of structured log…

Author: JStickler

_topic_map.yml

@@ -1871,6 +1871,8 @@ Topics:
 - Name: Forwarding logs to third party systems
   File: cluster-logging-external
   Distros: openshift-enterprise,openshift-origin
+- Name: Enabling JSON logging
+  File: cluster-logging-enabling-json-logging
 - Name: Collecting and storing Kubernetes events
   File: cluster-logging-eventrouter
   Distros: openshift-enterprise,openshift-origin

logging/cluster-logging-enabling-json-logging.adoc

@@ -0,0 +1,16 @@
+:context: cluster-logging-enabling-json-logging
+[id="cluster-logging-enabling-json-logging"]
+= Enabling JSON logging
+include::modules/common-attributes.adoc[]
+
+toc::[]
+
+You can configure the Log Forwarding API to parse JSON strings into a structured object.
+
+include::modules/cluster-logging-json-log-forwarding.adoc[leveloffset=+1]
+include::modules/cluster-logging-configuration-of-json-log-data-for-default-elasticsearch.adoc[leveloffset=+1]
+include::modules/cluster-logging-forwarding-json-logs-to-the-default-elasticsearch.adoc[leveloffset=+1]
+
+.Additional resources
+
+* xref:../logging/cluster-logging-external.adoc#cluster-logging-external[Forwarding logs to third-party systems]

logging/cluster-logging-exported-fields.adoc

@@ -5,11 +5,12 @@ include::modules/common-attributes.adoc[]
 
 toc::[]
 
-The following fields can be present in log records exported by OpenShift Logging system. Although log records are typically formatted as JSON objects, the same data model can be applied to other encodings.
+The following fields can be present in log records exported by OpenShift Logging. Although log records are typically formatted as JSON objects, the same data model can be applied to other encodings.
 
 To search these fields from Elasticsearch and Kibana, use the full dotted field name when searching. For example, with an Elasticsearch */_search URL*, to look for a Kubernetes pod name, use `/_search/q=kubernetes.pod_name:name-of-my-pod`.
 
-// The logging system can forward JSON-formatted log entries to external systems. These log entries are formatted as a fluentd message with extra fields such as `kubernetes`. The fields exported by the logging system and available for searching from Elasticsearch and Kibana are documented at the end of this document.
+// The logging system can parse JSON-formatted log entries to external systems. These log entries are formatted as a fluentd message with extra fields such as `kubernetes`. The fields exported by the logging system and available for searching from Elasticsearch and Kibana are documented at the end of this document.
 
 include::modules/cluster-logging-exported-fields-top-level-fields.adoc[leveloffset=0]
 include::modules/cluster-logging-exported-fields-kubernetes.adoc[leveloffset=0]
+// add modules/cluster-logging-exported-fields-openshift when available

logging/cluster-logging-external.adoc

@@ -1,13 +1,13 @@
 :context: cluster-logging-external
 [id="cluster-logging-external"]
-= Forwarding logs to third party systems
+= Forwarding logs to third-party systems
 include::modules/common-attributes.adoc[]
 
 toc::[]
 
 By default, OpenShift Logging sends container and infrastructure logs to the default internal Elasticsearch log store defined in the `ClusterLogging` custom resource. However, it does not send audit logs to the internal store because it does not provide secure storage. If this default configuration meets your needs, you do not need to configure the Cluster Log Forwarder.
 
-To send logs to other log aggregators, you use the {product-title} Cluster Log Forwarder. This API enables you to send container, infrastructure, and audit logs to specific endpoints within or outside your cluster. In addition, you can send different types of logs to various systems so that various individuals can access each type. You can also enable TLS support to send logs securely, as required by your organization.
+To send logs to other log aggregators, you use the {product-title} Cluster Log Forwarder. This API enables you to send container, infrastructure, and audit logs to specific endpoints within or outside your cluster. In addition, you can send different types of logs to various systems so that various individuals can access each type. You can also enable Transport Layer Security (TLS) support to send logs securely, as required by your organization.
 
 [NOTE]
 ====

logging/cluster-logging.adoc

@@ -68,4 +68,4 @@ For information, see xref:../logging/cluster-logging-eventrouter.adoc#cluster-lo
 
 include::modules/cluster-logging-forwarding-about.adoc[leveloffset=+2]
 
-For information, see xref:../logging/cluster-logging-external.adoc#cluster-logging-external[Forwarding logs to third party systems].
+For information, see xref:../logging/cluster-logging-external.adoc#cluster-logging-external[Forwarding logs to third-party systems].

logging/config/cluster-logging-collector.adoc

@@ -26,4 +26,4 @@ include::modules/cluster-logging-removing-unused-components-if-no-elasticsearch.
 
 .Additional resources
 
-* xref:../../logging/cluster-logging-external.adoc#cluster-logging-external[Forwarding logs to third party systems]
+* xref:../../logging/cluster-logging-external.adoc#cluster-logging-external[Forwarding logs to third-party systems]

modules/cluster-logging-collector-legacy-syslog.adoc

@@ -57,7 +57,7 @@ You can configure the following `syslog` parameters. For more information, see t
 ** `15` or `solaris-cron` for the scheduling daemon
 ** `16`–`23` or `local0` – `local7` for locally used facilities
 * payloadKey: The record field to use as payload for the syslog message.
-* rfc: The RFC to be used for sending log using syslog.
+* rfc: The RFC to be used for sending logs using syslog.
 * severity: The link:https://tools.ietf.org/html/rfc3164#section-4.1.1[syslog severity] to set on outgoing syslog records. The value can be a decimal integer or a case-insensitive keyword:
 ** `0` or `Emergency` for messages indicating the system is unusable
 ** `1` or `Alert` for messages indicating action must be taken immediately
@@ -67,7 +67,7 @@ You can configure the following `syslog` parameters. For more information, see t
 ** `5` or `Notice` for messages indicating normal but significant conditions
 ** `6` or `Informational` for messages indicating informational messages
 ** `7` or `Debug` for messages indicating debug-level messages, the default
-* tag: The record field to use as tag on the syslog message.
+* tag: The record field to use as a tag on the syslog message.
 * trimPrefix: The prefix to remove from the tag.
 
 .Procedure

modules/cluster-logging-collector-log-forward-es.adoc

@@ -45,9 +45,10 @@ spec:
      outputRefs:
      - elasticsearch-secure <9>
      - default <10>
+     parse: json <11>
      labels:
-       logs: application <11>
-   - name: infrastructure-audit-logs <12>
+       myLabel: myValue <12>
+   - name: infrastructure-audit-logs <13>
      inputRefs:
      - infrastructure
      outputRefs:
@@ -65,8 +66,9 @@ spec:
 <8> Specify which log types should be forwarded using that pipeline: `application,` `infrastructure`, or `audit`.
 <9> Specify the output to use with that pipeline for forwarding the logs.
 <10> Optional: Specify the `default` output to send the logs to the internal Elasticsearch instance.
-<11> Optional: One or more labels to add to the logs.
-<12> Optional: Configure multiple outputs to forward logs to other external log aggregators of any supported type:
+<11> Optional: Forward structured JSON log entries as JSON objects in the `structured` field. The log entry must contain valid structured JSON; otherwise, OpenShift Logging removes the `structured` field and instead sends the log entry to the default index, `app-00000x`.
+<12> Optional: One or more labels to add to the logs.
+<13> Optional: Configure multiple outputs to forward logs to other external log aggregtors of any supported type:
 ** Optional. A name to describe the pipeline.
 ** The `inputRefs` is the log type to forward using that pipeline: `application,` `infrastructure`, or `audit`.
 ** The `outputRefs` is the name of the output to use.

modules/cluster-logging-collector-log-forward-fluentd.adoc

@@ -5,7 +5,7 @@
 [id="cluster-logging-collector-log-forward-fluentd_{context}"]
 = Forwarding logs using the Fluentd forward protocol
 
-You can use the Fluentd *forward* protocol to send a copy of your logs to an external log aggregator that you have configured to accept the protocol. You can do this in addition to, or instead of, using the default Elasticsearch log store. You must also configure the external log aggregator to receive log data from {product-title}.
+You can use the Fluentd *forward* protocol to send a copy of your logs to an external log aggregator configured to accept the protocol instead of, or in addition to, the default Elasticsearch log store. You are responsible for configuring the external log aggregator to receive the logs from {product-title}.
 
 To configure log forwarding using the *forward* protocol, create a `ClusterLogForwarder` custom resource (CR) with one or more outputs to the Fluentd servers and pipelines that use those outputs. The Fluentd output can use a TCP (insecure) or TLS (secure TCP) connection.
 
@@ -14,10 +14,6 @@ To configure log forwarding using the *forward* protocol, create a `ClusterLogFo
 Alternately, you can use a config map to forward logs using the *forward* protocols. However, this method is deprecated in {product-title} and will be removed in a future release.
 ====
 
-.Prerequisites
-
-*  An external log aggregator that is configured to receive log data from {product-title} using the Fluentd *forward* protocol.
-
 .Procedure
 
 . Create a `ClusterLogForwarder` CR YAML file similar to the following:
@@ -47,9 +43,10 @@ spec:
      outputRefs:
      - fluentd-server-secure <9>
      - default <10>
+     parse: json <11>
      labels:
-       clusterId: C1234 <11>
-   - name: forward-to-fluentd-insecure <12>
+       clusterId: C1234 <12>
+   - name: forward-to-fluentd-insecure <13>
      inputRefs:
      - infrastructure
      outputRefs:
@@ -67,8 +64,9 @@ spec:
 <8> Specify which log types should be forwarded using that pipeline: `application,` `infrastructure`, or `audit`.
 <9> Specify the output to use  with that pipeline for forwarding the logs.
 <10> Optional. Specify the `default` output to forward logs to the internal Elasticsearch instance.
-<11> Optional. One or more labels to add to the logs.
-<12> Optional: Configure multiple outputs to forward logs to other external log aggregators of any supported type:
+<11> Optional: Forward structured JSON log entries as JSON objects in the `structured` field. The log entry must contain valid structured JSON; otherwise, OpenShift Logging removes the `structured` field and instead sends the log entry to the default index, `app-00000x`.
+<12> Optional. One or more labels to add to the logs.
+<13> Optional: Configure multiple outputs to forward logs to other external log aggregtors of any supported type:
 ** Optional. A name to describe the pipeline.
 ** The `inputRefs` is the log type to forward using that pipeline: `application,` `infrastructure`, or `audit`.
 ** The `outputRefs` is the name of the output to use.

modules/cluster-logging-collector-log-forward-kafka.adoc

@@ -41,9 +41,10 @@ spec:
      - application
      outputRefs: <10>
      - app-logs
+     parse: json <11>
      labels:
-       logType: application <11>
-   - name: infra-topic <12>
+       logType: application <12>
+   - name: infra-topic <13>
      inputRefs:
      - infrastructure
      outputRefs:
@@ -55,7 +56,7 @@ spec:
      - audit
      outputRefs:
      - audit-logs
-     - default <13>
+     - default <14>
      labels:
        logType: audit
 ----
@@ -69,15 +70,16 @@ spec:
 <8> Optional: Specify a name for the pipeline.
 <9> Specify which log types should be forwarded using that pipeline: `application,` `infrastructure`, or `audit`.
 <10> Specify the output to use with that pipeline for forwarding the logs.
-<11> Optional: One or more labels to add to the logs.
-<12> Optional: Configure multiple outputs to forward logs to other external log aggregators of any supported type:
+<11> Optional: Forward structured JSON log entries as JSON objects in the `structured` field. The log entry must contain valid structured JSON; otherwise, OpenShift Logging removes the `structured` field and instead sends the log entry to the default index, `app-00000x`.
+<12> Optional: One or more labels to add to the logs.
+<13> Optional: Configure multiple outputs to forward logs to other external log aggregtors of any supported type:
 ** Optional. A name to describe the pipeline.
 ** The `inputRefs` is the log type to forward using that pipeline: `application,` `infrastructure`, or `audit`.
 ** The `outputRefs` is the name of the output to use.
 ** Optional: One or more labels to add to the logs.
-<13> Optional: Specify `default` to forward logs to the internal Elasticsearch instance.
+<14> Optional: Specify `default` to forward logs to the internal Elasticsearch instance.
 
-. Optional: To forward a single output to multiple kafka brokers, specify an array of kafka brokers as shown in this example:
+. Optional: To forward a single output to multiple Kafka brokers, specify an array of Kafka brokers as shown in this example:
 +
 [source,yaml]
 ----