Skip to content

Commit 1cf462d

Browse files
abrennan89abrennan89
committed
OBSDOCS-206: Add multi ClusterLogForwarder docs
1 parent 33ae51d commit 1cf462d

23 files changed

+285
-56
lines changed

‎_topic_maps/_topic_map.yml

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2544,8 +2544,10 @@ Topics:
25442544
- Name: Log collection and forwarding
25452545
Dir: log_collection_forwarding
25462546
Topics:
2547-
- Name: About log forwarding
2547+
- Name: About log collection and forwarding
25482548
File: log-forwarding
2549+
- Name: Log output types
2550+
File: logging-output-types
25492551
- Name: Enabling JSON log forwarding
25502552
File: cluster-logging-enabling-json-logging
25512553
- Name: Configuring the logging collector
@@ -2573,6 +2575,8 @@ Topics:
25732575
- Name: Exported fields
25742576
File: cluster-logging-exported-fields
25752577
Distros: openshift-enterprise,openshift-origin
2578+
- Name: Glossary
2579+
File: logging-common-terms
25762580
---
25772581
Name: Monitoring
25782582
Dir: monitoring

‎_topic_maps/_topic_map_osd.yml

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -577,8 +577,10 @@ Topics:
577577
- Name: Log collection and forwarding
578578
Dir: log_collection_forwarding
579579
Topics:
580-
- Name: About log forwarding
580+
- Name: About log collection and forwarding
581581
File: log-forwarding
582+
- Name: Log output types
583+
File: logging-output-types
582584
- Name: Enabling JSON log forwarding
583585
File: cluster-logging-enabling-json-logging
584586
- Name: Configuring the logging collector
@@ -604,6 +606,8 @@ Topics:
604606
File: cluster-logging-uninstall
605607
- Name: Exported fields
606608
File: cluster-logging-exported-fields
609+
- Name: Glossary
610+
File: logging-common-terms
607611
---
608612
Name: Monitoring
609613
Dir: monitoring

‎_topic_maps/_topic_map_rosa.yml

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -731,8 +731,10 @@ Topics:
731731
- Name: Log collection and forwarding
732732
Dir: log_collection_forwarding
733733
Topics:
734-
- Name: About log forwarding
734+
- Name: About log collection and forwarding
735735
File: log-forwarding
736+
- Name: Log output types
737+
File: logging-output-types
736738
- Name: Enabling JSON log forwarding
737739
File: cluster-logging-enabling-json-logging
738740
- Name: Configuring the logging collector
@@ -758,6 +760,8 @@ Topics:
758760
File: cluster-logging-uninstall
759761
- Name: Exported fields
760762
File: cluster-logging-exported-fields
763+
- Name: Glossary
764+
File: logging-common-terms
761765
---
762766
Name: Monitoring
763767
Dir: monitoring

‎logging/cluster-logging.adoc

Lines changed: 13 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -2,33 +2,37 @@
22
include::_attributes/common-attributes.adoc[]
33
include::_attributes/attributes-openshift-dedicated.adoc[]
44
[id="cluster-logging"]
5-
= Understanding the {logging-title}
5+
= About Logging
66
:context: cluster-logging
77

88
toc::[]
99

10-
ifdef::openshift-enterprise,openshift-rosa,openshift-dedicated,openshift-webscale,openshift-origin[]
11-
As a cluster administrator, you can deploy the {logging} to aggregate all the logs from your {product-title} cluster, such as node system audit logs, application container logs, and infrastructure logs. The {logging} aggregates these logs from throughout your cluster and stores them in a default log store. You can xref:../logging/cluster-logging-visualizer.adoc#cluster-logging-visualizer[use the Kibana web console to visualize log data].
10+
As a cluster administrator, you can deploy {logging} on an {product-title} cluster, and use it to collect and aggregate node system audit logs, application container logs, and infrastructure logs. You can forward logs to your chosen log outputs, including on-cluster, Red{nbsp}Hat managed log storage. You can also visualize your log data in the {product-title} web console, or xref:../logging/cluster-logging-visualizer.adoc#cluster-logging-visualizer[the Kibana web console], depending on your deployed log storage solution.
11+
12+
[IMPORTANT]
13+
====
14+
The Kibana web console is now deprecated and will be removed in a future logging release.
15+
====
16+
17+
{product-title} cluster administrators can deploy the {logging} by using Operators. For information, see xref:../logging/cluster-logging-deploying.adoc#cluster-logging-deploying[Installing the {logging-title}].
18+
19+
The Operators are responsible for deploying, upgrading, and maintaining the {logging}. After the Operators are installed, you can create a `ClusterLogging` custom resource (CR) to schedule {logging} pods and other resources necessary to support the {logging}. You can also create a `ClusterLogForwarder` CR to specify which logs are collected, how they are transformed, and where they are forwarded to.
1220

1321
[NOTE]
1422
====
1523
Because the internal {product-title} Elasticsearch log store does not provide secure storage for audit logs, audit logs are not stored in the internal Elasticsearch instance by default. If you want to send the audit logs to the default internal Elasticsearch log store, for example to view the audit logs in Kibana, you must use the Log Forwarding API as described in xref:../logging/config/cluster-logging-log-store.adoc#cluster-logging-elasticsearch-audit_cluster-logging-log-store[Forward audit logs to the log store].
1624
====
17-
endif::[]
1825

1926
include::modules/logging-architecture-overview.adoc[leveloffset=+1]
2027

28+
include::modules/cluster-logging-about.adoc[leveloffset=+1]
29+
2130
ifdef::openshift-rosa,openshift-dedicated[]
2231
include::modules/cluster-logging-cloudwatch.adoc[leveloffset=+1]
2332
.Next steps
2433
* See xref:../logging/log_collection_forwarding/log-forwarding.adoc#cluster-logging-collector-log-forward-cloudwatch_log-forwarding[Forwarding logs to Amazon CloudWatch] for instructions.
2534
endif::[]
2635

27-
include::modules/logging-common-terms.adoc[leveloffset=+1]
28-
include::modules/cluster-logging-about.adoc[leveloffset=+1]
29-
30-
For information, see xref:../logging/cluster-logging-deploying.adoc#cluster-logging-deploying[Installing the {logging-title}].
31-
3236
include::modules/cluster-logging-json-logging-about.adoc[leveloffset=+2]
3337

3438
include::modules/cluster-logging-collecting-storing-kubernetes-events.adoc[leveloffset=+2]

‎logging/config/cluster-logging-configuring-cr.adoc

Lines changed: 31 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -8,9 +8,35 @@ toc::[]
88

99
To configure {logging-title} you customize the `ClusterLogging` custom resource (CR).
1010

11-
// The following include statements pull in the module files that comprise
12-
// the assembly. Include any combination of concept, procedure, or reference
13-
// modules required to cover the user story. You can also include other
14-
// assemblies.
15-
1611
include::modules/cluster-logging-about-crd.adoc[leveloffset=+1]
12+
13+
////
14+
// collecting this information here for a future PR
15+
16+
If you want to specify collector resources or scheduling, you must create a `ClusterLogging` CR:
17+
18+
.ClusterLogging resource example
19+
[source,yaml]
20+
----
21+
apiVersion: "logging.openshift.io/v1"
22+
kind: "ClusterLogging"
23+
metadata:
24+
name: audit-collector <1>
25+
namespace: openshift-kube-apiserver <2>
26+
spec:
27+
collection:
28+
type: "vector" <3>
29+
resources:
30+
limits:
31+
memory: 2G
32+
# ...
33+
----
34+
<1> The name of the `ClusterLogging` CR must be the same as the `ClusterLogForwarder` CR.
35+
<2> The namespace of the `ClusterLogging` CR must be the same as the `ClusterLogForwarder` CR.
36+
<3> The collector type that you want to use. This example uses the Vector collector.
37+
38+
[NOTE]
39+
====
40+
The relevant `spec` fields for this CR in multiple log forwarder mode are the `managmentState` and `collection` fields. All other `spec` fields are ignored.
41+
====
42+
////

‎logging/log_collection_forwarding/_attributes

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
../../_attributes/

‎logging/log_collection_forwarding/images

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
../../images/

‎logging/log_collection_forwarding/log-forwarding.adoc

Lines changed: 28 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -2,42 +2,51 @@
22
include::_attributes/common-attributes.adoc[]
33
include::_attributes/attributes-openshift-dedicated.adoc[]
44
[id="log-forwarding"]
5-
= About log forwarding
5+
= About log collection and forwarding
66
:context: log-forwarding
77

88
toc::[]
99

10-
By default, the {logging} sends container and infrastructure logs to the default internal log store defined in the `ClusterLogging` custom resource. However, it does not send audit logs to the internal store because it does not provide secure storage. If this default configuration meets your needs, you do not need to configure the Cluster Log Forwarder.
10+
Administrators can create `ClusterLogForwarder` resources that specify which logs are collected, how they are transformed, and where they are forwarded to.
1111

12-
To send logs to other log aggregators, you use the {product-title} Cluster Log Forwarder. This API enables you to send container, infrastructure, and audit logs to specific endpoints within or outside your cluster. In addition, you can send different types of logs to various systems so that various individuals can access each type. You can also enable Transport Layer Security (TLS) support to send logs securely, as required by your organization.
12+
`ClusterLogForwarder` resources can be used up to forward container, infrastructure, and audit logs to specific endpoints within or outside of a cluster. Transport Layer Security (TLS) is supported so that log forwarders can be configured to send logs securely.
1313

14-
[NOTE]
15-
====
16-
To send audit logs to the default internal Elasticsearch log store, use the Cluster Log Forwarder as described in xref:../../logging/config/cluster-logging-log-store.adoc#cluster-logging-elasticsearch-audit_cluster-logging-log-store[Forward audit logs to the log store].
17-
====
14+
Administrators can also authorize RBAC permissions that define which service accounts and users can access and forward which types of logs.
1815

19-
When you forward logs externally, the {logging} creates or modifies a Fluentd config map to send logs using your desired protocols. You are responsible for configuring the protocol on the external log aggregator.
16+
////
17+
include::modules/log-forwarding-modes.adoc[leveloffset=+1]
2018

21-
// unused files - either include or delete
22-
// cluster-logging-log-forwarding-disable.adoc
19+
[id="log-forwarding-enabling-multi-clf-mode"]
20+
== Enabling multi log forwarder mode for a cluster
2321

24-
include::modules/cluster-logging-collector-log-forwarding-about.adoc[leveloffset=+1]
22+
To use multi log forwarder mode, you must create a service account and cluster role bindings for that service account. You can then reference the service account in the `ClusterLogForwarder` resource to control access permissions.
2523

26-
include::modules/cluster-logging-forwarding-separate-indices.adoc[leveloffset=+1]
24+
include::modules/log-collection-rbac-permissions.adoc[leveloffset=+2]
2725

28-
include::modules/cluster-logging-collector-log-forwarding-supported-plugins-5-1.adoc[leveloffset=+1]
26+
[role="_additional-resources"]
27+
.Additional resources
28+
ifdef::openshift-enterprise[]
29+
* xref:../../authentication/using-rbac.adoc#using-rbac[Using RBAC to define and apply permissions]
30+
* xref:../../authentication/using-service-accounts-in-applications.adoc#using-service-accounts-in-applications[Using service accounts in applications]
31+
endif::[]
32+
* link:https://kubernetes.io/docs/reference/access-authn-authz/rbac/[Using RBAC Authorization Kubernetes documentation]
2933

30-
include::modules/cluster-logging-collector-log-forwarding-supported-plugins-5-2.adoc[leveloffset=+1]
34+
include::modules/logging-create-clf.adoc[leveloffset=+1]
35+
////
3136
32-
include::modules/cluster-logging-collector-log-forwarding-supported-plugins-5-3.adoc[leveloffset=+1]
37+
[id="log-forwarding-audit-logs"]
38+
== Sending audit logs to the internal log store
3339
34-
include::modules/cluster-logging-collector-log-forwarding-supported-plugins-5-4.adoc[leveloffset=+1]
40+
By default, the {logging} sends container and infrastructure logs to the default internal log store defined in the `ClusterLogging` custom resource. However, it does not send audit logs to the internal store because it does not provide secure storage. If this default configuration meets your needs, you do not need to configure the Cluster Log Forwarder.
3541
36-
include::modules/cluster-logging-collector-log-forwarding-supported-plugins-5-5.adoc[leveloffset=+1]
42+
[NOTE]
43+
====
44+
To send audit logs to the internal Elasticsearch log store, use the Cluster Log Forwarder as described in xref:../../logging/config/cluster-logging-log-store.adoc#cluster-logging-elasticsearch-audit_cluster-logging-log-store[Forward audit logs to the log store].
45+
====
3746
38-
include::modules/cluster-logging-collector-log-forwarding-supported-plugins-5-6.adoc[leveloffset=+1]
47+
include::modules/cluster-logging-collector-log-forwarding-about.adoc[leveloffset=+1]
3948
40-
include::modules/cluster-logging-collector-log-forwarding-supported-plugins-5-7.adoc[leveloffset=+1]
49+
include::modules/cluster-logging-forwarding-separate-indices.adoc[leveloffset=+1]
4150
4251
include::modules/cluster-logging-collector-log-forward-es.adoc[leveloffset=+1]
4352

‎logging/log_collection_forwarding/logging-output-types.adoc

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
:_content-type: ASSEMBLY
2+
include::_attributes/common-attributes.adoc[]
3+
include::_attributes/attributes-openshift-dedicated.adoc[]
4+
[id="logging-output-types"]
5+
= Log output types
6+
:context: logging-output-types
7+
8+
toc::[]
9+
10+
Log outputs specified in the `ClusterLogForwarder` CR can be any of the following types:
11+
12+
`default`:: The on-cluster, Red{nbsp}Hat managed log store. You are not required to configure the default output.
13+
+
14+
[NOTE]
15+
====
16+
If you configure a `default` output, you receive an error message, because the `default` output name is reserved for referencing the on-cluster, Red{nbsp}Hat managed log store.
17+
====
18+
`loki`:: Loki, a horizontally scalable, highly available, multi-tenant log aggregation system.
19+
`kafka`:: A Kafka broker. The `kafka` output can use a TCP or TLS connection.
20+
`elasticsearch`:: An external Elasticsearch instance. The `elasticsearch` output can use a TLS connection.
21+
`fluentdForward`:: An external log aggregation solution that supports Fluentd. This option uses the Fluentd *forward* protocols. The `fluentForward` output can use a TCP or TLS connection and supports shared-key authentication by providing a *shared_key* field in a secret. Shared-key authentication can be used with or without TLS.
22+
+
23+
[IMPORTANT]
24+
====
25+
The `fluentdForward` output is only supported if you are using the Fluentd collector. It is not supported if you are using the Vector collector. If you are using the Vector collector, you can forward logs to Fluentd by using the `http` output.
26+
====
27+
`syslog`:: An external log aggregation solution that supports the syslog link:https://tools.ietf.org/html/rfc3164[RFC3164] or link:https://tools.ietf.org/html/rfc5424[RFC5424] protocols. The `syslog` output can use a UDP, TCP, or TLS connection.
28+
`cloudwatch`:: Amazon CloudWatch, a monitoring and log storage service hosted by Amazon Web Services (AWS).
29+
30+
// supported outputs by version
31+
include::modules/cluster-logging-collector-log-forwarding-supported-plugins-5-7.adoc[leveloffset=+1]
32+
include::modules/cluster-logging-collector-log-forwarding-supported-plugins-5-6.adoc[leveloffset=+1]
33+
include::modules/cluster-logging-collector-log-forwarding-supported-plugins-5-5.adoc[leveloffset=+1]
34+
include::modules/cluster-logging-collector-log-forwarding-supported-plugins-5-4.adoc[leveloffset=+1]
35+
include::modules/cluster-logging-collector-log-forwarding-supported-plugins-5-3.adoc[leveloffset=+1]
36+
include::modules/cluster-logging-collector-log-forwarding-supported-plugins-5-2.adoc[leveloffset=+1]
37+
include::modules/cluster-logging-collector-log-forwarding-supported-plugins-5-1.adoc[leveloffset=+1]

‎logging/log_collection_forwarding/modules

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
../../modules/

0 commit comments

Comments
 (0)