Skip to content

Commit ba22603

Browse files
committed
OBSDOCS-134: Restructure log forwarding content
1 parent 4917767 commit ba22603

18 files changed

+241
-202
lines changed

_topic_maps/_topic_map.yml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2587,10 +2587,14 @@ Topics:
25872587
File: logging-output-types
25882588
- Name: Enabling JSON log forwarding
25892589
File: cluster-logging-enabling-json-logging
2590+
- Name: Configuring log forwarding
2591+
File: configuring-log-forwarding
25902592
- Name: Configuring the logging collector
25912593
File: cluster-logging-collector
25922594
- Name: Collecting and storing Kubernetes events
25932595
File: cluster-logging-eventrouter
2596+
- Name: Troubleshooting log forwarding
2597+
File: log-forwarding-troubleshooting
25942598
- Name: Log storage
25952599
Dir: log_storage
25962600
Topics:

_topic_maps/_topic_map_osd.yml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1072,10 +1072,14 @@ Topics:
10721072
File: logging-output-types
10731073
- Name: Enabling JSON log forwarding
10741074
File: cluster-logging-enabling-json-logging
1075+
- Name: Configuring log forwarding
1076+
File: configuring-log-forwarding
10751077
- Name: Configuring the logging collector
10761078
File: cluster-logging-collector
10771079
- Name: Collecting and storing Kubernetes events
10781080
File: cluster-logging-eventrouter
1081+
- Name: Troubleshooting log forwarding
1082+
File: log-forwarding-troubleshooting
10791083
- Name: Log storage
10801084
Dir: log_storage
10811085
Topics:

_topic_maps/_topic_map_rosa.yml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1302,10 +1302,14 @@ Topics:
13021302
File: logging-output-types
13031303
- Name: Enabling JSON log forwarding
13041304
File: cluster-logging-enabling-json-logging
1305+
- Name: Configuring log forwarding
1306+
File: configuring-log-forwarding
13051307
- Name: Configuring the logging collector
13061308
File: cluster-logging-collector
13071309
- Name: Collecting and storing Kubernetes events
13081310
File: cluster-logging-eventrouter
1311+
- Name: Troubleshooting log forwarding
1312+
File: log-forwarding-troubleshooting
13091313
- Name: Log storage
13101314
Dir: log_storage
13111315
Topics:

adding_service_cluster/adding-service.adoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,5 +25,5 @@ include::modules/deleting-service.adoc[leveloffset=+1]
2525
ifdef::openshift-rosa[]
2626
[role="_additional-resources"]
2727
== Additional resources
28-
* xref:../logging/log_collection_forwarding/log-forwarding.adoc#cluster-logging-collector-log-forward-cloudwatch_log-forwarding[Forwarding logs to Amazon CloudWatch]
28+
* xref:../logging/log_collection_forwarding/configuring-log-forwarding.adoc#cluster-logging-collector-log-forward-cloudwatch_configuring-log-forwarding[Forwarding logs to Amazon CloudWatch]
2929
endif::[]

adding_service_cluster/rosa-available-services.adoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ include::modules/aws-cloudwatch.adoc[leveloffset=+1]
1616
.Additional resources
1717

1818
* link:https://aws.amazon.com/cloudwatch/[Amazon CloudWatch product information]
19-
* xref:../logging/log_collection_forwarding/log-forwarding.adoc#cluster-logging-collector-log-forward-cloudwatch_log-forwarding[Forwarding logs to Amazon CloudWatch]
19+
* xref:../logging/log_collection_forwarding/configuring-log-forwarding.adoc#cluster-logging-collector-log-forward-cloudwatch_configuring-log-forwarding[Forwarding logs to Amazon CloudWatch]
2020
2121
include::modules/osd-rhoam.adoc[leveloffset=+1]
2222

logging/cluster-logging.adoc

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ include::modules/cluster-logging-about.adoc[leveloffset=+1]
3131
ifdef::openshift-rosa,openshift-dedicated[]
3232
include::modules/cluster-logging-cloudwatch.adoc[leveloffset=+1]
3333
.Next steps
34-
* See xref:../logging/log_collection_forwarding/log-forwarding.adoc#cluster-logging-collector-log-forward-cloudwatch_log-forwarding[Forwarding logs to Amazon CloudWatch] for instructions.
34+
* See xref:../logging/log_collection_forwarding/configuring-log-forwarding.adoc#cluster-logging-collector-log-forward-cloudwatch_configuring-log-forwarding[Forwarding logs to Amazon CloudWatch] for instructions.
3535
endif::[]
3636

3737
include::modules/cluster-logging-json-logging-about.adoc[leveloffset=+2]

logging/log_collection_forwarding/cluster-logging-enabling-json-logging.adoc

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ You can configure the Log Forwarding API to parse JSON strings into a structured
1111
include::modules/cluster-logging-json-log-forwarding.adoc[leveloffset=+1]
1212
include::modules/cluster-logging-configuration-of-json-log-data-for-default-elasticsearch.adoc[leveloffset=+1]
1313
include::modules/cluster-logging-forwarding-json-logs-to-the-default-elasticsearch.adoc[leveloffset=+1]
14+
include::modules/cluster-logging-forwarding-separate-indices.adoc[leveloffset=+1]
1415

1516
[role="_additional-resources"]
1617
.Additional resources
Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
:_mod-docs-content-type: ASSEMBLY
2+
include::_attributes/common-attributes.adoc[]
3+
include::_attributes/attributes-openshift-dedicated.adoc[]
4+
[id="configuring-log-forwarding"]
5+
= Configuring log forwarding
6+
:context: configuring-log-forwarding
7+
8+
toc::[]
9+
10+
By default, the {logging} sends container and infrastructure logs to the default internal log store defined in the `ClusterLogging` custom resource. However, it does not send audit logs to the internal store because it does not provide secure storage. If this default configuration meets your needs, you do not need to configure the Cluster Log Forwarder.
11+
12+
[NOTE]
13+
====
14+
To send audit logs to the internal Elasticsearch log store, use the Cluster Log Forwarder as described in xref:../../logging/log_storage/logging-config-es-store.adoc#cluster-logging-elasticsearch-audit_logging-config-es-store[Forwarding audit logs to the log store].
15+
====
16+
17+
include::modules/cluster-logging-collector-log-forwarding-about.adoc[leveloffset=+1]
18+
19+
include::modules/logging-create-clf.adoc[leveloffset=+1]
20+
21+
include::modules/logging-multiline-except.adoc[leveloffset=+1]
22+
23+
ifndef::openshift-rosa[]
24+
include::modules/cluster-logging-collector-log-forward-gcp.adoc[leveloffset=+1]
25+
endif::openshift-rosa[]
26+
27+
include::modules/logging-forward-splunk.adoc[leveloffset=+1]
28+
29+
include::modules/logging-http-forward.adoc[leveloffset=+1]
30+
31+
include::modules/cluster-logging-collector-log-forward-project.adoc[leveloffset=+1]
32+
33+
include::modules/cluster-logging-collector-log-forward-logs-from-application-pods.adoc[leveloffset=+1]
34+
35+
[role="_additional-resources"]
36+
.Additional resources
37+
38+
ifdef::openshift-enterprise,openshift-origin[]
39+
* xref:../../networking/ovn_kubernetes_network_provider/logging-network-policy.adoc#logging-network-policy[Logging for egress firewall and network policy rules]
40+
endif::[]
41+
ifdef::openshift-rosa,openshift-dedicated[]
42+
* link:https://docs.openshift.com/container-platform/latest/networking/ovn_kubernetes_network_provider/logging-network-policy.html#logging-network-policy[Logging for egress firewall and network policy rules]
43+
endif::[]
44+
45+
include::modules/cluster-logging-collector-log-forward-loki.adoc[leveloffset=+1]
46+
47+
[role="_additional-resources"]
48+
.Additional resources
49+
* link:https://grafana.com/docs/loki/latest/configuration/[Configuring Loki server]
50+
51+
include::modules/cluster-logging-collector-log-forward-es.adoc[leveloffset=+1]
52+
53+
include::modules/cluster-logging-collector-log-forward-fluentd.adoc[leveloffset=+1]
54+
55+
include::modules/cluster-logging-collector-log-forward-syslog.adoc[leveloffset=+1]
56+
57+
include::modules/cluster-logging-collector-log-forward-kafka.adoc[leveloffset=+1]
58+
59+
// Cloudwatch docs
60+
include::modules/cluster-logging-collector-log-forward-cloudwatch.adoc[leveloffset=+1]
61+
include::modules/cluster-logging-collector-log-forward-secret-cloudwatch.adoc[leveloffset=+1]
62+
63+
ifdef::openshift-rosa[]
64+
include::modules/rosa-cluster-logging-collector-log-forward-sts-cloudwatch.adoc[leveloffset=+1]
65+
endif::[]
66+
67+
ifdef::openshift-enterprise,openshift-origin,openshift-dedicated[]
68+
include::modules/cluster-logging-collector-log-forward-sts-cloudwatch.adoc[leveloffset=+1]
69+
endif::[]
70+
71+
[role="_additional-resources"]
72+
.Additional resources
73+
* link:https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html[AWS STS API Reference]
Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
:_mod-docs-content-type: ASSEMBLY
2+
include::_attributes/common-attributes.adoc[]
3+
include::_attributes/attributes-openshift-dedicated.adoc[]
4+
[id="log-forwarding-troubleshooting"]
5+
= Troubleshooting log forwarding
6+
:context: log-forwarding-troubleshooting
7+
8+
toc::[]
9+
10+
include::modules/redeploying-fluentd-pods.adoc[leveloffset=+1]
11+
include::modules/loki-rate-limit-errors.adoc[leveloffset=+1]

logging/log_collection_forwarding/log-forwarding.adoc

Lines changed: 3 additions & 192 deletions
Original file line numberDiff line numberDiff line change
@@ -26,10 +26,10 @@ Administrators can create `ClusterLogForwarder` resources that specify which log
2626

2727
Administrators can also authorize RBAC permissions that define which service accounts and users can access and forward which types of logs.
2828

29-
include::modules/log-forwarding-implementations.adoc[leveloffset=+1]
29+
include::modules/log-forwarding-implementations.adoc[leveloffset=+2]
3030

3131
[id="log-forwarding-enabling-multi-clf-feature"]
32-
== Enabling the multi log forwarder feature for a cluster
32+
=== Enabling the multi log forwarder feature for a cluster
3333

3434
To use the multi log forwarder feature, you must create a service account and cluster role bindings for that service account. You can then reference the service account in the `ClusterLogForwarder` resource to control access permissions.
3535

@@ -38,7 +38,7 @@ To use the multi log forwarder feature, you must create a service account and cl
3838
In order to support multi log forwarding in additional namespaces other than the `openshift-logging` namespace, you must xref:../../logging/cluster-logging-upgrading.adoc#logging-operator-upgrading-all-ns_cluster-logging-upgrading[update the {clo} to watch all namespaces]. This functionality is supported by default in new {clo} version 5.8 installations.
3939
====
4040

41-
include::modules/log-collection-rbac-permissions.adoc[leveloffset=+2]
41+
include::modules/log-collection-rbac-permissions.adoc[leveloffset=+3]
4242

4343
[role="_additional-resources"]
4444
.Additional resources
@@ -47,192 +47,3 @@ ifdef::openshift-enterprise[]
4747
* xref:../../authentication/using-service-accounts-in-applications.adoc#using-service-accounts-in-applications[Using service accounts in applications]
4848
endif::[]
4949
* link:https://kubernetes.io/docs/reference/access-authn-authz/rbac/[Using RBAC Authorization Kubernetes documentation]
50-
51-
include::modules/logging-create-clf.adoc[leveloffset=+1]
52-
53-
include::modules/logging-multiline-except.adoc[leveloffset=+1]
54-
55-
[id="log-forwarding-audit-logs"]
56-
== Sending audit logs to the internal log store
57-
58-
By default, the {logging} sends container and infrastructure logs to the default internal log store defined in the `ClusterLogging` custom resource. However, it does not send audit logs to the internal store because it does not provide secure storage. If this default configuration meets your needs, you do not need to configure the Cluster Log Forwarder.
59-
60-
[NOTE]
61-
====
62-
To send audit logs to the internal Elasticsearch log store, use the Cluster Log Forwarder as described in xref:../../logging/log_storage/logging-config-es-store.adoc#cluster-logging-elasticsearch-audit_logging-config-es-store[Forwarding audit logs to the log store].
63-
====
64-
65-
include::modules/cluster-logging-collector-log-forwarding-about.adoc[leveloffset=+1]
66-
67-
include::modules/cluster-logging-forwarding-separate-indices.adoc[leveloffset=+1]
68-
69-
include::modules/cluster-logging-collector-log-forward-es.adoc[leveloffset=+1]
70-
71-
include::modules/cluster-logging-collector-log-forward-fluentd.adoc[leveloffset=+1]
72-
73-
include::modules/cluster-logging-collector-log-forward-syslog.adoc[leveloffset=+1]
74-
75-
include::modules/cluster-logging-collector-log-forward-kafka.adoc[leveloffset=+1]
76-
77-
include::modules/cluster-logging-collector-log-forward-cloudwatch.adoc[leveloffset=+1]
78-
79-
ifdef::openshift-rosa[]
80-
include::modules/rosa-cluster-logging-collector-log-forward-sts-cloudwatch.adoc[leveloffset=+2]
81-
endif::[]
82-
83-
ifdef::openshift-enterprise,openshift-origin,openshift-dedicated[]
84-
[id="cluster-logging-collector-log-forward-sts-cloudwatch_{context}"]
85-
=== Forwarding logs to Amazon CloudWatch from STS enabled clusters
86-
87-
For clusters with AWS Security Token Service (STS) enabled, you can create an AWS service account manually or create a credentials request by using the
88-
ifdef::openshift-enterprise,openshift-origin[]
89-
xref:../../authentication/managing_cloud_provider_credentials/about-cloud-credential-operator.adoc[Cloud Credential Operator(CCO)]
90-
endif::[]
91-
ifdef::openshift-dedicated[]
92-
link:https://docs.openshift.com/container-platform/latest/authentication/managing_cloud_provider_credentials/about-cloud-credential-operator.html[Cloud Credential Operator(CCO)]
93-
endif::[]
94-
utility `ccoctl`.
95-
96-
.Prerequisites
97-
98-
* {logging-title-uc}: 5.5 and later
99-
100-
.Procedure
101-
102-
. Create a `CredentialsRequest` custom resource YAML by using the template below:
103-
+
104-
.CloudWatch credentials request template
105-
[source,yaml]
106-
----
107-
apiVersion: cloudcredential.openshift.io/v1
108-
kind: CredentialsRequest
109-
metadata:
110-
name: <your_role_name>-credrequest
111-
namespace: openshift-cloud-credential-operator
112-
spec:
113-
providerSpec:
114-
apiVersion: cloudcredential.openshift.io/v1
115-
kind: AWSProviderSpec
116-
statementEntries:
117-
- action:
118-
- logs:PutLogEvents
119-
- logs:CreateLogGroup
120-
- logs:PutRetentionPolicy
121-
- logs:CreateLogStream
122-
- logs:DescribeLogGroups
123-
- logs:DescribeLogStreams
124-
effect: Allow
125-
resource: arn:aws:logs:*:*:*
126-
secretRef:
127-
name: <your_role_name>
128-
namespace: openshift-logging
129-
serviceAccountNames:
130-
- logcollector
131-
----
132-
+
133-
. Use the `ccoctl` command to create a role for AWS using your `CredentialsRequest` CR. With the `CredentialsRequest` object, this `ccoctl` command creates an IAM role with a trust policy that is tied to the specified OIDC identity provider, and a permissions policy that grants permissions to perform operations on CloudWatch resources. This command also creates a YAML configuration file in `/<path_to_ccoctl_output_dir>/manifests/openshift-logging-<your_role_name>-credentials.yaml`. This secret file contains the `role_arn` key/value used during authentication with the AWS IAM identity provider.
134-
+
135-
[source,terminal]
136-
----
137-
$ ccoctl aws create-iam-roles \
138-
--name=<name> \
139-
--region=<aws_region> \
140-
--credentials-requests-dir=<path_to_directory_with_list_of_credentials_requests>/credrequests \
141-
--identity-provider-arn=arn:aws:iam::<aws_account_id>:oidc-provider/<name>-oidc.s3.<aws_region>.amazonaws.com <1>
142-
----
143-
<1> <name> is the name used to tag your cloud resources and should match the name used during your STS cluster install
144-
+
145-
. Apply the secret created:
146-
[source,terminal]
147-
+
148-
----
149-
$ oc apply -f output/manifests/openshift-logging-<your_role_name>-credentials.yaml
150-
----
151-
+
152-
. Create or edit a `ClusterLogForwarder` custom resource:
153-
+
154-
[source,yaml]
155-
----
156-
apiVersion: logging.openshift.io/v1
157-
kind: ClusterLogForwarder
158-
metadata:
159-
name: <log_forwarder_name> <1>
160-
namespace: <log_forwarder_namespace> <2>
161-
spec:
162-
serviceAccountName: clf-collector <3>
163-
outputs:
164-
- name: cw <4>
165-
type: cloudwatch <5>
166-
cloudwatch:
167-
groupBy: logType <6>
168-
groupPrefix: <group prefix> <7>
169-
region: us-east-2 <8>
170-
secret:
171-
name: <your_secret_name> <9>
172-
pipelines:
173-
- name: to-cloudwatch <10>
174-
inputRefs: <11>
175-
- infrastructure
176-
- audit
177-
- application
178-
outputRefs:
179-
- cw <12>
180-
----
181-
<1> In legacy implementations, the CR name must be `instance`. In multi log forwarder implementations, you can use any name.
182-
<2> In legacy implementations, the CR namespace must be `openshift-logging`. In multi log forwarder implementations, you can use any namespace.
183-
<3> Specify the `clf-collector` service account. The service account is only required in multi log forwarder implementations if the log forwarder is not deployed in the `openshift-logging` namespace.
184-
<4> Specify a name for the output.
185-
<5> Specify the `cloudwatch` type.
186-
<6> Optional: Specify how to group the logs:
187-
+
188-
* `logType` creates log groups for each log type.
189-
* `namespaceName` creates a log group for each application name space. Infrastructure and audit logs are unaffected, remaining grouped by `logType`.
190-
* `namespaceUUID` creates a new log groups for each application namespace UUID. It also creates separate log groups for infrastructure and audit logs.
191-
<7> Optional: Specify a string to replace the default `infrastructureName` prefix in the names of the log groups.
192-
<8> Specify the AWS region.
193-
<9> Specify the name of the secret that contains your AWS credentials.
194-
<10> Optional: Specify a name for the pipeline.
195-
<11> Specify which log types to forward by using the pipeline: `application,` `infrastructure`, or `audit`.
196-
<12> Specify the name of the output to use when forwarding logs with this pipeline.
197-
endif::[]
198-
199-
[role="_additional-resources"]
200-
.Additional resources
201-
* link:https://docs.aws.amazon.com/STS/latest/APIReference/welcome.html[AWS STS API Reference]
202-
203-
include::modules/cluster-logging-collector-log-forward-secret-cloudwatch.adoc[leveloffset=+2]
204-
205-
include::modules/cluster-logging-collector-log-forward-loki.adoc[leveloffset=+1]
206-
207-
include::modules/loki-rate-limit-errors.adoc[leveloffset=+2]
208-
209-
[role="_additional-resources"]
210-
.Additional resources
211-
212-
* xref:../../logging/cluster-logging-exported-fields.adoc#cluster-logging-exported-fields-kubernetes_cluster-logging-exported-fields[Log Record Fields]
213-
214-
* link:https://grafana.com/docs/loki/latest/configuration/[Configuring Loki server]
215-
216-
ifndef::openshift-rosa[]
217-
include::modules/cluster-logging-collector-log-forward-gcp.adoc[leveloffset=+1]
218-
endif::openshift-rosa[]
219-
220-
include::modules/logging-forward-splunk.adoc[leveloffset=+1]
221-
222-
include::modules/logging-http-forward.adoc[leveloffset=+1]
223-
224-
include::modules/cluster-logging-collector-log-forward-project.adoc[leveloffset=+1]
225-
226-
include::modules/cluster-logging-collector-log-forward-logs-from-application-pods.adoc[leveloffset=+1]
227-
228-
[role="_additional-resources"]
229-
.Additional resources
230-
231-
ifdef::openshift-enterprise,openshift-origin[]
232-
* xref:../../networking/ovn_kubernetes_network_provider/logging-network-policy.adoc#logging-network-policy[Logging for egress firewall and network policy rules]
233-
endif::[]
234-
ifdef::openshift-rosa,openshift-dedicated[]
235-
* link:https://docs.openshift.com/container-platform/latest/networking/ovn_kubernetes_network_provider/logging-network-policy.html#logging-network-policy[Logging for egress firewall and network policy rules]
236-
endif::[]
237-
238-
include::modules/cluster-logging-troubleshooting-log-forwarding.adoc[leveloffset=+1]

0 commit comments

Comments
 (0)