Skip to content

Commit 46670ce

Browse files
libanderlibander
committed
RHDEVDOCS-4064-loki final review - SMEs & Peer v2
1 parent 0f16f3b commit 46670ce

9 files changed

+400
-32
lines changed

‎_topic_maps/_topic_map.yml‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -76,7 +76,7 @@ Topics:
7676
- Name: Installation and update
7777
Distros: openshift-enterprise,openshift-origin
7878
File: architecture-installation
79-
- Name: Red Hat OpenShift Cluster Manager
79+
- Name: Red Hat OpenShift Cluster Manager
8080
Distros: openshift-enterprise
8181
File: ocm-overview-ocp
8282
- Name: Control plane architecture
@@ -2117,6 +2117,8 @@ Topics:
21172117
File: cluster-logging-systemd
21182118
- Name: Maintenance and support
21192119
File: cluster-logging-maintenance-support
2120+
- Name: Logging with the LokiStack
2121+
File: cluster-logging-loki
21202122
- Name: Viewing logs for a specific resource
21212123
File: viewing-resource-logs
21222124
- Name: Viewing cluster logs in Kibana

‎logging/cluster-logging-external.adoc‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,7 @@ include::modules/cluster-logging-collector-log-forward-syslog.adoc[leveloffset=+
4242
include::modules/cluster-logging-collector-log-forward-cloudwatch.adoc[leveloffset=+1]
4343

4444
include::modules/cluster-logging-collector-log-forward-loki.adoc[leveloffset=+1]
45+
4546
include::modules/cluster-logging-troubleshooting-loki-entry-out-of-order-errors.adoc[leveloffset=+2]
4647

4748

‎logging/cluster-logging-loki.adoc‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
:_content-type: ASSEMBLY
2+
:context: cluster-logging-loki
3+
[id="cluster-logging-loki"]
4+
= Loki
5+
include::_attributes/common-attributes.adoc[]
6+
7+
toc::[]
8+
9+
include::modules/cluster-logging-loki-about.adoc[leveloffset=+1]
10+
11+
include::modules/cluster-logging-loki-deploy.adoc[leveloffset=+1]
12+
13+
include::modules/cluster-logging-forwarding-lokistack.adoc[leveloffset=+1]
14+
15+
include::modules/cluster-logging-troubleshooting-loki-entry-out-of-order-errors.adoc[leveloffset=+2]
16+
17+
include::modules/cluster-logging-feature-reference.adoc[leveloffset=+1]
18+
19+
== Additional Resources
20+
* link:https://grafana.com/docs/loki/latest/logql/[Loki Query Language (LogQL) Documentation]
21+
* link:https://loki-operator.dev/docs/howto_connect_grafana.md/[Grafana Dashboard Documentation]
22+
* link:https://loki-operator.dev/docs/object_storage.md/[Loki Object Storage Documentation]
23+
* link:https://grafana.com/docs/loki/latest/operations/storage/schema/#changing-the-schema[Loki Storage Schema Documentation]

‎modules/cluster-logging-collector-log-forward-loki.adoc‎

Lines changed: 34 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -16,44 +16,47 @@ To configure log forwarding to Loki, you must create a `ClusterLogForwarder` cus
1616
+
1717
[source,yaml]
1818
----
19-
apiVersion: "logging.openshift.io/v1"
20-
kind: ClusterLogForwarder
21-
metadata:
22-
name: instance <1>
23-
namespace: openshift-logging <2>
24-
spec:
25-
outputs:
26-
- name: loki-insecure <3>
27-
type: "loki" <4>
28-
url: http://loki.insecure.com:9200 <5>
29-
- name: loki-secure
30-
type: "loki"
31-
url: https://loki.secure.com:9200 <6>
32-
secret:
33-
name: loki-secret <7>
34-
pipelines:
35-
- name: application-logs <8>
36-
inputRefs: <9>
37-
- application
38-
- audit
39-
outputRefs:
40-
- loki-secure <10>
41-
loki:
42-
tenantKey: kubernetes.namespace_name <11>
43-
labelKeys: kubernetes.labels.foo <12>
19+
apiVersion: "logging.openshift.io/v1"
20+
kind: ClusterLogForwarder
21+
metadata:
22+
name: instance <1>
23+
namespace: openshift-logging <2>
24+
spec:
25+
outputs:
26+
- name: loki-insecure <3>
27+
type: "loki" <4>
28+
url: http://loki.insecure.com:3100 <5>
29+
loki:
30+
tenantKey: kubernetes.namespace_name
31+
labelKeys: kubernetes.labels.foo
32+
- name: loki-secure <6>
33+
type: "loki"
34+
url: https://loki.secure.com:3100
35+
secret:
36+
name: loki-secret <7>
37+
loki:
38+
tenantKey: kubernetes.namespace_name <8>
39+
labelKeys: kubernetes.labels.foo <9>
40+
pipelines:
41+
- name: application-logs <10>
42+
inputRefs: <11>
43+
- application
44+
- audit
45+
outputRefs: <12>
46+
- loki-secure
4447
----
4548
<1> The name of the `ClusterLogForwarder` CR must be `instance`.
4649
<2> The namespace for the `ClusterLogForwarder` CR must be `openshift-logging`.
4750
<3> Specify a name for the output.
4851
<4> Specify the type as `"loki"`.
49-
<5> Specify the URL and port of the Loki system as a valid absolute URL. You can use the `http` (insecure) or `https` (secure HTTP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP Address.
52+
<5> Specify the URL and port of the Loki system as a valid absolute URL. You can use the `http` (insecure) or `https` (secure HTTP) protocol. If the cluster-wide proxy using the CIDR annotation is enabled, the output must be a server name or FQDN, not an IP Address. Loki's default port for HTTP(S) communication is 3100.
5053
<6> For a secure connection, you can specify an `https` or `http` URL that you authenticate by specifying a `secret`.
5154
<7> For an `https` prefix, specify the name of the secret required by the endpoint for TLS communication. The secret must exist in the `openshift-logging` project, and must have keys of: *tls.crt*, *tls.key*, and *ca-bundle.crt* that point to the respective certificates that they represent. Otherwise, for `http` and `https` prefixes, you can specify a secret that contains a username and password. For more information, see the following "Example: Setting secret that contains a username and password."
52-
<8> Optional: Specify a name for the pipeline.
53-
<9> Specify which log types to forward by using the pipeline: `application,` `infrastructure`, or `audit`.
54-
<10> Specify the name of the output to use when forwarding logs with this pipeline.
55-
<11> Optional: Specify a meta-data key field to generate values for the `TenantID` field in Loki. For example, setting `tenantKey: kubernetes.namespace_name` uses the names of the Kubernetes namespaces as values for tenant IDs in Loki. To see which other log record fields you can specify, see the "Log Record Fields" link in the following "Additional resources" section.
56-
<12> Optional: Specify a list of meta-data field keys to replace the default Loki labels. Loki label names must match the regular expression `[a-zA-Z_:][a-zA-Z0-9_:]*`. Illegal characters in meta-data keys are replaced with `_` to form the label name. For example, the `kubernetes.labels.foo` meta-data key becomes Loki label `kubernetes_labels_foo`. If you do not set `labelKeys`, the default value is: `[log_type, kubernetes.namespace_name, kubernetes.pod_name, kubernetes_host]`. Keep the set of labels small because Loki limits the size and number of labels allowed. See link:https://grafana.com/docs/loki/latest/configuration/#limits_config[Configuring Loki, limits_config]. You can still query based on any log record field using query filters.
55+
<8> Optional: Specify a meta-data key field to generate values for the `TenantID` field in Loki. For example, setting `tenantKey: kubernetes.namespace_name` uses the names of the Kubernetes namespaces as values for tenant IDs in Loki. To see which other log record fields you can specify, see the "Log Record Fields" link in the following "Additional resources" section.
56+
<9> Optional: Specify a list of meta-data field keys to replace the default Loki labels. Loki label names must match the regular expression `[a-zA-Z_:][a-zA-Z0-9_:]*`. Illegal characters in meta-data keys are replaced with `_` to form the label name. For example, the `kubernetes.labels.foo` meta-data key becomes Loki label `kubernetes_labels_foo`. If you do not set `labelKeys`, the default value is: `[log_type, kubernetes.namespace_name, kubernetes.pod_name, kubernetes_host]`. Keep the set of labels small because Loki limits the size and number of labels allowed. See link:https://grafana.com/docs/loki/latest/configuration/#limits_config[Configuring Loki, limits_config]. You can still query based on any log record field using query filters.
57+
<10> Optional: Specify a name for the pipeline.
58+
<11> Specify which log types to forward by using the pipeline: `application,` `infrastructure`, or `audit`.
59+
<12> Specify the name of the output to use when forwarding logs with this pipeline.
5760
+
5861
[NOTE]
5962
====

‎modules/cluster-logging-feature-reference.adoc‎

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
// Module is included in the following assemblies:
2+
//cluster-logging-loki.adoc
3+
:_content-type: REFERENCE
4+
[id="logging-feature-ref_{context}"]
5+
= Collector features
6+
7+
.Log Sources
8+
[options="header"]
9+
|===============================================================
10+
| Feature | Fluentd | Vector
11+
| App container logs | &#10003; | &#10003;
12+
| App-specific routing | &#10003; | &#10003;
13+
| App-specific routing by namespace | &#10003; | &#10003;
14+
| Infra container logs | &#10003; | &#10003;
15+
| Infra journal logs | &#10003; | &#10003;
16+
| Kube API audit logs | &#10003; | &#10003;
17+
| Openshift API audit logs | &#10003; | &#10003;
18+
| Open Virtual Network (OVN) audit logs| &#10003; | &#10003;
19+
|===============================================================
20+
21+
.Outputs
22+
[options="header"]
23+
|==========================================================
24+
| Feature | Fluentd | Vector
25+
| Elasticsearch v5-v7 | &#10003; | &#10003;
26+
| Fluent forward | &#10003; |
27+
| Syslog RFC3164 | &#10003; |
28+
| Syslog RFC5424 | &#10003; |
29+
| Kafka | &#10003; | &#10003;
30+
| Cloudwatch | &#10003; | &#10003;
31+
| Loki | &#10003; | &#10003;
32+
|==========================================================
33+
34+
.Authorization and Authentication
35+
[options="header"]
36+
|=================================================================
37+
| Feature | Fluentd | Vector
38+
| Elasticsearch certificates | &#10003; | &#10003;
39+
| Elasticsearch username / password | &#10003; | &#10003;
40+
| Cloudwatch keys | &#10003; | &#10003;
41+
| Cloudwatch STS | &#10003; |
42+
| Kafka certificates | &#10003; | &#10003;
43+
| Kafka username / password | &#10003; | &#10003;
44+
| Kafka SASL | &#10003; | &#10003;
45+
| Loki bearer token | &#10003; | &#10003;
46+
|=================================================================
47+
48+
.Normalizations and Transformations
49+
[options="header"]
50+
|============================================================================
51+
| Feature | Fluentd | Vector
52+
| Viaq data model - app | &#10003; | &#10003;
53+
| Viaq data model - infra | &#10003; | &#10003;
54+
| Viaq data model - infra(journal) | &#10003; | &#10003;
55+
| Viaq data model - Linux audit | &#10003; | &#10003;
56+
| Viaq data model - kube-apiserver audit | &#10003; | &#10003;
57+
| Viaq data model - OpenShift API audit | &#10003; | &#10003;
58+
| Viaq data model - OVN | &#10003; | &#10003;
59+
| Loglevel Normalization | &#10003; | &#10003;
60+
| JSON parsing | &#10003; | &#10003;
61+
| Structured Index | &#10003; | &#10003;
62+
| Multiline error detection | &#10003; |
63+
| Multicontainer / split indices | &#10003; | &#10003;
64+
| Flatten labels | &#10003; | &#10003;
65+
| CLF static labels | &#10003; | &#10003;
66+
|============================================================================
67+
68+
.Tuning
69+
[options="header"]
70+
|==========================================================
71+
| Feature | Fluentd | Vector
72+
| Fluentd readlinelimit | &#10003; |
73+
| Fluentd buffer | &#10003; |
74+
| - chunklimitsize | &#10003; |
75+
| - totallimitsize | &#10003; |
76+
| - overflowaction | &#10003; |
77+
| - flushthreadcount | &#10003; |
78+
| - flushmode | &#10003; |
79+
| - flushinterval | &#10003; |
80+
| - retrywait | &#10003; |
81+
| - retrytype | &#10003; |
82+
| - retrymaxinterval | &#10003; |
83+
| - retrytimeout | &#10003; |
84+
|==========================================================
85+
86+
.Visibility
87+
[options="header"]
88+
|=====================================================
89+
| Feature | Fluentd | Vector
90+
| Metrics | &#10003; | &#10003;
91+
| Dashboard | &#10003; | &#10003;
92+
| Alerts | &#10003; |
93+
|=====================================================
94+
95+
.Miscellaneous
96+
[options="header"]
97+
|===========================================================
98+
| Feature | Fluentd | Vector
99+
| Global proxy support | &#10003; | &#10003;
100+
| x86 support | &#10003; | &#10003;
101+
| ARM support | &#10003; | &#10003;
102+
| PowerPC support | &#10003; | &#10003;
103+
| IBM Z support | &#10003; | &#10003;
104+
| IPV6 support | &#10003; | &#10003;
105+
| Log event buffering | &#10003; |
106+
| Disconnected Cluster | &#10003; | &#10003;
107+
|===========================================================

‎modules/cluster-logging-forwarding-lokistack.adoc‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
// Module is included in the following assemblies:
2+
//cluster-logging-loki.adoc
3+
:_content-type: PROCEDURE
4+
[id="cluster-logging-forwarding-lokistack_{context}"]
5+
= Forwarding logs to LokiStack
6+
7+
To configure log forwarding to the LokiStack gateway, you must create a ClusterLogging custom resource (CR).
8+
9+
.Prerequisites
10+
11+
* {logging-title-uc}: 5.5 and later
12+
* `LokiOperator` Operator
13+
14+
.Procedure
15+
16+
. Create or edit a YAML file that defines the `ClusterLogging` custom resource (CR):
17+
18+
[source,yaml]
19+
----
20+
apiVersion: logging.openshift.io/v1
21+
kind: ClusterLogging
22+
metadata:
23+
name: instance
24+
namespace: openshift-logging
25+
spec:
26+
logStore:
27+
type: lokistack
28+
lokistack:
29+
name: lokistack-dev
30+
collection:
31+
type: vector
32+
----

‎modules/cluster-logging-logcli-reference.adoc‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
// Module is included in the following assemblies:
2+
//cluster-logging-loki.adoc
3+
:_content-type: REFERENCE
4+
[id="logging-logcli-about_{context}"]
5+
6+
= Querying Loki
7+
8+
You can use Loki's command-line interface `logcli` to query logs.
9+
10+
.Example Application Log Query
11+
[source,terminal]
12+
----
13+
$ oc extract cm/lokistack-sample-ca-bundle --to=lokistack --confirm
14+
$ cat lokistack/*.crt >lokistack_ca.crt
15+
$ logcli -o raw --bearer-token="${bearer_token}" --ca-cert="lokistack_ca.crt" --addr xxxxxx
16+
----
17+
18+
.Example Infrastructure Log Query
19+
[source,terminal]
20+
----
21+
$ logcli --bearer-token="$(oc whoami -t)" --addr https://lokistack-dev-openshift-logging.apps.devcluster.openshift.com/api/logs/v1/infrastructure labels
22+
----
23+
24+
.Example Audit log Query
25+
[source,terminal]
26+
----
27+
$ logcli --bearer-token="$(oc whoami -t)" --addr https://lokistack-dev-openshift-logging.apps.devcluster.openshift.com/api/logs/v1/audit labels
28+
----
29+
30+
.Additional Resources
31+
* link:https://grafana.com/docs/loki/latest/tools/logcli/[LogCLI Documentation]

‎modules/cluster-logging-loki-about.adoc‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
// Module is included in the following assemblies:
2+
//cluster-logging-loki.adoc
3+
:_content-type: CONCEPT
4+
[id="about-logging-loki_{context}"]
5+
= About the LokiStack
6+
7+
In {logging} documentation, *LokiStack* refers to the {logging} supported combination of Loki, and web proxy with {product-title} authentication integration. LokiStack's proxy uses {product-title} authentication to enforce multi-tenancy. *Loki* refers to the log store as either the individual component or an external store.
8+
9+
Loki is a horizontally scalable, highly available, multi-tenant log aggregation system currently offered as an alternative to Elasticsearch as a log store for the {logging}. Elasticsearch indexes incoming log records completely during ingestion. Loki only indexes a few fixed labels during ingestion, and defers more complex parsing until after the logs have been stored. This means Loki can collect logs more quickly. As with Elasticsearch, you can query Loki link:https://grafana.com/docs/loki/latest/[using JSON paths or regular expressions].
10+
11+
[id="deployment-sizing_{context}"]
12+
== Deployment Sizing
13+
Sizing for Loki follows the format of `N<x>._<size>_` where the value `<N>` is number of instances and `<size>` specifies performance capabilities.
14+
15+
.Loki Sizing
16+
[options="header"]
17+
|========================================================================================
18+
| | 1x.extra-small | 1x.small | 1x.medium
19+
| *Data transfer* | Demo use only. | 500GB/day | 2TB/day
20+
| *Queries per second (QPS)* | Demo use only. | 25-50 QPS at 200ms | 25-75 QPS at 200ms
21+
| *Replication factor* | None | 2 | 3
22+
| *Total CPU requests* | 5 vCPUs | 36 vCPUs | 54 vCPUs
23+
| *Total Memory requests* | 7.5Gi | 63Gi | 139Gi
24+
| *Total Disk requests* | 150Gi | 300Gi | 450Gi
25+
|========================================================================================
26+
27+
[id="CRD-API-support_{context}"]
28+
== Supported API Custom Resource Definitions
29+
LokiStack development is ongoing, not all APIs are supported currently supported.
30+
31+
[options="header"]
32+
|=====================================================================
33+
| CustomResourceDefinition (CRD)| ApiVersion | Support state
34+
| LokiStack | lokistack.loki.grafana.com/v1 | Supported in 5.5
35+
| RulerConfig | rulerconfig.loki.grafana/v1beta1 | Technology Preview
36+
| AlertingRule | alertingrule.loki.grafana/v1beta1 | Technology Preview
37+
| RecordingRule | recordingrule.loki.grafana/v1beta1 | Technology Preview
38+
|=====================================================================
39+
40+
:FeatureName: Usage of `RulerConfig`, `AlertingRule` and `RecordingRule` custom resource definitions (CRDs).
41+
include::snippets/technology-preview.adoc[]

0 commit comments

Comments
 (0)