Skip to content

Commit 4f6f4d2

Browse files
MirzWeissMirzWeiss
committed
TELCODOCS:412 - Document enhanced logging for must-gather in sandboxed containers using the debug log level in CRI-O
1 parent d63f3a4 commit 4f6f4d2

6 files changed

+176
-25
lines changed

‎_topic_maps/_topic_map.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2033,7 +2033,7 @@ Topics:
20332033
File: uninstalling-sandboxed-containers
20342034
- Name: Upgrade OpenShift sandboxed containers
20352035
File: upgrade-sandboxed-containers
2036-
- Name: Collecting OpenShift sandboxed containers data for Red Hat Support
2036+
- Name: Collecting OpenShift sandboxed containers data
20372037
File: troubleshooting-sandboxed-containers
20382038
---
20392039
Name: Logging

‎modules/sandboxed-containers-collect-data-support.adoc

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * sandboxed_containers/troubleshooting-sandboxed-containers.adoc
4+
5+
:_content-type: CONCEPT
6+
[id="sandboxed-containers-collect_data_rh_support_{context}"]
7+
= Collecting {sandboxed-containers-first} data for Red Hat Support
8+
9+
When opening a support case, it is helpful to provide debugging information about your cluster to Red Hat Support.
10+
11+
The `must-gather` tool enables you to collect diagnostic information about your {product-title} cluster, including virtual machines and other data related to {sandboxed-containers-first}.
12+
13+
For prompt support, supply diagnostic information for both {product-title} and {sandboxed-containers-first}.

‎modules/sandboxed-containers-collecting-data.adoc

Lines changed: 6 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -6,26 +6,17 @@
66

77
:_content-type: CONCEPT
88
[id="sandboxed-containers-collecting-data_{context}"]
9-
= About collecting {sandboxed-containers-first} data
9+
= About {sandboxed-containers-first} log data
1010

11-
You can use the `oc adm must-gather` CLI command to collect information about your cluster. The following features and objects are associated with {sandboxed-containers-first}:
11+
When you collect log data about your cluster, the following features and objects are associated with {sandboxed-containers-first}:
1212

13-
* All namespaces and their child objects that belong to any {sandboxed-containers-first}
14-
resources
13+
* All namespaces and their child objects that belong to any {sandboxed-containers-first} resources
1514
* All {sandboxed-containers-first} custom resource definitions (CRDs)
1615
17-
The `oc adm must-gather` CLI command collects the following component logs:
16+
The following {sandboxed-containers-first} component logs are collected for each pod running with the `kata` runtime:
1817

18+
* Kata agent logs
19+
* Kata runtime logs
1920
* QEMU logs
2021
* Audit logs
21-
* {sandboxed-containers-first} logs
2222
* CRI-O logs
23-
24-
These component logs are collected as long as there is at least one pod running with the `kata` runtime.
25-
26-
To collect {sandboxed-containers-first} data with `must-gather`, you must specify the
27-
{sandboxed-containers-first} image:
28-
[source,terminal]
29-
----
30-
--image=registry.redhat.io/openshift-sandboxed-containers-tech-preview/osc-must-gather-rhel8:1.1.0
31-
----

‎modules/sandboxed-containers-enable-debug-crio.adoc

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,98 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * sandboxed_containers/troubleshooting-sandboxed-containers.adoc
4+
5+
:_content-type: PROCEDURE
6+
[id="sandboxed-containers-enable-debug-logs_{context}"]
7+
= Enabling debug logs for {sandboxed-containers-first}
8+
9+
As a cluster administrator, you can collect a more detailed level of logs for {sandboxed-containers-first}. Enhance logging by changing the `log_level` in the CRI-O runtime for the worker nodes running {sandboxed-containers-first}.
10+
11+
.Procedure
12+
13+
. Create a YAML file for the `ContainerRuntimeConfig` CR with the following manifest:
14+
+
15+
[source,yaml]
16+
----
17+
apiVersion: machineconfiguration.openshift.io/v1
18+
kind: ContainerRuntimeConfig
19+
metadata:
20+
name: crio-debug
21+
spec:
22+
machineConfigPoolSelector:
23+
matchLabels:
24+
pools.operator.machineconfiguration.openshift.io/worker: '' <1>
25+
containerRuntimeConfig:
26+
logLevel: debug
27+
----
28+
<1> Specify a label for the machine config pool that you want you want to modify.
29+
30+
. Create the `ContainerRuntimeConfig` CR:
31+
+
32+
[source,terminal]
33+
----
34+
$ oc create -f ctrcfg.yaml
35+
----
36+
37+
+
38+
[NOTE]
39+
====
40+
The file name listed above is a suggestion. You can create this file using another name.
41+
====
42+
43+
. Verify the CR is created:
44+
+
45+
[source,terminal]
46+
----
47+
$ oc get ctrcfg
48+
----
49+
+
50+
.Example output
51+
[source,terminal]
52+
----
53+
NAME AGE
54+
crio-debug 3m19s
55+
----
56+
57+
.Verification
58+
59+
. Monitor the machine config pool until the `UPDATED` field for all worker nodes appears as `True`:
60+
+
61+
[source,terminal]
62+
----
63+
$ oc get mcp worker
64+
----
65+
+
66+
.Example output
67+
[source,terminal]
68+
----
69+
NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE
70+
worker rendered-worker-169 False True False 3 1 1 0 9h
71+
----
72+
73+
. Verify that the `log_level` was updated in CRI-O:
74+
75+
.. Open an `oc debug` session to a node in the machine config pool and run `chroot /host`.
76+
+
77+
[source,terminal]
78+
----
79+
$ oc debug node/<node_name>
80+
----
81+
+
82+
[source,terminal]
83+
----
84+
sh-4.4# chroot /host
85+
----
86+
87+
.. Verify the changes in the `crio.conf` file:
88+
+
89+
[source,terminal]
90+
----
91+
sh-4.4# crio config | egrep 'log_level
92+
----
93+
+
94+
.Example output
95+
[source,terminal]
96+
----
97+
log_level = "debug"
98+
----

‎modules/sandboxed-containers-viewing-logs.adoc

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * sandboxed_containers/troubleshooting-sandboxed-containers.adoc
4+
5+
:_content-type: PROCEDURE
6+
[id="sandboxed-containers-view-debug-logs_{context}"]
7+
= Viewing debug logs for {sandboxed-containers-first}
8+
9+
Cluster administrators can use the enhanced debug logs for {sandboxed-containers-first} to troubleshoot issues. The logs for each node are printed to the node journal.
10+
11+
You can review the logs for the following {sandboxed-containers-first} components:
12+
13+
* Kata agent
14+
* Kata runtime (`containerd-shim-kata-v2`)
15+
* virtiofsd
16+
17+
Logs for QEMU do not print to the node journal. However, a QEMU failure is reported to the runtime, and the console of the QEMU guest is printed to the node journal. You can view these logs together with the Kata agent logs.
18+
19+
.Prerequisites
20+
21+
* You have installed the OpenShift CLI (`oc`).
22+
* You have access to the cluster as a user with the `cluster-admin` role.
23+
24+
.Procedure
25+
26+
* To review the Kata agent logs and guest console logs, run:
27+
+
28+
[source,terminal]
29+
----
30+
$ oc debug node/<nodename> -- journalctl -D /host/var/log/journal -t kata -g “reading guest console”
31+
----
32+
33+
* To review the kata runtime logs, run:
34+
+
35+
[source,terminal]
36+
----
37+
$ oc debug node/<nodename> -- journalctl -D /host/var/log/journal -t kata
38+
----
39+
40+
* To review the virtiofsd logs, run:
41+
+
42+
[source,terminal]
43+
----
44+
$ oc debug node/<nodename> -- journalctl -D /host/var/log/journal -t virtiofsd
45+
----

‎sandboxed_containers/troubleshooting-sandboxed-containers.adoc

Lines changed: 13 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,26 +1,30 @@
11
:_content-type: ASSEMBLY
22
[id="troubleshooting-sandboxed-containers"]
3-
= Collecting {sandboxed-containers-first} data for Red Hat Support
3+
= Collecting {sandboxed-containers-first} data
44
include::_attributes/common-attributes.adoc[]
55
:context: troubleshooting-sandboxed-containers
66

77
toc::[]
88

99
[role="_abstract"]
1010

11-
When opening a support case, it is helpful to provide debugging
12-
information about your cluster to Red Hat Support.
11+
When troubleshooting {sandboxed-containers-first}, you can open a support case and provide debugging information using the `must-gather` tool.
1312

14-
The `must-gather` tool enables you to collect diagnostic information about your
15-
{product-title} cluster, including virtual machines and other data related to
16-
{sandboxed-containers-first}.
13+
If you are a cluster administrator, you can also review logs on your own, enabling a more detailed level of logs.
1714

18-
For prompt support, supply diagnostic information for both {product-title}
19-
and {sandboxed-containers-first}.
15+
include::modules/sandboxed-containers-collect-data-support.adoc[leveloffset=+1]
16+
include::modules/about-must-gather.adoc[leveloffset=+2]
2017

21-
include::modules/about-must-gather.adoc[leveloffset=+1]
18+
To collect {sandboxed-containers-first} data with `must-gather`, you must specify the
19+
{sandboxed-containers-first} image:
20+
[source,terminal]
21+
----
22+
--image=registry.redhat.io/openshift-sandboxed-containers/osc-must-gather-rhel8:1.2.0
23+
----
2224

2325
include::modules/sandboxed-containers-collecting-data.adoc[leveloffset=+1]
26+
include::modules/sandboxed-containers-enable-debug-crio.adoc[leveloffset=+1]
27+
include::modules/sandboxed-containers-viewing-logs.adoc[leveloffset=+2]
2428

2529
[id="troubleshooting-sandboxed-containers_additional-resources"]
2630
[role="_additional-resources"]

0 commit comments

Comments
 (0)