Skip to content

Commit 2999b3a

Browse files
apinnickapinnick
committed
CNV-20433: Loki logging queries
Signed-off-by: Avital Pinnick <[email protected]>
1 parent 1a4cce7 commit 2999b3a

File tree

5 files changed

+218
-44
lines changed

5 files changed

+218
-44
lines changed

‎modules/virt-configuring-pod-log-verbosity.adoc

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * virt/support/virt-troubleshooting.adoc
4+
5+
:_content-type: PROCEDURE
6+
[id="virt-configuring-pod-log-verbosity_{context}"]
7+
= Configuring {VirtProductName} pod log verbosity
8+
9+
You can configure the verbosity level of {VirtProductName} pod logs by editing the `HyperConverged` custom resource (CR).
10+
11+
.Procedure
12+
13+
. To set log verbosity for specific components, open the `HyperConverged` CR in your default text editor by running the following command:
14+
+
15+
[source,terminal]
16+
----
17+
$ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv
18+
----
19+
20+
. Set the log level for one or more components by editing the `spec.logVerbosityConfig` stanza. For example:
21+
+
22+
[source,yaml]
23+
----
24+
apiVersion: hco.kubevirt.io/v1beta1
25+
kind: HyperConverged
26+
metadata:
27+
name: kubevirt-hyperconverged
28+
spec:
29+
logVerbosityConfig:
30+
kubevirt:
31+
virtAPI: 5 <1>
32+
virtController: 4
33+
virtHandler: 3
34+
virtLauncher: 2
35+
virtOperator: 6
36+
----
37+
<1> The log verbosity value must be an integer in the range `1–9`, where a higher number indicates a more detailed log. In this example, the `virtAPI` component logs are exposed if their priority level is `5` or higher.
38+
39+
. Apply your changes by saving and exiting the editor.

‎modules/virt-loki-log-queries.adoc

Lines changed: 119 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,119 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * virt/support/virt-troubleshooting.adoc
4+
5+
:_content-type: reference
6+
[id="virt-loki-log-queries_{context}"]
7+
= {VirtProductName} LogQL queries
8+
9+
You can view and filter aggregated logs for {VirtProductName} components by running Loki Query Language (LogQL) queries on the *Observe* -> *Logs* page in the web console.
10+
11+
The default log type is _infrastructure_. The `virt-launcher` log type is _application_.
12+
13+
Optional: You can include or exclude strings or regular expressions by using line filter expressions.
14+
15+
[NOTE]
16+
====
17+
If the query matches a large number of logs, the query might time out.
18+
====
19+
20+
.{VirtProductName} LogQL example queries
21+
[cols="1a,6a",options="header"]
22+
|====
23+
|Component
24+
|LogQL query
25+
26+
|All
27+
|[source,text]
28+
----
29+
{log_type=~".+"}\|json
30+
\|kubernetes_labels_app_kubernetes_io_part_of="hyperconverged-cluster"
31+
----
32+
33+
|`cdi-apiserver`
34+
35+
`cdi-deployment`
36+
37+
`cdi-operator`
38+
|[source,text]
39+
----
40+
{log_type=~".+"}\|json
41+
\|kubernetes_labels_app_kubernetes_io_part_of="hyperconverged-cluster"
42+
\|kubernetes_labels_app_kubernetes_io_component="storage"
43+
----
44+
45+
|`hco-operator`
46+
|[source,text]
47+
----
48+
{log_type=~".+"}\|json
49+
\|kubernetes_labels_app_kubernetes_io_part_of="hyperconverged-cluster"
50+
\|kubernetes_labels_app_kubernetes_io_component="deployment"
51+
----
52+
53+
|`kubemacpool`
54+
|[source,text]
55+
----
56+
{log_type=~".+"}\|json
57+
\|kubernetes_labels_app_kubernetes_io_part_of="hyperconverged-cluster"
58+
\|kubernetes_labels_app_kubernetes_io_component="network"
59+
----
60+
61+
|`virt-api`
62+
63+
`virt-controller`
64+
65+
`virt-handler`
66+
67+
`virt-operator`
68+
|[source,text]
69+
----
70+
{log_type=~".+"}\|json
71+
\|kubernetes_labels_app_kubernetes_io_part_of="hyperconverged-cluster"
72+
\|kubernetes_labels_app_kubernetes_io_component="compute"
73+
----
74+
75+
|`ssp-operator`
76+
|[source,text]
77+
----
78+
{log_type=~".+"}\|json
79+
\|kubernetes_labels_app_kubernetes_io_part_of="hyperconverged-cluster"
80+
\|kubernetes_labels_app_kubernetes_io_component="schedule"
81+
----
82+
83+
|Container|[source,text]
84+
----
85+
{log_type=~".+",kubernetes_container_name=~"<container>\|<container>"} <1>
86+
\|json\|kubernetes_labels_app_kubernetes_io_part_of="hyperconverged-cluster"
87+
----
88+
<1> Specify one or more containers separated by a pipe (`\|`).
89+
90+
|`virt-launcher`
91+
|You must select *application* from the log type list before running this query.
92+
93+
[source,text]
94+
----
95+
{log_type=~".+", kubernetes_container_name="compute"}\|json
96+
\|!= "custom-ga-command" <1>
97+
----
98+
<1> `\|!= "custom-ga-command"` excludes libvirt logs that contain the string `custom-ga-command`. (https://bugzilla.redhat.com/show_bug.cgi?id=2177684[*BZ#2177684*])
99+
|====
100+
101+
You can filter log lines to include or exclude strings or regular expressions by using line filter expressions.
102+
103+
.Line filter expressions
104+
[cols="1a,2",options="header"]
105+
|====
106+
|Line filter expression|Description
107+
|`\|= "<string>"` |Log line contains string
108+
|`!= "<string>"` |Log line does not contain string
109+
|`\|~ "<regex>"` |Log line contains regular expression
110+
|`!~ "<regex>"` |Log line does not contain regular expression
111+
|====
112+
113+
.Example line filter expression
114+
[source,text]
115+
----
116+
{log_type=~".+"}|json
117+
|kubernetes_labels_app_kubernetes_io_part_of="hyperconverged-cluster"
118+
|= "error" != "timeout"
119+
----

‎modules/virt-viewing-logs-cli.adoc

Lines changed: 3 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -4,40 +4,12 @@
44

55
:_content-type: PROCEDURE
66
[id="virt-viewing-logs-cli_{context}"]
7-
= Viewing {VirtProductName} logs with the CLI
7+
= Viewing {VirtProductName} pod logs with the CLI
88

9-
You can configure the verbosity level of {VirtProductName} component logs by editing the `HyperConverged` custom resource (CR). Then, you can view logs for the component pods by using the `oc` CLI tool.
9+
You can view logs for the {VirtProductName} pods by using the `oc` CLI tool.
1010

1111
.Procedure
1212

13-
. To set log verbosity for specific components, open the `HyperConverged` CR in your default text editor by running the following command:
14-
+
15-
[source,terminal]
16-
----
17-
$ oc edit hyperconverged kubevirt-hyperconverged -n openshift-cnv
18-
----
19-
20-
. Set the log level for one or more components by editing the `spec.logVerbosityConfig` stanza. For example:
21-
+
22-
[source,yaml]
23-
----
24-
apiVersion: hco.kubevirt.io/v1beta1
25-
kind: HyperConverged
26-
metadata:
27-
name: kubevirt-hyperconverged
28-
spec:
29-
logVerbosityConfig:
30-
kubevirt:
31-
virtAPI: 5 <1>
32-
virtController: 4
33-
virtHandler: 3
34-
virtLauncher: 2
35-
virtOperator: 6
36-
----
37-
<1> The log verbosity value must be an integer in the range `1–9`, where a higher number indicates a more detailed log. In this example, the `virtAPI` component logs are exposed if their priority level is `5` or higher.
38-
39-
. Apply your changes by saving and exiting the editor.
40-
4113
. View a list of pods in the {VirtProductName} namespace by running the following command:
4214
+
4315
[source,terminal]
@@ -64,20 +36,13 @@ virt-operator-7ccfdbf65f-vllz8 1/1 Running 0 32m
6436
----
6537
====
6638
67-
. To view logs for a component pod, run the following command:
39+
. View the pod log by running the following command:
6840
+
6941
[source,terminal]
7042
----
7143
$ oc logs -n openshift-cnv <pod_name>
7244
----
7345
+
74-
For example:
75-
+
76-
[source,terminal]
77-
----
78-
$ oc logs -n openshift-cnv virt-handler-2m86x
79-
----
80-
+
8146
[NOTE]
8247
====
8348
If a pod fails to start, you can use the `--previous` option to view logs from the last attempt.

‎modules/virt-viewing-logs-loki.adoc

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
// Module included in the following assemblies:
2+
//
3+
// * virt/support/virt-troubleshooting.adoc
4+
5+
:_content-type: PROCEDURE
6+
[id="virt-viewing-logs-loki_{context}"]
7+
= Viewing aggregated {VirtProductName} logs with the LokiStack
8+
9+
You can view aggregated logs for {VirtProductName} pods and containers by using the LokiStack in the web console.
10+
11+
.Prerequisites
12+
13+
* You deployed the LokiStack.
14+
15+
.Procedure
16+
17+
. Navigate to *Observe* -> *Logs* in the web console.
18+
. Select *application*, for `virt-launcher` pod logs, or *infrastructure*, for {VirtProductName} control plane pods and containers, from the log type list.
19+
. Click *Show Query* to display the query field.
20+
. Enter the LogQL query in the query field and click *Run Query* to display the filtered logs.

‎virt/support/virt-troubleshooting.adoc

Lines changed: 37 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -3,19 +3,24 @@
33
= Troubleshooting
44
include::_attributes/common-attributes.adoc[]
55
:context: virt-troubleshooting
6+
:toclevels: 4
7+
8+
{VirtProductName} provides tools and logs for troubleshooting virtual machines and virtualization components.
69

710
toc::[]
811

912
You can troubleshoot {VirtProductName} components by using the xref:../../virt/support/virt-support-overview.adoc#virt-web-console_virt-support-overview[tools provided in the web console] or by using the `oc` CLI tool.
1013

14+
// This section is in an assembly so that we can use xrefs.
1115
[id="events_{context}"]
1216
== Events
1317

14-
xref:../../nodes/clusters/nodes-containers-events.adoc#nodes-containers-events[{product-title} events] are records of important life-cycle information and are useful for monitoring and troubleshooting resource issues. You can gather information about the following events:
18+
xref:../../nodes/clusters/nodes-containers-events.adoc#nodes-containers-events[{product-title} events] are records of important life-cycle information and are useful for monitoring and troubleshooting virtual machine, namespace, and resource issues.
1519

1620
* VM events: Navigate to the xref:../../virt/virt-web-console-overview.adoc#ui-virtualmachine-details-events[*Events* tab] of the *VirtualMachine details* page in the web console.
1721

18-
* Namespace events: Use the `oc get` command with the namespace:
22+
Namespace events::
23+
You can view namespace events by running the following command:
1924
+
2025
[source,terminal]
2126
----
@@ -24,7 +29,8 @@ $ oc get events -n <namespace>
2429
+
2530
See the xref:../../nodes/clusters/nodes-containers-events.adoc#nodes-containers-events-list_nodes-containers-events[list of events] for details about specific events.
2631

27-
* Resource events: Use the `oc describe` command with the resource:
32+
Resource events::
33+
You can view resource events by running the following command:
2834
+
2935
[source,terminal]
3036
----
@@ -34,13 +40,38 @@ $ oc describe <resource> <resource_name>
3440
[id="virt-logs_{context}"]
3541
== Logs
3642

37-
You can view logs for {VirtProductName} components and VMs by using the web console or the `oc` CLI tool. You can retrieve virtual machine logs from the `virt-launcher` pod. To control log verbosity, edit the `HyperConverged` custom resource.
43+
You can review the following logs for troubleshooting:
44+
45+
* xref:../../virt/support/virt-troubleshooting.adoc#viewing-virt-component-pods_virt-troubleshooting[Virtual machine]
46+
* xref:../../virt/support/virt-troubleshooting.adoc#virt-viewing-logs-cli_virt-troubleshooting[{VirtProductName} pod]
47+
* xref:../../virt/support/virt-troubleshooting.adoc#virt-viewing-logs-loki_virt-troubleshooting[Aggregated {VirtProductName} logs]
3848

3949
include::modules/virt-viewing-virtual-machine-logs-web.adoc[leveloffset=+2]
4050

41-
include::modules/virt-viewing-logs-cli.adoc[leveloffset=+2]
51+
[id="viewing-virt-component-pods_{context}"]
52+
=== Viewing {VirtProductName} pod logs
53+
54+
You can view logs for {VirtProductName} pods by using the `oc` CLI tool.
55+
56+
You can configure the verbosity level of the logs by editing the `HyperConverged` custom resource (CR).
57+
58+
include::modules/virt-viewing-logs-cli.adoc[leveloffset=+3]
59+
60+
include::modules/virt-configuring-pod-log-verbosity.adoc[leveloffset=+3]
61+
62+
include::modules/virt-common-error-messages.adoc[leveloffset=+3]
63+
64+
include::modules/virt-viewing-logs-loki.adoc[leveloffset=+2]
65+
66+
include::modules/virt-loki-log-queries.adoc[leveloffset=+3]
67+
4268

43-
include::modules/virt-common-error-messages.adoc[leveloffset=+2]
69+
[role="_additional-resources"]
70+
[id="additional-resources_{context}"]
71+
==== Additional resources for LokiStack and LogQL
72+
* xref:../../logging/cluster-logging-loki.adoc#about-logging-loki_cluster-logging-loki[About the LokiStack]
73+
* xref:../../logging/cluster-logging-loki.adoc#logging-loki-deploy_cluster-logging-loki[Deploying the LokiStack] on {product-title}
74+
* link:https://grafana.com/docs/loki/latest/logql/log_queries/[LogQL log queries] in the Grafana documentation
4475

4576
[id="troubleshooting-data-volumes_{context}"]
4677
== Troubleshooting data volumes

0 commit comments

Comments
 (0)