You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Application logs can help you understand what is happening inside your application. The logs are particularly useful for debugging problems and monitoring cluster activity. Most modern applications have some kind of logging mechanism. Likewise, container engines are designed to support logging. The easiest and most adopted logging method for containerized applications is writing to standard output and standard error streams.
12
+
Application logs can help you understand what is happening inside your application. The
13
+
logs are particularly useful for debugging problems and monitoring cluster activity. Most
14
+
modern applications have some kind of logging mechanism. Likewise, container engines
15
+
are designed to support logging. The easiest and most adopted logging method for
16
+
containerized applications is writing to standard output and standard error streams.
13
17
14
-
However, the native functionality provided by a container engine or runtime is usually not enough for a complete logging solution.
18
+
However, the native functionality provided by a container engine or runtime is usually
19
+
not enough for a complete logging solution.
15
20
16
-
For example, you may want to access your application's logs if a container crashes, a pod gets evicted, or a node dies.
21
+
For example, you may want to access your application's logs if a container crashes,
22
+
a pod gets evicted, or a node dies.
17
23
18
-
In a cluster, logs should have a separate storage and lifecycle independent of nodes, pods, or containers. This concept is called _cluster-level logging_.
24
+
In a cluster, logs should have a separate storage and lifecycle independent of nodes,
A container engine handles and redirects any output generated to a containerized application's `stdout` and `stderr` streams.
75
-
For example, the Docker container engine redirects those two streams to [a logging driver](https://docs.docker.com/engine/admin/logging/overview), which is configured in Kubernetes to write to a file in JSON format.
84
+
A container runtime handles and redirects any output generated to a containerized application's `stdout` and `stderr` streams.
85
+
Different container runtimes implement this in different ways; however, the integration with the kubelet is standardized
86
+
as the _CRI logging format_.
76
87
77
-
{{< note >}}
78
-
The Docker JSON logging driver treats each line as a separate message. When using the Docker logging driver, there is no direct support for multi-line messages. You need to handle multi-line messages at the logging agent level or higher.
79
-
{{< /note >}}
88
+
By default, if a container restarts, the kubelet keeps one terminated container with its logs. If a pod is evicted from the node,
89
+
all corresponding containers are also evicted, along with their logs.
80
90
81
-
By default, if a container restarts, the kubelet keeps one terminated container with its logs. If a pod is evicted from the node, all corresponding containers are also evicted, along with their logs.
91
+
The kubelet makes logs available to clients via a special feature of the Kubernetes API. The usual way to access this is
92
+
by running `kubectl logs`.
82
93
83
-
An important consideration in node-level logging is implementing log rotation,
84
-
so that logs don't consume all available storage on the node. Kubernetes
85
-
is not responsible for rotating logs, but rather a deployment tool
86
-
should set up a solution to address that.
87
-
For example, in Kubernetes clusters, deployed by the `kube-up.sh` script,
88
-
there is a [`logrotate`](https://linux.die.net/man/8/logrotate)
89
-
tool configured to run each hour. You can also set up a container runtime to
90
-
rotate an application's logs automatically.
94
+
### Log rotation
91
95
92
-
As an example, you can find detailed information about how `kube-up.sh` sets
93
-
up logging for COS image on GCP in the corresponding
When using a **CRI container runtime**, the kubelet is responsible for rotating the logs and managing the logging directory structure.
97
-
The kubelet sends this information to the CRI container runtime and the runtime writes the container logs to the given location.
98
-
The two kubelet parameters [`containerLogMaxSize` and `containerLogMaxFiles`](/docs/reference/config-api/kubelet-config.v1beta1/#kubelet-config-k8s-io-v1beta1-KubeletConfiguration)
99
-
in [kubelet config file](/docs/tasks/administer-cluster/kubelet-config-file/)
100
-
can be used to configure the maximum size for each log file and the maximum number of files allowed for each container respectively.
98
+
You can configure the kubelet to rotate logs automatically.
99
+
100
+
If you configure rotation, the kubelet is responsible for rotating container logs and managing the logging directory structure.
101
+
The kubelet sends this information to the container runtime (using CRI),
102
+
and the runtime writes the container logs to the given location.
103
+
104
+
You can configure two kubelet [configuration settings](/docs/reference/config-api/kubelet-config.v1beta1/#kubelet-config-k8s-io-v1beta1-KubeletConfiguration),
105
+
`containerLogMaxSize` and `containerLogMaxFiles`,
106
+
using the [kubelet configuration file](/docs/tasks/administer-cluster/kubelet-config-file/).
107
+
These settings let you configure the maximum size for each log file and the maximum number of files allowed for each container respectively.
101
108
102
109
When you run [`kubectl logs`](/docs/reference/generated/kubectl/kubectl-commands#logs) as in
103
110
the basic logging example, the kubelet on the node handles the request and
104
111
reads directly from the log file. The kubelet returns the content of the log file.
105
112
113
+
106
114
{{< note >}}
107
-
If an external system has performed the rotation or a CRI container runtime is used,
108
-
only the contents of the latest log file will be available through
109
-
`kubectl logs`. For example, if there's a 10MB file, `logrotate` performs
110
-
the rotation and there are two files: one file that is 10MB in size and a second file that is empty.
111
-
`kubectl logs` returns the latest log file which in this example is an empty response.
115
+
Only the contents of the latest log file are available through
116
+
`kubectl logs`.
117
+
118
+
For example, if a Pod writes 40 MiB of logs and the kubelet rotates logs
119
+
after 10 MiB, running `kubectl logs` returns at most 10MiB of data.
112
120
{{< /note >}}
113
121
114
-
### System component logs
122
+
## System component logs
123
+
124
+
There are two types of system components: those that typically run in a container,
125
+
and those components directly involved in running containers. For example:
126
+
127
+
* The kubelet and container runtime do not run in containers. The kubelet runs
128
+
your containers (grouped together in {{< glossary_tooltip text="pods" term_id="pod" >}})
129
+
* The Kubernetes scheduler, controller manager, and API server run within pods
The etcd component runs in the control plane, and most commonly also as a static pod.
132
+
If your cluster uses kube-proxy, you typically run this as a `DaemonSet`.
133
+
134
+
### Log locations {#log-location-node}
135
+
136
+
The way that the kubelet and container runtime write logs depends on the operating
137
+
system that the node uses:
138
+
139
+
{{< tabs name="log_location_node_tabs" >}}
140
+
{{% tab name="Linux" %}}
141
+
142
+
On Linux nodes that use systemd, the kubelet and container runtime write to journald
143
+
by default. You use `journalctl` to read the systemd journal; for example:
144
+
`journalctl -u kubelet`.
145
+
146
+
If systemd is not present, the kubelet and container runtime write to `.log` files in the
147
+
`/var/log` directory. If you want to have logs written elsewhere, you can indirectly
148
+
run the kubelet via a helper tool, `kube-log-runner`, and use that tool to redirect
149
+
kubelet logs to a directory that you choose.
150
+
151
+
You can also set a logging directory using the deprecated kubelet command line
152
+
argument `--log-dir`. However, the kubelet always directs your container runtime to
153
+
write logs into directories within `/var/log/pods`.
115
154
116
-
There are two types of system components: those that run in a container and those
117
-
that do not run in a container. For example:
155
+
For more information on `kube-log-runner`, read [System Logs](/docs/concepts/cluster-administration/system-logs/#klog).
118
156
119
-
* The Kubernetes scheduler and kube-proxy run in a container.
120
-
* The kubelet and container runtime do not run in containers.
157
+
{{% /tab %}}
158
+
{{% tab name="Windows" %}}
121
159
122
-
On machines with systemd, the kubelet and container runtime write to journald. If
123
-
systemd is not present, the kubelet and container runtime write to `.log` files
124
-
in the `/var/log` directory. System components inside containers always write
125
-
to the `/var/log` directory, bypassing the default logging mechanism.
126
-
They use the [`klog`](https://github.com/kubernetes/klog)
127
-
logging library. You can find the conventions for logging severity for those
128
-
components in the [development docs on logging](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-instrumentation/logging.md).
160
+
By default, the kubelet writes logs to files within the directory `C:\var\logs`
161
+
(notice that this is not `C:\var\log`).
129
162
130
-
Similar to the container logs, system component logs in the `/var/log`
131
-
directory should be rotated. In Kubernetes clusters brought up by
132
-
the `kube-up.sh` script, those logs are configured to be rotated by
133
-
the `logrotate` tool daily or once the size exceeds 100MB.
163
+
Although `C:\var\log` is the Kubernetes default location for these logs, several
164
+
cluster deployment tools set up Windows nodes to log to `C:\var\log\kubelet` instead.
165
+
166
+
If you want to have logs written elsewhere, you can indirectly
167
+
run the kubelet via a helper tool, `kube-log-runner`, and use that tool to redirect
168
+
kubelet logs to a directory that you choose.
169
+
170
+
However, the kubelet always directs your container runtime to write logs within the
171
+
directory `C:\var\log\pods`.
172
+
173
+
For more information on `kube-log-runner`, read [System Logs](/docs/concepts/cluster-administration/system-logs/#klog).
174
+
{{% /tab %}}
175
+
{{< /tabs >}}
176
+
177
+
<br /><!-- work around rendering nit -->
178
+
179
+
For Kubernetes cluster components that run in pods, these write to files inside
180
+
the `/var/log` directory, bypassing the default logging mechanism (the components
181
+
do not write to the systemd journal). You can use Kubernetes' storage mechanisms
182
+
to map persistent storage into the container that runs the component.
183
+
184
+
For details about etcd and its logs, view the [etcd documentation](https://etcd.io/docs/).
185
+
Again, you can use Kubernetes' storage mechanisms to map persistent storage into
186
+
the container that runs the component.
187
+
188
+
{{< note >}}
189
+
If you deploy Kubernetes cluster components (such as the scheduler) to log to
190
+
a volume shared from the parent node, you need to consider and ensure that those
191
+
logs are rotated. **Kubernetes does not manage that log rotation**.
192
+
193
+
Your operating system may automatically implement some log rotation - for example,
194
+
if you share the directory `/var/log` into a static Pod for a component, node-level
195
+
log rotation treats a file in that directory the same as a file written by any component
196
+
outside Kubernetes.
197
+
198
+
Some deploy tools account for that log rotation and automate it; others leave this
199
+
as your responsibility.
200
+
{{< /note >}}
134
201
135
202
## Cluster-level logging architectures
136
203
@@ -178,7 +245,7 @@ like `kubectl logs`.
178
245
179
246
For example, a pod runs a single container, and the container
180
247
writes to two different log files using two different formats. Here's a
In the sample configurations, you can replace fluentd with any logging agent, reading from any source inside an application container.
270
-
271
339
### Exposing logs directly from the application
272
340
273
341
274
342
275
343
Cluster-logging that exposes or pushes logs directly from every application is outside the scope of Kubernetes.
344
+
345
+
## {{% heading "whatsnext" %}}
346
+
347
+
* Read about [Kubernetes system logs](/docs/concepts/cluster-administration/system-logs/)
348
+
* Learn about [Traces For Kubernetes System Components](/docs/concepts/cluster-administration/system-traces/)
349
+
* Learn how to [customise the termination message](/docs/tasks/debug/debug-application/determine-reason-pod-failure/#customizing-the-termination-message) that Kubernetes records when a Pod fails
0 commit comments