@@ -96,16 +96,16 @@ This would work for:
96
96
## Proposal
97
97
98
98
### Implement client for logs endpoint viewer (OS agnostic)
99
- - Extend ` kubectl logs ` to work with node objects.
99
+ - Implement a new ` kubectl node- logs ` to work with node objects.
100
100
- Implement a client for the ` /var/log/ ` kubelet endpoint viewer.
101
101
102
102
### Linux distros with systemd / journald
103
103
Supplement the the ` /var/log/ ` endpoint viewer on the kubelet with a thin shim
104
- over the ` journal ` directory that shells out to journalctl. Then extend
105
- ` kubectl logs ` to also work with node objects.
104
+ over the ` journal ` directory that shells out to journalctl. Then implement
105
+ ` kubectl node- logs ` to also work with node objects.
106
106
107
107
### Linux distributions without systemd / journald
108
- Running the new "kubectl logs nodes " command against services on nodes that do
108
+ Running the new "kubectl node- logs" command against services on nodes that do
109
109
not use systemd / journald should return "OS not supported". However getting
110
110
logs from ` /var/log/ ` should work on all systems.
111
111
@@ -118,22 +118,22 @@ Reuse the kubelet API for querying the Linux journal for invoking the
118
118
Consider a scenario where pods / containers are refusing to come up on certain
119
119
nodes. As mentioned in the motivation section, troubleshooting this scenario
120
120
involves the cluster administrator to SSH into nodes to scan the logs. Allowing
121
- them to use ` kubectl logs ` to do the same as they would to debug issues with a
121
+ them to use ` kubectl node- logs ` to do the same as they would to debug issues with a
122
122
pod / container would greatly simply their debug workflow. This also opens up
123
123
opportunities for tooling and simplifying automated log gathering. The feature
124
124
can also be used to debug issues with Kubernetes services especially in Windows
125
125
nodes that run as native Windows services and not as DaemonSets or Deployments.
126
126
127
- Here are some example of how a cluser administrator would use this feature:
127
+ Here are some example of how a cluster administrator would use this feature:
128
128
```
129
129
# Show kubelet and crio journal logs from all masters
130
- kubectl logs nodes --role master -s kubelet -s crio
130
+ kubectl node- logs --role master -q kubelet -q crio
131
131
132
132
# Show kubelet log file (/var/log/kubelet/kubelet.log) from all Windows worker nodes
133
- kubectl logs nodes --label kubernetes.io/os=windows -s kubelet
133
+ kubectl node- logs --label kubernetes.io/os=windows -q kubelet
134
134
135
135
# Display docker runtime WinEvent log entries from a specific Windows worker node
136
- kubectl logs nodes <node-name> --service docker
136
+ kubectl node- logs <node-name> --query docker
137
137
```
138
138
139
139
### Risks and Mitigations
@@ -163,7 +163,7 @@ that is lacking a client. Given its existence we can supplement that with a
163
163
wafer thin shim over the /journal directory that shells out to journalctl. This
164
164
allows us to extend the endpoint for getting logs from the system journal on
165
165
Linux systems that support systemd. To enable filtering of logs, we can reuse
166
- the existing filters supported by journalctl. The ` kubectl logs ` will have
166
+ the existing filters supported by journalctl. The ` kubectl node- logs ` will have
167
167
command line options for specifying these filters when interacting with node
168
168
objects.
169
169
@@ -196,8 +196,8 @@ you need to enable both options to get access to this new feature and disabling
196
196
` enableSystemLogHandler ` will disable the new feature irrespective of the
197
197
` NodeLogViewer ` feature gate.
198
198
199
- A reference implementation of this feature without the feature gate is
200
- available [ here] ( https://github.com/kubernetes/kubernetes/pull/96120 ) .
199
+ A reference implementation of this feature is available
200
+ [ here] ( https://github.com/kubernetes/kubernetes/pull/96120 ) .
201
201
202
202
#### kubectl
203
203
@@ -209,36 +209,42 @@ to appropriate resource type and associated endpoints, it will allow us to
209
209
restrict node logs access to only cluster administrators as long as the cluster
210
210
is setup in that manner. Access to the ` node/logs ` sub-resource needs to be
211
211
explicitly granted as a user with access to ` nodes ` will not automatically have
212
- access to ` node/logs ` .
212
+ access to ` node/logs ` . In the alpha phase the functionality will be behind
213
+ ` kubectl alpha node-logs ` sub-command. The functionality will be moved to
214
+ ` kubectl node-logs ` in the beta phase. However the examples will reference the
215
+ final destination i.e. ` kubectl node-logs ` .
213
216
214
- The ` logs ` sub-command for node objects will follow a heuristics approach when
217
+ The ` logs --query ` sub-command for node objects will follow a heuristics approach when
215
218
asked to query for logs from a Windows or Linux service. If asked to get the
216
219
logs from a service ` foobar ` , it will first assume ` foobar ` logs to the Linux
217
220
journal / Windows eventing mechanisms (Application, System, and ETW). If unable
218
221
to get logs from these, it will attempt to get logs from ` /var/log/foobar.log ` ,
219
222
` /var/log/foobar/foobar.log ` , ` /var/log/foobar*INFO ` or
220
- ` /var/log/foobar/foobar*INFO ` in that order.
223
+ ` /var/log/foobar/foobar*INFO ` in that order. Alternatively an explicit file
224
+ location can be passed to the ` --query ` option.
221
225
Here are some examples and explanation of the options that will be added.
222
226
```
223
227
Examples:
224
228
# Show kubelet logs from all masters
225
- kubectl logs nodes --role master -s kubelet
229
+ kubectl node- logs --role master -q kubelet
226
230
227
231
# Show docker logs from Windows nodes
228
- kubectl logs nodes -l kubernetes.io/os=windows -s docker
232
+ kubectl node-logs -l kubernetes.io/os=windows -q docker
233
+
234
+ # Show foo.log from Windows nodes
235
+ kubectl node-logs -l kubernetes.io/os=windows -q /foo/foo.log
229
236
230
237
Options:
231
- --case-sensitive=true: Filters are case sensitive by default. Pass --case-sensitive=false to do a case insensitive filter.
232
238
-g, --grep='': Filter log entries by the provided regex pattern. Only applies to node journal logs.
233
239
-o, --output='': Display journal logs in an alternate format (short, cat, json, short-unix). Only applies to node journal logs.
234
240
--raw=false: Perform no transformation of the returned data.
235
241
--role='': Set a label selector by node role.
236
242
-l, --selector='': Selector (label query) to filter on.
237
- --since='': Return logs after a specific ISO timestamp or relative date. Only applies to node journal or Get-WinEvent logs .
238
- --tail=0 : Return up to this many lines (not more than 100k) from the end of the log. Only applies to node journal or Get-WinEvent logs .
243
+ --since-time ='': Return logs after a specific ISO timestamp.
244
+ --tail=-1 : Return up to this many lines (not more than 100k) from the end of the log.
239
245
--sort=timestamp: Interleave logs by sorting the output. Defaults on when viewing node journal logs.
240
- -s , --service =[]: Return log entries from the specified service(s).
241
- --until='': Return logs before a specific ISO timestamp or relative date .
246
+ -q , --query =[]: Return log entries that matches any of the specified service(s).
247
+ --until-time ='': Return logs before a specific ISO timestamp.
242
248
```
243
249
244
250
The ` --sort=timestamp ` feature will introduce log unification across node
@@ -247,11 +253,12 @@ to see logs across nodes from the same time. Similarly for pods, it will allow
247
253
seeing logs across containers aligned by time.
248
254
249
255
Given that the feature will be introduced behind a feature gate, by default
250
- ` kubectl logs nodes ` will return a feature not enabled message. When the
251
- feature is enabled in alpha phase, ` kubectl logs nodes ` will display a
252
- warning message that the feature is in alpha. When the ` --service ` option
256
+ ` kubectl node- logs ` will return a functionality not available message. When the
257
+ feature is enabled in alpha phase, ` kubectl node- logs ` will display a
258
+ warning message that the feature is in alpha. When the ` --query ` option
253
259
is used against Linux nodes that do not support systemd/journald and the service
254
- does not log to ` /var/log ` , an OS not supported message will be returned.
260
+ does not log to ` /var/log ` , the same functionality not available message will be
261
+ returned.
255
262
256
263
### Test Plan
257
264
Add unit tests to kubelet and kubectl that exercise the new arguments that
@@ -261,7 +268,8 @@ have been added. A reference implementation of the tests can be seen
261
268
### Graduation Criteria
262
269
263
270
The plan is to introduce the feature as alpha in the v1.25 time frame behind the
264
- ` NodeLogs ` feature gate.
271
+ ` NodeLogViewer ` kubelet feature gate and using the ` kubectl alpha node-logs `
272
+ sub-command.
265
273
266
274
#### Alpha -> Beta Graduation
267
275
@@ -273,6 +281,8 @@ logs. In addition we will garner feedback on the heuristic approach and based on
273
281
that we will decide if we need introduce options to explicitly differentiate
274
282
between file vs journal / WinEvent logs.
275
283
284
+ The kubectl implementation will move from ` kubectl alpha node-logs ` to
285
+ ` kubectl node-logs ` .
276
286
#### Beta -> GA Graduation
277
287
278
288
The plan is to graduate the feature to GA in the v1.27 time frame at which point
@@ -283,7 +293,7 @@ beta phases.
283
293
284
294
### Version Skew Strategy
285
295
286
- If a kubectl version that has the new ` logs nodes ` option is used against a node
296
+ If a kubectl version that has the new ` node- logs` option is used against a node
287
297
that is using a kubelet that does not have the extended ` /var/log ` endpoint
288
298
viewer, the result should be "feature not supported".
289
299
@@ -373,5 +383,5 @@ logs. The Windows side would require privileged container support. However this
373
383
would not help scenarios where containers are not launching successfully on the
374
384
nodes.
375
385
376
- For the kubectl changes an alternative to extending ` kubect logs` would be to
377
- introduce a plugin or add a new sub-command under ` kubectl alpha ` .
386
+ For the kubectl changes an alternative to introducing ` kubectl node- logs` would be to
387
+ introduce a plugin.
0 commit comments