Skip to content

Commit 5b727ac

Browse files
committed
KEP-2258: Simply API and kubectl details
- Simplify the API as per thockin's feedback in kubernetes/kubernetes#96120 - Introduce feature during alpha behind `kubectl alpha node-logs` - Use `kubectl node-logs` instead of `kubectl logs nodes`
1 parent 73bd848 commit 5b727ac

File tree

1 file changed

+40
-30
lines changed
  • keps/sig-windows/2258-node-service-log-viewer

1 file changed

+40
-30
lines changed

keps/sig-windows/2258-node-service-log-viewer/README.md

Lines changed: 40 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -96,16 +96,16 @@ This would work for:
9696
## Proposal
9797

9898
### Implement client for logs endpoint viewer (OS agnostic)
99-
- Extend `kubectl logs` to work with node objects.
99+
- Implement a new `kubectl node-logs` to work with node objects.
100100
- Implement a client for the `/var/log/` kubelet endpoint viewer.
101101

102102
### Linux distros with systemd / journald
103103
Supplement the the `/var/log/` endpoint viewer on the kubelet with a thin shim
104-
over the `journal` directory that shells out to journalctl. Then extend
105-
`kubectl logs` to also work with node objects.
104+
over the `journal` directory that shells out to journalctl. Then implement
105+
`kubectl node-logs` to also work with node objects.
106106

107107
### Linux distributions without systemd / journald
108-
Running the new "kubectl logs nodes" command against services on nodes that do
108+
Running the new "kubectl node-logs" command against services on nodes that do
109109
not use systemd / journald should return "OS not supported". However getting
110110
logs from `/var/log/` should work on all systems.
111111

@@ -118,22 +118,22 @@ Reuse the kubelet API for querying the Linux journal for invoking the
118118
Consider a scenario where pods / containers are refusing to come up on certain
119119
nodes. As mentioned in the motivation section, troubleshooting this scenario
120120
involves the cluster administrator to SSH into nodes to scan the logs. Allowing
121-
them to use `kubectl logs` to do the same as they would to debug issues with a
121+
them to use `kubectl node-logs` to do the same as they would to debug issues with a
122122
pod / container would greatly simply their debug workflow. This also opens up
123123
opportunities for tooling and simplifying automated log gathering. The feature
124124
can also be used to debug issues with Kubernetes services especially in Windows
125125
nodes that run as native Windows services and not as DaemonSets or Deployments.
126126

127-
Here are some example of how a cluser administrator would use this feature:
127+
Here are some example of how a cluster administrator would use this feature:
128128
```
129129
# Show kubelet and crio journal logs from all masters
130-
kubectl logs nodes --role master -s kubelet -s crio
130+
kubectl node-logs --role master -q kubelet -q crio
131131
132132
# Show kubelet log file (/var/log/kubelet/kubelet.log) from all Windows worker nodes
133-
kubectl logs nodes --label kubernetes.io/os=windows -s kubelet
133+
kubectl node-logs --label kubernetes.io/os=windows -q kubelet
134134
135135
# Display docker runtime WinEvent log entries from a specific Windows worker node
136-
kubectl logs nodes <node-name> --service docker
136+
kubectl node-logs <node-name> --query docker
137137
```
138138

139139
### Risks and Mitigations
@@ -163,7 +163,7 @@ that is lacking a client. Given its existence we can supplement that with a
163163
wafer thin shim over the /journal directory that shells out to journalctl. This
164164
allows us to extend the endpoint for getting logs from the system journal on
165165
Linux systems that support systemd. To enable filtering of logs, we can reuse
166-
the existing filters supported by journalctl. The `kubectl logs` will have
166+
the existing filters supported by journalctl. The `kubectl node-logs` will have
167167
command line options for specifying these filters when interacting with node
168168
objects.
169169

@@ -196,8 +196,8 @@ you need to enable both options to get access to this new feature and disabling
196196
`enableSystemLogHandler` will disable the new feature irrespective of the
197197
`NodeLogViewer` feature gate.
198198

199-
A reference implementation of this feature without the feature gate is
200-
available [here](https://github.com/kubernetes/kubernetes/pull/96120).
199+
A reference implementation of this feature is available
200+
[here](https://github.com/kubernetes/kubernetes/pull/96120).
201201

202202
#### kubectl
203203

@@ -209,36 +209,42 @@ to appropriate resource type and associated endpoints, it will allow us to
209209
restrict node logs access to only cluster administrators as long as the cluster
210210
is setup in that manner. Access to the `node/logs` sub-resource needs to be
211211
explicitly granted as a user with access to `nodes` will not automatically have
212-
access to `node/logs`.
212+
access to `node/logs`. In the alpha phase the functionality will be behind
213+
`kubectl alpha node-logs` sub-command. The functionality will be moved to
214+
`kubectl node-logs` in the beta phase. However the examples will reference the
215+
final destination i.e. `kubectl node-logs`.
213216

214-
The `logs` sub-command for node objects will follow a heuristics approach when
217+
The `logs --query` sub-command for node objects will follow a heuristics approach when
215218
asked to query for logs from a Windows or Linux service. If asked to get the
216219
logs from a service `foobar`, it will first assume `foobar` logs to the Linux
217220
journal / Windows eventing mechanisms (Application, System, and ETW). If unable
218221
to get logs from these, it will attempt to get logs from `/var/log/foobar.log`,
219222
`/var/log/foobar/foobar.log`, `/var/log/foobar*INFO` or
220-
`/var/log/foobar/foobar*INFO` in that order.
223+
`/var/log/foobar/foobar*INFO` in that order. Alternatively an explicit file
224+
location can be passed to the `--query` option.
221225
Here are some examples and explanation of the options that will be added.
222226
```
223227
Examples:
224228
# Show kubelet logs from all masters
225-
kubectl logs nodes --role master -s kubelet
229+
kubectl node-logs --role master -q kubelet
226230
227231
# Show docker logs from Windows nodes
228-
kubectl logs nodes -l kubernetes.io/os=windows -s docker
232+
kubectl node-logs -l kubernetes.io/os=windows -q docker
233+
234+
# Show foo.log from Windows nodes
235+
kubectl node-logs -l kubernetes.io/os=windows -q /foo/foo.log
229236
230237
Options:
231-
--case-sensitive=true: Filters are case sensitive by default. Pass --case-sensitive=false to do a case insensitive filter.
232238
-g, --grep='': Filter log entries by the provided regex pattern. Only applies to node journal logs.
233239
-o, --output='': Display journal logs in an alternate format (short, cat, json, short-unix). Only applies to node journal logs.
234240
--raw=false: Perform no transformation of the returned data.
235241
--role='': Set a label selector by node role.
236242
-l, --selector='': Selector (label query) to filter on.
237-
--since='': Return logs after a specific ISO timestamp or relative date. Only applies to node journal or Get-WinEvent logs.
238-
--tail=0: Return up to this many lines (not more than 100k) from the end of the log. Only applies to node journal or Get-WinEvent logs.
243+
--since-time='': Return logs after a specific ISO timestamp.
244+
--tail=-1: Return up to this many lines (not more than 100k) from the end of the log.
239245
--sort=timestamp: Interleave logs by sorting the output. Defaults on when viewing node journal logs.
240-
-s, --service=[]: Return log entries from the specified service(s).
241-
--until='': Return logs before a specific ISO timestamp or relative date.
246+
-q, --query=[]: Return log entries that matches any of the specified service(s).
247+
--until-time='': Return logs before a specific ISO timestamp.
242248
```
243249

244250
The `--sort=timestamp` feature will introduce log unification across node
@@ -247,11 +253,12 @@ to see logs across nodes from the same time. Similarly for pods, it will allow
247253
seeing logs across containers aligned by time.
248254

249255
Given that the feature will be introduced behind a feature gate, by default
250-
`kubectl logs nodes` will return a feature not enabled message. When the
251-
feature is enabled in alpha phase, `kubectl logs nodes` will display a
252-
warning message that the feature is in alpha. When the `--service` option
256+
`kubectl node-logs` will return a functionality not available message. When the
257+
feature is enabled in alpha phase, `kubectl node-logs` will display a
258+
warning message that the feature is in alpha. When the `--query` option
253259
is used against Linux nodes that do not support systemd/journald and the service
254-
does not log to `/var/log`, an OS not supported message will be returned.
260+
does not log to `/var/log`, the same functionality not available message will be
261+
returned.
255262

256263
### Test Plan
257264
Add unit tests to kubelet and kubectl that exercise the new arguments that
@@ -261,7 +268,8 @@ have been added. A reference implementation of the tests can be seen
261268
### Graduation Criteria
262269

263270
The plan is to introduce the feature as alpha in the v1.25 time frame behind the
264-
`NodeLogs` feature gate.
271+
`NodeLogViewer` kubelet feature gate and using the `kubectl alpha node-logs`
272+
sub-command.
265273

266274
#### Alpha -> Beta Graduation
267275

@@ -273,6 +281,8 @@ logs. In addition we will garner feedback on the heuristic approach and based on
273281
that we will decide if we need introduce options to explicitly differentiate
274282
between file vs journal / WinEvent logs.
275283

284+
The kubectl implementation will move from `kubectl alpha node-logs` to
285+
`kubectl node-logs`.
276286
#### Beta -> GA Graduation
277287

278288
The plan is to graduate the feature to GA in the v1.27 time frame at which point
@@ -283,7 +293,7 @@ beta phases.
283293

284294
### Version Skew Strategy
285295

286-
If a kubectl version that has the new `logs nodes` option is used against a node
296+
If a kubectl version that has the new `node-logs` option is used against a node
287297
that is using a kubelet that does not have the extended `/var/log` endpoint
288298
viewer, the result should be "feature not supported".
289299

@@ -373,5 +383,5 @@ logs. The Windows side would require privileged container support. However this
373383
would not help scenarios where containers are not launching successfully on the
374384
nodes.
375385

376-
For the kubectl changes an alternative to extending `kubect logs` would be to
377-
introduce a plugin or add a new sub-command under `kubectl alpha`.
386+
For the kubectl changes an alternative to introducing `kubectl node-logs` would be to
387+
introduce a plugin.

0 commit comments

Comments
 (0)