fix(nodes): nodes_log query and tailLines arguments #409

ngopalak-redhat · 2025-10-27T04:40:40Z

I'm using the recently added Node log query to get Kubelet logs. It didn't work even after enabling the feature gate NodeLogQuery.
A small fix was required to using the "query" param as per the docs: https://kubernetes.io/docs/concepts/cluster-administration/system-logs/

# Fetch kubelet logs from a node named node-1.example
kubectl get --raw "/api/v1/nodes/node-1.example/proxy/logs/?query=kubelet"

Tested with OpenShift Cluster. Here are the test results:
** Before **

Using kubernetes-local-mcp-server mcp server get the node logs of ip-10-0-125-163.ec2.internal with the query "kubelet" as log_path

⏺ I'll get the node logs from ip-10-0-125-163.ec2.internal with the kubelet log path.

⏺ kubernetes-local-mcp-server - Node: Log (MCP)(name: "ip-10-0-125-163.ec2.internal", log_path: "kubelet")
  ⎿  Error: failed to get node log for ip-10-0-125-163.ec2.internal: failed to get node logs: the server could not find the requested resource

** After **


> Using kubernetes-local-mcp-server mcp server get the node logs of ip-10-0-125-163.ec2.internal with the query "kubelet" as log_path

⏺ I'll get the node logs for you using the kubernetes-local-mcp-server.

⏺ kubernetes-local-mcp-server - Node: Log (MCP)(name: "ip-10-0-125-163.ec2.internal", log_path: "kubelet")
  ⎿  ⚠ Large MCP response (~475.7k tokens), this can fill up context quickly
  ⎿ Oct 27 01:26:39.357782 ip-10-0-125-163 systemd[1]: Starting Kubernetes Kubelet...
    Oct 27 01:26:39.789882 ip-10-0-125-163 kubenswrapper[2501]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the c
    onfig file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
    … +16108 lines (ctrl+o to expand)

Error 413 is expected. Its a claude issue. But with this fix, MCP server should be able to fetch kubelet logs using the proxy API.

Signed-off-by: Neeraj Krishna Gopalakrishna <[email protected]>

ngopalak-redhat · 2025-10-27T04:45:42Z

/assign @manusa as you recently worked on this file

ngopalak-redhat · 2025-10-27T05:19:09Z

cc: @blublinsky

pkg/kubernetes/accesscontrol_clientset.go

matzew · 2025-10-27T08:22:34Z

@blublinsky mind checking the PR also in the context of these comment: https://github.com/containers/kubernetes-mcp-server/blob/main/pkg/kubernetes/nodes.go#L8-L13

To reflect the correct usage pattern?

manusa · 2025-10-27T08:27:40Z

This was originally implemented by @blublinsky
In the original implementation there was no support added for query, we might actually want to repurpose to use query instead of logpath but that should be reviewed by their team.
I found some gaps in kind's implementation while reviewing this, especially in the URL query parameter handling. The docs also have the query params mid path, which doesn't clarify either what's the expected behavior.

I'd advise that @blublinsky and others who actually rely on this, chimed in and described what do they exactly need.
Then we can actually properly define the feature, the tool, its accepted arguments, etc. Implementation would be the easiest part.

blublinsky · 2025-10-27T08:51:30Z

I think it's fine as long as the capability does not change. We rely on LLM to do the right invocation, so it should not matter

Signed-off-by: Neeraj Krishna Gopalakrishna <[email protected]>

ngopalak-redhat · 2025-10-27T09:39:12Z

This was originally implemented by @blublinsky In the original implementation there was no support added for query, we might actually want to repurpose to use query instead of logpath but that should be reviewed by their team. I found some gaps in kind's implementation while reviewing this, especially in the URL query parameter handling. The docs also have the query params mid path, which doesn't clarify either what's the expected behavior.

I'd advise that @blublinsky and others who actually rely on this, chimed in and described what do they exactly need. Then we can actually properly define the feature, the tool, its accepted arguments, etc. Implementation would be the easiest part.

@manusa Thanks for reviewing. The query parameter is added based on the public docs https://kubernetes.io/docs/concepts/cluster-administration/system-logs/ and I tested it to be working with Kubernetes cluster. We might need to broaden the usage itself later to use the many other options it supports. For now this is a bug fix

matzew · 2025-10-27T15:30:04Z

Thanks for the test fix.

LGTM

@blublinsky any final review, or are we good to go with this?

manusa · 2025-10-27T16:18:21Z

This was originally implemented by @blublinsky In the original implementation there was no support added for query, we might actually want to repurpose to use query instead of logpath but that should be reviewed by their team. I found some gaps in kind's implementation while reviewing this, especially in the URL query parameter handling. The docs also have the query params mid path, which doesn't clarify either what's the expected behavior.
I'd advise that @blublinsky and others who actually rely on this, chimed in and described what do they exactly need. Then we can actually properly define the feature, the tool, its accepted arguments, etc. Implementation would be the easiest part.

@manusa Thanks for reviewing. The query parameter is added based on the public docs https://kubernetes.io/docs/concepts/cluster-administration/system-logs/ and I tested it to be working with Kubernetes cluster. We might need to broaden the usage itself later to use the many other options it supports. For now this is a bug fix

Yes, I know. I linked those in the PR I created (#399) as a follow up of the original implementation #384

The problem I find here is that the docs are nor clear since the path:

/api/v1/nodes/node-1.example/proxy/logs/$logFileName

does work and is what was implemented in the first place.

The other problem I see is that what's stated in the docs (logFileName is part of the URL query params -not to confuse with the query parameter-):

/api/v1/nodes/node-1.example/proxy/logs/?query=/$logFileName

doesn't behave as expected at least with a Kind cluster.

Let's proceed this way.
I'll add an extra-commit to your branch to refactor the tool so that the LLM is actually responsible of passing the arguments as the documentation reflects.

Once it's ready, I'll let you know, and both @blublinsky and you can confirm that your use-cases are still functional.

Signed-off-by: Marc Nuri <[email protected]>

manusa · 2025-10-27T17:34:05Z

@ngopalak-redhat @blublinsky I added a subsequent commit to the branch with the changes to make the tool resemble the official API exposed in the docs.

Please, check if these changes work with your use-cases. It should be compatible with both your approaches.

ngopalak-redhat · 2025-10-28T00:52:26Z

@ngopalak-redhat @blublinsky I added a subsequent commit to the branch with the changes to make the tool resemble the official API exposed in the docs.

Please, check if these changes work with your use-cases. It should be compatible with both your approaches.

@manusa I have tested with my cluster and its working now:

Using kubernetes-local-mcp-server mcp server get the node logs of ip-10-0-54-152.ec2.internal with the query "kubelet" as log_path

⏺ I'll get the kubelet logs from the node ip-10-0-54-152.ec2.internal using the kubernetes-local-mcp-server.

⏺ kubernetes-local-mcp-server - Node: Log (MCP)(name: "ip-10-0-54-152.ec2.internal", query: "kubelet")
  ⎿  ⚠ Large MCP response (~206.4k tokens), this can fill up context quickly
  ⎿  Oct 27 23:32:39.634961 ip-10-0-54-152 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
     Oct 27 23:32:39.634972 ip-10-0-54-152 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
     Oct 27 23:32:39.634980 ip-10-0-54-152 systemd[1]: kubelet.service: Failed with result 'resources'.
     … +6856 lines (ctrl+o to expand)
  ⎿  API Error: 413 {"type":"error","error":{"type":"invalid_request_error","message":"Prompt is too long"},"request_id":"req_vrtx_011CUYgCKY8siZAGFKj4ARbJ"}

Also tested the tailLines:

Using kubernetes-local-mcp-server mcp server get the node logs of ip-10-0-54-152.ec2.internal with the query "kubelet" as log_path and tailLines 10

⏺ I'll get the kubelet logs from the node ip-10-0-54-152.ec2.internal with the last 10 lines.

⏺ kubernetes-local-mcp-server - Node: Log (MCP)(name: "ip-10-0-54-152.ec2.internal", query: "kubelet", tailLines: 10)
  ⎿ Oct 28 00:39:17.629107 ip-10-0-54-152 kubenswrapper[2499]: I1028 00:39:17.627790    2499 log.go:25] "Finished parsing log file" path="/var/log/pods/opens
    hift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-54-152.ec2.internal_c85ffeb3c02f5aa8ca578dc3d61838ec/kube-rbac-proxy-crio/3.log"
    Oct 28 00:39:17.629107 ip-10-0-54-152 kubenswrapper[2499]: I1028 00:39:17.628771    2499 log.go:25] "Finished parsing log file" path="/var/log/pods/opens
    … +17 lines (ctrl+o to expand)



⏺ Here are the last 10 lines of kubelet logs from node ip-10-0-54-152.ec2.internal:

  Oct 28 00:39:17.629107 ip-10-0-54-152 kubenswrapper[2499]: I1028 00:39:17.627790    2499 log.go:25] "Finished parsing log file" path="/var/log/pods/openshif
  t-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-54-152.ec2.internal_c85ffeb3c02f5aa8ca578dc3d61838ec/kube-rbac-proxy-crio/3.log"
  Oct 28 00:39:17.629107 ip-10-0-54-152 kubenswrapper[2499]: I1028 00:39:17.628771    2499 log.go:25] "Finished parsing log file" path="/var/log/pods/openshif
  t-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-54-152.ec2.internal_c85ffeb3c02f5aa8ca578dc3d61838ec/kube-rbac-proxy-crio/3.log"
  Oct 28 00:44:17.652803 ip-10-0-54-152 kubenswrapper[2499]: I1028 00:44:17.652692    2499 log.go:25] "Finished parsing log file"
  path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zpwdl_d5a65282-f58c-4695-b5ad-3a5dfb13b0f9/ovn-acl-logging/0.log"
  Oct 28 00:44:17.657069 ip-10-0-54-152 kubenswrapper[2499]: I1028 00:44:17.652872    2499 log.go:25] "Finished parsing log file"
  path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zpwdl_d5a65282-f58c-4695-b5ad-3a5dfb13b0f9/ovn-acl-logging/0.log"
  Oct 28 00:44:17.657069 ip-10-0-54-152 kubenswrapper[2499]: I1028 00:44:17.655646    2499 log.go:25] "Finished parsing log file" path="/var/log/pods/openshif
  t-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-54-152.ec2.internal_c85ffeb3c02f5aa8ca578dc3d61838ec/kube-rbac-proxy-crio/3.log"
  Oct 28 00:44:17.657069 ip-10-0-54-152 kubenswrapper[2499]: I1028 00:44:17.655911    2499 log.go:25] "Finished parsing log file" path="/var/log/pods/openshif
  t-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-54-152.ec2.internal_c85ffeb3c02f5aa8ca578dc3d61838ec/kube-rbac-proxy-crio/3.log"
  Oct 28 00:49:17.678207 ip-10-0-54-152 kubenswrapper[2499]: I1028 00:49:17.678103    2499 log.go:25] "Finished parsing log file"
  path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zpwdl_d5a65282-f58c-4695-b5ad-3a5dfb13b0f9/ovn-acl-logging/0.log"
  Oct 28 00:49:17.682994 ip-10-0-54-152 kubenswrapper[2499]: I1028 00:49:17.678629    2499 log.go:25] "Finished parsing log file"
  path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zpwdl_d5a65282-f58c-4695-b5ad-3a5dfb13b0f9/ovn-acl-logging/0.log"
  Oct 28 00:49:17.682994 ip-10-0-54-152 kubenswrapper[2499]: I1028 00:49:17.681227    2499 log.go:25] "Finished parsing log file" path="/var/log/pods/openshif
  t-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-54-152.ec2.internal_c85ffeb3c02f5aa8ca578dc3d61838ec/kube-rbac-proxy-crio/3.log"
  Oct 28 00:49:17.682994 ip-10-0-54-152 kubenswrapper[2499]: I1028 00:49:17.681451    2499 log.go:25] "Finished parsing log file" path="/var/log/pods/openshif
  t-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-54-152.ec2.internal_c85ffeb3c02f5aa8ca578dc3d61838ec/kube-rbac-proxy-crio/3.log"

  The logs show kubenswrapper (kubelet wrapper) parsing log files for pods running on this node, including:
  - openshift-machine-config-operator (kube-rbac-proxy-crio)
  - openshift-ovn-kubernetes (ovn-acl-logging)

  All entries are informational messages showing normal log parsing activity between 00:39 and 00:49 UTC on October 28.

blublinsky · 2025-10-28T09:57:22Z

/lgtm

Fix the node log query

43c1762

Signed-off-by: Neeraj Krishna Gopalakrishna <[email protected]>

ngopalak-redhat force-pushed the ngopalak/fix_node_path branch from 39b4f3d to 43c1762 Compare October 27, 2025 04:42

ngopalak-redhat marked this pull request as ready for review October 27, 2025 04:44

matzew reviewed Oct 27, 2025

View reviewed changes

pkg/kubernetes/accesscontrol_clientset.go Outdated Show resolved Hide resolved

Fix the node log query

14fa2b3

Signed-off-by: Neeraj Krishna Gopalakrishna <[email protected]>

ngopalak-redhat requested a review from matzew October 27, 2025 09:55

manusa self-assigned this Oct 27, 2025

manusa mentioned this pull request Oct 27, 2025

feat(nodes): nodes_stats_summary tool to get PSI metrics #410

Merged

manusa changed the title ~~Fix the node log query using the proxy~~ fix(nodes): nodes_log query and tailLines arguments Oct 27, 2025

fix(nodes): nodes_log query and tailLines arguments

5553a4e

Signed-off-by: Marc Nuri <[email protected]>

manusa added this to the 0.1.0 milestone Oct 28, 2025

manusa approved these changes Oct 28, 2025

View reviewed changes

manusa merged commit 44053f1 into containers:main Oct 28, 2025
6 checks passed

fix(nodes): nodes_log query and tailLines arguments #409

fix(nodes): nodes_log query and tailLines arguments #409

Uh oh!

Conversation

ngopalak-redhat commented Oct 27, 2025

Uh oh!

ngopalak-redhat commented Oct 27, 2025

Uh oh!

ngopalak-redhat commented Oct 27, 2025

Uh oh!

Uh oh!

matzew commented Oct 27, 2025

Uh oh!

manusa commented Oct 27, 2025

Uh oh!

blublinsky commented Oct 27, 2025

Uh oh!

ngopalak-redhat commented Oct 27, 2025

Uh oh!

matzew commented Oct 27, 2025

Uh oh!

manusa commented Oct 27, 2025

Uh oh!

manusa commented Oct 27, 2025

Uh oh!

ngopalak-redhat commented Oct 28, 2025

Uh oh!

blublinsky commented Oct 28, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants