Kmsg channel closed

[kerus1024]

After deploying or rolling out deployment node-problem-detector, i encounter a kmsg channel closed error on some nodes.
As a result, kernel monitor based metrics are not collected on those nodes.
The affected nodes are not under any significant load, and there are essentially no kernel log messages being generated.

Environment

• Ubuntu Jammy (kernel 5.15.0-x)
• Ubuntu Noble (kernel 6.8.0-x)

log

I1030 06:41:19.605611       1 log_watchers.go:40] Use log watcher of plugin "kmsg"
<REDACTED>
I1030 06:41:19.605732       1 log_watchers.go:40] Use log watcher of plugin "filelog"
I1030 06:41:19.606512       1 k8s_exporter.go:54] Waiting for kube-apiserver to be ready (timeout 5m0s)...
I1030 06:41:19.614361       1 node_problem_detector.go:63] K8s exporter started.
I1030 06:41:19.614493       1 node_problem_detector.go:67] Prometheus exporter started.
I1030 06:41:19.614504       1 log_monitor.go:111] Start log monitor /custom-config/additional-filelog.json
I1030 06:41:19.614541       1 log_watcher.go:80] Start watching filelog
I1030 06:41:19.614549       1 log_monitor.go:111] Start log monitor /config/kernel-monitor.json
I1030 06:41:19.614613       1 log_monitor.go:236] Initialize condition generated: []
I1030 06:41:19.615573       1 log_monitor.go:111] Start log monitor /config/docker-monitor.json
I1030 06:41:19.615599       1 log_monitor.go:236] Initialize condition generated: [{Type:KernelDeadlock Status:False Transition:2025-10-30 06:41:19.615589877 +0000 UTC m=+0.055785295 Reason:KernelHasNoDeadlock Message:kernel has no deadlock} {Type:ReadonlyFilesystem Status:False Transition:2025-10-30 06:41:19.61558997 +0000 UTC m=+0.055785381 Reason:FilesystemIsNotReadOnly Message:Filesystem is not read-only}]
E1030 06:41:19.615656       1 log_watcher_linux.go:105] Kmsg channel closed
E1030 06:41:19.615696       1 log_monitor.go:137] Log channel closed: /config/kernel-monitor.json
I1030 06:41:19.619274       1 log_watcher.go:80] Start watching journald
I1030 06:41:19.619292       1 log_monitor.go:111] Start log monitor /config/systemd-monitor.json
I1030 06:41:19.619325       1 log_monitor.go:236] Initialize condition generated: [{Type:CorruptDockerOverlay2 Status:False Transition:2025-10-30 06:41:19.619318108 +0000 UTC m=+0.059513518 Reason:NoCorruptDockerOverlay2 Message:docker overlay2 is functioning properly}]
I1030 06:41:19.621986       1 log_watcher.go:80] Start watching journald
I1030 06:41:19.622011       1 log_monitor.go:111] Start log monitor /custom-config/additional.json
I1030 06:41:19.622120       1 log_monitor.go:236] Initialize condition generated: []
I1030 06:41:19.623024       1 problem_detector.go:76] Problem detector started
I1030 06:41:19.623053       1 log_monitor.go:236] Initialize condition generated: []
E1030 06:41:19.623115       1 log_watcher_linux.go:105] Kmsg channel closed
E1030 06:41:19.623138       1 log_monitor.go:137] Log channel closed: /custom-config/additional.json

reproduce

On the affected nodes, cat /dev/kmsg exits immediately

root@hostname:~# cat /dev/kmsg
6,2047,25658836,-;microcode: CPU 171: patch_level=0x0a0011d5
cat: /dev/kmsg: Broken pipe

What I've tried

• Running dmesg -C to consume kmsg has no effect.
• journalctl -kf is working normally.
• running echo -n "kerustest" > /dev/kmsg can sometimes resolve the broken pipe problem.

[hakman]

@SergeyKanzhelev @wangzhen127 did you encounter this issue in the past?

[kerus1024]

@fruitbox12

Im seeing EPIPE failures from /dev/kmsg, which indicates a ring buffer overrun scenario.

In my view, the proper fix here is not switching the message tailer to journald, but handling kmsg ring buffer overruns correctly in the parser implementation. Specifically, this code path should treat EPIPE as a recoverable condition (log/drop notification and continue reading, or reopen/retry as needed) rather than terminating the stream permanently:

node-problem-detector/vendor/github.com/euank/go-kmsg-parser/kmsgparser/kmsgparser.go

Lines 129 to 135 in c3913bb

	n, err := p.kmsgReader.Read(msg)
	if err != nil {
	if err == syscall.EPIPE {
	p.log.Warningf("short read from kmsg; skipping")
	continue
	}