feat: reduce interference from kernel_write to profiled applications

[totoroyyb]

This is merely a proposal ticket, and I haven't thought it through yet.

hiresperf uses kernel_write to dump ring buffer data to the file. However, under the circumstance where the polling rate and logging rate are both high, kernel_write may also significantly compete for cachelines with profiled applications.

The kernel_write is essentially a buffered file I/O. However, in this case, since we are mainly dumping data, the buffered I/O seems to be unnecessary.

One option we can consider is using Direct I/O.

[yizhuoliang]

Haha, that's actually a bit tricky.

Naively, open the log file in the kernel module with O_DIRECT flag can move us to direct I/O, but that's a blocking operation, which prevents us from achieving high polling frequency 💔.

So, we need not only direct, but also async I/O. But "async" intrinsically implies some buffering and thus conflicting with the goal of "direct". Asking ChatGPT, one possible way to hack is like, we directly write into the DMA pages of the block device, so the values are buffered on the device, not our memory.

Fortunately, the logging shouldn't be super costly, especially if we dedicate a core to the logger, right?

[totoroyyb]

Yea, a simple flag won’t get us to anywhere. Logging “computation" itself is not that expensive. However, at high frequency and huge volume, all buffered writes are squeezed into cachelines L1-L3. Whenever it lands on L3, it will compete for cachelines with other apps, especially our profiled apps.

…

________________________________ From: Yizhuo (Coulson) Liang ***@***.***> Sent: Wednesday, July 2, 2025 10:33:35 AM To: yizhuoliang/hiresperf ***@***.***> Cc: Yibo Yan ***@***.***>; Author ***@***.***> Subject: Re: [yizhuoliang/hiresperf] feat: reduce interference from kernel_write to profiled applications (Issue #15) [https://avatars.githubusercontent.com/u/70337586?s=20&v=4]yizhuoliang left a comment (yizhuoliang/hiresperf#15)<#15 (comment)> Haha, that's actually a bit tricky. Naively, open the log file in the kernel module with O_DIRECT flag can move us to direct I/O, but that's a blocking operation, which prevents us from achieving high polling frequency 💔. So, we need not only direct, but also async I/O. But "async" intrinsically implies some buffering and thus conflicting with the goal of "direct". Asking ChatGPT, one possible way to hack is like, we directly write into the DMA pages of the block device, so the values are buffered on the device, not our memory. Fortunately, the logging shouldn't be super costly, especially if we dedicate a core to the logger, right? — Reply to this email directly, view it on GitHub<#15 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AHHAJ4GKTXT4V4QLFXNTKJL3GQJW7AVCNFSM6AAAAACATIQK3SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTAMRYGY4TINBQGQ>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[totoroyyb]

We had some preliminary results, showing that a reasonable ring buffer size (<10k) and pretty intense logging rate don't really have too much interference on the profiled applications. At least in terms of profiled counter values.

As for the task execution duration, there is a significant slowdown when the load itself takes substantial time to finish (~20s). With a very large ring buffer and intensive logging, it can increase the completion time by >= 1min.

Meanwhile, kernel_write does cause some spikes in IMC readings.