Optimize I/O operations for better performance on MAC #1390
Conversation
- Remove redundant Flush() call before ForceSync() in periodic sync - Skip sync operations when using direct I/O (no page cache to flush) - Optimize macOS writes: use buffered I/O with /dev/rdisk for faster writes - Enable direct I/O only for verification reads to ensure accuracy These optimizations improve write performance while maintaining data integrity.
mcyork
force-pushed
the
optimize-io-operations
branch
from
98afd57 to
9afc647
Compare
| // Convert /dev/diskX to /dev/rdiskX for raw device access (like dd uses) | ||
| // Raw devices (/dev/rdisk) have better buffering behavior than block devices (/dev/disk) | ||
| std::string rawPath = path; | ||
| size_t diskPos = rawPath.find("/dev/disk"); | ||
| if (diskPos != std::string::npos) { | ||
| rawPath.replace(diskPos + 5, 4, "rdisk"); // Replace "disk" with "rdisk" | ||
| std::cout << "Using raw device path: " << rawPath << " (instead of " << path << ")" << std::endl; | ||
| } | ||
|
|
There was a problem hiding this comment.
Huh? This is already happening:
rpi-imager/src/downloadthread.cpp
Line 323 in 179efaf
| // ForceSync() already handles both flushing and syncing to disk | ||
| // No need for redundant Flush() call - ForceSync() does everything |
There was a problem hiding this comment.
The point of this functionality is to avoid excessive page cache usage, which was cited in early testing as causing problems on memory-constrained systems.
| // Use buffered I/O for writes (like v1.9.6) - much faster! | ||
| // Direct I/O (F_NOCACHE) will be enabled only for verification reads | ||
| // in PrepareForSequentialRead() to ensure accurate verification. | ||
| // This gives us: fast writes + accurate verification | ||
| using_direct_io_ = false; |
There was a problem hiding this comment.
As below, the intent here was to have more control over the memory pressure Imager was exacting on the system, but clearly on macOS that comes with a substantial write cost.
At this stage, I'm going to have to consider the balance of harms - and that might take a while.
|
Thanks for the contribution, @mcyork! I particularly appreciate someone taking the time to model out their ideas in a PR, and measure the output! (Honestly, given the scarcity of community-sourced PRs, I nearly fell out of my chair when I saw this!) That said, I think there's a block you can probably remove - and the Buffered I/O change will need to be considered against earlier testing results (where systems under memory pressure behaved poorly - extremely long 'flush' cycles, allocation thrashing in other applications, etc). Imager 2.0 tries to be very deliberate about it's memory allocation and management - mostly so that if it's causing problems, I can hold my hands up and consider biasing the algorithm further. tldr; Not an immediate merge, but a valued contribution that I'll evaluate further. |
|
Yeah I wasn't watching the other branches, I just had a write problem so
tried to get my copy going faster. And it worked. Yeah I had to push the PR
just to say clearly what I had done. If you see it in code, it's in code.
You know what it is. Yeah I don't pretend to understand the caching
behavior. I did see that it ran a lot faster. I might have missed another
option where you can increase the buffer. I think there was some line that
said how much to write into the buffer and one of the things that slowed
the Mac down was putting too small a chunk in there and it would use it too
quickly and I think it was waiting for the program to feed it more
chunks. I don't remember but long story short I'm glad you reviewed it and
you seem to be on a stable path. I'm personally not concerned about memory
constraints so if there's a way to say how much memory do I have... then do
this other thing that's nutty :)
Ian McCutcheon
https://workshop.esoup.net/
…On Mon, Dec 15, 2025 at 2:42 AM Tom Dewey ***@***.***> wrote:
*tdewey-rpi* left a comment (raspberrypi/rpi-imager#1390)
<#1390 (comment)>
Thanks for the contribution, @mcyork <https://github.com/mcyork>!
I particularly appreciate someone taking the time to model out their ideas
in a PR, and *measure* the output! (Honestly, given the scarcity of
community-sourced PRs, I nearly fell out of my chair when I saw this!)
That said, I think there's a block you can probably remove - and the
Buffered I/O change will need to be considered against earlier testing
results (where systems under memory pressure behaved poorly - extremely
long 'flush' cycles, allocation thrashing in other applications, etc).
Imager 2.0 tries to be very deliberate about it's memory allocation and
management - mostly so that if it's causing problems, I can hold my hands
up and consider biasing the algorithm further.
tldr; Not an immediate merge, but a valued contribution that I'll evaluate
further.
—
Reply to this email directly, view it on GitHub
<#1390 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAOKKVLJYNV6J7IZZUFQMDT4B2GBXAVCNFSM6AAAAACO5F5322VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTMNJUHE2TMMRWHA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
2 participants
These optimizations improve write performance while maintaining data integrity of verify.