Fastest way to write a fixed value (like memset)?

[Swiftb0y]

I sometimes find myself writing vec![0u8; count].write(/*...*/); (where .write is BinWrite::write) to manually specify some padding in BinWrite impls. The temporary allocation here is obviously suboptimal. Is there any better way to do this with a similarly concise API?

[Swiftb0y]

Ideally, BinWrite had a blanket impl for Iterators, then I could do std::iter::repeat_n(0u8, count).write(/*...*/);

[Swiftb0y]

I implemented a quick PoC through a wrapper trait. Is this something that would be desirable to have in binrw? master...Swiftb0y:binrw:feat/itered-write

[csnover]

Hi, thanks for your question! I’m a little confused about why std doesn’t have a mechanism like this, but I guess I am also confused about why you can’t use the existing pad directives, or else just write to the stream directly? Why does BinWrite need to be implemented for this?

[Swiftb0y]

I’m a little confused about why std doesn’t have a mechanism like this

Well it has, but its only specifically on slices (slice::fill) and not on the io::Write trait.

why you can’t use the existing pad directives

Because pad_* doesn't let me specify what to pad with (I'm working on a format where entire sections are initialized with a non-zero sentinel value).

or else just write to the stream directly?

I assume you're referring to the function on the std::io::Write trait? As far as I can tell only lets me write data its copying from a buffer, which I don't want to allocate just for that.

Why does BinWrite need to be implemented for this?

You're right, it doesn't really need to be, it was just a generic extension to the memset idea of writing a copy of any type N times without having to allocate N copies in a container that implements BinWrite.

I'm hoping this answers your questions.

[csnover]

I assume you're referring to the function on the std::io::Write trait? As far as I can tell only lets me write data its copying from a buffer, which I don't want to allocate just for that.

This must happen no matter what, but it can be a fixed array on the stack to avoid allocation overhead. Otherwise you are doing one call per fill byte and that is going to be worse, especially if your output stream is not buffered since that will be one syscall per byte. You do not have to allocate the exact size of the output, just loop full writes of the buffer for 0..byte_count/buffer_size + slice the remainder.

Because pad_* doesn't let me specify what to pad with (I'm working on a format where entire sections are initialized with a non-zero sentinel value).

OK, this would be a better thing to fix and is tracked by #205.

[Swiftb0y]

This must happen no matter what, but it can be a fixed array on the stack to avoid allocation overhead. Otherwise you are doing one call per fill byte and that is going to be worse, especially if your output stream is not buffered since that will be one syscall per byte.

Fully agree, though I assume memset handles that behind the scenes automatically in the most efficient way possible (depending on available SIMD features for instance).

You do not have to allocate the exact size of the output, just loop full writes of the buffer for 0..byte_count/buffer_size + slice the remainder.

Right, fully agree that there are solutions for this, I'm just asking whether it would make sense to have a primitive for this implemented in binrw.

OK, this would be a better thing to fix and is tracked by #205.

Yes, I fully agree, though that doesn't address the generic container/iterator usecase. Did you take a look at the implementation sketch I posted above?

[csnover]

Fully agree, though I assume memset handles that behind the scenes automatically in the most efficient way possible (depending on available SIMD features for instance).

memset, probably, but we are not talking about memset :-)

Yes, I fully agree, though that doesn't address the generic container/iterator usecase. Did you take a look at the implementation sketch I posted above?

Yes, I did. I have no useful feedback to offer as I am over-encumbered. So far I am not sure what this generic case actually is; so far the only concrete example is padding, which is going to be slow and bad this way. If #205 were implemented, would you be thinking about this at all? If not, seem imprudent to spend more time thinking about it for now.

Thanks,

[Swiftb0y]

Yeah probably not. Most usecases I can think about could use pad_after as a workaround. It would be nice if whatever performant mechanism it uses were also available in handwritten BinWrite implementations.