How to do multi-block writes?

[peterkrull]

I feel like I am missing something here, but are multi-block writes not possible through the public API?

SdCardInner::write(..) seems to have the support for writing multiple blocks properly, but all the calling functions (specifically the public VolumeManager::write()) seem to just use a single [Block::new()] multiple times, instead of an actual multi-element array of Block. Is this correct?

[peterkrull]

In my own fork I have been playing with creating an array of Blocks in the VolumeManager::write() function to get multi-block writes. It works, but it is quite inefficient memory-wise, since most of the data in the blocks is just a copy from an already contiguous slice of bytes. I believe it could be possible to make an array (or iterator?) of an alternative RefBlock, which as the name suggests just holds a reference to some data corresponding to a block. Take the example illustrated:

1 : Data from existing block
- : Input buffer to write
0 : Zeroed blank data

   owned     slice     slice     owned
|1111|----|---------|---------|----|0000|
| block 0 | block 1 | block 2 | block 3 |

Block 0 and block 3 would in this case need to be owned, since they must be constructed partially from some of the input buffer, and in the case of block 0, some previously written data. Block 1 and block 2 can be constructed as a 512-byte slice of the input buffer, without copying anything. So in any case, we only need to create two owned blocks, and the rest can be referenced from the input buffer, and all could be turned into a series of RefBlock. I have not yet implemented anything yet, but thought it could be done. Would like to hear from others on this

[thejpster]

I think the trick is ensuring any byte slices which are passed straight to the disk API as if they were Blocks are aligned as a Block would be. But yes if the input is correctly aligned and is at least 512 bytes, a memcpy could be avoided. Or I guess more importantly, a multi-block write could be sent to the SD Card.

I believe the README says that this crate chooses readability over performance, but this is, I think, a sensible optimisation for most scenarios. Although not on the nRF line where DMA requires data to be in RAM and passing along slices supplied by a user means it's possible to pass in a slice located in flash. But the nRF HAL gets to deal with that.

[thejpster]

Another option is a feature which enables the VolumeManager to hold a 4K internal buffer instead of a 512 byte one. User data alignment then doesn't matter, and we can send eight blocks at a time to the disk.

[peterkrull]

I like the idea of the 4K buffer, since that is essentially what I have ended up doing external to this library. If the entire buffer could be handled internally, ensuring optimal alignment, that would be really nice. Though shouldn't it be each File (or RawFile) which should hold an internal buffer?

[thejpster]

So far, I've had one buffer that the files and FAT32 code can temporarily access. This is designed for microcontrollers with like 8K of RAM total.

[thejpster]

A BufferedFile object, would be a cool, but different thing.

[peterkrull]

So far, I've had one buffer that the files and FAT32 code can temporarily access. This is designed for microcontrollers with like 8K of RAM total.

How would that work? If you have multiple files open, that are all being written to, wouldn't the buffer get filled with data from different sources? Would you then write the buffer contents, which contains data from file A, whenever file B wants to write?

[thejpster]

Because every disk write is either a complete block, so the prior contents are overwritten, or a read/modify/write.

There is no file buffering. If you call write and pass 3 bytes, it will read one block, change 3 bytes, and write it back.

[peterkrull]

I am aware of that, maybe I misunderstood your previous statement. Hmm, are there any parts of the library, except from write, which would benefit from having a large buffer? I.e. why does it makes sense for VolumeManager to have a shared buffer, compared to just creating it when needed in write?

I guess what I would really want for my project is the BufferedFile which does not actually write data until the 4K buffer is full, or as full as can be while ensuring block alignment. Mostly to increase the effective throughput when doing many small writes. To the user of this type, it should behave the same as the regular File, since any data which is written, but not flushed, is practically the same as data which resides in the buffer.

[thejpster]

I recall thinking that the VolumeManager might be statically allocated, saving you precious stack space.

[Mirkopoj]

Hello, I wrote a BufferedFile for my implementation where I hold a File and a 16KiB buffer. Even tho it improved preformance now I'm wondering if my naive aproach is as effective as it could be. If I create a file and only perform wirtes 16KiB at a time will this be allways byte alligned or may the file creation fall inside a block (not at the start I mean) making all my wirtes be 1 RMW a 30 full block writes and 1 extra RMW? If the later where to be the case, would reading the offset and "shrinking" the buffer for the first wirte (so that now I'm block alligned) be enough? Or is it possible that I lose the allignment due to internal SD management?

[thejpster]

As long as both your file offsets and the amount you write are a multiple of 512 you should be fine.