Serialization performance tuning how-to?

[brent-statsig]

I'm working through writing a service that downloads n files concurrently - then writes them to a single avro file.

• Is there a recommended way to parallel serialize data? I see Writer calls maybe_write_header in all public append APIs, along with into_inner, which makes it hard to just get the raw bytes of serialized rows without the header attached. It doesn't look like the raw Serializer impl in ser.rs is public either. What is the recommended way to split serialization work across cores?
• Schema validation per-value appended is expensive - it would be really nice to have compile flags around it so it can be stripped out for production, or have a sampling rate attached to it to retain some runtime safety?

[martin-g]

Hi,

I don't have a good solution for you.
The current stable version (0.20) is synchronous...
At #238 we are discussing how to add both sync and async APIs.

#246 is a possible solution that provides async APIs and uses proc macros to remove the async part to produce sync APIs. You may give it a try, e.g. by using tokio::spawn(async_writer.append(...)) but I have the feeling this will lead to corrupted avro file due to the interleaved writes to the writer.

Another option is to use GenericSingleObjectWriter to produce the bytes for a single record in parallel and then append to a file behind a mutex or similar.

• Schema validation per-value appended is expensive - it would be really nice to have compile flags around it so it can be stripped out for production, or have a sampling rate attached to it to retain some runtime safety?

You are the second person mentioning this - #310 (comment)
Adding a feature flag to disable the validation is easy but risky.
@Kriskras99 What do you think about this idea ?
CC @mina-asham

[Kriskras99]

It would be nice if we could have it as a configuration option on the writer, but a feature flag would be easier to implement. If we do put it behind a feature flag I think it should be on by default, and disabling it would require using default-features = false.

[brent-statsig]

I put up #326 to try at least make it easy to concurrently serialize, which in my experience is the bottleneck with large amounts of data.

[brent-statsig]

The problem with Tokio::spawn(async_writer.append(...)) is that you need a mut in order to use any of the append functions, which introduces really long periods of locking. The serialization and encoding steps are really heavy, but don't actually need a mut, which is why I split it out in my linked PR above

[brent-statsig]

Also thank you for the fast response :) its very appreciated

[Kriskras99]

If it is the serialisation from a T to a Value that is slow, you can use to_value.
If it is also the Value to actual bytes parts, then your PR might be needed.