Avro and Parquet enhancements

[jlambatl]

We have several use cases where we need to write events out as Avro- or Parquet-encoded data (e.g. to S3 and Kafka) so that downstream systems in our data platform can consume them directly.

Right now, the Avro support in Vector appears to be limited to using to_avro_datum. From the apache-avro docs:

NOTE: This function has a quite small niche of usage and does NOT generate headers and sync markers; use https://docs.rs/apache-avro/latest/apache_avro/struct.Writer.html to be fully Avro-compatible if you don’t know what you are doing, instead.

Because to_avro_datum only produces a single Avro-encoded value (no container file header, no sync markers, no codec metadata), the output isn’t directly compatible with the Avro Object Container File (OCF) format or with many “big data” tools that expect standard Avro files.
What we’d like to contribute

We’d like to improve Vector’s Avro compatibility so that it can:

• Write fully compatible Avro object container files (OCF)
  • Use apache_avro::Writer (or an equivalent abstraction) so that:
  • The Avro file header is written, including:
    • avro.schema with the JSON schema
    • Sync markers are generated and written between blocks.
  • Target use case: writing Avro files to S3 / object storage that are consumable by Spark/Flink/other Avro tooling out of the box.
  • Keep the existing implementation so that it's not going to break existing users

Our goal is to make Vector’s Avro output “just work” with the broader Avro ecosystem (Java Avro, avro-tools, Spark, etc.), not only with custom consumers that know about the current to_avro_datum usage.

Proposed approach

If you’re open to this, we’d like to contribute this work as a series of small, incremental PRs that preserve existing behavior by default. Roughly:

• Introduce an OCF writer path that uses apache_avro::Writer for file-style outputs, controlled via configuration (so the existing raw datum behavior can be kept for backward compatibility).

• Add configuration options for:

  • datum (existing/default)
  • Avro Object Container (new)

• Add compatibility tests, for example:

  • Golden Avro files produced by Vector that are validated with avro-tools / Java Avro.
  • Round-trip tests where data written by Vector is read back with the Rust apache-avro reader.

We’d really appreciate feedback on:

• Whether you’d be willing to accept PRs in this direction.
• Any existing design constraints we should be aware of (beyond the contributing guidelines, maybe this has been done before?).

(rewrote this to be more explicit)

[jlambatl]

Because the batching/buffering for these sinks is done, using to_avro_datum is the right approach. Using the Writer as suggested by the documentation would be closer to the BulkSerialization type in the arrow/clickhouse sink.

[pront]

Our goal is to make Vector’s Avro output “just work” with the broader Avro ecosystem

Thanks for the detailed proposal, @jlambatl! We're cautiously supportive of this enhancement. You're correct that the current implementation uses to_avro_datum. Adding OCF support would indeed improve interoperability with Spark, Flink, and other tools in the Avro ecosystem.

if you’re open to this, we’d like to contribute this work as a series of small, incremental PRs that preserve existing behavior by default.

This is a great approach.

[jlambatl]

I submitted a PR for this change, #24585