feat(schema): Add support to write shredded variants for HoodieRecordType.AVRO

[voonhous]

Describe the issue this Pull Request addresses

Closes: #18066

This PR introduces full read and write support for Shredded Variant types in Hudi, enabling the storage of Spark 4.0 Variant data in an optimized Parquet format.

Previously, Variant types were stored only as unshredded binary blobs. Shredding decomposes the variant data into typed columns (typed_value), allowing for significantly better compression and query performance (column pruning) when querying specific sub-fields of the Variant.

Summary and Changelog

This PR implements the "Shredding" mechanism at the Avro write layer, allowing Hudi to transform Variant records into the shredded structure required by Parquet. It abstracts the shredding logic behind a provider interface to maintain separation between Hudi's core Avro support and Spark's Variant implementation.

Key Changes:

1. Configuration:

   • Added hoodie.parquet.variant.shredding.provider.class in HoodieStorageConfig to specify the implementation used to parse/shred variants.
   • Added logic to auto-detect the provider (org.apache.hudi.variant.Spark4VariantShreddingProvider) if present on the classpath.

2. Core Write Support (hudi-hadoop-common):

   • Introduced VariantShreddingProvider: An interface for parsing variant binary data into shredded GenericRecords.
   • Updated HoodieAvroWriteSupport: Added logic to identify shredded variant fields in the schema and utilize the provider to populate the typed_value column during the write phase.
   • Updated HoodieAvroFileWriterFactory: Added logic to instantiate the shredding provider via reflection.

3. Spark 4 Integration (hudi-spark4-common):

   • Added Spark4VariantShreddingProvider: A concrete implementation utilizing org.apache.spark.types.variant.VariantShreddingWriter to perform the actual shredding.
   • Updated BaseSpark4Adapter: Modified isDataTypeEqualForParquet to recognize the shredded physical schema (structs with 3 fields: value, metadata, typed_value) as a valid Variant type.
   • Updated HoodieHadoopFsRelationFactory: Propagates the hoodie.parquet.variant.allow.reading.shredded config to Spark's SQLConf to ensure the reader handles the schema correctly.

4. Testing:

   • Added TestVariantDataType: Comprehensive tests verifying:
     • Writing with shredding enabled (verifying Parquet schema contains typed_value).
     • Writing with shredding disabled (verifying standard binary structure).
     • Reading back data in both scenarios.

Impact

• Performance: Users on Spark 4.0+ utilizing the Variant type will experience improved query performance on Hudi tables due to the ability to leverage Parquet column shredding.
• Compatibility: This feature is specific to Spark 4.0+. The implementation uses reflection and separate modules to ensure no regressions for older Spark versions.

Risk Level

Low

• The changes are isolated to the Variant data type paths.
• The shredding logic is guarded by configuration flags (hoodie.parquet.variant.write.shredding.enabled).
• The provider is loaded via reflection, preventing classpath issues in environments without Spark 4 libraries.

Documentation Update

• The new configuration hoodie.parquet.variant.shredding.provider.class needs to be documented (marked as Advanced).
• Updates to Variant data type documentation to mention support for shredding in Spark 4.0.

Contributor's checklist

• Read through contributor's guide
• Enough context is provided in the sections above
• Adequate tests were added if applicable

[voonhous]

Oops, accidentally pushed this to hudi repo instead of my own fork. Should be fine, need to remind myself to delete the branch after merge then.

[hudi-bot]

CI report:

• 2d2790c UNKNOWN
• c4b0b3c UNKNOWN
• dbc778a UNKNOWN
• ccb1ff4 UNKNOWN
• fb48ca6 UNKNOWN
• 48a01e9 UNKNOWN
• f79a417 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[the-other-tim-brown]

Is it possible to use the struct's metadata to be more direct about what is or isn't a variant?