[SPARK-53659][SQL] Infer Variant shredding schema when writing to Parquet

### What changes were proposed in this pull request?

When writing Variant to Parquet, we want the shredding schema to adapt to the data being written on a per-file basis. This PR adds a new output writer that buffers the first few rows before starting the write, then uses the content of those rows to determine a shredding schema, and only then creates the Parquet writer with that schema.

The heuristics for determining the shredding schema are currently fairly simple: if a field appears consistently with a consistent type, we create `value` and `typed_value`, and if it appears with an inconsistent type, we only create `value`. We drop fields that occur in less than 10% of sampled rows, and have an upper bound of 300 total fields (counting `value` and `typed_value` separately) to avoid creating excessively wide Parquet schemas, which can cause performance issues.

### Why are the changes needed?

Allows Spark to make use of the [Variant shredding spec](https://github.com/apache/parquet-format/blob/master/VariantShredding.md) without requiring the user to manually set a shredding schema.

### Does this PR introduce _any_ user-facing change?

Only if `spark.sql.variant.inferShreddingSchema` and `spark.sql.variant.writeShredding.enabled` are both set to true. They currently false by default.

### How was this patch tested?

Unit tests in PR.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #52406 from cashmand/variant_shredding_inference.

Lead-authored-by: cashmand <david.cashman@databricks.com>
Co-authored-by: Wenchen Fan <cloud0fan@gmail.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>

Author: cashmand

common/variant/src/main/java/org/apache/spark/types/variant/Variant.java

@@ -99,6 +99,11 @@ public BigDecimal getDecimal() {
     return VariantUtil.getDecimal(value, pos);
   }
 
+  // Get the decimal value, including trailing zeros
+  public BigDecimal getDecimalWithOriginalScale() {
+    return VariantUtil.getDecimalWithOriginalScale(value, pos);
+  }
+
   // Get a float value from the variant.
   public float getFloat() {
     return VariantUtil.getFloat(value, pos);

sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala

@@ -5422,6 +5422,31 @@ object SQLConf {
       .stringConf
       .createWithDefault("")
 
+  val VARIANT_SHREDDING_MAX_SCHEMA_WIDTH =
+    buildConf("spark.sql.variant.shredding.maxSchemaWidth")
+      .internal()
+      .doc("Maximum number of shredded fields to create when inferring a schema for Variant")
+      .version("4.1.0")
+      .intConf
+      .createWithDefault(300)
+
+  val VARIANT_SHREDDING_MAX_SCHEMA_DEPTH =
+    buildConf("spark.sql.variant.shredding.maxSchemaDepth")
+      .internal()
+      .doc("Maximum depth in Variant value to traverse when inferring a schema. " +
+           "Any array/object below this depth will be shredded as a single binary.")
+      .version("4.1.0")
+      .intConf
+      .createWithDefault(50)
+
+  val VARIANT_INFER_SHREDDING_SCHEMA =
+    buildConf("spark.sql.variant.inferShreddingSchema")
+      .internal()
+      .doc("Infer shredding schema when writing Variant columns in Parquet tables.")
+      .version("4.1.0")
+      .booleanConf
+      .createWithDefault(false)
+
   val LEGACY_CSV_ENABLE_DATE_TIME_PARSING_FALLBACK =
     buildConf("spark.sql.legacy.csv.enableDateTimeParsingFallback")
       .internal()