[Variant] Add variant_get and Shredded VariantArray (apache#8021)

# Which issue does this PR close?

- Part of apache#6736
- Closes apache#7941
- Closes apache#7965

# Rationale for this change

This is has a proposal for how to structure shredded `VariantArray`s and
the `variant_get` kernel

If people like the basic idea I will file some more tickets to track
additional follow on work

It is based on ideas ideas from @carpecodeum in
apache#7946 and @scovich in
apache#7915

I basically took the tests from
apache#7965 and the conversation with
@scovich recorded from
apache#7941 (comment)
and I bashed out how this might look

# What changes are included in this PR?

1. Update `VariantArray` to represent shredding
2. Add code to `variant_get` to support extracting paths as both
variants and typed fields
3. A pattern that I think can represent shredding and extraction
4. Tests for same

Note there are many things that are NOT in this PR that I envision doing
as follow on PRs:
1. Support and implementing `Path`s
2. Support for shredded objects
3. Support shredded lists
4. Support nested objects / lists 
5. Full casting support
6. Support for other output types: `StringArray`, `StringViewArray`, etc
8. Many performance improvements 


# Are these changes tested?

Yes

# Are there any user-facing changes?

New feature

---------

Co-authored-by: Samyak Sarnayak <[email protected]>
Co-authored-by: Ryan Johnson <[email protected]>

Author: 3 people

parquet-variant-compute/src/from_json.rs

@@ -52,7 +52,7 @@ pub fn batch_json_string_to_variant(input: &ArrayRef) -> Result<VariantArray, Ar
 #[cfg(test)]
 mod test {
     use crate::batch_json_string_to_variant;
-    use arrow::array::{Array, ArrayRef, AsArray, StringArray};
+    use arrow::array::{Array, ArrayRef, StringArray};
     use arrow_schema::ArrowError;
     use parquet_variant::{Variant, VariantBuilder};
     use std::sync::Arc;
@@ -69,8 +69,8 @@ mod test {
         let array_ref: ArrayRef = Arc::new(input);
         let variant_array = batch_json_string_to_variant(&array_ref).unwrap();
 
-        let metadata_array = variant_array.metadata_field().as_binary_view();
-        let value_array = variant_array.value_field().as_binary_view();
+        let metadata_array = variant_array.metadata_field();
+        let value_array = variant_array.value_field().expect("value field");
 
         // Compare row 0
         assert!(!variant_array.is_null(0));

parquet-variant-compute/src/lib.rs

@@ -15,14 +15,34 @@
 // specific language governing permissions and limitations
 // under the License.
 
+//! [`VariantArray`] and compute kernels for the [Variant Binary Encoding] from [Apache Parquet].
+//!
+//! ## Main APIs
+//! - [`VariantArray`] : Represents an array of `Variant` values.
+//! - [`VariantArrayBuilder`]: For building [`VariantArray`]
+//! - [`batch_json_string_to_variant`]: Function to convert a batch of JSON strings to a `VariantArray`.
+//! - [`batch_variant_to_json_string`]: Function to convert a `VariantArray` to a batch of JSON strings.
+//! - [`cast_to_variant`]: Module to cast other Arrow arrays to `VariantArray`.
+//! - [`variant_get`]: Module to get values from a `VariantArray` using a specified [`VariantPath`]
+//!
+//! ## 🚧 Work In Progress
+//!
+//! This crate is under active development and is not yet ready for production use.
+//! If you are interested in helping, you can find more information on the GitHub [Variant issue]
+//!
+//! [Variant Binary Encoding]: https://github.com/apache/parquet-format/blob/master/VariantEncoding.md
+//! [Apache Parquet]: https://parquet.apache.org/
+//! [`VariantPath`]: parquet_variant::VariantPath
+//! [Variant issue]: https://github.com/apache/arrow-rs/issues/6736
+
 pub mod cast_to_variant;
 mod from_json;
 mod to_json;
 mod variant_array;
 mod variant_array_builder;
 pub mod variant_get;
 
-pub use variant_array::VariantArray;
+pub use variant_array::{ShreddingState, VariantArray};
 pub use variant_array_builder::{VariantArrayBuilder, VariantArrayVariantBuilder};
 
 pub use from_json::batch_json_string_to_variant;