Skip to content

Commit 0366140

Browse files
carpecodeumalamb
andauthored
Variant: Write Variant Values as JSON (#7670)
# Which issue does this PR close? Closes [Variant: Write Variant Values as JSON](#7426) #7426 Part of [[EPIC] [Parquet] Implement Variant type support in Parquet](#6736) #6736 # Rationale for this change This is an initial version, serving as a simple interface between the Variant implementation and the Serde JSON library. A huge thank you to @PinkCrow007, @mprammer, @alamb, the rest of the CMU variant team, and everyone else we've interacted with who has helped me get started with contributing to this project. This is my first Arrow-related PR, and I thank you all for your insight and support. # What changes are included in this PR? This PR implements a comprehensive JSON conversion API for Variant types with three main functions (`variant_to_json`, `variant_to_json_string`, and `variant_to_json_value`) that convert different Variant types to JSON format, including primitives, decimals, dates, timestamps, and binary data with proper escaping and base64 encoding. The implementation adds missing methods to `VariantObject` and `VariantArray` for field/element access, includes two new dependencies (`serde_json` and `base64`), and provides comprehensive test coverage with unit, integration, and documentation test suites. Open to input for improving any part of this implementation. # Are there any user-facing changes? The new API's added in parquet-variant will be user-facing. --------- Co-authored-by: Andrew Lamb <[email protected]>
1 parent 71ac9bd commit 0366140

File tree

5 files changed

+1371
-11
lines changed

5 files changed

+1371
-11
lines changed

parquet-variant/Cargo.toml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,5 +35,7 @@ rust-version = "1.83"
3535
[dependencies]
3636
arrow-schema = { workspace = true }
3737
chrono = { workspace = true }
38+
serde_json = "1.0"
39+
base64 = "0.21"
3840

3941
[lib]
Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
//! Example showing how to convert Variant values to JSON
19+
20+
use parquet_variant::{
21+
variant_to_json, variant_to_json_string, variant_to_json_value, VariantBuilder,
22+
};
23+
24+
fn main() -> Result<(), Box<dyn std::error::Error>> {
25+
let mut builder = VariantBuilder::new();
26+
27+
{
28+
let mut person = builder.new_object();
29+
person.append_value("name", "Alice");
30+
person.append_value("age", 30i32);
31+
person.append_value("email", "[email protected]");
32+
person.append_value("is_active", true);
33+
person.append_value("score", 95.7f64);
34+
person.append_value("department", "Engineering");
35+
person.finish();
36+
}
37+
38+
let (metadata, value) = builder.finish();
39+
let variant = parquet_variant::Variant::try_new(&metadata, &value)?;
40+
41+
let json_string = variant_to_json_string(&variant)?;
42+
let json_value = variant_to_json_value(&variant)?;
43+
let pretty_json = serde_json::to_string_pretty(&json_value)?;
44+
println!("{}", pretty_json);
45+
46+
let mut buffer = Vec::new();
47+
variant_to_json(&mut buffer, &variant)?;
48+
let buffer_result = String::from_utf8(buffer)?;
49+
50+
// Verify all methods produce the same result
51+
assert_eq!(json_string, buffer_result);
52+
assert_eq!(json_string, serde_json::to_string(&json_value)?);
53+
54+
Ok(())
55+
}

parquet-variant/src/lib.rs

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,8 +33,10 @@ mod decoder;
3333
mod variant;
3434
// TODO: dead code removal
3535
mod builder;
36+
mod to_json;
3637
#[allow(dead_code)]
3738
mod utils;
3839

3940
pub use builder::*;
41+
pub use to_json::{variant_to_json, variant_to_json_string, variant_to_json_value};
4042
pub use variant::*;

0 commit comments

Comments
 (0)