Skip to content

Commit ef2c1a3

Browse files
examples: replace sql_dialect with custom_sql_parser example (#19383)
## Which issue does this PR close? Related to #16756 ## Rationale for this change The existing `sql_dialect.rs` example demonstrates `COPY ... STORED AS ...`, which is actually already fully supported by the standard `DFParser`. This PR replaces it with the example from #16756: `CREATE EXTERNAL CATALOG ... STORED AS ... LOCATION ...` with automatic table discovery. ## What changes are included in this PR? The first commit updates `dialect.rs` to show that `DFParser` already handles `COPY ... STORED AS`, making it clear this syntax doesn't need customization. Example output from `cargo run --example sql_ops -- dialect`: ``` Query: COPY source_table TO 'file.fasta' STORED AS FASTA --- Parsing without extension --- Standard DFParser: Parsed as Statement::CopyTo: COPY source_table TO file.fasta STORED AS FASTA --- Parsing with extension --- Custom MyParser: Parsed as MyStatement::MyCopyTo: COPY source_table TO 'file.fasta' STORED AS FASTA ``` The second commit adds a new `custom_sql_parser.rs` example that implements `CREATE EXTERNAL CATALOG my_catalog STORED AS <format> LOCATION '<url>'` with automatic table discovery from object storage. It also removes the old `dialect.rs` example. ## Are these changes tested? Yes, the new example is runnable with `cargo run --example sql_ops -- custom_sql_parser` and demonstrates the full flow from parsing custom DDL through registering the catalog to querying discovered tables. Example output: ``` === Part 1: Standard DataFusion Parser === Parsing: CREATE EXTERNAL CATALOG parquet_testing STORED AS parquet LOCATION 'local://workspace/parquet-testing/data' OPTIONS ( 'schema_name' = 'staged_data', 'format.pruning' = 'true' ) Error: SQL error: ParserError("Expected: TABLE, found: CATALOG at Line: 1, Column: 17") === Part 2: Custom Parser === Parsing: CREATE EXTERNAL CATALOG parquet_testing STORED AS parquet LOCATION 'local://workspace/parquet-testing/data' OPTIONS ( 'schema_name' = 'staged_data', 'format.pruning' = 'true' ) Target Catalog: parquet_testing Data Location: local://workspace/parquet-testing/data Resolved Schema: staged_data Registered 69 tables into schema: staged_data Executing: SELECT id, bool_col, tinyint_col FROM parquet_testing.staged_data.alltypes_plain LIMIT 5 +----+----------+-------------+ | id | bool_col | tinyint_col | +----+----------+-------------+ | 4 | true | 0 | | 5 | false | 1 | | 6 | true | 0 | | 7 | false | 1 | | 2 | true | 0 | +----+----------+-------------+ ``` ## Are there any user-facing changes? Documentation only. I replaced the `sql_dialect.rs` example with `custom_sql_parser.rs` and updated the README. No API changes.
1 parent 6ce2374 commit ef2c1a3

File tree

4 files changed

+433
-146
lines changed

4 files changed

+433
-146
lines changed

datafusion-examples/README.md

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -191,12 +191,12 @@ cargo run --example dataframe -- dataframe
191191

192192
#### Category: Single Process
193193

194-
| Subcommand | File Path | Description |
195-
| ---------- | ----------------------------------------------------- | ------------------------------ |
196-
| analysis | [`sql_ops/analysis.rs`](examples/sql_ops/analysis.rs) | Analyze SQL queries |
197-
| dialect | [`sql_ops/dialect.rs`](examples/sql_ops/dialect.rs) | Implement a custom SQL dialect |
198-
| frontend | [`sql_ops/frontend.rs`](examples/sql_ops/frontend.rs) | Build LogicalPlans from SQL |
199-
| query | [`sql_ops/query.rs`](examples/sql_ops/query.rs) | Query data using SQL |
194+
| Subcommand | File Path | Description |
195+
| ----------------- | ----------------------------------------------------------------------- | -------------------------------------------------- |
196+
| analysis | [`sql_ops/analysis.rs`](examples/sql_ops/analysis.rs) | Analyze SQL queries |
197+
| custom_sql_parser | [`sql_ops/custom_sql_parser.rs`](examples/sql_ops/custom_sql_parser.rs) | Implement a custom SQL parser to extend DataFusion |
198+
| frontend | [`sql_ops/frontend.rs`](examples/sql_ops/frontend.rs) | Build LogicalPlans from SQL |
199+
| query | [`sql_ops/query.rs`](examples/sql_ops/query.rs) | Query data using SQL |
200200

201201
## UDF Examples
202202

0 commit comments

Comments
 (0)