Skip to content

Commit 1b60ce6

Browse files
Add YAML SQL example using calcite_connection_properties (#37303)
This PR adds documentation and examples for using calcite_connection_properties in YAML SQL transforms. This enables users to use dialect-specific SQL functions such as PostgreSQL's INITCAP and STRING_TO_ARRAY functions. Changes: - Add new example file: sql/sql_calcite_connection_properties.yaml showing how to configure PostgreSQL functions in YAML SQL queries - Add test case in tests/sql.yaml to verify calcite_connection_properties works with PostgreSQL functions Fixes #36614
1 parent f14f1ce commit 1b60ce6

File tree

2 files changed

+74
-0
lines changed

2 files changed

+74
-0
lines changed
Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
# coding=utf-8
2+
#
3+
# Licensed to the Apache Software Foundation (ASF) under one or more
4+
# contributor license agreements. See the NOTICE file distributed with
5+
# this work for additional information regarding copyright ownership.
6+
# The ASF licenses this file to You under the Apache License, Version 2.0
7+
# (the "License"); you may not use this file except in compliance with
8+
# the License. You may obtain a copy of the License at
9+
#
10+
# http://www.apache.org/licenses/LICENSE-2.0
11+
#
12+
# Unless required by applicable law or agreed to in writing, software
13+
# distributed under the License is distributed on an "AS IS" BASIS,
14+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
15+
# See the License for the specific language governing permissions and
16+
# limitations under the License.
17+
18+
# This example demonstrates using calcite_connection_properties to enable
19+
# dialect-specific SQL functions. By setting "fun" to "postgresql", you can
20+
# use PostgreSQL-specific functions like STRING_TO_ARRAY in your SQL queries.
21+
#
22+
# The calcite_connection_properties option is passed to the underlying Calcite
23+
# SQL engine to configure its behavior. Common properties include:
24+
# - "fun": SQL function library to use ("standard", "postgresql", "bigquery", etc.)
25+
# - "lex": Lexical policy for identifiers
26+
#
27+
# See Apache Calcite documentation for all available connection properties.
28+
pipeline:
29+
transforms:
30+
- type: Create
31+
name: CreateSampleData
32+
config:
33+
elements:
34+
- {id: 1, tags: "java python go"}
35+
- {id: 2, tags: "rust cpp"}
36+
- {id: 3, tags: "javascript typescript"}
37+
- type: Sql
38+
name: TransformWithPostgresFunction
39+
input: CreateSampleData
40+
config:
41+
query: "SELECT id, STRING_TO_ARRAY(tags, ' ') as tag_list FROM PCOLLECTION"
42+
- type: LogForTesting
43+
input: TransformWithPostgresFunction
44+
45+
options:
46+
calcite_connection_properties: {"fun": "postgresql"}
47+
48+
# Expected:
49+
# Row(id=1, tag_list=['java', 'python', 'go'])
50+
# Row(id=2, tag_list=['rust', 'cpp'])
51+
# Row(id=3, tag_list=['javascript', 'typescript'])

sdks/python/apache_beam/yaml/tests/sql.yaml

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -93,3 +93,26 @@ pipelines:
9393
- type: PyTransform
9494
config:
9595
constructor: apache_beam.transforms.util.LogElements
96+
97+
# Test calcite_connection_properties option with PostgreSQL functions
98+
- pipeline:
99+
type: chain
100+
transforms:
101+
- type: Create
102+
name: CreateData
103+
config:
104+
elements:
105+
- {id: 1, text: "hello world"}
106+
- {id: 2, text: "foo bar baz"}
107+
- type: Sql
108+
name: SqlWithPostgresFunction
109+
config:
110+
query: "SELECT id, INITCAP(text) as title_text FROM PCOLLECTION"
111+
- type: AssertEqual
112+
config:
113+
elements:
114+
- {id: 1, title_text: "Hello World"}
115+
- {id: 2, title_text: "Foo Bar Baz"}
116+
options:
117+
calcite_connection_properties:
118+
fun: "postgresql"

0 commit comments

Comments
 (0)