Improve SQL transform documentation for calcite_connection_properties in YAML

- Add comprehensive SQL transform examples showing calcite_connection_properties usage
- Create dedicated sql/ directory under yaml/examples/transforms with 5 examples:
  - sql_basic_example.yaml: Basic SQL without special configuration
  - sql_postgresql_functions.yaml: PostgreSQL functions like SPLIT_PART
  - sql_bigquery_functions.yaml: BigQuery syntax and functions
  - sql_mysql_functions.yaml: MySQL date/string functions
  - sql_advanced_configuration.yaml: Multiple configuration options
- Add detailed README.md explaining calcite_connection_properties options
- Update yaml/tests/sql.yaml with calcite_connection_properties test cases
- Update examples/README.md to reference new SQL documentation

This addresses the issue where calcite_connection_properties configuration
was 'tricky to get right' by providing clear examples and documentation
for different SQL dialects and use cases.

Fixes: SQL options in YAML/xlang pipelines need better documentation

Author: Arunodoy18

sdks/python/apache_beam/yaml/examples/README.md

@@ -100,6 +100,18 @@ These examples leverage the built-in mapping transforms including `MapToFields`,
 `Filter` and `Explode`. More information can be found about mapping transforms
 [here](https://beam.apache.org/documentation/sdks/yaml-udf/).
 
+### SQL
+
+Examples that demonstrate SQL transforms with various database dialect configurations:
+
+- [Basic SQL Transform](transforms/sql/sql_basic_example.yaml) - Simple SQL queries without special configuration
+- [PostgreSQL Functions](transforms/sql/sql_postgresql_functions.yaml) - Using PostgreSQL-specific functions like SPLIT_PART
+- [BigQuery Functions](transforms/sql/sql_bigquery_functions.yaml) - BigQuery syntax and functions with proper calcite_connection_properties
+- [MySQL Functions](transforms/sql/sql_mysql_functions.yaml) - MySQL-specific date and string functions
+- [Advanced Configuration](transforms/sql/sql_advanced_configuration.yaml) - Multiple calcite_connection_properties options
+
+These examples show how to use the `calcite_connection_properties` pipeline option to configure SQL transforms for different database dialects and enable dialect-specific functions and syntax.
+
 ### IO
 
 #### Spanner

sdks/python/apache_beam/yaml/examples/transforms/sql/README.md

@@ -0,0 +1,104 @@
+# SQL Transform calcite_connection_properties Configuration Guide
+
+This directory contains examples demonstrating how to use `calcite_connection_properties` in Beam YAML pipelines to configure SQL transforms for different database dialects and use cases.
+
+## Overview
+
+The `calcite_connection_properties` option in pipeline options allows you to configure Apache Calcite's SQL parser and function library to support database-specific SQL syntax and functions. This is particularly useful when you need to use SQL functions or syntax that are specific to certain databases like PostgreSQL, BigQuery, MySQL, or Oracle.
+
+## Configuration Options
+
+The most commonly used `calcite_connection_properties` include:
+
+### Function Libraries (`fun`)
+- `"standard"` - Standard SQL functions (default)
+- `"postgresql"` - PostgreSQL-specific functions (e.g., SPLIT_PART, STRING_AGG)
+- `"bigquery"` - BigQuery-specific functions (e.g., FORMAT_TIMESTAMP, ARRAY_TO_STRING)
+- `"mysql"` - MySQL-specific functions (e.g., DATEDIFF, SUBSTRING_INDEX)
+- `"oracle"` - Oracle-specific functions (e.g., NVL, SUBSTR)
+
+### Lexical Rules (`lex`)
+- `"standard"` - Standard SQL lexical rules (default)
+- `"big_query"` - BigQuery lexical rules and syntax
+- `"mysql"` - MySQL lexical rules
+- `"oracle"` - Oracle lexical rules
+
+### Other Properties
+- `conformance` - SQL conformance level ("LENIENT", "STRICT", etc.)
+- `caseSensitive` - Whether identifiers are case sensitive ("true"/"false")
+- `quotedCasing` - How to handle quoted identifiers ("UNCHANGED", "TO_UPPER", "TO_LOWER")
+- `unquotedCasing` - How to handle unquoted identifiers
+
+## Usage Patterns
+
+### Basic Configuration
+```yaml
+options:
+  calcite_connection_properties:
+    fun: "postgresql"
+```
+
+### Advanced Configuration
+```yaml
+options:
+  calcite_connection_properties:
+    fun: "bigquery"
+    lex: "big_query"
+    conformance: "LENIENT"
+    caseSensitive: "false"
+```
+
+## Examples in this Directory
+
+1. **sql_basic_example.yaml** - Basic SQL transform without special configuration
+2. **sql_postgresql_functions.yaml** - Using PostgreSQL functions like SPLIT_PART
+3. **sql_bigquery_functions.yaml** - BigQuery syntax and functions
+4. **sql_mysql_functions.yaml** - MySQL-specific date and string functions
+5. **sql_advanced_configuration.yaml** - Multiple configuration options
+
+## Common Use Cases
+
+### PostgreSQL Functions
+Useful for string manipulation and array operations:
+```yaml
+options:
+  calcite_connection_properties:
+    fun: "postgresql"
+```
+
+### BigQuery Compatibility
+For BigQuery-style syntax and functions:
+```yaml
+options:
+  calcite_connection_properties:
+    fun: "bigquery"
+    lex: "big_query"
+```
+
+### Lenient SQL Parsing
+For more flexible SQL parsing:
+```yaml
+options:
+  calcite_connection_properties:
+    conformance: "LENIENT"
+```
+
+## Important Notes
+
+- These properties affect only the SQL parsing and function availability, not the actual data processing semantics
+- Some database-specific functions may not be available depending on the Calcite version used
+- Always test your SQL queries with the intended configuration before deploying to production
+- The `calcite_connection_properties` must be specified in the pipeline `options` section, not in individual transform configurations
+
+## Troubleshooting
+
+If you encounter SQL parsing errors:
+
+1. Check that the function you're using is supported by the specified function library
+2. Verify that the lexical rules (`lex`) match your SQL syntax style
+3. Try using `conformance: "LENIENT"` for more flexible parsing
+4. Refer to the Apache Calcite documentation for supported functions in each dialect
+
+For more information about Beam SQL and supported functions, see:
+- [Beam SQL Documentation](https://beam.apache.org/documentation/dsls/sql/overview/)
+- [Apache Calcite SQL Reference](https://calcite.apache.org/docs/reference.html)

sdks/python/apache_beam/yaml/examples/transforms/sql/sql_advanced_configuration.yaml

@@ -0,0 +1,87 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Advanced SQL Transform Configuration Examples
+# This example demonstrates multiple calcite_connection_properties and their effects.
+
+pipeline:
+  transforms:
+    - type: Create
+      name: CreateComplexData
+      config:
+        elements:
+          - {id: 1, name: "Product A", price: 29.99, tags: ["electronics", "gadget"], metadata: '{"brand": "TechCorp", "warranty": 12}'}
+          - {id: 2, name: "Product B", price: 15.50, tags: ["books", "fiction"], metadata: '{"author": "John Doe", "pages": 320}'}
+          - {id: 3, name: "Product C", price: 199.99, tags: ["electronics", "computer"], metadata: '{"brand": "CompuTech", "warranty": 24}'}
+
+    # Example 1: Standard SQL with strict conformance
+    - type: Sql
+      name: StandardSQL
+      input: CreateComplexData
+      config:
+        query: |
+          SELECT 
+            id,
+            name,
+            price,
+            CASE 
+              WHEN price < 20 THEN 'Budget'
+              WHEN price < 100 THEN 'Mid-range'
+              ELSE 'Premium'
+            END as price_category
+          FROM PCOLLECTION
+          WHERE price > 10
+          ORDER BY price
+
+    # Example 2: Using Oracle-style functions
+    - type: Sql
+      name: OracleStyleSQL
+      input: CreateComplexData
+      config:
+        query: |
+          SELECT 
+            id,
+            name,
+            price,
+            -- Oracle-style string functions
+            SUBSTR(name, 1, 10) as short_name,
+            LENGTH(name) as name_length,
+            NVL(name, 'Unknown') as safe_name
+          FROM PCOLLECTION
+
+    - type: LogForTesting
+      input: StandardSQL
+      
+    - type: LogForTesting
+      input: OracleStyleSQL
+
+# Multiple calcite_connection_properties can be configured:
+# - conformance: Controls SQL conformance level (LENIENT, STRICT, etc.)
+# - caseSensitive: Whether identifiers are case sensitive
+# - quotedCasing: How to handle quoted identifiers (UNCHANGED, TO_UPPER, TO_LOWER)
+# - unquotedCasing: How to handle unquoted identifiers
+# - fun: SQL function library (standard, oracle, mysql, postgresql, bigquery, etc.)
+# - lex: Lexical analysis rules (standard, oracle, mysql, big_query, etc.)
+options:
+  calcite_connection_properties:
+    conformance: "LENIENT"
+    fun: "oracle"
+    lex: "oracle" 
+    caseSensitive: "false"
+    quotedCasing: "UNCHANGED"
+    unquotedCasing: "TO_UPPER"
+  streaming: false

sdks/python/apache_beam/yaml/examples/transforms/sql/sql_basic_example.yaml

@@ -0,0 +1,50 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Basic SQL Transform Example
+# This example demonstrates basic SQL transform usage with default Calcite configuration.
+
+pipeline:
+  transforms:
+    - type: Create
+      name: CreateData
+      config:
+        elements:
+          - {id: 1, name: "Alice", age: 30, city: "Seattle"}
+          - {id: 2, name: "Bob", age: 25, city: "Portland"}
+          - {id: 3, name: "Charlie", age: 35, city: "San Francisco"}
+          - {id: 4, name: "Diana", age: 28, city: "Seattle"}
+
+    - type: Sql
+      name: FilterAndGroup
+      input: CreateData
+      config:
+        query: |
+          SELECT 
+            city,
+            COUNT(*) as person_count,
+            AVG(age) as avg_age
+          FROM PCOLLECTION 
+          WHERE age >= 25
+          GROUP BY city
+          ORDER BY city
+
+    - type: LogForTesting
+      input: FilterAndGroup
+
+options:
+  streaming: false

sdks/python/apache_beam/yaml/examples/transforms/sql/sql_bigquery_functions.yaml

@@ -0,0 +1,65 @@
+# coding=utf-8
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# SQL Transform with BigQuery Functions and Syntax
+# This example demonstrates using BigQuery-specific SQL syntax and functions.
+# The calcite_connection_properties enable BigQuery function library and lexical rules.
+
+pipeline:
+  transforms:
+    - type: Create
+      name: CreateSalesData
+      config:
+        elements:
+          - {transaction_id: "txn_001", customer_id: 101, amount: 250.75, timestamp: "2024-01-15T10:30:00Z", product_categories: ["electronics", "accessories"]}
+          - {transaction_id: "txn_002", customer_id: 102, amount: 89.99, timestamp: "2024-01-15T11:45:00Z", product_categories: ["books", "education"]}
+          - {transaction_id: "txn_003", customer_id: 103, amount: 1250.00, timestamp: "2024-01-15T14:20:00Z", product_categories: ["electronics", "computers"]}
+          - {transaction_id: "txn_004", customer_id: 101, amount: 45.50, timestamp: "2024-01-16T09:15:00Z", product_categories: ["food", "groceries"]}
+
+    - type: Sql
+      name: AnalyzeSalesData
+      input: CreateSalesData
+      config:
+        query: |
+          SELECT 
+            customer_id,
+            COUNT(*) as transaction_count,
+            SUM(amount) as total_spent,
+            AVG(amount) as avg_transaction_amount,
+            -- BigQuery-style date/time functions
+            FORMAT_TIMESTAMP('%Y-%m-%d', PARSE_TIMESTAMP('%Y-%m-%dT%H:%M:%SZ', timestamp)) as transaction_date,
+            -- BigQuery array functions (when available)
+            ARRAY_TO_STRING(product_categories, ', ') as categories_str,
+            -- Conditional aggregation using BigQuery syntax
+            COUNTIF(amount > 100) as high_value_transactions,
+            -- BigQuery mathematical functions
+            ROUND(amount, 2) as rounded_amount
+          FROM PCOLLECTION
+          GROUP BY customer_id, transaction_date, categories_str, rounded_amount
+          ORDER BY customer_id, total_spent DESC
+
+    - type: LogForTesting
+      input: AnalyzeSalesData
+
+# Configure Calcite to use BigQuery function library and syntax
+# 'fun': 'bigquery' enables BigQuery-specific functions
+# 'lex': 'big_query' enables BigQuery lexical rules and syntax
+options:
+  calcite_connection_properties:
+    fun: "bigquery"
+    lex: "big_query"
+  streaming: false