Skip to content

Commit 407936e

Browse files
committed
feat: IRIS pgwire compatibility fixes for DDL, parameters, and timestamps
- Fixed multi-statement DDL with comments by enhancing DdlSplitter and stripping comments - Consolidated parameter translation ( -> ?) in SQLTranslator across all query paths - Implemented DefaultValuesTranslator to rewrite INSERT statements with DEFAULT in VALUES - Normalized ISO 8601 timestamps to IRIS-compatible format - Translated PostgreSQL ALTER TABLE SET DATA TYPE/DROP NOT NULL to IRIS-compatible syntax - Enhanced IRIS data type mapping for PostgreSQL OID compatibility - Added comprehensive unit and E2E integration tests - Bumped version to 1.0.6
1 parent 770bee0 commit 407936e

38 files changed

+1427
-371
lines changed

AGENTS.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,8 @@ Auto-generated from all feature plans. Last updated: 2026-01-02
55
## Active Technologies
66
- Python 3.11 + python>=3.11, psycopg[binary], iris-devtester, intersystems-irispython (026-address-gaps-in)
77
- PostgreSQL (via InterSystems IRIS) (026-address-gaps-in)
8+
- Python 3.11 + intersystems-irispython, psycopg[binary], iris-devtester (034-issues-that-likely)
9+
- InterSystems IRIS (via PostgreSQL wire protocol) (034-issues-that-likely)
810

911
- Python 3.11+ + `iris-devtester`, `intersystems-irispython`, `psycopg[binary]` (033-devtester-skills)
1012

@@ -24,6 +26,7 @@ cd src [ONLY COMMANDS FOR ACTIVE TECHNOLOGIES][ONLY COMMANDS FOR ACTIVE TECHNOLO
2426
Python 3.11+: Follow standard conventions
2527

2628
## Recent Changes
29+
- 034-issues-that-likely: Added Python 3.11 + intersystems-irispython, psycopg[binary], iris-devtester
2730
- 026-address-gaps-in: Added Python 3.11 + python>=3.11, psycopg[binary], iris-devtester, intersystems-irispython
2831

2932
- 033-devtester-skills: Added Python 3.11+ + `iris-devtester`, `intersystems-irispython`, `psycopg[binary]`

CHANGELOG.md

Lines changed: 10 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -5,42 +5,22 @@ All notable changes to this project will be documented in this file.
55
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
66
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
77

8-
## [Unreleased]
8+
## [1.0.6] - 2026-01-16
99

1010
### Added
11-
- **Multi-Action ALTER TABLE Splitting**: Automatically decomposes PostgreSQL-style multi-column `ALTER TABLE` statements (e.g., multiple `ADD COLUMN` or `DROP COLUMN` actions) into individual IRIS-compatible statements.
12-
- **IRIS Bridge Gaps** (Feature 026): Comprehensive performance and functionality enhancements for IRIS integration
13-
- **Fast Path Bulk Insert**: Protocol-level batching achieving 3,700+ rows/second (11× improvement)
14-
- **HNSW Index Translation**: Support for PostgreSQL `USING hnsw` translated to IRIS `AS HNSW`
15-
- **Recursive JSON Pathing**: Support for nested `->` and `->>` operators translated to `JSON_VALUE`
16-
- **DDL Idempotency**: Full `IF NOT EXISTS` support for `CREATE TABLE` and `CREATE INDEX`
17-
- **Simple Query Translation**: Enabled full SQL translation for the standard PostgreSQL query protocol
18-
- **P6 COPY Protocol** (Feature 023): PostgreSQL COPY FROM STDIN and COPY TO STDOUT for bulk data operations
19-
- Bulk data import/export with CSV processing and streaming
20-
- 1000-row batching for memory efficiency (<100MB for 1M rows)
21-
- Transaction integration with automatic rollback on errors
22-
- Query-based export support (`COPY (SELECT ...) TO STDOUT`)
23-
- Performance: 600+ rows/second sustained throughput
11+
- **Multi-statement DDL with comments**: Updated `DdlSplitter` to be fully comment-aware and strip comments before execution to prevent IRIS parsing errors.
12+
- **Prepared statement translation ($n → ?)**: Consolidated parameter translation logic into `SQLTranslator` for consistency across all query paths.
13+
- **Default keyword in VALUES clause**: Implemented `DefaultValuesTranslator` to rewrite `INSERT` statements using `DEFAULT` within `VALUES` lists.
14+
- **Timestamp binding normalization**: Updated `DATETranslator` and `IRISExecutor` to normalize ISO 8601 timestamps (stripping `T`, `Z`, and offsets) into IRIS-accepted ODBC formats.
15+
- **ALTER TABLE translation**: Updated `DdlSplitter` to translate PostgreSQL `SET DATA TYPE` and `DROP NOT NULL` syntax to IRIS-compatible `ALTER COLUMN` commands.
2416

2517
### Fixed
26-
- Fixed critical bug in translation cache causing `TypeError` with mixed naive/aware datetimes
27-
- Standardized all internal timestamps to timezone-aware UTC
28-
- Fixed `CREATE INDEX` idempotency via specialized comment marker `/* IF_NOT_EXISTS */`
29-
- Dynamic versioning recognition in package metadata validation
30-
- Python bytecode cleanup (95+ artifacts removed from git)
31-
- Black code formatting (20 files reformatted to compliance)
32-
- asyncpg parameter type OID inference from CAST expressions
33-
- PostgreSQL compatibility documentation improvements
18+
- Fixed IRIS execution error "Input encountered after end of query" by improving semicolon and comment handling in the DDL splitter.
19+
- Resolved "LITERAL (1) found" errors during migrations by avoiding no-op SELECT injections for skipped DDL.
20+
- Enhanced IRIS data type mapping to better handle PostgreSQL OIDs (e.g., `BIGINT`, `TINYINT`).
3421

35-
### Security
36-
- Upgraded authlib to 1.6.5 (fixes 3 HIGH severity CVEs)
37-
- Upgraded cryptography to 46.0.3 (fixes 1 HIGH severity CVE)
22+
## [1.0.5] - 2026-01-15
3823

39-
### Performance
40-
- High-performance DML Fast Path: Buffering and collapsing `Sync` cycles into bulk IRIS calls
41-
- IRIS executemany() optimization for 4-10× performance improvement in bulk operations
42-
- COPY protocol optimized for 600+ rows/second sustained throughput
43-
- Memory-efficient streaming for large result sets
4424

4525
## [0.1.0] - 2025-01-05
4626

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
# Specification Quality Checklist: IRIS pgwire compatibility fixes
2+
3+
**Purpose**: Validate specification completeness and quality before proceeding to planning
4+
**Created**: 2026-01-16
5+
**Feature**: [Link to spec.md](../spec.md)
6+
7+
## Content Quality
8+
9+
- [x] No implementation details (languages, frameworks, APIs)
10+
- [x] Focused on user value and business needs
11+
- [x] Written for non-technical stakeholders
12+
- [x] All mandatory sections completed
13+
14+
## Requirement Completeness
15+
16+
- [x] No [NEEDS CLARIFICATION] markers remain
17+
- [x] Requirements are testable and unambiguous
18+
- [x] Success criteria are measurable
19+
- [x] Success criteria are technology-agnostic (no implementation details)
20+
- [x] All acceptance scenarios are defined
21+
- [x] Edge cases are identified
22+
- [x] Scope is clearly bounded
23+
- [x] Dependencies and assumptions identified
24+
25+
## Feature Readiness
26+
27+
- [x] All functional requirements have clear acceptance criteria
28+
- [x] User scenarios cover primary flows
29+
- [x] Feature meets measurable outcomes defined in Success Criteria
30+
- [x] No implementation details leak into specification
31+
32+
## Notes
33+
34+
- Items marked incomplete require spec updates before `/speckit.clarify` or `/speckit.plan`
35+
- Validation pass after 1 iteration; no clarifications required.
Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
# Contract: ALTER TABLE Compatibility
2+
3+
## Purpose
4+
Define how ALTER TABLE SET DATA TYPE and DROP NOT NULL behave for IRIS.
5+
6+
## Inputs
7+
- ALTER TABLE statements that change column type or nullability.
8+
9+
## Outputs
10+
- IRIS-compatible ALTER TABLE statements or clear errors when unsupported.
11+
12+
## Rules
13+
1. **SET DATA TYPE translation**: `ALTER COLUMN x SET DATA TYPE T` MUST be translated to `ALTER COLUMN x T` when supported.
14+
2. **DROP NOT NULL translation**: `ALTER COLUMN x DROP NOT NULL` MUST be translated to `ALTER COLUMN x NULL` when supported.
15+
3. **Best-effort behavior**: When IRIS rejects the operation due to constraints, return a clear, actionable error.
16+
17+
## Error Handling
18+
- Errors MUST indicate the column and operation that failed and suggest IRIS constraints as the cause.
19+
20+
## Acceptance Criteria
21+
- Supported ALTER TABLE changes execute successfully in IRIS.
22+
- Unsupported changes return clear, actionable errors.
Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
# Contract: DDL Splitting and Comment Handling
2+
3+
## Purpose
4+
Ensure multi-statement SQL and DDL with comments are split safely without corrupting statement boundaries.
5+
6+
## Inputs
7+
- SQL text potentially containing multiple statements, comments, and string literals.
8+
9+
## Outputs
10+
- Ordered list of individual SQL statements, preserving semantics.
11+
12+
## Rules
13+
1. **Comment-aware splitting**: Semicolons inside `--` or `/* */` comments MUST NOT split statements.
14+
2. **String-aware splitting**: Semicolons inside single or double-quoted strings MUST NOT split statements.
15+
3. **Leading comments**: Leading comment blocks MUST be preserved with their associated statement.
16+
4. **ALTER TABLE multi-action**: Multi-action `ALTER TABLE` statements MUST be decomposed into single-action statements for IRIS execution.
17+
18+
## Error Handling
19+
- If DDL decomposition fails, return a clear error that includes the original statement context.
20+
21+
## Acceptance Criteria
22+
- DDL scripts with comments and multiple statements execute in order without corruption.
23+
- Multi-action ALTER TABLE statements are split into valid IRIS statements.
Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
# Contract: DEFAULT in VALUES Handling
2+
3+
## Purpose
4+
Provide IRIS-compatible behavior for INSERT statements that use `DEFAULT` within a VALUES list.
5+
6+
## Inputs
7+
- INSERT statement with column list and VALUES list that may contain `DEFAULT`.
8+
9+
## Outputs
10+
- INSERT statement rewritten to omit DEFAULT column/value pairs.
11+
12+
## Rules
13+
1. **DEFAULT removal**: Columns paired with `DEFAULT` in VALUES MUST be removed from both column list and values list.
14+
2. **Row integrity**: Remaining columns and values MUST preserve ordering and alignment.
15+
3. **Row-level DEFAULT VALUES**: `INSERT INTO table DEFAULT VALUES` passes through unchanged.
16+
17+
## Error Handling
18+
- If DEFAULT removal would empty the column list while values remain, return a clear error.
19+
20+
## Acceptance Criteria
21+
- INSERT statements with per-column DEFAULT values execute successfully in IRIS.
22+
- Defaults are applied as defined by table schema.
Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
# Contract: Prepared Statement Parameter Translation
2+
3+
## Purpose
4+
Guarantee that prepared statements use IRIS-compatible parameter placeholders across all protocol paths.
5+
6+
## Inputs
7+
- SQL statement containing PostgreSQL positional parameters (`$1`, `$2`, ...).
8+
9+
## Outputs
10+
- SQL statement with IRIS-compatible `?` placeholders and translated casts.
11+
12+
## Rules
13+
1. **Placeholder translation**: All `$n` placeholders MUST be translated to `?`.
14+
2. **Type cast translation**: PostgreSQL `::type` casts MUST be rewritten as IRIS `CAST(... AS type)`.
15+
3. **Apply to all query paths**: Translation MUST run for simple and extended protocol flows.
16+
17+
## Error Handling
18+
- If translation encounters unsupported casts, return a clear error specifying the cast.
19+
20+
## Acceptance Criteria
21+
- Prepared statements using `$n` placeholders execute successfully on IRIS.
22+
- Parameter description inference aligns with translated casts.
Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
# Contract: SQL Normalization Pipeline
2+
3+
## Purpose
4+
Define the required ordering and behaviors for SQL normalization so that IRIS receives compatible statements without client-side rewrites.
5+
6+
## Inputs
7+
- Raw SQL statement from PostgreSQL client.
8+
9+
## Outputs
10+
- Normalized SQL statement suitable for IRIS execution.
11+
12+
## Rules
13+
1. **Parameter translation precedes normalization**: `$n` placeholders MUST be translated to `?` before any normalization steps run.
14+
2. **Schema mapping**: Unqualified `public` schema references MUST map to the configured IRIS schema.
15+
3. **Identifier normalization**: Unquoted identifiers MUST be normalized for IRIS compatibility.
16+
4. **Date literal translation**: ISO date literals MUST be translated to IRIS-accepted forms.
17+
5. **JSON operator translation**: `->` and `->>` MUST be mapped to IRIS JSON equivalents.
18+
6. **Vector type normalization**: Vector type declarations MUST be normalized to IRIS vector syntax.
19+
20+
## Error Handling
21+
- If translation fails, return a clear error indicating the unsupported construct.
22+
23+
## Acceptance Criteria
24+
- Given a SQL statement with parameters and supported constructs, the output is valid IRIS SQL.
25+
- Given a statement with unsupported constructs, a clear, actionable error is returned.
Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
# Contract: Timestamp Normalization
2+
3+
## Purpose
4+
Ensure timestamp values are accepted by IRIS when clients provide ISO 8601 strings or binary-encoded timestamps.
5+
6+
## Inputs
7+
- Timestamp literals in SQL or bound parameter values.
8+
9+
## Outputs
10+
- IRIS-compatible timestamp strings (`YYYY-MM-DD HH:MM:SS[.fff]`).
11+
12+
## Rules
13+
1. **Timezone suffix stripping**: `Z` or offset suffixes MUST be removed for standard timestamp handling.
14+
2. **Binary decoding**: Binary timestamp parameters MUST be converted to IRIS-compatible strings.
15+
3. **Microseconds support**: Preserve fractional seconds when present.
16+
17+
## Error Handling
18+
- If a timestamp cannot be normalized, return a clear error with the offending value.
19+
20+
## Acceptance Criteria
21+
- ISO 8601 timestamps with `T`/`Z` or offsets are accepted after normalization.
22+
- Binary timestamp parameters execute successfully on IRIS.
Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
# Data Model: IRIS pgwire compatibility fixes
2+
3+
## Entities
4+
5+
### SQL Statement
6+
- **Description**: Client-submitted SQL text that may contain comments, parameters, and multiple statements.
7+
- **Key Fields**:
8+
- `raw_text`: Original SQL text from client.
9+
- `normalized_text`: SQL after normalization/translation steps.
10+
- `statement_list`: Parsed list of individual statements (order-preserving).
11+
- **Relationships**:
12+
- Contains zero or more `Parameter Binding` entries.
13+
14+
### Parameter Binding
15+
- **Description**: Positional parameter placeholders and runtime values supplied by the client.
16+
- **Key Fields**:
17+
- `placeholder_style`: `$n` (PostgreSQL) or `?` (IRIS).
18+
- `values`: Ordered list of bound parameter values.
19+
- `types`: Optional inferred types for parameter descriptions.
20+
- **Relationships**:
21+
- Belongs to a `SQL Statement`.
22+
23+
### Timestamp Value
24+
- **Description**: Temporal values passed as literals or bound parameters that must be compatible with IRIS.
25+
- **Key Fields**:
26+
- `input_format`: ISO 8601 variants (with/without timezone suffix).
27+
- `normalized_format`: IRIS-accepted ODBC-like timestamp string.
28+
- **Validation Rules**:
29+
- Timezone suffixes are stripped for standard timestamp columns.
30+
31+
## State Transitions
32+
- **Statement Lifecycle**: `raw_text``normalized_text``statement_list` → executed.
33+
- **Parameter Lifecycle**: `$n` placeholders + values → translated placeholders + normalized values.

0 commit comments

Comments
 (0)