Skip to content

Latest commit

 

History

History
328 lines (268 loc) · 20.1 KB

File metadata and controls

328 lines (268 loc) · 20.1 KB

Development Roadmap

Current Version: v1.1.0


Phase 1: Core Features ✅ (Completed)

Build the skeleton of a basic ODBC driver.

Item Status Notes
DLL project structure (VS / CMake) C++17, x64
ODBC handle management (Env/Dbc/Stmt) Allocation / deallocation
ClickHouse HTTP connection Using cpp-httplib
Connection string parsing SQLDriverConnect / SQLConnect
Query execution (SQLExecDirect) POST + JSONCompact
Result retrieval (SQLFetch / SQLGetData) Forward-only scrolling
Basic type mapping Major ClickHouse types supported
ANSI + Unicode dual support W-suffixed function exports
Catalog functions SQLTables, SQLColumns, etc.
Driver registration script Direct registry registration
Diagnostic functions SQLGetDiagRec/Field

Phase 2: MS Access Compatibility Enhancement ✅ (Completed)

Enable ClickHouse tables to be used as linked tables in MS Access.

Item Status Notes
Comprehensive SQLGetInfo implementation Full coverage of info types required by Access
Enhanced SQLGetTypeInfo Mapping for all ClickHouse types
SQLStatistics implementation Index information (empty result set)
SQLSpecialColumns implementation Row identifier information (empty result set)
SQLForeignKeys implementation Foreign key information (stub)
SQLDescribeCol accuracy improvement Accurate column size and precision reporting
Full SQLColAttribute attribute support Comprehensive coverage of attributes used by Access
Full SQLFetchScroll implementation FIRST/LAST/ABSOLUTE/RELATIVE support
Connection dialog SQLDriverConnect dialog display (GUI not implemented, DriverCompletion handling done)
NULL handling improvement Accurate SQL_NULL_DATA reporting, 22002 error

Phase 3: Complete Data Type Support ✅ (Completed)

Accurately map all ClickHouse data types to ODBC types.

Item Status Notes
Integer types (Int8–Int256, UInt) Signed/unsigned support, Int128/256 as VARCHAR
Floating point (Float32/64) SQL_REAL / SQL_DOUBLE mapping
Decimal (Decimal32/64/128/256) Precision/scale preserved, SQL_C_NUMERIC conversion
String types (String, FixedString) SQL_VARCHAR / SQL_CHAR mapping
Date/time types (Date, DateTime, DateTime64) SQL_C_TYPE_TIME added, DateTime64 fractional seconds
UUID SQL_GUID mapping, SQL_C_GUID conversion
Nullable types Nullable(T) unwrapping, IsNullableType detection
Enum types VARCHAR mapping
Array / Tuple / Map String representation (JSON dump)
LowCardinality Wrapped type unwrapping
Nothing / Interval / Geo types VARCHAR mapping
SimpleAggregateFunction Automatic inner type extraction
JSON / Nested SQL_LONGVARCHAR mapping
Overflow detection SQLSTATE 22003 (Numeric value out of range)
Enhanced type info metadata Comprehensive BuildTypeInfoResultSet

Phase 4: Performance & Reliability ✅ (Completed)

Item Status Notes
Connection pooling SQL_ATTR_CONNECTION_DEAD support (Driver Manager integration)
Large result set streaming Chunked reading (future)
Timeout handling Configurable connection/query timeout, SQLSTATE HYT00
Robust error handling Comprehensive SQLSTATE coverage (08002/HY024/01004/24000/HYT00, etc.)
Thread safety recursive_mutex + HandleLock RAII + g_clients_mutex
Memory leak prevention Defensive cleanup (ENV→DBC→STMT cascade release)
Connection retry Configurable retry_count/retry_delay_ms

Phase 5: Advanced Features ✅ (Completed)

Item Status Notes
Parameterized queries (SQLBindParameter) Client-side parameter substitution, all C types supported
SQLDescribeParam Type info for bound parameters, default SQL_VARCHAR/255
SQLNumParams improvement Accurate count skipping ? inside quotes
DSN registry reading (SQLConnect) Retrieves connection info from ODBC.INI via SQLGetPrivateProfileString
Cursor name (SQLGetCursorName/SQLSetCursorName) ANSI/Wide versions, auto-generated names, truncation support
SQLSetPos (SQL_POSITION/SQL_REFRESH) Cursor positioning, refresh support
SQLFreeStmt SQL_CLOSE Result set reset only, prepared query preserved
Batch INSERT SQLBulkOperations (future)
TLS/SSL support SChannel (future)
DSN configuration (ConfigDSN) ANSI/Wide versions, registry read/write, ADD/CONFIG/REMOVE support
ODBC trace log TraceLog singleton, OutputDebugString + file output, level control
32-bit build support Win32 configuration (future)
Installer (MSI/NSIS) One-click install (future)
CI/CD pipeline GitHub Actions (future)
Unit tests Google Test migration (future)
Integration tests Real ClickHouse server (future)

Phase 6: ODBC 2.x Compatibility & Full MS Access Support ✅ (Completed)

Item Status Notes
SQLExtendedFetch ODBC 2.x backward compat, delegates to SQLFetchScroll, RowCount/RowStatus output
SQLColAttributes (ODBC 2.x) 2.x field ID → 3.x mapping, delegates to SQLColAttribute
SQLSetScrollOptions (ODBC 2.x) Deprecated stub, cursor_type/rowset_size configuration
SQL_ROWSET_SIZE / SQL_ATTR_ROW_ARRAY_SIZE SQLGetStmtAttr/SQLSetStmtAttr support
SQL_ATTR_ROW_STATUS_PTR / SQL_ATTR_ROWS_FETCHED_PTR Statement attribute added
SQLGetFunctions update ODBC 2.x functions + catalog functions bitmap/individual check coverage
SQLGetInfo update SQL_DESCRIBE_PARAMETER → "Y"
DLL export additions SQLExtendedFetch, SQLColAttributes, SQLSetScrollOptions
DLL vcxproj fix Exclude test files, add /FS option

Phase 7: ODBC 2.x Legacy Functions & Data-at-Execution ✅ (Completed)

Item Status Notes
SQLParamData/SQLPutData Full Data-at-Execution implementation, chunk accumulation, DAE state management
SQLExecute DAE detection SQL_DATA_AT_EXEC / SQL_LEN_DATA_AT_EXEC indicator detection, SQL_NEED_DATA return
SQLError/SQLErrorW (ODBC 2.x) Iterative diagnostic record consumption, handle priority (stmt > dbc > env)
SQLGetConnectOption/SQLSetConnectOption (ODBC 2.x) Delegates to SQLGetConnectAttr/SQLSetConnectAttr
SQLGetStmtOption/SQLSetStmtOption (ODBC 2.x) Delegates to SQLGetStmtAttr/SQLSetStmtAttr
SQLTransact (ODBC 2.x) Delegates to SQLEndTran
DLL export additions SQLError/W, SQLGet/SetConnectOption, SQLGet/SetStmtOption, SQLTransact
SQLGetFunctions update All legacy functions added to bitmap/individual checks

Phase 8: Descriptor Handles & SQLBulkOperations ✅ (Completed)

Item Status Notes
DescriptorType enum APD/IPD/ARD/IRD (4 types)
DescriptorRecord struct Type, name, length, precision, scale, nullable, pointers, etc. (~25 fields)
OdbcDescriptor handle Record map, count, type, auto/manual distinction
Statement auto-descriptors auto_apd/ipd/ard/ird (unique_ptr) + current pointers
SQLGetDescField/SQLGetDescFieldW Header fields (COUNT, ALLOC_TYPE) + record fields (~25 IDs)
SQLSetDescField/SQLSetDescFieldW IRD read-only protection, all settable fields supported
SQLGetDescRec/SQLGetDescRecW Composite record retrieval (name, type, subtype, length, precision, scale, nullable)
SQLSetDescRec Composite record setting, IRD protection
SQLCopyDesc Descriptor copy, IRD write protection
SyncDescriptorsFromStatement IRD←result set, ARD←bound columns, APD/IPD←bound parameters sync
SQLGetStmtAttr descriptor attributes SQL_ATTR_APP/IMP_ROW/PARAM_DESC returns actual pointer
SQLSetStmtAttr descriptor attributes Application descriptor set/reset, IMP returns HY017 error
SQLBulkOperations SQL_ADD Builds and executes INSERT from bound columns
SQLGetFunctions update 5 descriptor functions added to bitmap/individual checks
DLL export additions SQLGetDescField/W, SQLSetDescField/W, SQLGetDescRec/W, SQLSetDescRec, SQLCopyDesc

Phase 9: SQLBrowseConnect & Row Array Binding ✅ (Completed)

Item Status Notes
SQLBrowseConnect/SQLBrowseConnectW Iterative connection browsing, required attribute detection, SQL_NEED_DATA/SQL_SUCCESS
BrowseConnectImpl helper DSN reading, HOST/UID required checks, connection attempt
Row Array Binding (column-wise) Multi-row fetch when rowset_size > 1 with SQL_BIND_BY_COLUMN
Row Array Binding (row-wise) Struct size binding via SQL_ATTR_ROW_BIND_TYPE
SQL_ATTR_ROW_BIND_TYPE SQLGetStmtAttr/SQLSetStmtAttr support, default SQL_BIND_BY_COLUMN
SQL_ATTR_PARAMSET_SIZE SQLGetStmtAttr/SQLSetStmtAttr support, default 1, 0→1 clamping
SQLFetch multi-row support Fetch rowset_size rows, row_status_ptr/rows_fetched_ptr output, partial rowset support
SQLGetFunctions update SQLBrowseConnect added to bitmap/individual checks
DLL export additions SQLBrowseConnect/SQLBrowseConnectW

Phase 10: Parameter Array Execution & Compatibility Attribute Enhancement ✅ (Completed)

Item Status Notes
Parameter array execution (paramset_size > 1) Column-wise/row-wise binding, param_status_ptr/params_processed_ptr output
GetCTypeSize()/OffsetParameter() helpers C type size calculation + column/row-wise parameter offset computation
SQLExtendedFetch multi-row support stmt-level pointer save/restore, delegates to SQLFetch multi-row
SQL_ATTR_PARAM_BIND_TYPE SQLGetStmtAttr/SQLSetStmtAttr support, default SQL_PARAM_BIND_BY_COLUMN
SQL_ATTR_PARAM_STATUS_PTR Per-parameter-set status output
SQL_ATTR_PARAMS_PROCESSED_PTR Processed parameter set count output
SQL_ATTR_NOSCAN Default SQL_NOSCAN_OFF
SQL_ATTR_CONCURRENCY Default SQL_CONCUR_READ_ONLY
SQL_ATTR_MAX_LENGTH Default 0 (unlimited)
SQL_ATTR_RETRIEVE_DATA Default SQL_RD_ON
SQL_ATTR_USE_BOOKMARKS Default SQL_UB_OFF

Phase 11: Connection Attribute Completion & SQLGetInfo Enhancement ✅ (Completed)

Item Status Notes
8 additional connection attributes metadata_id, txn_isolation, packet_size, async_enable, quiet_mode, odbc_cursors, translate_lib, translate_option
SQLGetConnectAttr attribute additions 9 attributes supported, default handler returns SQL_SUCCESS
SQLGetConnectAttrW string attributes Wide version for current_catalog, tracefile, translate_lib
SQLSetConnectAttr attribute additions Set support for 9 attributes
SQLSetConnectAttrW string attributes Wide→UTF-8 conversion for 3 string attributes
SQLGetInfo 20+ new info types FETCH_DIRECTION, FILE_USAGE, POS_OPERATIONS, LOCK_TYPES, DATETIME_LITERALS, etc.
Driver version update SQL_DRIVER_VER "01.00.0000"
Improved unknown attribute default handler SQL_ERROR → SQL_SUCCESS (improved compatibility)

Phase 12: SQLGetData Piecemeal Retrieval & RowCount Improvement & Driver Quality Enhancement ✅ (Completed)

Item Status Notes
SQLGetData piecemeal retrieval Partial reads for SQL_C_CHAR/SQL_C_WCHAR/SQL_C_BINARY, offset tracking, SQL_NO_DATA completion
SQLRowCount improvement Non-SELECT returns affected_rows, SELECT returns rows.size()
affected_rows tracking INSERT/UPDATE/DELETE detection in ExecDirectImpl, affected_rows setting
SQLCancel enhancement DAE state reset, getdata state reset, handle validation
SQL_ATTR_CURSOR_SCROLLABLE SQLGetStmtAttr/SQLSetStmtAttr support, default SQL_NONSCROLLABLE
SQL_ATTR_CURSOR_SENSITIVITY SQLGetStmtAttr/SQLSetStmtAttr support, default SQL_INSENSITIVE
SQLGetDiagField enhancement SQL_DIAG_CURSOR_ROW_COUNT, SQL_DIAG_DYNAMIC_FUNCTION, SQL_DIAG_DYNAMIC_FUNCTION_CODE, SQL_DIAG_ROW_COUNT improvement

Phase 13: ODBC Escape Sequence Processing & Environment Attribute Enhancement & Driver Compatibility Improvement ✅ (Completed)

Item Status Notes
ODBC escape sequence parser ProcessOdbcEscapeSequences(){fn}, {d}, {t}, {ts}, {oj}, {escape}, {call} support
Scalar function conversion (60+ functions) UCASE→upper, LCASE→lower, NOW→now, CURDATE→today, CONVERT→toXxx, etc.
String function mapping CONCAT, LENGTH, SUBSTRING, LTRIM, RTRIM, LEFT, RIGHT, REPLACE, LOCATE, SPACE, REPEAT, ASCII, CHAR
Numeric function mapping ABS, CEILING, FLOOR, ROUND, TRUNCATE, SQRT, POWER, EXP, LOG, MOD, PI, SIN, COS, TAN, etc.
Date/time function mapping YEAR→toYear, MONTH→toMonth, DAYOFMONTH→toDayOfMonth, HOUR, MINUTE, SECOND, WEEK, QUARTER, etc.
CONVERT function SQL type conversion SQL_VARCHAR→toString, SQL_INTEGER→toInt32, SQL_DOUBLE→toFloat64, SQL_DATE→toDate, etc.
Nested escape sequences Recursive processing of {fn UCASE({fn SUBSTRING(...)})}
ExecDirectImpl integration Automatic escape sequence conversion before query execution
SQLPrepare/SQLPrepareW integration Automatic escape sequence conversion at prepare time
SQLNativeSql/SQLNativeSqlW integration Escape processing added to native SQL conversion
SQL_ATTR_CONNECTION_POOLING improvement Field added to OdbcEnvironment, full Get/Set support
SQL_ATTR_CP_MATCH SQLGetEnvAttr/SQLSetEnvAttr support, default SQL_CP_STRICT_MATCH
SQLGetDiagField SERVER_NAME improvement SQL_DIAG_SERVER_NAME/CONNECTION_NAME returns actual host name (STMT/DBC support)
SQLGetInfo function bitmask enhancement SQL_STRING_FUNCTIONS 18 bits, SQL_NUMERIC_FUNCTIONS 22 bits, SQL_TIMEDATE_FUNCTIONS 20 bits
SQLGetInfo system functions update SQL_FN_SYS_DBNAME, SQL_FN_SYS_USERNAME added
SQLGetInfo conversion functions update SQL_FN_CVT_CONVERT added

MS Access Workflow Tests ✅ (Completed)

Integration tests based on real use cases using accdb files.

Item Status Notes
Table link creation workflow SQLTables→SQLColumns→SQLStatistics→SQLSpecialColumns→SQLPrimaryKeys→SQLForeignKeys sequence
Driver info & capability checks SQLGetInfo key values, SQLGetFunctions full function bitmap
Datasheet view reading SQLNumResultCols→SQLDescribeCol→SQLBindCol→SQLFetch flow
Parameterized queries (filtering) SQLPrepare + SQLBindParameter + WHERE clause
NULL value handling SQL_NULL_DATA detection, mixed NULL/non-NULL result sets
ODBC 2.x compatible fetch SQLExtendedFetch, SQLColAttributes
ODBC escape sequences Date/time/function/CONVERT/outer join/LIKE escape
Data type conversion Integer/float/date/timestamp/Memo piecemeal
Connection & statement attributes AUTOCOMMIT/CURSOR_TYPE/CONCURRENCY/QUERY_TIMEOUT
Transactions (no-op) Both COMMIT/ROLLBACK return SQL_SUCCESS
Diagnostic info retrieval SQLGetDiagRec/SQLGetDiagField
Column metadata (3.x + 2.x API) SQLColAttribute + SQLColAttributes
Multiple simultaneous statements Independent statements on the same connection
End-to-End integration test All phases (info check→catalog→data read→escape)
Report aggregation queries YEAR/MONTH + GROUP BY + BETWEEN escape
ClickHouse type mapping validation Including Nullable/LowCardinality
Shutdown sequence Correct release order: stmt→dbc→env

Phase 14: Bug Fixes & Driver Quality Enhancement ✅ (Completed)

Bug fixes found during code review and ODBC spec compliance improvements.

Item Status Notes
ExecDirectImpl escape sequence bug fix Use processed_query for actual query execution (previously sent unprocessed query)
SQLGetFunctions SQL_API_ALL_FUNCTIONS support Added ODBC 2.x 100-element array (FunctionId=0) support
SQLGetFunctions individual check fix Added SQLGETENVATTR/SQLSETENVATTR/SQLGETSTMTATTR/SQLSETSTMTATTR/SQLGETDIAGREC/SQLGETDIAGFIELD/SQLNATIVESQL/legacy alloc functions
SQLDriverConnect DriverCompletion handling SQL_DRIVER_PROMPT→HYC00, SQL_DRIVER_COMPLETE/REQUIRED→required parameter check
SQLCloseCursor 24000 check Returns SQLSTATE 24000 when cursor is not open
SQLNativeSqlW output length fix Correctly returns TextLength2Ptr even when OutStatementText is null

Phase 15: Driver Quality Enhancement (Additional Bug Fixes) ✅ (Completed)

Additional bug fixes found during detailed examination of all source files.

Item Status Notes
SQLColumns SQL_DATA_TYPE date/time fix Returns SQL_DATETIME(9) for date/time/timestamp types (previously returned concise type)
ExecuteQuery URL encoding fix Safe URL parameter encoding via httplib::Params (special character handling)
SQLGetDiagFieldW Wide conversion fix Wide conversion for 7 string field types
SQLGetDiagRecW TextLength fix Fixed from byte count to character count (ODBC spec compliance)
SQLErrorW TextLength fix Fixed from UTF-8 byte count to Wide character count
LOCATE escape function argument order fix Swapped ODBC LOCATE(needle,haystack) arguments to positionUTF8(haystack,needle)

Version Target Schedule
v0.1.0 Core ODBC features, basic connection & query execution ✅ Completed
v0.2.0 MS Access linked table basic operation ✅ Completed
v0.3.0 Full data type support, accurate NULL handling ✅ Completed
v0.5.0 Performance improvement, robust error handling ✅ Completed
v0.6.0 Parameterized queries, trace log, ConfigDSN ✅ Completed
v0.7.0 SQLDescribeParam, cursor name, DSN reading, SQLSetPos ✅ Completed
v0.8.0 ODBC 2.x compatibility (SQLExtendedFetch, etc.), full MS Access support ✅ Completed
v0.9.0 Full ODBC 2.x legacy function support, Data-at-Execution ✅ Completed
v1.0.0-rc Descriptor handles, SQLBulkOperations ✅ Completed
v1.0.0-rc2 SQLBrowseConnect, Row Array Binding ✅ Completed
v1.0.0-rc3 Parameter array execution, compatibility attribute enhancement ✅ Completed
v1.0.0-rc4 Connection attribute completion, SQLGetInfo enhancement ✅ Completed
v1.0.0-rc5 SQLGetData piecemeal retrieval, RowCount improvement, driver quality enhancement ✅ Completed
v1.0.0-rc6 ODBC escape sequence processing, environment attribute enhancement, driver compatibility improvement ✅ Completed
v1.0.0-rc7 MS Access workflow tests — accdb use case validation (46 tests) ✅ Completed
v1.0.0-rc8 Bug fixes & driver quality — escape bug fix, SQLGetFunctions completion, DriverCompletion support ✅ Completed
v1.0.0-rc9 Driver quality — SQLColumns date/time, URL encoding, DiagFieldW Wide conversion, LOCATE argument order fix ✅ Completed
v1.0.0 Production-ready (Debug/Release build fully supported, 623 tests passed) ✅ Completed
v1.1.0 MS Access real-environment testing (catalog bypass, search pattern escape, SQL_C_DEFAULT resolution, trace infrastructure, 678 tests) ✅ Completed

v1.0.0 Release ✅ (Completed)

Quality assurance and final adjustments for release.