odbc fetch batch #40
Merged
odbc fetch batch #40
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Collaborator
lovasoa
commented
Sep 25, 2025
lovasoa
commented
- refactor(odbc): implement bulk fetch using columnar buffers
- refactor(odbc): enhance OdbcValue and ColumnData structures
- fix tests
- refactor(odbc): update text length constants and improve buffer handling
- refactor(odbc): improve binary data length handling in buffer descriptions
This commit introduces a new bulk fetch implementation in the ODBC bridge, utilizing columnar buffers for improved performance. The changes include the addition of a `build_bindings` function to create column bindings, a new `stream_rows` function to handle batch processing, and the extraction of values from buffers. This refactor enhances the efficiency of data retrieval by reducing the overhead of row-by-row fetching.
This commit introduces significant changes to the ODBC module, including the addition of the `OdbcValueVec` enum to encapsulate various ODBC types and the `ColumnData` struct to hold column values along with their type information. The `OdbcValue` and `OdbcValueRef` structures are updated to utilize `ColumnData`, improving memory management through the use of `Arc`. Additionally, utility methods for value extraction and conversion are added, streamlining the handling of ODBC data types and enhancing overall code clarity.
This commit modifies the text length constants in the ODBC bridge, changing `MAX_TEXT_LEN` to 4096 and introducing `MIN_TEXT_LEN` as 1024. Additionally, it enhances the handling of various data types by ensuring that the maximum string length is constrained between these new limits, improving the robustness of buffer descriptions.
…tions This commit modifies the handling of binary data types in the ODBC bridge by adjusting the length constraints for `Binary`, `Varbinary`, and `LongVarbinary`. The new implementation ensures that the length is capped between `MIN_TEXT_LEN` and `MAX_TEXT_LEN`, enhancing the robustness of buffer descriptions.
This commit introduces methods to retrieve raw ODBC Date, Time, and Timestamp values in the OdbcValueRef structure. Additionally, the Decode implementations for NaiveDate, NaiveTime, NaiveDateTime, and DateTime types are updated to prioritize these raw values, enhancing the handling of date and time data in the ODBC bridge.
This commit refactors the error handling in the Decode implementations for NaiveDate, NaiveTime, NaiveDateTime, and DateTime types by simplifying the syntax for creating date and time objects. The changes enhance code readability and maintainability while ensuring robust error reporting for invalid date and time values.
This commit refactors the OdbcValueVec enum by removing nullable variants and consolidating the handling of null values into a separate nulls vector. The conversion function for AnySlice is updated to return both the value vector and the nulls vector, improving the overall structure and clarity of the code. Additionally, the value retrieval methods are simplified to utilize the new nulls handling approach.
This commit updates the test for handling mixed null and non-null values in ODBC queries. The test now verifies the retrieval of multiple rows, including cases with null values, ensuring accurate validation of the query results.
This commit refactors the OdbcRow structure to utilize an OdbcBatch, allowing for better management of column data and row indices. The OdbcBatch encapsulates the columns and their associated data, streamlining the retrieval process and enhancing the overall architecture of the ODBC module.
…ry efficiency This commit updates the OdbcBatch structure to store columns as an Arc array, enhancing memory management and reducing cloning overhead. The change is reflected in both the OdbcBatch definition and its instantiation within the ODBC bridge.
This commit adds OdbcBufferSettings to configure buffer settings for ODBC connections, allowing users to optimize memory usage and performance during data fetching. The new settings include batch size and maximum column size, with corresponding methods in OdbcConnectOptions for easy configuration.
This commit refactors the OdbcValueVec enum to include nullable variants for BigInt and Double types, enhancing the handling of null values. Additionally, the instantiation of ColumnData within tests is updated to utilize OdbcBatch, improving memory efficiency and reducing cloning overhead. The changes streamline the management of column data across various ODBC types.
This commit introduces generic helper functions for handling non-nullable and nullable slices in the conversion of AnySlice to OdbcValueVec. The refactor improves code readability and maintainability by reducing redundancy in the conversion logic. Additionally, the OdbcBridge is updated to utilize a helper function for determining buffer lengths, streamlining the buffer description process for various data types.
This commit introduces an unbuffered mode in OdbcBufferSettings, allowing for more memory-efficient row processing by fetching rows one at a time. The implementation includes updates to the OdbcBridge and related functions to handle both buffered and unbuffered modes effectively. Additionally, tests are added to verify the correct functionality of the unbuffered mode.
…logic This commit refines the OdbcBufferSettings structure to support both buffered and unbuffered modes more effectively. It updates the handling of max_column_size and batch_size, ensuring better memory management and performance tuning. The OdbcBridge is modified to streamline data fetching based on the new settings, and tests are adjusted to reflect these changes, particularly for unbuffered mode.
This commit refactors the OdbcBridge to streamline the usage of OdbcValueVec by removing redundant crate paths and enhancing type handling. The changes improve code clarity and maintainability, particularly in the initialization and data retrieval processes for various data types.
This commit introduces a new error type, MismatchedTypeError, to provide detailed information when a Rust type is incompatible with a SQL type. The error includes fields for the Rust type, its SQL representation, the actual SQL type, and an optional source error. Additionally, the mismatched_types function is updated to utilize this new error type, enhancing error reporting in type decoding scenarios. The Any module is also refactored to remove the obsolete error handling code.
This commit enhances the error handling in OdbcValueRef by utilizing the new MismatchedTypeError for better type mismatch reporting. It also simplifies the data fetching logic in OdbcBridge by consolidating the handling of default values and null checks, improving code clarity and maintainability. Additionally, tests are updated to reflect these changes, ensuring robust functionality across buffered and unbuffered modes.
This commit refactors the error handling in PgRecordDecoder by removing the redundant use of the MismatchedTypeError, streamlining the return of mismatched type errors. The changes enhance code clarity and maintainability in type decoding scenarios.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.