Commit 0c5a597
committed
refactor(be): Phase 1 - Extract JniDataBridge utility class from JniConnector
Extract all static data exchange methods from JniConnector into a new
JniDataBridge utility class. JniDataBridge is a pure stateless utility
for C++ Block <-> JNI shared memory data exchange, data-source agnostic.
Moved to JniDataBridge:
- TableMetaAddress helper class
- fill_block / fill_column (Java -> C++ Block)
- to_java_table / parse_table_schema (C++ Block -> Java)
- get_jni_type / get_jni_type_with_different_string (type mapping)
- All private _fill_*_column helpers and templates
JniConnector retains backward-compatible inline wrappers that delegate
to JniDataBridge, so all existing callers (vjdbc_connector,
vjni_format_transformer, UDF functions, 6 JNI readers) continue to
work without changes.
This is Phase 1 of the JNI read/write architecture refactoring plan.
refactor(be/fe): Phase 2 - Unify JDBC writer with VJniFormatTransformer
Refactor VJdbcTableWriter to use VJniFormatTransformer instead of
inheriting from JdbcConnector. This follows the same pattern as
VMCPartitionWriter for MaxCompute writes.
C++ changes:
- VJdbcTableWriter no longer inherits JdbcConnector; holds a
VJniFormatTransformer instance instead
- open() creates VJniFormatTransformer with JdbcJniWriter class name
- write() delegates to VJniFormatTransformer::write()
- Build writer params from TDataSink instead of JdbcConnectorParam
- VJniFormatTransformer now uses JniDataBridge directly
Java changes:
- New JdbcJniWriter class extending JniWriter
- Handles JDBC connection pool, PreparedStatement batch inserts,
and transaction control (openTrans/commitTrans/rollbackTrans)
- insertColumn/insertNullColumn logic adapted from BaseJdbcExecutor
- Independent class, does not modify existing BaseJdbcExecutor
This is Phase 2 of the JNI read/write architecture refactoring plan.
refactor(be/fe): Phase 3 - JDBC reader integration via JniReader (transitional)
Integrate JDBC reading into the unified JniReader/GenericReader framework
while keeping the existing JdbcScanner as a transitional pipeline entry
point (no FE changes required).
C++ changes:
- New JdbcJniReader (extends JniReader) delegates to JdbcJniScanner
on the Java side via JniConnector, following the PaimonJniReader pattern
- JdbcScanner rewritten to delegate to JdbcJniReader instead of
JdbcConnector. Builds JDBC params from TupleDescriptor and passes
them to JdbcJniReader. Old JdbcConnector/JdbcConnectorParam removed.
- No FE changes needed: JDBCScanOperatorX still creates JdbcScanner
using TJdbcScanNode params
Java changes:
- New JdbcJniScanner (extends JniScanner) manages its own connection
pool, PreparedStatement, ResultSet lifecycle
- Reads column values from ResultSet into VectorTable using the
standard JniScanner getNext/getNextBatchMeta protocol
- Independent of BaseJdbcExecutor, uses JdbcDataSource for pool mgmt
This is Phase 3 of the JNI read/write architecture refactoring plan.
refactor(be): Phase 4 - Strip JdbcConnector to utility-only class
Remove all read/write functionality from JdbcConnector, reducing it
from ~690 lines to ~200 lines. It now only serves as a utility for:
- test_connection() (used by PInternalService::test_jdbc_connection)
- clean_datasource() (used after test_connection)
Removed:
- TableConnector inheritance (no longer needed)
- All read methods: query(), get_next(), _get_reader_params(),
_cast_string_to_hll/bitmap/json()
- All write methods: append(), exec_stmt_write(),
begin_trans(), abort_trans(), finish_trans()
- JdbcStatistic struct (was only used by old JdbcScanner)
- All JNI method IDs for read/write operations
- Data type casting maps and related member variables
- TupleDescriptor dependency (not needed for test_connection)
Updated:
- internal_service.cpp: removed use_transaction param (no longer exists)
JDBC read now flows: JdbcScanner -> JdbcJniReader -> JniConnector -> JdbcJniScanner
JDBC write now flows: VJdbcTableWriter -> VJniFormatTransformer -> JdbcJniWriter
This completes Phase 4 of the JNI read/write architecture refactoring.
refactor(be/fe): Remove JdbcConnector entirely
Replace JdbcConnector with JniConnector + JdbcConnectionTester for
the last remaining use case (test_jdbc_connection in internal_service).
Changes:
- New JdbcConnectionTester.java (extends JniScanner): lightweight
connection tester that verifies connectivity in open()
- internal_service.cpp: test_jdbc_connection now uses JniConnector
with JdbcConnectionTester instead of JdbcConnector
- Extracted _resolve_jdbc_driver_url() as standalone function for
driver URL resolution (was inside JdbcConnector)
- Deleted vjdbc_connector.h and vjdbc_connector.cpp entirely
JdbcConnector, JdbcConnectorParam, and all JNI method registrations
for the old BaseJdbcExecutor path are now gone from the codebase.
refactor(fe): Mark legacy BaseJdbcExecutor hierarchy as @deprecated
After JdbcConnector removal, BaseJdbcExecutor and JdbcExecutorFactory
are no longer called from any C++ code. They are retained temporarily
because JdbcJniScanner does not yet implement all database-specific
type conversions from the executor subclasses (MySQL, Oracle, PG, etc.).
Added @deprecated annotations pointing to the new replacements:
- Read: JdbcJniScanner (extends JniScanner)
- Write: JdbcJniWriter (extends JniWriter)
- Connection test: JdbcConnectionTester (extends JniScanner)
refactor(fe/be): Add JdbcTypeHandler strategy for database-specific type handling
Introduce the JdbcTypeHandler strategy pattern to replace the monolithic
BaseJdbcExecutor class hierarchy with pluggable, per-database type handlers.
New files (Java):
- JdbcTypeHandler: strategy interface with 6 extension points
(getColumnValue, getOutputConverter, setValidationQuery,
initializeStatement, abortReadConnection, setSystemProperties)
- DefaultTypeHandler: generic JDBC type reading + shared utilities
- JdbcTypeHandlerFactory: maps table_type string to handler
- 9 database-specific handlers:
MySQLTypeHandler, OracleTypeHandler, PostgreSQLTypeHandler,
ClickHouseTypeHandler, SQLServerTypeHandler, DB2TypeHandler,
SapHanaTypeHandler, TrinoTypeHandler, GbaseTypeHandler
Modified files:
- JdbcJniScanner: refactored to delegate to JdbcTypeHandler
instead of hardcoded getColumnValue(). Also initializes
per-column output converters once in open().
- jdbc_scanner.h: added _odbc_table_type_to_string() helper
- jdbc_scanner.cpp: passes table_type param to JdbcJniScanner
Key design decisions:
- Type handling logic ported 1:1 from BaseJdbcExecutor subclasses
- MySQLTypeHandler uses Integer.MIN_VALUE fetchSize for streaming
- MySQL/SQLServer handlers abort connection on incomplete reads
- Oracle/DB2/SapHana have database-specific validation queries
- Output converters cached per-column for performance
refactor(fe/be): Integrate JDBC into FileScanner path (FileQueryScanNode)
Migrate JdbcScanNode from ExternalScanNode to FileQueryScanNode, unifying
JDBC reads with the same framework used by Paimon/Hudi/Iceberg/MaxCompute.
Architecture change:
Before: JdbcScanNode(ExternalScanNode) → JDBC_SCAN_NODE → JDBCScanOperatorX
After: JdbcScanNode(FileQueryScanNode) → FILE_SCAN_NODE → FileScanOperatorX
FE changes:
- JdbcScanNode: extends FileQueryScanNode instead of ExternalScanNode
- Implements getSplits() returning single JdbcSplit
- Implements setScanParams() → TTableFormatFileDesc with jdbc_params
- getFileFormatType() returns FORMAT_JNI
- getPathPartitionKeys()/getLocationProperties() return empty
- Preserves all predicate push-down logic unchanged
- New JdbcSplit extends FileSplit with JDBC connection parameters
- PhysicalPlanTranslator: updated visitPhysicalJdbcScan with proper
TableRefInfo setup for FileQueryScanNode compatibility
Thrift changes:
- TTableFormatFileDesc: added field 11 'jdbc_params' (map<string,string>)
BE changes:
- file_scanner.cpp: added 'jdbc' branch in FORMAT_JNI reader creation
- Extracts jdbc_params from TTableFormatFileDesc
- Creates JdbcJniReader with params map
- Added jdbc_jni_reader.h include
format
[refactor](be) Merge JniConnector into JniReader and remove Avro/LakeSoul
- Merge all JniConnector instance methods into JniReader base class
(open/get_next_block/close/fill_block/get_table_schema/get_statistics)
- Update all 6 JniReader subclasses to pass connector_class/params/column_names
to JniReader base constructor instead of creating JniConnector
- Move JniConnector static method callers to JniDataBridge
(UDF/UDAF/UDTF: to_java_table, parse_table_schema, fill_block)
- Adapt internal_service.cpp JDBC connection test to use JniReader
- Remove Avro JNI reader (avro_jni_reader.h/cpp)
- Remove LakeSoul JNI reader (lakesoul_jni_reader.h/cpp)
- Delete jni_connector.h/cpp entirely
Files deleted:
be/src/vec/exec/jni_connector.h
be/src/vec/exec/jni_connector.cpp
be/src/vec/exec/format/avro/avro_jni_reader.h
be/src/vec/exec/format/avro/avro_jni_reader.cpp
be/src/vec/exec/format/table/lakesoul_jni_reader.h
be/src/vec/exec/format/table/lakesoul_jni_reader.cpp
format 2
fix compile
compile bug
fix compile
fix compile
fix compile
fix compile
format
fix review
fix 1
fix 2
fix 3
format
fix mysql array
fix jdbc 2
oracle test sql
format
format 2
deprecated
fix ut
2
support doris special type
format
pg date
fix jdbc table
fix oracle timestamp tz
fix oracle sqlserver test
fix pg binary1 parent 10b052f commit 0c5a597
File tree
67 files changed
+4591
-2679
lines changed- be
- src
- service
- util
- vec
- aggregate_functions
- exec
- format
- avro
- table
- scan
- exprs/table_function
- functions
- runtime
- sink/writer
- test/vec/exec
- fe
- be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc
- fe-core/src/main/java/org/apache/doris
- datasource/jdbc/source
- nereids/glue/translator
- gensrc/thrift
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
67 files changed
+4591
-2679
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
41 | 41 | | |
42 | 42 | | |
43 | 43 | | |
44 | | - | |
| 44 | + | |
45 | 45 | | |
46 | 46 | | |
47 | 47 | | |
| |||
107 | 107 | | |
108 | 108 | | |
109 | 109 | | |
| 110 | + | |
110 | 111 | | |
111 | 112 | | |
112 | 113 | | |
| |||
119 | 120 | | |
120 | 121 | | |
121 | 122 | | |
122 | | - | |
123 | 123 | | |
124 | 124 | | |
125 | 125 | | |
| |||
867 | 867 | | |
868 | 868 | | |
869 | 869 | | |
870 | | - | |
871 | | - | |
872 | | - | |
873 | | - | |
874 | | - | |
| 870 | + | |
875 | 871 | | |
876 | 872 | | |
877 | 873 | | |
| |||
991 | 987 | | |
992 | 988 | | |
993 | 989 | | |
994 | | - | |
995 | 990 | | |
996 | 991 | | |
997 | 992 | | |
| |||
1004 | 999 | | |
1005 | 1000 | | |
1006 | 1001 | | |
1007 | | - | |
1008 | | - | |
1009 | | - | |
1010 | | - | |
1011 | | - | |
1012 | | - | |
1013 | | - | |
1014 | | - | |
1015 | | - | |
1016 | | - | |
1017 | | - | |
1018 | | - | |
1019 | | - | |
1020 | | - | |
1021 | | - | |
1022 | | - | |
1023 | | - | |
1024 | | - | |
1025 | | - | |
1026 | | - | |
1027 | | - | |
1028 | 1002 | | |
1029 | | - | |
1030 | | - | |
1031 | | - | |
| 1003 | + | |
| 1004 | + | |
| 1005 | + | |
| 1006 | + | |
| 1007 | + | |
| 1008 | + | |
| 1009 | + | |
| 1010 | + | |
| 1011 | + | |
| 1012 | + | |
| 1013 | + | |
| 1014 | + | |
| 1015 | + | |
| 1016 | + | |
| 1017 | + | |
| 1018 | + | |
| 1019 | + | |
| 1020 | + | |
| 1021 | + | |
| 1022 | + | |
| 1023 | + | |
| 1024 | + | |
| 1025 | + | |
| 1026 | + | |
| 1027 | + | |
| 1028 | + | |
| 1029 | + | |
| 1030 | + | |
| 1031 | + | |
| 1032 | + | |
| 1033 | + | |
| 1034 | + | |
| 1035 | + | |
| 1036 | + | |
| 1037 | + | |
| 1038 | + | |
| 1039 | + | |
| 1040 | + | |
| 1041 | + | |
| 1042 | + | |
| 1043 | + | |
| 1044 | + | |
| 1045 | + | |
| 1046 | + | |
| 1047 | + | |
| 1048 | + | |
| 1049 | + | |
| 1050 | + | |
| 1051 | + | |
| 1052 | + | |
| 1053 | + | |
| 1054 | + | |
| 1055 | + | |
| 1056 | + | |
| 1057 | + | |
| 1058 | + | |
| 1059 | + | |
| 1060 | + | |
| 1061 | + | |
| 1062 | + | |
| 1063 | + | |
| 1064 | + | |
| 1065 | + | |
| 1066 | + | |
| 1067 | + | |
| 1068 | + | |
| 1069 | + | |
| 1070 | + | |
| 1071 | + | |
| 1072 | + | |
| 1073 | + | |
| 1074 | + | |
| 1075 | + | |
| 1076 | + | |
1032 | 1077 | | |
1033 | | - | |
| 1078 | + | |
| 1079 | + | |
| 1080 | + | |
| 1081 | + | |
| 1082 | + | |
| 1083 | + | |
| 1084 | + | |
| 1085 | + | |
| 1086 | + | |
| 1087 | + | |
| 1088 | + | |
| 1089 | + | |
1034 | 1090 | | |
1035 | | - | |
| 1091 | + | |
1036 | 1092 | | |
1037 | 1093 | | |
1038 | 1094 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
Lines changed: 5 additions & 5 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
38 | 38 | | |
39 | 39 | | |
40 | 40 | | |
41 | | - | |
| 41 | + | |
42 | 42 | | |
43 | 43 | | |
44 | 44 | | |
| |||
110 | 110 | | |
111 | 111 | | |
112 | 112 | | |
113 | | - | |
114 | | - | |
| 113 | + | |
| 114 | + | |
115 | 115 | | |
116 | 116 | | |
117 | 117 | | |
| |||
190 | 190 | | |
191 | 191 | | |
192 | 192 | | |
193 | | - | |
| 193 | + | |
194 | 194 | | |
195 | 195 | | |
196 | 196 | | |
| |||
205 | 205 | | |
206 | 206 | | |
207 | 207 | | |
208 | | - | |
| 208 | + | |
209 | 209 | | |
210 | 210 | | |
211 | 211 | | |
| |||
0 commit comments