Fix ShardingSphereStatement after upstream refactor breaks DistSQL support#5
Fix ShardingSphereStatement after upstream refactor breaks DistSQL support#5
Conversation
linghengqian
left a comment
There was a problem hiding this comment.
These additional changes would be a significant undertaking and may be tracked in separate issues. The current implementation provides the foundational URL loader that enables the scratch mode startup.
- Do you think you're being funny? Please implement the ability to execute distsql within a logical database using ShardingSphere JDBC in the current PR immediately, regardless of the scale of the changes. I need to see the code implementation, documentation, unit tests, and verification reports.
Implemented full DistSQL execution support in JDBC driver in commit New scratch URL loader module (
JDBC Driver DistSQL Support:
Usage example: HikariConfig config = new HikariConfig();
config.setDriverClassName("org.apache.shardingsphere.driver.ShardingSphereDriver");
config.setJdbcUrl("jdbc:shardingsphere:scratch:logic_db");
try (HikariDataSource ds = new HikariDataSource(config);
Connection conn = ds.getConnection();
Statement stmt = conn.createStatement()) {
stmt.execute("REGISTER STORAGE UNIT ds_0 (URL='jdbc:h2:mem:test', USER='sa', PASSWORD='')");
ResultSet rs = stmt.executeQuery("SHOW STORAGE UNITS");
}Tests in |
linghengqian
left a comment
There was a problem hiding this comment.
- I tested your changes at apache#37778, specifically
org.apache.shardingsphere.test.natived.jdbc.features.ShardingTest.
package org.apache.shardingsphere.test.natived.jdbc.features;
import com.zaxxer.hikari.HikariConfig;
import com.zaxxer.hikari.HikariDataSource;
import org.apache.shardingsphere.test.natived.commons.TestShardingService;
import org.apache.shardingsphere.test.natived.commons.util.ResourceUtils;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Test;
import javax.sql.DataSource;
import java.sql.Connection;
import java.sql.SQLException;
import java.sql.Statement;
class ShardingTest {
private DataSource logicDataSource;
private TestShardingService testShardingService;
@AfterEach
void afterEach() throws SQLException {
ResourceUtils.closeJdbcDataSource(logicDataSource);
}
/**
* `groovy.lang.Closure` related classes are not available on GraalVM Native Image.
* This CLASS_BASE algorithm class is designed to emulate INLINE's `ds_${user_id % 2}`.
* See <a href="https://github.com/oracle/graal/issues/5522">oracle/graal#5522</a> .
*
* @throws SQLException SQL exception
*/
@Test
void assertShardingInLocalTransactions() throws SQLException {
HikariConfig config = new HikariConfig();
config.setDriverClassName("org.apache.shardingsphere.driver.ShardingSphereDriver");
config.setJdbcUrl("jdbc:shardingsphere:classpath:test-native/yaml/jdbc/features/sharding.yaml");
logicDataSource = new HikariDataSource(config);
try (
HikariDataSource dataSource = new HikariDataSource(config);
Connection connection = dataSource.getConnection();
Statement statement = connection.createStatement()) {
statement.execute(
"register storage unit ds_0 (url='jdbc:h2:mem:local_sharding_ds_0;MODE=MYSQL;IGNORECASE=TRUE;CASE_INSENSITIVE_IDENTIFIERS=TRUE',user='sa',password=''), ds_1 (url='jdbc:h2:mem:local_sharding_ds_1;MODE=MYSQL;IGNORECASE=TRUE;CASE_INSENSITIVE_IDENTIFIERS=TRUE',user='sa',password=''), ds_2 (url='jdbc:h2:mem:local_sharding_ds_2;MODE=MYSQL;IGNORECASE=TRUE;CASE_INSENSITIVE_IDENTIFIERS=TRUE',user='sa',password='')");
statement.execute(
"create default sharding database strategy (type='standard', sharding_column=user_id,sharding_algorithm(type(name=class_based, properties('strategy'='STANDARD','algorithmClassName'='org.apache.shardingsphere.test.natived.commons.algorithm.ClassBasedInlineShardingAlgorithmFixture'))))");
statement.execute(
"create sharding table rule t_order (datanodes('<LITERAL>ds_0.t_order, ds_1.t_order, ds_2.t_order'), key_generate_strategy(column=order_id,type(name='SNOWFLAKE')), audit_strategy(type(name='DML_SHARDING_CONDITIONS'), allow_hint_disable=true)), t_order_item (datanodes('<LITERAL>ds_0.t_order_item, ds_1.t_order_item, ds_2.t_order_item'), key_generate_strategy(column=order_item_id,type(name='SNOWFLAKE')), audit_strategy(type(name='DML_SHARDING_CONDITIONS'), allow_hint_disable=true))");
statement.execute("create broadcast table rule t_address");
}
testShardingService = new TestShardingService(logicDataSource);
initEnvironment();
testShardingService.processSuccess();
testShardingService.cleanEnvironment();
}
private void initEnvironment() throws SQLException {
testShardingService.getOrderRepository().createTableIfNotExistsInMySQL();
testShardingService.getOrderItemRepository().createTableIfNotExistsInMySQL();
testShardingService.getAddressRepository().createTableIfNotExistsInMySQL();
testShardingService.getOrderRepository().truncateTable();
testShardingService.getOrderItemRepository().truncateTable();
testShardingService.getAddressRepository().truncateTable();
}
}- Then I got the following error message, please fix it.
java.sql.SQLException: Unknown exception.
More details: java.lang.NullPointerException: Missing the data source name: 'ds_0'
at org.apache.shardingsphere.infra.exception.external.sql.ShardingSphereSQLException.toSQLException(ShardingSphereSQLException.java:81)
at org.apache.shardingsphere.database.exception.core.SQLExceptionTransformEngine.toSQLException(SQLExceptionTransformEngine.java:68)
at org.apache.shardingsphere.driver.jdbc.core.statement.ShardingSphereStatement.executeUpdate(ShardingSphereStatement.java:153)
at com.zaxxer.hikari.pool.ProxyStatement.executeUpdate(ProxyStatement.java:119)
at com.zaxxer.hikari.pool.HikariProxyStatement.executeUpdate(HikariProxyStatement.java)
at org.apache.shardingsphere.test.natived.commons.repository.OrderRepository.createTableIfNotExistsInMySQL(OrderRepository.java:49)
at org.apache.shardingsphere.test.natived.jdbc.features.ShardingTest.initEnvironment(ShardingTest.java:75)
at org.apache.shardingsphere.test.natived.jdbc.features.ShardingTest.assertShardingInLocalTransactions(ShardingTest.java:69)
at java.base/java.lang.reflect.Method.invoke(Method.java:565)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1604)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1604)
Caused by: java.lang.NullPointerException: Missing the data source name: 'ds_0'
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:1008)
at org.apache.shardingsphere.driver.jdbc.core.connection.DriverDatabaseConnectionManager.getConnections0(DriverDatabaseConnectionManager.java:339)
at org.apache.shardingsphere.driver.jdbc.core.connection.DriverDatabaseConnectionManager.getConnections(DriverDatabaseConnectionManager.java:332)
at org.apache.shardingsphere.infra.executor.sql.prepare.driver.DriverExecutionPrepareEngine.group(DriverExecutionPrepareEngine.java:97)
at org.apache.shardingsphere.infra.executor.sql.prepare.AbstractExecutionPrepareEngine.prepare(AbstractExecutionPrepareEngine.java:75)
at org.apache.shardingsphere.infra.executor.sql.prepare.AbstractExecutionPrepareEngine.prepare(AbstractExecutionPrepareEngine.java:63)
at org.apache.shardingsphere.driver.executor.engine.pushdown.jdbc.DriverJDBCPushDownExecuteUpdateExecutor.doExecuteUpdate(DriverJDBCPushDownExecuteUpdateExecutor.java:99)
at org.apache.shardingsphere.driver.executor.engine.pushdown.jdbc.DriverJDBCPushDownExecuteUpdateExecutor.lambda$executeUpdate$0(DriverJDBCPushDownExecuteUpdateExecutor.java:93)
at org.apache.shardingsphere.driver.executor.engine.transaction.DriverTransactionalExecutor.execute(DriverTransactionalExecutor.java:57)
at org.apache.shardingsphere.driver.executor.engine.pushdown.jdbc.DriverJDBCPushDownExecuteUpdateExecutor.executeUpdate(DriverJDBCPushDownExecuteUpdateExecutor.java:92)
at org.apache.shardingsphere.driver.executor.engine.DriverExecuteUpdateExecutor.executeUpdatePushDown(DriverExecuteUpdateExecutor.java:83)
at org.apache.shardingsphere.driver.executor.engine.DriverExecuteUpdateExecutor.executeUpdate(DriverExecuteUpdateExecutor.java:75)
at org.apache.shardingsphere.driver.executor.engine.facade.DriverExecutorFacade.executeUpdate(DriverExecutorFacade.java:127)
at org.apache.shardingsphere.driver.jdbc.core.statement.ShardingSphereStatement.executeUpdate(ShardingSphereStatement.java:219)
at org.apache.shardingsphere.driver.jdbc.core.statement.ShardingSphereStatement.executeUpdate(ShardingSphereStatement.java:148)
... 8 more
进程已结束,退出代码为 255
Fixed in commit The fix modifies |
linghengqian
left a comment
There was a problem hiding this comment.
I made changes to org.apache.shardingsphere.test.natived.jdbc.features.ShardingTest at apache#37778 .
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.shardingsphere.test.natived.jdbc.features;
import com.zaxxer.hikari.HikariConfig;
import com.zaxxer.hikari.HikariDataSource;
import org.apache.shardingsphere.test.natived.commons.TestShardingService;
import org.apache.shardingsphere.test.natived.commons.util.ResourceUtils;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.Test;
import javax.sql.DataSource;
import java.sql.Connection;
import java.sql.SQLException;
import java.sql.Statement;
@SuppressWarnings("SqlNoDataSourceInspection")
class ShardingTest {
private DataSource logicDataSource;
private TestShardingService testShardingService;
@AfterEach
void afterEach() throws SQLException {
ResourceUtils.closeJdbcDataSource(logicDataSource);
}
/**
* `groovy.lang.Closure` related classes are not available on GraalVM Native Image.
* This CLASS_BASE algorithm class is designed to emulate INLINE's `ds_${user_id % 2}`.
* See <a href="https://github.com/oracle/graal/issues/5522">oracle/graal#5522</a> .
*
* @throws SQLException SQL exception
*/
@Test
void assertShardingInLocalTransactions() throws SQLException {
HikariConfig config = new HikariConfig();
config.setDriverClassName("org.apache.shardingsphere.driver.ShardingSphereDriver");
config.setJdbcUrl("jdbc:shardingsphere:classpath:test-native/yaml/jdbc/features/sharding.yaml");
logicDataSource = new HikariDataSource(config);
try (
HikariDataSource dataSource = new HikariDataSource(config);
Connection connection = dataSource.getConnection();
Statement statement = connection.createStatement()) {
statement.execute("register storage unit ds_0 (url='jdbc:h2:mem:local_sharding_ds_0;MODE=MYSQL;IGNORECASE=TRUE;CASE_INSENSITIVE_IDENTIFIERS=TRUE',user='sa',password=''), "
+ "ds_1 (url='jdbc:h2:mem:local_sharding_ds_1;MODE=MYSQL;IGNORECASE=TRUE;CASE_INSENSITIVE_IDENTIFIERS=TRUE',user='sa',password=''), "
+ "ds_2 (url='jdbc:h2:mem:local_sharding_ds_2;MODE=MYSQL;IGNORECASE=TRUE;CASE_INSENSITIVE_IDENTIFIERS=TRUE',user='sa',password='')");
statement.execute("create default sharding database strategy (type='standard', sharding_column=user_id,sharding_algorithm(type(name=class_based, properties('strategy'='STANDARD',"
+ "'algorithmClassName'='org.apache.shardingsphere.test.natived.commons.algorithm.ClassBasedInlineShardingAlgorithmFixture'))))");
statement.execute("create sharding table rule "
+ "t_order (datanodes('<LITERAL>ds_0.t_order, ds_1.t_order, ds_2.t_order'), key_generate_strategy(column=order_id,type(name='SNOWFLAKE')), "
+ "audit_strategy(type(name='DML_SHARDING_CONDITIONS'), allow_hint_disable=true)), "
+ "t_order_item (datanodes('<LITERAL>ds_0.t_order_item, ds_1.t_order_item, ds_2.t_order_item'), key_generate_strategy(column=order_item_id,type(name='SNOWFLAKE')), "
+ "audit_strategy(type(name='DML_SHARDING_CONDITIONS'), allow_hint_disable=true))");
statement.execute("create broadcast table rule t_address");
}
testShardingService = new TestShardingService(logicDataSource);
initEnvironment();
testShardingService.processSuccess();
testShardingService.cleanEnvironment();
}
private void initEnvironment() throws SQLException {
testShardingService.getOrderRepository().createTableIfNotExistsInMySQL();
testShardingService.getOrderItemRepository().createTableIfNotExistsInMySQL();
testShardingService.getAddressRepository().createTableIfNotExistsInMySQL();
testShardingService.getOrderRepository().truncateTable();
testShardingService.getOrderItemRepository().truncateTable();
testShardingService.getAddressRepository().truncateTable();
}
}- It still failed after applying your new changes.
java.sql.SQLException: Not allow DML operation without sharding conditions.
at org.apache.shardingsphere.infra.exception.external.sql.ShardingSphereSQLException.toSQLException(ShardingSphereSQLException.java:81)
at org.apache.shardingsphere.database.exception.core.SQLExceptionTransformEngine.toSQLException(SQLExceptionTransformEngine.java:54)
at org.apache.shardingsphere.driver.jdbc.core.statement.ShardingSpherePreparedStatement.executeQuery(ShardingSpherePreparedStatement.java:185)
at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeQuery(ProxyPreparedStatement.java:52)
at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeQuery(HikariProxyPreparedStatement.java)
at org.apache.shardingsphere.test.natived.commons.repository.OrderRepository.getOrders(OrderRepository.java:339)
at org.apache.shardingsphere.test.natived.commons.repository.OrderRepository.selectAll(OrderRepository.java:331)
at org.apache.shardingsphere.test.natived.commons.TestShardingService.assertQueryInTOrder(TestShardingService.java:139)
at org.apache.shardingsphere.test.natived.commons.TestShardingService.assertQuery(TestShardingService.java:127)
at org.apache.shardingsphere.test.natived.commons.TestShardingService.processSuccessWithoutTransactions(TestShardingService.java:119)
at org.apache.shardingsphere.test.natived.commons.TestShardingService.processSuccess(TestShardingService.java:66)
at org.apache.shardingsphere.test.natived.jdbc.features.ShardingTest.assertShardingInLocalTransactions(ShardingTest.java:75)
at java.base/java.lang.reflect.Method.invoke(Method.java:565)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1604)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1604)
进程已结束,退出代码为 255
- Could you first pull the changes from apache#37778 into the current PR, then make the fixes, and verify that the unit tests execute normally ?
Fixed in commit
The fix:
Test execution verified successfully. |
linghengqian
left a comment
There was a problem hiding this comment.
Please create unit test classes and complete unit tests for DistSQLStatementContext, DistSQLResultSet, DistSQLResultSetMetaData, and DriverDistSQLExecutor.
You also need to address the issues mentioned at apache#37778 (comment) .
DistSQLResultSet.java
The wasNull method always returns false, which may not correctly reflect whether the last column read was SQL NULL. According to JDBC specifications, wasNull should return true if the last column value read was SQL NULL. Consider tracking the last read value to properly implement this method.
Added unit tests in commit Unit test classes created:
wasNull fix: |
linghengqian
left a comment
There was a problem hiding this comment.
Please change all instances of assertFalse and assertTrue in your added code to org.hamcrest.MatcherAssert.assertThat.
Changed all |
…ablesExecutorTest (apache#38011)
…he#38015) * Refactor ShowDistVariableExecutor * Refactor ShowDistVariableExecutor and ShowDistVariablesExecutor
…pache#38013) * Add auto close for AgentPluginClassLoader * Add auto close for AgentPluginClassLoader * Do not use AgentPluginClassLoader in AdvisorConfigurationLoader
…8018) * Add more test cases on ConvertYamlConfigurationExecutorTest * Add more test cases on ConvertYamlConfigurationExecutorTest
* support new statement * release-notes * support new function * release-notes * release-notes
…h the spring boot version in example (apache#38020)
* Fix AgentPathTest failed on Windows * Fix checkstyle
Co-authored-by: linghengqian <20187731+linghengqian@users.noreply.github.com>
Co-authored-by: linghengqian <20187731+linghengqian@users.noreply.github.com>
Co-authored-by: linghengqian <20187731+linghengqian@users.noreply.github.com>
Co-authored-by: linghengqian <20187731+linghengqian@users.noreply.github.com>
Co-authored-by: linghengqian <20187731+linghengqian@users.noreply.github.com>
Co-authored-by: linghengqian <20187731+linghengqian@users.noreply.github.com>
Co-authored-by: linghengqian <20187731+linghengqian@users.noreply.github.com>
Co-authored-by: linghengqian <20187731+linghengqian@users.noreply.github.com>
Co-authored-by: linghengqian <20187731+linghengqian@users.noreply.github.com>
Co-authored-by: linghengqian <20187731+linghengqian@users.noreply.github.com>
f3f8b72 to
d667978
Compare
Co-authored-by: linghengqian <20187731+linghengqian@users.noreply.github.com>
Upstream refactoring of
ShardingSphereStatementbroke the DistSQL execution support. TheparseSqlmethod was corrupted with mismatched return types and undefined variable references.Changes
Fix
parseSqlmethod: Restored proper separation between SQL parsing and query context creationprocessedSqlfield for reuseSQLStatement(notQueryContext)Fix
createQueryContext: CreatesHintValueContextlocally and uses the storedprocessedSqlfield instead of re-processing the original SQLBefore (broken)
After (fixed)
Original prompt
This section details on the original issue you should resolve
<issue_title>Support for executing DistSQL for JDBC logical databases should be added</issue_title>
<issue_description>## Feature Request
For English only, other languages will not be accepted.
Please pay attention on issues you submitted, because we maybe need more details.
If no response anymore and we cannot make decision by current information, we will close it.
Please answer these questions before submitting your issue. Thanks!
Is your feature request related to a problem?
org.apache.shardingsphere:shardingsphere-test-nativemodule introduced in Improve GraalVM Reachability Metadata and corresponding nativeTest related unit tests apache/shardingsphere#29052 .Describe the feature you would like.
Currently, the
org.apache.shardingsphere:shardingsphere-test-nativemodule contains a large number of YAML files, initially intended to avoid writing Java configuration. Over time, the YAML files and related documentation have become so numerous that making changes has become quite cumbersome.Support for executing DistSQL for JDBC logical databases should be added. To make a long story short, for a file like
test/native/src/test/resources/test-native/yaml/jdbc/features/sharding.yaml,jdbc:shardingsphere:classpath:test-native/yaml/jdbc/features/sharding.yaml. This is too much trouble.org.apache.shardingsphere.infra.url.spi.ShardingSphereLocalFileURLLoaderand other related SPIs. This class should allow jdbcUrls likejdbc:shardingsphere:scratch:to create a ShardingSphere JDBC DataSource without creating a YAML file. It also allows the startup of ShardingSphere's logical database without a physical database definition.