how to config actual-data-nodes while my actual-data-nodes tables is dynamic。

[wlklnn]

My shard-table will be created everyday。 It can't be configured by
<sharding:table-rule logic-table="t_mt" table-strategy-ref="self_hint_shard" actual-data-nodes="——"/>
my maven dependency is

org.apache.shardingsphere
shardingsphere-jdbc-core-spring-namespace
5.1.0

[wlklnn]

if the actual-data-nodes was set as not exists table,there is a NPE.

[linghengqian]

• This answer in Apache ShardingSphere 5.1.0, the best answer is obviously to rely on Zookeeper/Etcd, in Apache ShardingSphere Proxy to dynamically change through DistSQL.

• Of course, you can do some things via CI/CD without using a Proxy, because ActualDatanode is actually a Groovy expression that can call static variables and methods of Java classes. If you know Gitlab Runner, I'm sure you know what I'm talking about, there are many applications for this.

• Second, ShardingSphere seems to have an agenda to create tables that don't exist at startup. It is pointless to configure a table that does not exist in the ShardingSphere, which will cause metadata checking to fail.

[wlklnn]

The actualDatanode is dynamic ,for example, today's ActualDatanode is [table_name_20220401..table_name_20220411], and become [table_name_20220402..table_name_20220412] tomorrow,should i use a Scheduling to set ActualDatanode everyday ? but i can't find how to do this.

[linghengqian]

@wlklnn

• The answer maybe is yes. Anyway, it's just changing the date. If you don't use DistSQL, you can call a static method in actualdatanodes.

actual-data-nodes: ds0.$->{com.lingh.LocalShardingDatabasesAndTablesUtil.getActualDataNodes()}

 public class LocalShardingDatabasesAndTablesUtil { 
     public static List<String> getActualDataNodes() { 
         LocalDate startTime = LocalDate.now().minusDays(10); 
         LocalDate endTime = LocalDate.now(); 
         return LongStream.range(0, ChronoUnit.DAYS.between(startTime, endTime)) 
                 .mapToObj(startTime::plusDays) 
                 .map(localDate -> "table_name_" + localDate.format(DateTimeFormatter.ofPattern("yyyyMMdd"))) 
                 .collect(Collectors.toList()); 
     }
}

• The simple choice is to set it directly in ShardingSphere JDBC, and then configure a cron pipeline task in gitlab to re-execute .gitlab-ci.yml every day, which requires a stage in the build and deploy projects (well, it seems very Boring, after all, everyone has been exposed to CI/CD).

• Therefore, a good practice without restarting the project is to use ShardingSphere's DistSQL, and splicing the strings of actualdatanodes does not belong to the scope of how to do.

• The reason why I still use JDK8's API demo is because it's very intuitive. I prefer to use java.time.LocalDate#datesUntil after JDK9 to simplify this long Stream() operation, but that's not what I mainly want to express.

• I just wanted to express that there is absolutely no one-size-fits-all answer, but there are many kinds of answers, no?🕺🤳

[wlklnn]

@linghengqian
Thanks very much.
I just use ShardingSphere's jdbc ,not proxy ,so there is no DistSQL, and the most hard for me is :how to refresh actualdatanodes without restart my application. how to get the [actualdatanodes ] bean object and reset it's value is still unknown to me.
thanks agin for your reply.

[linghengqian]

There is also some experiment in ShardingSphere 4.x to implement SPI in the registry to dynamically modify actualDatanodes, this process becomes more complex in 5.x as the data distribution changes and the ShardingRule is rebuilt, I wonder if anyone's willing to do that. If you dispense with ShardingSphere data sources altogether, you can use its algorithm classes to encapsulate util yourself, bypassing the problems actualDatanodes needs to find.

[wlklnn]

I try to refresh it by ContextManager.alterRuleConfiguration(final String schemaName, final Collection ruleConfigs),It seems to be working properly.
InitActualDataNodesExample.txt

[linghengqian]

@wlklnn

• Your sharing looks very cool! It's the first time I see such an application.

@strongduanmu

• Is it possible to provide a Util class within ShardingSphere JDBC that wraps the method updateShardRuleActualDataNodes() for updating actual-data-nodes mentioned by @wlklnn ? I believe many people are looking for how to update it dynamically, and it is convenient to simplify the process.

package com.lingh.ao;

import org.apache.shardingsphere.driver.jdbc.core.datasource.ShardingSphereDataSource;
import org.apache.shardingsphere.infra.config.RuleConfiguration;
import org.apache.shardingsphere.sharding.algorithm.config.AlgorithmProvidedShardingRuleConfiguration;
import org.apache.shardingsphere.sharding.api.config.rule.ShardingTableRuleConfiguration;
import org.springframework.scheduling.annotation.EnableScheduling;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;

import javax.annotation.Resource;
import java.util.Collection;
import java.util.LinkedList;

@Component
public class InitActualDataNodesAO {
    @Resource
    private ShardingSphereDataSource shardingSphereDataSource;
    private final String logicTableNameForTOrder = "t_order";
    private final String schemaNameForTOrder = "sharding_db";

    public void testSharding() {
        // generate actualDataNodes
        String actualDataNodes = "ds-0.t_order_$->{[1..8]}";

        this.updateShardRuleActualDataNodes(shardingSphereDataSource, schemaNameForTOrder, logicTableNameForTOrder, actualDataNodes);
    }

    private void updateShardRuleActualDataNodes(ShardingSphereDataSource dataSource, String schemaName, String logicTableName, String newActualDataNodes) {
        // Context manager.
        org.apache.shardingsphere.mode.manager.ContextManager contextManager = dataSource.getContextManager();

        // Rule configuration.
        Collection<RuleConfiguration> newRuleConfigList = new LinkedList<>();
        Collection<RuleConfiguration> oldRuleConfigList = dataSource.getContextManager()
                .getMetaDataContexts()
                .getMetaData(schemaName)
                .getRuleMetaData()
                .getConfigurations();

        for (RuleConfiguration oldRuleConfig : oldRuleConfigList) {
            if (oldRuleConfig instanceof AlgorithmProvidedShardingRuleConfiguration) {

                // Algorithm provided sharding rule configuration
                AlgorithmProvidedShardingRuleConfiguration oldAlgorithmConfig = (AlgorithmProvidedShardingRuleConfiguration) oldRuleConfig;
                AlgorithmProvidedShardingRuleConfiguration newAlgorithmConfig = new AlgorithmProvidedShardingRuleConfiguration();

                // Sharding table rule configuration Collection
                Collection<ShardingTableRuleConfiguration> newTableRuleConfigList = new LinkedList<>();
                Collection<ShardingTableRuleConfiguration> oldTableRuleConfigList = oldAlgorithmConfig.getTables();

                oldTableRuleConfigList.forEach(oldTableRuleConfig -> {
                    if (logicTableName.equals(oldTableRuleConfig.getLogicTable())) {
                        ShardingTableRuleConfiguration newTableRuleConfig = new ShardingTableRuleConfiguration(oldTableRuleConfig.getLogicTable(), newActualDataNodes);
                        newTableRuleConfig.setTableShardingStrategy(oldTableRuleConfig.getTableShardingStrategy());
                        newTableRuleConfig.setDatabaseShardingStrategy(oldTableRuleConfig.getDatabaseShardingStrategy());
                        newTableRuleConfig.setKeyGenerateStrategy(oldTableRuleConfig.getKeyGenerateStrategy());

                        newTableRuleConfigList.add(newTableRuleConfig);
                    } else {
                        newTableRuleConfigList.add(oldTableRuleConfig);
                    }
                });

                newAlgorithmConfig.setTables(newTableRuleConfigList);
                newAlgorithmConfig.setAutoTables(oldAlgorithmConfig.getAutoTables());
                newAlgorithmConfig.setBindingTableGroups(oldAlgorithmConfig.getBindingTableGroups());
                newAlgorithmConfig.setBroadcastTables(oldAlgorithmConfig.getBroadcastTables());
                newAlgorithmConfig.setDefaultDatabaseShardingStrategy(oldAlgorithmConfig.getDefaultDatabaseShardingStrategy());
                newAlgorithmConfig.setDefaultTableShardingStrategy(oldAlgorithmConfig.getDefaultTableShardingStrategy());
                newAlgorithmConfig.setDefaultKeyGenerateStrategy(oldAlgorithmConfig.getDefaultKeyGenerateStrategy());
                newAlgorithmConfig.setDefaultShardingColumn(oldAlgorithmConfig.getDefaultShardingColumn());
                newAlgorithmConfig.setShardingAlgorithms(oldAlgorithmConfig.getShardingAlgorithms());
                newAlgorithmConfig.setKeyGenerators(oldAlgorithmConfig.getKeyGenerators());

                newRuleConfigList.add(newAlgorithmConfig);
            }
        }

        // update context
        contextManager.alterRuleConfiguration(schemaName, newRuleConfigList);
    }
}

[strongduanmu]

Is it possible to provide a Util class within ShardingSphere JDBC that wraps the method updateShardRuleActualDataNodes() for updating actual-data-nodes mentioned by @wlklnn ? I believe many people are looking for how to update it dynamically, and it is convenient to simplify the process.

@linghengqian Thank you for your remind, I will take a look at this way.

[xieyongchao2]

@strongduanmu @linghengqian
I found that I used version 5.1.1. The actual nodes are {[20220612,20220613]}. Only 20220612 nodes are configured through sharding JDBC. However, when I query, the node I landed on is 20220613 through the fragmentation algorithm, but there seems to be no error, and the data can be found.

[linghengqian]

@xieyongchao2 Replying to the closed issue is an inexplicable thing for the owner of this issue. I think it would be better if you open a new issue and provide a minimal replica demo, and then point out the association with this issue.

[caol64]

@wlklnn

• Your sharing looks very cool! It's the first time I see such an application.

@strongduanmu

• Is it possible to provide a Util class within ShardingSphere JDBC that wraps the method updateShardRuleActualDataNodes() for updating actual-data-nodes mentioned by @wlklnn ? I believe many people are looking for how to update it dynamically, and it is convenient to simplify the process.

package com.lingh.ao;

import org.apache.shardingsphere.driver.jdbc.core.datasource.ShardingSphereDataSource;
import org.apache.shardingsphere.infra.config.RuleConfiguration;
import org.apache.shardingsphere.sharding.algorithm.config.AlgorithmProvidedShardingRuleConfiguration;
import org.apache.shardingsphere.sharding.api.config.rule.ShardingTableRuleConfiguration;
import org.springframework.scheduling.annotation.EnableScheduling;
import org.springframework.scheduling.annotation.Scheduled;
import org.springframework.stereotype.Component;

import javax.annotation.Resource;
import java.util.Collection;
import java.util.LinkedList;

@Component
public class InitActualDataNodesAO {
    @Resource
    private ShardingSphereDataSource shardingSphereDataSource;
    private final String logicTableNameForTOrder = "t_order";
    private final String schemaNameForTOrder = "sharding_db";

    public void testSharding() {
        // generate actualDataNodes
        String actualDataNodes = "ds-0.t_order_$->{[1..8]}";

        this.updateShardRuleActualDataNodes(shardingSphereDataSource, schemaNameForTOrder, logicTableNameForTOrder, actualDataNodes);
    }

    private void updateShardRuleActualDataNodes(ShardingSphereDataSource dataSource, String schemaName, String logicTableName, String newActualDataNodes) {
        // Context manager.
        org.apache.shardingsphere.mode.manager.ContextManager contextManager = dataSource.getContextManager();

        // Rule configuration.
        Collection<RuleConfiguration> newRuleConfigList = new LinkedList<>();
        Collection<RuleConfiguration> oldRuleConfigList = dataSource.getContextManager()
                .getMetaDataContexts()
                .getMetaData(schemaName)
                .getRuleMetaData()
                .getConfigurations();

        for (RuleConfiguration oldRuleConfig : oldRuleConfigList) {
            if (oldRuleConfig instanceof AlgorithmProvidedShardingRuleConfiguration) {

                // Algorithm provided sharding rule configuration
                AlgorithmProvidedShardingRuleConfiguration oldAlgorithmConfig = (AlgorithmProvidedShardingRuleConfiguration) oldRuleConfig;
                AlgorithmProvidedShardingRuleConfiguration newAlgorithmConfig = new AlgorithmProvidedShardingRuleConfiguration();

                // Sharding table rule configuration Collection
                Collection<ShardingTableRuleConfiguration> newTableRuleConfigList = new LinkedList<>();
                Collection<ShardingTableRuleConfiguration> oldTableRuleConfigList = oldAlgorithmConfig.getTables();

                oldTableRuleConfigList.forEach(oldTableRuleConfig -> {
                    if (logicTableName.equals(oldTableRuleConfig.getLogicTable())) {
                        ShardingTableRuleConfiguration newTableRuleConfig = new ShardingTableRuleConfiguration(oldTableRuleConfig.getLogicTable(), newActualDataNodes);
                        newTableRuleConfig.setTableShardingStrategy(oldTableRuleConfig.getTableShardingStrategy());
                        newTableRuleConfig.setDatabaseShardingStrategy(oldTableRuleConfig.getDatabaseShardingStrategy());
                        newTableRuleConfig.setKeyGenerateStrategy(oldTableRuleConfig.getKeyGenerateStrategy());

                        newTableRuleConfigList.add(newTableRuleConfig);
                    } else {
                        newTableRuleConfigList.add(oldTableRuleConfig);
                    }
                });

                newAlgorithmConfig.setTables(newTableRuleConfigList);
                newAlgorithmConfig.setAutoTables(oldAlgorithmConfig.getAutoTables());
                newAlgorithmConfig.setBindingTableGroups(oldAlgorithmConfig.getBindingTableGroups());
                newAlgorithmConfig.setBroadcastTables(oldAlgorithmConfig.getBroadcastTables());
                newAlgorithmConfig.setDefaultDatabaseShardingStrategy(oldAlgorithmConfig.getDefaultDatabaseShardingStrategy());
                newAlgorithmConfig.setDefaultTableShardingStrategy(oldAlgorithmConfig.getDefaultTableShardingStrategy());
                newAlgorithmConfig.setDefaultKeyGenerateStrategy(oldAlgorithmConfig.getDefaultKeyGenerateStrategy());
                newAlgorithmConfig.setDefaultShardingColumn(oldAlgorithmConfig.getDefaultShardingColumn());
                newAlgorithmConfig.setShardingAlgorithms(oldAlgorithmConfig.getShardingAlgorithms());
                newAlgorithmConfig.setKeyGenerators(oldAlgorithmConfig.getKeyGenerators());

                newRuleConfigList.add(newAlgorithmConfig);
            }
        }

        // update context
        contextManager.alterRuleConfiguration(schemaName, newRuleConfigList);
    }
}

@linghengqian In 5.1.2 version, ShardingSphereDataSource doesn't have getContextManager method. Is this solution needs to update?

[linghengqian]

@caol64

• Hi, if you feel the need, you should open a new issue instead of staying in this one.

• According to Confusion about changes to ShardingSphereDataSource and ContextManager #18494, the metadata module is undergoing a massive adjustment, which means that every version uses this method differently. I don't really recommend doing this, DistSQL is currently my solution.

• If you wish to do this, you'll need to track the git changelog after each version update, like the issue I pointed out. Unless you are a source code reading enthusiast, this is actually very troublesome. Major changes to metadata in the matser branch will facilitate resolution at https://github.com/apache/shardingsphere/discussions/19703.