[Bug?] 1.2.2 Docker Compose standalone fails to start #11712

slawr · 2023-12-13T14:29:01Z

slawr
Dec 13, 2023

Hi, Posted this on Slack but it seems not so much activity there. So I will repost here.

Summary

Trying to update from earlier IoTDB version.

I can successfully run 1.2.2 standalone docker image from Docker hub, then sanity test the container by connecting to the IoTDB client and interacting with the server.

However if I switch to the upstream docker compose the standalone compose yaml builds but the system fails to start.

As creating a container of the upstream image via docker run succeeds, but Compose using the same image fails my first reaction is that it is an environment issue, but it is hard for me to determine if there is just a setup/documentation gap or an error in the compose yaml. Deployment using compose is a primary target for my use case so this is currently a blocker.

Detail

Two presumably related errors stand out in the logs:

023-12-13 13:39:35,874 [main] ERROR o.a.i.db.service.DataNode:264 - Cannot pull system configurations from ConfigNode-leader after 10 retries
2023-12-13 13:38:47,631 [main] ERROR o.a.i.c.c.s.SyncConfigNodeClientPool:106 - REGISTER_CONFIG_NODE failed on ConfigNode TEndPoint(ip:127.0.0.1, port:10710)
org.apache.iotdb.commons.client.exception.ClientManagerException: net.sf.cglib.core.CodeGenerationException: org.apache.thrift.transport.TTransportException-->java.net.ConnectException: Connection refused (Connection refused)

A search of PR and issues before making this post I can see that others seem to have had similar issues with recent versions, but they do not include a solution:

Debug info below.

iotdb/docker$ sudo docker compose -f src/main/DockerCompose/docker-compose-standalone.yml up -d
[+] Running 1/1
 ✔ Container iotdb-service  Started

docker-compose-standalone.yml:

version: "3"
services:
  iotdb-service:
    image: apache/iotdb:latest
    hostname: iotdb-service
    container_name: iotdb-service
    ports:
      - "6667:6667"
    environment:
      - cn_internal_address=iotdb-service
      - cn_internal_port=10710
      - cn_consensus_port=10720
      - cn_seed_config_node=iotdb-service:10710
      - dn_rpc_address=iotdb-service
      - dn_internal_address=iotdb-service
      - dn_rpc_port=6667
      - dn_mpp_data_exchange_port=10740
      - dn_schema_region_consensus_port=10750
      - dn_data_region_consensus_port=10760
      - dn_seed_config_node=iotdb-service:10710
    volumes:
        - ./data:/iotdb/data
        - ./logs:/iotdb/logs
    networks:
      iotdb:
        ipv4_address: 172.18.0.6

networks:
  iotdb:
    external: true

$ sudo docker compose -f src/main/DockerCompose/docker-compose-standalone.yml logs
iotdb-service  | update dn_internal_address /iotdb/conf/iotdb-datanode.properties
iotdb-service  | update dn_data_region_consensus_port /iotdb/conf/iotdb-datanode.properties
iotdb-service  | update cn_internal_port /iotdb/conf/iotdb-confignode.properties
iotdb-service  | update dn_rpc_address /iotdb/conf/iotdb-datanode.properties
iotdb-service  | update dn_mpp_data_exchange_port /iotdb/conf/iotdb-datanode.properties
iotdb-service  | update dn_rpc_port /iotdb/conf/iotdb-datanode.properties
iotdb-service  | update cn_consensus_port /iotdb/conf/iotdb-confignode.properties
iotdb-service  | update cn_internal_address /iotdb/conf/iotdb-confignode.properties
iotdb-service  | update dn_schema_region_consensus_port /iotdb/conf/iotdb-datanode.properties
iotdb-service  | Checking whether the ports are already occupied...
iotdb-service  | ---------------------
iotdb-service  | Starting IoTDB DataNode
iotdb-service  | ---------------------
iotdb-service  | WARN:
iotdb-service  | WARN: the value of net.core.somaxconn (=4096) is too small, please set it to a larger value using the following command.
iotdb-service  | WARN:     sudo sysctl -w net.core.somaxconn=65535
iotdb-service  | WARN: The original net.core.somaxconn value will be set back when the os reboots.
iotdb-service  | WARN:
iotdb-service  | setting local JMX...
iotdb-service  | Maximum memory allocation pool = 1531MB, initial memory allocation pool = 200MB
iotdb-service  | If you want to change this configuration, please check conf/datanode-env.sh.
iotdb-service  | 2023-12-13 13:38:45,583 [main] INFO  o.a.i.d.c.IoTDBDescriptor:161 - Start to read config file file:/iotdb/conf/iotdb-common.properties 
iotdb-service  | 2023-12-13 13:38:45,587 [main] INFO  o.a.i.d.c.IoTDBDescriptor:178 - Start to read config file file:/iotdb/conf/iotdb-datanode.properties 
iotdb-service  | 2023-12-13 13:38:45,590 [main] INFO  o.a.i.d.c.IoTDBDescriptor:1610 - initial allocateMemoryForRead = 481925529 
iotdb-service  | 2023-12-13 13:38:45,591 [main] INFO  o.a.i.d.c.IoTDBDescriptor:1611 - initial allocateMemoryForWrite = 481925529 
iotdb-service  | 2023-12-13 13:38:45,591 [main] INFO  o.a.i.d.c.IoTDBDescriptor:1612 - initial allocateMemoryForSchema = 160641843 
iotdb-service  | 2023-12-13 13:38:45,591 [main] INFO  o.a.i.d.c.IoTDBDescriptor:1613 - initial allocateMemoryForConsensus = 160641843 
iotdb-service  | 2023-12-13 13:38:45,591 [main] INFO  o.a.i.d.c.IoTDBDescriptor:1778 - allocateMemoryForSchemaRegion = 80320921 
iotdb-service  | 2023-12-13 13:38:45,591 [main] INFO  o.a.i.d.c.IoTDBDescriptor:1782 - allocateMemoryForSchemaCache = 64256737 
iotdb-service  | 2023-12-13 13:38:45,591 [main] INFO  o.a.i.d.c.IoTDBDescriptor:1786 - allocateMemoryForPartitionCache = 16064184 
iotdb-service  | 2023-12-13 13:38:45,602 [main] INFO  o.a.i.t.c.c.TSFileDescriptor:126 - try loading iotdb-common.properties from /iotdb/conf/iotdb-common.properties 
iotdb-service  | 2023-12-13 13:38:45,609 [main] INFO  o.a.i.d.c.IoTDBDescriptor:403 - IoTDB enable memory control: true 
iotdb-service  | 2023-12-13 13:38:45,610 [main] INFO  o.a.i.d.c.IoTDBDescriptor:1347 - MQTT host is not configured, will use dn_rpc_address. 
iotdb-service  | 2023-12-13 13:38:45,645 [main] INFO  o.a.i.db.service.DataNode:150 - IoTDB-DataNode environment variables: 
iotdb-service  | 	IOTDB_HOME=/iotdb;
iotdb-service  | 	IOTDB_CONF=/iotdb/conf;
iotdb-service  | 	IOTDB_DATA_HOME=/iotdb; 
iotdb-service  | 2023-12-13 13:38:45,646 [main] INFO  o.a.i.db.service.DataNode:151 - IoTDB-DataNode default charset is: UTF-8 
iotdb-service  | 2023-12-13 13:38:45,672 [main] INFO  o.a.i.c.c.IoTDBThreadPoolFactory:203 - new single scheduled thread pool: Stateful-Trigger-Information-Updater 
iotdb-service  | 2023-12-13 13:38:45,682 [main] INFO  o.a.i.d.s.DataNodeServerCommandLine:77 - Running mode -s 
iotdb-service  | 2023-12-13 13:38:45,687 [main] INFO  o.a.i.d.c.IoTDBStartCheck:129 - Starting IoTDB 1.2.2 (Build: 5d0bfb0) 
iotdb-service  | 2023-12-13 13:38:45,688 [main] INFO  o.a.i.d.c.IoTDBStartCheck:138 -  /iotdb/data/datanode/system/schema dir has been created. 
iotdb-service  | 2023-12-13 13:38:45,689 [main] INFO  o.a.i.d.c.IoTDBStartCheck:165 -  /iotdb/data/datanode/system/system.properties has been created. 
iotdb-service  | 2023-12-13 13:38:45,694 [main] INFO  o.a.i.c.s.StartupChecks:59 - Start JMX locally. 
iotdb-service  | 2023-12-13 13:38:45,695 [main] INFO  o.a.i.c.s.StartupChecks:45 - JDK version is 11. 
iotdb-service  | 2023-12-13 13:38:45,695 [main] INFO  o.a.i.d.c.DataNodeStartupCheck:56 - DataNode port check successful. 
iotdb-service  | 2023-12-13 13:38:45,698 [main] INFO  o.a.i.d.p.c.ConfigNodeInfo:90 - Successfully update ConfigNode: [TEndPoint(ip:127.0.0.1, port:10710)]. 
iotdb-service  | 2023-12-13 13:38:45,698 [main] INFO  o.a.i.db.service.DataNode:234 - Pulling system configurations from the ConfigNode-leader... 
iotdb-service  | 2023-12-13 13:38:45,842 [main] WARN  o.a.i.d.p.c.ConfigNodeClient:264 - The current node may have been down TEndPoint(ip:127.0.0.1, port:10710),try next node 
iotdb-service  | 2023-12-13 13:38:45,843 [main] WARN  o.a.i.d.p.c.ConfigNodeClient:264 - The current node may have been down TEndPoint(ip:127.0.0.1, port:10710),try next node 
iotdb-service  | 2023-12-13 13:38:45,844 [main] WARN  o.a.i.db.service.DataNode:246 - Cannot pull system configurations from ConfigNode-leader, because: net.sf.cglib.core.CodeGenerationException: org.apache.thrift.TException-->Error in calling method init, because: Fail to connect to any config node. Please check status of ConfigNodes or logs of connected DataNode 
iotdb-service  | 2023-12-13 13:38:50,845 [main] WARN  o.a.i.d.p.c.ConfigNodeClient:264 - The current node may have been down TEndPoint(ip:127.0.0.1, port:10710),try next node 
iotdb-service  | 2023-12-13 13:38:50,845 [main] WARN  o.a.i.d.p.c.ConfigNodeClient:264 - The current node may have been down TEndPoint(ip:127.0.0.1, port:10710),try next node 
iotdb-service  | 2023-12-13 13:38:50,846 [main] WARN  o.a.i.db.service.DataNode:246 - Cannot pull system configurations from ConfigNode-leader, because: net.sf.cglib.core.CodeGenerationException: org.apache.thrift.TException-->Error in calling method init, because: Fail to connect to any config node. Please check status of ConfigNodes or logs of connected DataNode 
iotdb-service  | 2023-12-13 13:38:55,847 [main] WARN  o.a.i.d.p.c.ConfigNodeClient:264 - The current node may have been down TEndPoint(ip:127.0.0.1, port:10710),try next node 
iotdb-service  | 2023-12-13 13:38:55,848 [main] WARN  o.a.i.d.p.c.ConfigNodeClient:264 - The current node may have been down TEndPoint(ip:127.0.0.1, port:10710),try next node 
iotdb-service  | 2023-12-13 13:38:55,848 [main] WARN  o.a.i.db.service.DataNode:246 - Cannot pull system configurations from ConfigNode-leader, because: net.sf.cglib.core.CodeGenerationException: org.apache.thrift.TException-->Error in calling method init, because: Fail to connect to any config node. Please check status of ConfigNodes or logs of connected DataNode 
iotdb-service  | 2023-12-13 13:39:00,849 [main] WARN  o.a.i.d.p.c.ConfigNodeClient:264 - The current node may have been down TEndPoint(ip:127.0.0.1, port:10710),try next node 
iotdb-service  | 2023-12-13 13:39:00,850 [main] WARN  o.a.i.d.p.c.ConfigNodeClient:264 - The current node may have been down TEndPoint(ip:127.0.0.1, port:10710),try next node 
iotdb-service  | 2023-12-13 13:39:00,850 [main] WARN  o.a.i.db.service.DataNode:246 - Cannot pull system configurations from ConfigNode-leader, because: net.sf.cglib.core.CodeGenerationException: org.apache.thrift.TException-->Error in calling method init, because: Fail to connect to any config node. Please check status of ConfigNodes or logs of connected DataNode 
iotdb-service  | 2023-12-13 13:39:05,853 [main] WARN  o.a.i.d.p.c.ConfigNodeClient:264 - The current node may have been down TEndPoint(ip:127.0.0.1, port:10710),try next node 
iotdb-service  | 2023-12-13 13:39:05,855 [main] WARN  o.a.i.d.p.c.ConfigNodeClient:264 - The current node may have been down TEndPoint(ip:127.0.0.1, port:10710),try next node 
iotdb-service  | 2023-12-13 13:39:05,856 [main] WARN  o.a.i.db.service.DataNode:246 - Cannot pull system configurations from ConfigNode-leader, because: net.sf.cglib.core.CodeGenerationException: org.apache.thrift.TException-->Error in calling method init, because: Fail to connect to any config node. Please check status of ConfigNodes or logs of connected DataNode 
iotdb-service  | 2023-12-13 13:39:10,858 [main] WARN  o.a.i.d.p.c.ConfigNodeClient:264 - The current node may have been down TEndPoint(ip:127.0.0.1, port:10710),try next node 
iotdb-service  | 2023-12-13 13:39:10,858 [main] WARN  o.a.i.d.p.c.ConfigNodeClient:264 - The current node may have been down TEndPoint(ip:127.0.0.1, port:10710),try next node 
iotdb-service  | 2023-12-13 13:39:10,859 [main] WARN  o.a.i.db.service.DataNode:246 - Cannot pull system configurations from ConfigNode-leader, because: net.sf.cglib.core.CodeGenerationException: org.apache.thrift.TException-->Error in calling method init, because: Fail to connect to any config node. Please check status of ConfigNodes or logs of connected DataNode 
iotdb-service  | 2023-12-13 13:39:15,861 [main] WARN  o.a.i.d.p.c.ConfigNodeClient:264 - The current node may have been down TEndPoint(ip:127.0.0.1, port:10710),try next node 
iotdb-service  | 2023-12-13 13:39:15,862 [main] WARN  o.a.i.d.p.c.ConfigNodeClient:264 - The current node may have been down TEndPoint(ip:127.0.0.1, port:10710),try next node 
iotdb-service  | 2023-12-13 13:39:15,863 [main] WARN  o.a.i.db.service.DataNode:246 - Cannot pull system configurations from ConfigNode-leader, because: net.sf.cglib.core.CodeGenerationException: org.apache.thrift.TException-->Error in calling method init, because: Fail to connect to any config node. Please check status of ConfigNodes or logs of connected DataNode 
iotdb-service  | 2023-12-13 13:39:20,866 [main] WARN  o.a.i.d.p.c.ConfigNodeClient:264 - The current node may have been down TEndPoint(ip:127.0.0.1, port:10710),try next node 
iotdb-service  | 2023-12-13 13:39:20,867 [main] WARN  o.a.i.d.p.c.ConfigNodeClient:264 - The current node may have been down TEndPoint(ip:127.0.0.1, port:10710),try next node 
iotdb-service  | 2023-12-13 13:39:20,868 [main] WARN  o.a.i.db.service.DataNode:246 - Cannot pull system configurations from ConfigNode-leader, because: net.sf.cglib.core.CodeGenerationException: org.apache.thrift.TException-->Error in calling method init, because: Fail to connect to any config node. Please check status of ConfigNodes or logs of connected DataNode 
iotdb-service  | 2023-12-13 13:39:25,869 [main] WARN  o.a.i.d.p.c.ConfigNodeClient:264 - The current node may have been down TEndPoint(ip:127.0.0.1, port:10710),try next node 
iotdb-service  | 2023-12-13 13:39:25,870 [main] WARN  o.a.i.d.p.c.ConfigNodeClient:264 - The current node may have been down TEndPoint(ip:127.0.0.1, port:10710),try next node 
iotdb-service  | 2023-12-13 13:39:25,870 [main] WARN  o.a.i.db.service.DataNode:246 - Cannot pull system configurations from ConfigNode-leader, because: net.sf.cglib.core.CodeGenerationException: org.apache.thrift.TException-->Error in calling method init, because: Fail to connect to any config node. Please check status of ConfigNodes or logs of connected DataNode 
iotdb-service  | 2023-12-13 13:39:30,872 [main] WARN  o.a.i.d.p.c.ConfigNodeClient:264 - The current node may have been down TEndPoint(ip:127.0.0.1, port:10710),try next node 
iotdb-service  | 2023-12-13 13:39:30,873 [main] WARN  o.a.i.d.p.c.ConfigNodeClient:264 - The current node may have been down TEndPoint(ip:127.0.0.1, port:10710),try next node 
iotdb-service  | 2023-12-13 13:39:30,873 [main] WARN  o.a.i.db.service.DataNode:246 - Cannot pull system configurations from ConfigNode-leader, because: net.sf.cglib.core.CodeGenerationException: org.apache.thrift.TException-->Error in calling method init, because: Fail to connect to any config node. Please check status of ConfigNodes or logs of connected DataNode 
iotdb-service  | 2023-12-13 13:39:35,874 [main] ERROR o.a.i.db.service.DataNode:264 - Cannot pull system configurations from ConfigNode-leader after 10 retries 
iotdb-service  | 2023-12-13 13:39:35,879 [main] ERROR o.a.i.db.service.DataNode:193 - Fail to start server 
iotdb-service  | org.apache.iotdb.commons.exception.StartupException: Cannot pull system configurations from ConfigNode-leader
iotdb-service  | 	at org.apache.iotdb.db.service.DataNode.pullAndCheckSystemConfigurations(DataNode.java:267)
iotdb-service  | 	at org.apache.iotdb.db.service.DataNode.doAddNode(DataNode.java:165)
iotdb-service  | 	at org.apache.iotdb.db.service.DataNodeServerCommandLine.run(DataNodeServerCommandLine.java:81)
iotdb-service  | 	at org.apache.iotdb.commons.ServerCommandLine.doMain(ServerCommandLine.java:58)
iotdb-service  | 	at org.apache.iotdb.db.service.DataNode.main(DataNode.java:152)
iotdb-service  | 2023-12-13 13:39:35,880 [main] INFO  o.a.i.db.service.DataNode:893 - Deactivating IoTDB DataNode... 
iotdb-service  | 2023-12-13 13:39:35,881 [main] INFO  o.a.i.c.s.RegisterManager:67 - deregister all service. 
iotdb-service  | 2023-12-13 13:39:35,881 [main] INFO  o.a.i.db.service.DataNode:897 - IoTDB DataNode is deactivated. 
iotdb-service  | 2023-12-13 13:39:35,888 [main] INFO  o.a.i.c.s.m.MetricService:77 - MetricService try to stop. 
iotdb-service  | 2023-12-13 13:39:35,888 [main] INFO  o.a.i.c.s.m.MetricService:82 - MetricService stop successfully. 
iotdb-service  | 2023-12-13 13:39:35,985 [main] INFO  o.a.i.c.c.IoTDBThreadPoolFactory:203 - new single scheduled thread pool: RatisBgDiskGuardian 
iotdb-service  | 2023-12-13 13:39:36,108 [main] INFO  o.a.ratis.conf.ConfUtils:46 - raft.rpc.type = GRPC (default) 
iotdb-service  | 2023-12-13 13:39:36,113 [main] INFO  o.a.ratis.conf.ConfUtils:53 - raft.grpc.admin.host = null (fallback to raft.grpc.server.host) 
iotdb-service  | 2023-12-13 13:39:36,115 [main] INFO  o.a.ratis.conf.ConfUtils:53 - raft.grpc.admin.port = 10750 (fallback to raft.grpc.server.port) 
iotdb-service  | 2023-12-13 13:39:36,116 [main] INFO  o.a.ratis.conf.ConfUtils:53 - raft.grpc.client.host = null (fallback to raft.grpc.server.host) 
iotdb-service  | 2023-12-13 13:39:36,116 [main] INFO  o.a.ratis.conf.ConfUtils:53 - raft.grpc.client.port = 10750 (fallback to raft.grpc.server.port) 
iotdb-service  | 2023-12-13 13:39:36,117 [main] INFO  o.a.ratis.conf.ConfUtils:46 - raft.grpc.server.host = null (default) 
iotdb-service  | 2023-12-13 13:39:36,118 [main] INFO  o.a.ratis.conf.ConfUtils:46 - raft.grpc.server.port = 10750 (custom) 
iotdb-service  | 2023-12-13 13:39:36,118 [main] INFO  o.a.ratis.conf.ConfUtils:46 - raft.grpc.message.size.max = 512MB (=536870912) (custom) 
iotdb-service  | 2023-12-13 13:39:36,119 [main] INFO  o.a.ratis.conf.ConfUtils:46 - raft.server.log.appender.buffer.byte-limit = 16777216 (custom) 
iotdb-service  | 2023-12-13 13:39:36,122 [main] INFO  o.a.ratis.conf.ConfUtils:46 - raft.grpc.flow.control.window = 4194304 (custom) 
iotdb-service  | 2023-12-13 13:39:36,122 [main] INFO  o.a.ratis.conf.ConfUtils:46 - raft.server.rpc.request.timeout = 3000ms (default) 
iotdb-service  | 2023-12-13 13:39:36,136 [main] INFO  o.a.ratis.conf.ConfUtils:46 - raft.grpc.server.heartbeat.channel = true (default) 
iotdb-service  | 2023-12-13 13:39:36,139 [main] INFO  o.a.ratis.conf.ConfUtils:46 - raft.grpc.server.async.request.thread.pool.cached = true (default) 
iotdb-service  | 2023-12-13 13:39:36,140 [main] INFO  o.a.ratis.conf.ConfUtils:46 - raft.grpc.server.async.request.thread.pool.size = 32 (default) 
iotdb-service  | 2023-12-13 13:39:36,387 [main] INFO  o.a.ratis.conf.ConfUtils:46 - raft.datastream.type = DISABLED (default) 
iotdb-service  | 2023-12-13 13:39:36,390 [main] INFO  o.a.ratis.conf.ConfUtils:46 - raft.server.threadpool.proxy.cached = true (default) 
iotdb-service  | 2023-12-13 13:39:36,391 [main] INFO  o.a.ratis.conf.ConfUtils:46 - raft.server.threadpool.proxy.size = 0 (default) 
iotdb-service  | 2023-12-13 13:39:36,392 [main] INFO  o.a.ratis.conf.ConfUtils:46 - raft.server.sleep.deviation.threshold = 4000ms (custom) 
iotdb-service  | 2023-12-13 13:39:36,392 [main] INFO  o.a.ratis.conf.ConfUtils:46 - raft.server.rpc.slowness.timeout = 10min (custom) 
iotdb-service  | 2023-12-13 13:39:36,392 [main] INFO  o.a.ratis.conf.ConfUtils:46 - raft.server.leaderelection.leader.step-down.wait-time = 30s (custom) 
iotdb-service  | 2023-12-13 13:39:36,396 [main] INFO  o.a.ratis.conf.ConfUtils:46 - raft.server.storage.dir = [/iotdb/data/datanode/consensus/schema_region] (custom) 
iotdb-service  | 2023-12-13 13:39:36,407 [main] INFO  o.a.i.c.c.IoTDBThreadPoolFactory:203 - new single scheduled thread pool: RatisBgDiskGuardian 
iotdb-service  | 2023-12-13 13:39:36,407 [main] INFO  o.a.ratis.conf.ConfUtils:46 - raft.rpc.type = GRPC (default) 
iotdb-service  | 2023-12-13 13:39:36,408 [main] INFO  o.a.ratis.conf.ConfUtils:53 - raft.grpc.admin.host = null (fallback to raft.grpc.server.host) 
iotdb-service  | 2023-12-13 13:39:36,408 [main] INFO  o.a.ratis.conf.ConfUtils:53 - raft.grpc.admin.port = 10760 (fallback to raft.grpc.server.port) 
iotdb-service  | 2023-12-13 13:39:36,408 [main] INFO  o.a.ratis.conf.ConfUtils:53 - raft.grpc.client.host = null (fallback to raft.grpc.server.host) 
iotdb-service  | 2023-12-13 13:39:36,408 [main] INFO  o.a.ratis.conf.ConfUtils:53 - raft.grpc.client.port = 10760 (fallback to raft.grpc.server.port) 
iotdb-service  | 2023-12-13 13:39:36,408 [main] INFO  o.a.ratis.conf.ConfUtils:46 - raft.grpc.server.host = null (default) 
iotdb-service  | 2023-12-13 13:39:36,408 [main] INFO  o.a.ratis.conf.ConfUtils:46 - raft.grpc.server.port = 10760 (custom) 
iotdb-service  | 2023-12-13 13:39:36,408 [main] INFO  o.a.ratis.conf.ConfUtils:46 - raft.grpc.message.size.max = 512MB (=536870912) (custom) 
iotdb-service  | 2023-12-13 13:39:36,409 [main] INFO  o.a.ratis.conf.ConfUtils:46 - raft.server.log.appender.buffer.byte-limit = 16777216 (custom) 
iotdb-service  | 2023-12-13 13:39:36,409 [main] INFO  o.a.ratis.conf.ConfUtils:46 - raft.grpc.flow.control.window = 4194304 (custom) 
iotdb-service  | 2023-12-13 13:39:36,409 [main] INFO  o.a.ratis.conf.ConfUtils:46 - raft.server.rpc.request.timeout = 3000ms (default) 
iotdb-service  | 2023-12-13 13:39:36,409 [main] INFO  o.a.ratis.conf.ConfUtils:46 - raft.grpc.server.heartbeat.channel = true (default) 
iotdb-service  | 2023-12-13 13:39:36,409 [main] INFO  o.a.ratis.conf.ConfUtils:46 - raft.grpc.server.async.request.thread.pool.cached = true (default) 
iotdb-service  | 2023-12-13 13:39:36,409 [main] INFO  o.a.ratis.conf.ConfUtils:46 - raft.grpc.server.async.request.thread.pool.size = 32 (default) 
iotdb-service  | 2023-12-13 13:39:36,410 [main] INFO  o.a.ratis.conf.ConfUtils:46 - raft.datastream.type = DISABLED (default) 
iotdb-service  | 2023-12-13 13:39:36,411 [main] INFO  o.a.ratis.conf.ConfUtils:46 - raft.server.threadpool.proxy.cached = true (default) 
iotdb-service  | 2023-12-13 13:39:36,411 [main] INFO  o.a.ratis.conf.ConfUtils:46 - raft.server.threadpool.proxy.size = 0 (default) 
iotdb-service  | 2023-12-13 13:39:36,411 [main] INFO  o.a.ratis.conf.ConfUtils:46 - raft.server.sleep.deviation.threshold = 4000ms (custom) 
iotdb-service  | 2023-12-13 13:39:36,411 [main] INFO  o.a.ratis.conf.ConfUtils:46 - raft.server.rpc.slowness.timeout = 10min (custom) 
iotdb-service  | 2023-12-13 13:39:36,411 [main] INFO  o.a.ratis.conf.ConfUtils:46 - raft.server.leaderelection.leader.step-down.wait-time = 30s (custom) 
iotdb-service  | 2023-12-13 13:39:36,411 [main] INFO  o.a.ratis.conf.ConfUtils:46 - raft.server.storage.dir = [/iotdb/data/datanode/consensus/data_region] (custom)

$ cat src/main/DockerCompose/logs/log_datanode_error.log 
2023-12-13 13:39:35,874 [main] ERROR o.a.i.db.service.DataNode:264 - Cannot pull system configurations from ConfigNode-leader after 10 retries 
2023-12-13 13:39:35,879 [main] ERROR o.a.i.db.service.DataNode:193 - Fail to start server 
org.apache.iotdb.commons.exception.StartupException: Cannot pull system configurations from ConfigNode-leader
	at org.apache.iotdb.db.service.DataNode.pullAndCheckSystemConfigurations(DataNode.java:267)
	at org.apache.iotdb.db.service.DataNode.doAddNode(DataNode.java:165)
	at org.apache.iotdb.db.service.DataNodeServerCommandLine.run(DataNodeServerCommandLine.java:81)
	at org.apache.iotdb.commons.ServerCommandLine.doMain(ServerCommandLine.java:58)
	at org.apache.iotdb.db.service.DataNode.main(DataNode.java:152)

$ cat src/main/DockerCompose/logs/log_confignode_error.log 
2023-12-13 13:38:47,631 [main] ERROR o.a.i.c.c.s.SyncConfigNodeClientPool:106 - REGISTER_CONFIG_NODE failed on ConfigNode TEndPoint(ip:127.0.0.1, port:10710) 
org.apache.iotdb.commons.client.exception.ClientManagerException: net.sf.cglib.core.CodeGenerationException: org.apache.thrift.transport.TTransportException-->java.net.ConnectException: Connection refused (Connection refused)
	at org.apache.iotdb.commons.client.ClientManager.borrowClient(ClientManager.java:55)
	at org.apache.iotdb.confignode.client.sync.SyncConfigNodeClientPool.sendSyncRequestToConfigNodeWithRetry(SyncConfigNodeClientPool.java:72)
	at org.apache.iotdb.confignode.service.ConfigNode.sendRegisterConfigNodeRequest(ConfigNode.java:329)
	at org.apache.iotdb.confignode.service.ConfigNode.active(ConfigNode.java:202)
	at org.apache.iotdb.confignode.service.ConfigNodeCommandLine.run(ConfigNodeCommandLine.java:79)
	at org.apache.iotdb.commons.ServerCommandLine.doMain(ServerCommandLine.java:58)
	at org.apache.iotdb.confignode.service.ConfigNode.main(ConfigNode.java:107)
Caused by: net.sf.cglib.core.CodeGenerationException: org.apache.thrift.transport.TTransportException-->java.net.ConnectException: Connection refused (Connection refused)
	at net.sf.cglib.core.ReflectUtils.newInstance(ReflectUtils.java:235)
	at net.sf.cglib.core.ReflectUtils.newInstance(ReflectUtils.java:220)
	at net.sf.cglib.proxy.Enhancer.createUsingReflection(Enhancer.java:639)
	at net.sf.cglib.proxy.Enhancer.firstInstance(Enhancer.java:538)
	at net.sf.cglib.core.AbstractClassGenerator.create(AbstractClassGenerator.java:231)
	at net.sf.cglib.proxy.Enhancer.createHelper(Enhancer.java:377)
	at net.sf.cglib.proxy.Enhancer.create(Enhancer.java:304)
	at org.apache.iotdb.commons.client.sync.SyncThriftClientWithErrorHandler.newErrorHandler(SyncThriftClientWithErrorHandler.java:46)
	at org.apache.iotdb.commons.client.sync.SyncConfigNodeIServiceClient$Factory.makeObject(SyncConfigNodeIServiceClient.java:119)
	at org.apache.iotdb.commons.client.sync.SyncConfigNodeIServiceClient$Factory.makeObject(SyncConfigNodeIServiceClient.java:101)
	at org.apache.commons.pool2.impl.GenericKeyedObjectPool.create(GenericKeyedObjectPool.java:780)
	at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:439)
	at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:350)
	at org.apache.iotdb.commons.client.ClientManager.borrowClient(ClientManager.java:53)
	... 6 common frames omitted
Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
	at org.apache.thrift.transport.TSocket.open(TSocket.java:243)
	at org.apache.iotdb.rpc.TElasticFramedTransport.open(TElasticFramedTransport.java:92)
	at org.apache.iotdb.commons.client.sync.SyncConfigNodeIServiceClient.<init>(SyncConfigNodeIServiceClient.java:64)
	at org.apache.iotdb.commons.client.sync.SyncConfigNodeIServiceClient$$EnhancerByCGLIB$$191de7e.<init>(<generated>)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
	at java.base/java.lang.reflect.Constructor.newInstance(Unknown Source)
	at net.sf.cglib.core.ReflectUtils.newInstance(ReflectUtils.java:228)
	... 19 common frames omitted
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
	at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
	at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source)
	at java.base/java.net.SocksSocketImpl.connect(Unknown Source)
	at java.base/java.net.Socket.connect(Unknown Source)
	at org.apache.thrift.transport.TSocket.open(TSocket.java:238)
	... 27 common frames omitted
2023-12-13 13:38:47,636 [main] ERROR o.a.i.c.s.ConfigNode:232 - Meet error while starting up. 
org.apache.iotdb.commons.exception.StartupException: All retry failed due to: net.sf.cglib.core.CodeGenerationException: org.apache.thrift.transport.TTransportException-->java.net.ConnectException: Connection refused (Connection refused)
	at org.apache.iotdb.confignode.service.ConfigNode.sendRegisterConfigNodeRequest(ConfigNode.java:352)
	at org.apache.iotdb.confignode.service.ConfigNode.active(ConfigNode.java:202)
	at org.apache.iotdb.confignode.service.ConfigNodeCommandLine.run(ConfigNodeCommandLine.java:79)
	at org.apache.iotdb.commons.ServerCommandLine.doMain(ServerCommandLine.java:58)
	at org.apache.iotdb.confignode.service.ConfigNode.main(ConfigNode.java:107)

Answered by HTHou

Dec 14, 2023

Hi, in v1.2.2, cn_seed_config_node called cn_target_config_node_list and dn_seed_config_node called dn_target_config_node_list. You may need to try these.

View full answer

HTHou · 2023-12-14T02:45:27Z

HTHou
Dec 14, 2023
Collaborator

Hi, in v1.2.2, cn_seed_config_node called cn_target_config_node_list and dn_seed_config_node called dn_target_config_node_list. You may need to try these.

2 replies

slawr Dec 14, 2023
Author

Hi Haonan, thank you for the hint. I had used docker inspect trying to find an environmental difference between the containers but completely missed that for some reason. I expect that could be it then. I am on vacation today, but will try it tomorrow and report back. Thanks again.

slawr Dec 15, 2023
Author

I made the two changes to the compose.

I rebuilt the container. On startup the server initially still failed to start.
The datanode error log contained the following:

2023-12-15 11:42:22,130 [main] ERROR o.a.i.db.service.DataNode:193 - Fail to start server 
org.apache.commons.io.FileExistsException: File element in parameter 'null' already exists: '/iotdb/data/datanode/system/system.properties'
	at org.apache.commons.io.FileUtils.requireAbsent(FileUtils.java:2587)
	at org.apache.commons.io.FileUtils.moveFile(FileUtils.java:2305)
	at org.apache.commons.io.FileUtils.moveFile(FileUtils.java:2283)
	at org.apache.iotdb.db.conf.IoTDBStartCheck.checkOldSystemConfig(IoTDBStartCheck.java:241)
	at org.apache.iotdb.db.service.DataNode.prepareDataNode(DataNode.java:209)
	at org.apache.iotdb.db.service.DataNode.doAddNode(DataNode.java:159)
	at org.apache.iotdb.db.service.DataNodeServerCommandLine.run(DataNodeServerCommandLine.java:81)
	at org.apache.iotdb.commons.ServerCommandLine.doMain(ServerCommandLine.java:58)
	at org.apache.iotdb.db.service.DataNode.main(DataNode.java:152)

I mention that in case it is something else the community should be aware of.

If I deleted the docker data directory tree then re-ran the compose up the server now started and I could successfully connect to the CLI client and talk to the server.

For anyone reading this later:
Upstream 1.2.2 tagged standalone compose is correct. IoTDB master has changed since recent 1.2.2 release as mentioned by Haonan. cn_target_config_node_list changed to cn_seed_config_node after 1.2.2 not the other way around. 1.2.2 was the last tagged release and is pretty recent and I presumably missed the fact that it had changed on master.

MRIIOT · 2024-02-06T16:08:43Z

MRIIOT
Feb 6, 2024

For 1.3.0-standalone this is what works for me:

iotdb-service:
    hostname: iotdb-service
    container_name: iotdb-service
    image: apache/iotdb:1.3.0-standalone
    restart: unless-stopped
    networks:
      - sharc
    ports:
      - "6667:6667/tcp"
    #expose:
    #  - "10710"
    environment:
      - enable_rest_service=true
      - cn_internal_address=iotdb-service
      - cn_internal_port=10710
      - cn_consensus_port=10720
      - cn_seed_config_node=iotdb-service:10710
      - dn_rpc_address=iotdb-service
      - dn_internal_address=iotdb-service
      - dn_rpc_port=6667
      - dn_mpp_data_exchange_port=10740
      - dn_schema_region_consensus_port=10750
      - dn_data_region_consensus_port=10760
      #- dn_target_config_node_list=iotdb-service:10710
      - dn_seed_config_node=iotdb-service:10710
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro
      - $VOLUME_PATH/iotdb/data:/iotdb/data
      - $VOLUME_PATH/iotdb/logs:/iotdb/logs
    logging:
      driver: "json-file"
      options:
        max-size: "20m"

1 reply

slawr Feb 6, 2024
Author

Thank you for the input. I think your example demonstrates Haonan's answer that the config_node name changed after 1.2.2.
1.3.0 had not yet been released when the question was raised and I had not spotted the change when looking at the updated master doc/src. Good to see though 1.3.0 is working as expected.

Your reply makes me realise that I have not closed this question, despite getting a working answer. I'll do that now.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Bug?] 1.2.2 Docker Compose standalone fails to start #11712

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 2 comments 3 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

[Bug?] 1.2.2 Docker Compose standalone fails to start #11712

Uh oh!

Uh oh!

slawr Dec 13, 2023

Summary

Detail

Replies: 2 comments · 3 replies

Uh oh!

Uh oh!

HTHou Dec 14, 2023 Collaborator

Uh oh!

slawr Dec 14, 2023 Author

Uh oh!

Uh oh!

slawr Dec 15, 2023 Author

Uh oh!

MRIIOT Feb 6, 2024

Uh oh!

slawr Feb 6, 2024 Author

slawr
Dec 13, 2023

Replies: 2 comments 3 replies

HTHou
Dec 14, 2023
Collaborator

slawr Dec 14, 2023
Author

slawr Dec 15, 2023
Author

MRIIOT
Feb 6, 2024

slawr Feb 6, 2024
Author