Skip to content

[Bug]: Failed to start docker container #26

@quaff

Description

@quaff

Environment

  • OS Version and CPU Arch(uname -a): Darwin 22.5.0 Darwin Kernel Version 22.5.0: Thu Jun 8 22:22:22 PDT 2023; root:xnu-8796.121.3~7/RELEASE_X86_64 x86_64

  • Docker Desktop Version 4.20.1 (110738)

Fast Reproduce Steps(Required)

Steps to reproduce the behavior:

  1. docker pull oceanbase/oceanbase-ce
  2. docker run -p 2881:2881 --name obstandalone -e MINI_MODE=1 -d oceanbase/oceanbase-ce

Expected behavior
Container obstandalone keep running

Actual Behavior
Container obstandalone exited with failure

Additional context
docker exec -it obstandalone obd display-trace a0cbb15e-1c7c-11ee-acab-0242ac110002

[2023-07-07 04:13:29.429] [DEBUG] - mkdir /root/.obd/lock/
[2023-07-07 04:13:29.430] [DEBUG] - unknown lock mode 
[2023-07-07 04:13:29.430] [DEBUG] - try to get share lock /root/.obd/lock/global
[2023-07-07 04:13:29.430] [DEBUG] - share lock `/root/.obd/lock/global`, count 1
[2023-07-07 04:13:29.430] [DEBUG] - cmd: ['obcluster']
[2023-07-07 04:13:29.430] [DEBUG] - opts: {'servers': None, 'components': None, 'force_delete': None, 'strict_check': None, 'without_parameter': None}
[2023-07-07 04:13:29.430] [DEBUG] - Get Deploy by name
[2023-07-07 04:13:29.430] [DEBUG] - mkdir /root/.obd/cluster/
[2023-07-07 04:13:29.431] [DEBUG] - mkdir /root/.obd/config_parser/
[2023-07-07 04:13:29.431] [DEBUG] - try to get exclusive lock /root/.obd/lock/deploy_obcluster
[2023-07-07 04:13:29.431] [DEBUG] - exclusive lock `/root/.obd/lock/deploy_obcluster`, count 1
[2023-07-07 04:13:29.436] [DEBUG] - Deploy status judge
[2023-07-07 04:13:29.438] [INFO] Get local repositories
[2023-07-07 04:13:29.438] [DEBUG] - mkdir /root/.obd/repository
[2023-07-07 04:13:29.438] [DEBUG] - Get local repository oceanbase-ce-4.1.0.0-8439ecf8db5e0649bd49671b41ea9e8c85756b63
[2023-07-07 04:13:29.438] [DEBUG] - Search repository oceanbase-ce version: 4.1.0.0, tag: 8439ecf8db5e0649bd49671b41ea9e8c85756b63, release: None, package_hash: None
[2023-07-07 04:13:29.438] [DEBUG] - try to get share lock /root/.obd/lock/mirror_and_repo
[2023-07-07 04:13:29.438] [DEBUG] - share lock `/root/.obd/lock/mirror_and_repo`, count 1
[2023-07-07 04:13:29.438] [DEBUG] - mkdir /root/.obd/repository/oceanbase-ce
[2023-07-07 04:13:29.441] [DEBUG] - Found repository oceanbase-ce-4.1.0.0-100000192023032010.el7-8439ecf8db5e0649bd49671b41ea9e8c85756b63
[2023-07-07 04:13:29.441] [DEBUG] - Get local repository obagent-1.3.0-d57fbb4962b2fbecb6282358c59295fdfba4d6ac
[2023-07-07 04:13:29.441] [DEBUG] - Search repository obagent version: 1.3.0, tag: d57fbb4962b2fbecb6282358c59295fdfba4d6ac, release: None, package_hash: None
[2023-07-07 04:13:29.441] [DEBUG] - share lock `/root/.obd/lock/mirror_and_repo`, count 2
[2023-07-07 04:13:29.441] [DEBUG] - mkdir /root/.obd/repository/obagent
[2023-07-07 04:13:29.444] [DEBUG] - Found repository obagent-1.3.0-22.el7-d57fbb4962b2fbecb6282358c59295fdfba4d6ac
[2023-07-07 04:13:29.444] [DEBUG] - Get deploy config
[2023-07-07 04:13:29.465] [INFO] Search plugins
[2023-07-07 04:13:29.466] [DEBUG] - Searching start_check plugin for components ...
[2023-07-07 04:13:29.466] [DEBUG] - Searching start_check plugin for oceanbase-ce-4.1.0.0-100000192023032010.el7-8439ecf8db5e0649bd49671b41ea9e8c85756b63
[2023-07-07 04:13:29.466] [DEBUG] - mkdir /root/.obd/plugins
[2023-07-07 04:13:29.468] [DEBUG] - Found for oceanbase-ce-py_script_start_check-4.0.0.0 for oceanbase-ce-4.1.0.0
[2023-07-07 04:13:29.468] [DEBUG] - Searching start_check plugin for obagent-1.3.0-22.el7-d57fbb4962b2fbecb6282358c59295fdfba4d6ac
[2023-07-07 04:13:29.469] [DEBUG] - Found for obagent-py_script_start_check-1.3.0 for obagent-1.3.0
[2023-07-07 04:13:29.469] [DEBUG] - Searching create_tenant plugin for components ...
[2023-07-07 04:13:29.469] [DEBUG] - Searching create_tenant plugin for oceanbase-ce-4.1.0.0-100000192023032010.el7-8439ecf8db5e0649bd49671b41ea9e8c85756b63
[2023-07-07 04:13:29.470] [DEBUG] - Found for oceanbase-ce-py_script_create_tenant-4.0.0.0 for oceanbase-ce-4.1.0.0
[2023-07-07 04:13:29.470] [DEBUG] - Searching create_tenant plugin for obagent-1.3.0-22.el7-d57fbb4962b2fbecb6282358c59295fdfba4d6ac
[2023-07-07 04:13:29.470] [DEBUG] - No such create_tenant plugin for obagent-1.3.0
[2023-07-07 04:13:29.470] [DEBUG] - Searching start plugin for components ...
[2023-07-07 04:13:29.470] [DEBUG] - Searching start plugin for oceanbase-ce-4.1.0.0-100000192023032010.el7-8439ecf8db5e0649bd49671b41ea9e8c85756b63
[2023-07-07 04:13:29.470] [DEBUG] - Found for oceanbase-ce-py_script_start-4.0.0.0 for oceanbase-ce-4.1.0.0
[2023-07-07 04:13:29.470] [DEBUG] - Searching start plugin for obagent-1.3.0-22.el7-d57fbb4962b2fbecb6282358c59295fdfba4d6ac
[2023-07-07 04:13:29.471] [DEBUG] - Found for obagent-py_script_start-1.3.0 for obagent-1.3.0
[2023-07-07 04:13:29.471] [DEBUG] - Searching connect plugin for components ...
[2023-07-07 04:13:29.471] [DEBUG] - Searching connect plugin for oceanbase-ce-4.1.0.0-100000192023032010.el7-8439ecf8db5e0649bd49671b41ea9e8c85756b63
[2023-07-07 04:13:29.471] [DEBUG] - Found for oceanbase-ce-py_script_connect-3.1.0 for oceanbase-ce-4.1.0.0
[2023-07-07 04:13:29.471] [DEBUG] - Searching connect plugin for obagent-1.3.0-22.el7-d57fbb4962b2fbecb6282358c59295fdfba4d6ac
[2023-07-07 04:13:29.471] [DEBUG] - Found for obagent-py_script_connect-1.3.0 for obagent-1.3.0
[2023-07-07 04:13:29.471] [DEBUG] - Searching bootstrap plugin for components ...
[2023-07-07 04:13:29.471] [DEBUG] - Searching bootstrap plugin for oceanbase-ce-4.1.0.0-100000192023032010.el7-8439ecf8db5e0649bd49671b41ea9e8c85756b63
[2023-07-07 04:13:29.472] [DEBUG] - Found for oceanbase-ce-py_script_bootstrap-4.0.0.0 for oceanbase-ce-4.1.0.0
[2023-07-07 04:13:29.472] [DEBUG] - Searching bootstrap plugin for obagent-1.3.0-22.el7-d57fbb4962b2fbecb6282358c59295fdfba4d6ac
[2023-07-07 04:13:29.472] [DEBUG] - Found for obagent-py_script_bootstrap-0.1 for obagent-1.3.0
[2023-07-07 04:13:29.472] [DEBUG] - Searching display plugin for components ...
[2023-07-07 04:13:29.472] [DEBUG] - Searching display plugin for oceanbase-ce-4.1.0.0-100000192023032010.el7-8439ecf8db5e0649bd49671b41ea9e8c85756b63
[2023-07-07 04:13:29.472] [DEBUG] - Found for oceanbase-ce-py_script_display-3.1.0 for oceanbase-ce-4.1.0.0
[2023-07-07 04:13:29.473] [DEBUG] - Searching display plugin for obagent-1.3.0-22.el7-d57fbb4962b2fbecb6282358c59295fdfba4d6ac
[2023-07-07 04:13:29.473] [DEBUG] - Found for obagent-py_script_display-1.3.0 for obagent-1.3.0
[2023-07-07 04:13:29.473] [INFO] Open ssh connection
[2023-07-07 04:13:29.474] [INFO] Load cluster param plugin
[2023-07-07 04:13:29.474] [DEBUG] - Searching param plugin for components ...
[2023-07-07 04:13:29.474] [DEBUG] - Search param plugin for oceanbase-ce
[2023-07-07 04:13:29.474] [DEBUG] - Found for oceanbase-ce-param-4.0.0.0 for oceanbase-ce-4.1.0.0
[2023-07-07 04:13:29.474] [DEBUG] - Applying oceanbase-ce-param-4.0.0.0 for oceanbase-ce-4.1.0.0-100000192023032010.el7-8439ecf8db5e0649bd49671b41ea9e8c85756b63
[2023-07-07 04:13:30.001] [DEBUG] - Search param plugin for obagent
[2023-07-07 04:13:30.001] [DEBUG] - Found for obagent-param-1.3.0 for obagent-1.3.0
[2023-07-07 04:13:30.001] [DEBUG] - Applying obagent-param-1.3.0 for obagent-1.3.0-22.el7-d57fbb4962b2fbecb6282358c59295fdfba4d6ac
[2023-07-07 04:13:30.059] [DEBUG] - Call oceanbase-ce-py_script_start_check-4.0.0.0 for oceanbase-ce-4.1.0.0-100000192023032010.el7-8439ecf8db5e0649bd49671b41ea9e8c85756b63
[2023-07-07 04:13:30.059] [DEBUG] - import start_check
[2023-07-07 04:13:30.065] [DEBUG] - add start_check ref count to 1
[2023-07-07 04:13:30.066] [INFO] Check before start observer
[2023-07-07 04:13:30.066] [DEBUG] -- local execute: ls /root/ob/store/clog/tenant_1/ 
[2023-07-07 04:13:30.077] [DEBUG] -- exited code 2, error output:
[2023-07-07 04:13:30.077] [DEBUG] ls: cannot access /root/ob/store/clog/tenant_1/: No such file or directory
[2023-07-07 04:13:30.077] [DEBUG] 
[2023-07-07 04:13:30.077] [DEBUG] -- local execute: cat /root/ob/run/observer.pid 
[2023-07-07 04:13:30.081] [DEBUG] -- exited code 0
[2023-07-07 04:13:30.082] [DEBUG] -- local execute: ls /proc/221 
[2023-07-07 04:13:30.086] [DEBUG] -- exited code 2, error output:
[2023-07-07 04:13:30.086] [DEBUG] ls: cannot access /proc/221: No such file or directory
[2023-07-07 04:13:30.087] [DEBUG] 
[2023-07-07 04:13:30.087] [DEBUG] -- 127.0.0.1 port check
[2023-07-07 04:13:30.087] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B41' | awk -F' ' '{print $2}' | uniq 
[2023-07-07 04:13:30.093] [DEBUG] -- exited code 0
[2023-07-07 04:13:30.094] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{print $2,$10}' | grep '00000000:0B42' | awk -F' ' '{print $2}' | uniq 
[2023-07-07 04:13:30.101] [DEBUG] -- exited code 0
[2023-07-07 04:13:30.102] [DEBUG] -- local execute: ls /root/ob/store/sstable/block_file 
[2023-07-07 04:13:30.107] [DEBUG] -- exited code 2, error output:
[2023-07-07 04:13:30.107] [DEBUG] ls: cannot access /root/ob/store/sstable/block_file: No such file or directory
[2023-07-07 04:13:30.107] [DEBUG] 
[2023-07-07 04:13:30.107] [DEBUG] -- local execute: grep -e '^ *lo:' /proc/net/dev 
[2023-07-07 04:13:30.110] [DEBUG] -- exited code 0
[2023-07-07 04:13:30.110] [DEBUG] -- local execute: cat /proc/sys/fs/aio-max-nr /proc/sys/fs/aio-nr 
[2023-07-07 04:13:30.113] [DEBUG] -- exited code 0
[2023-07-07 04:13:30.114] [DEBUG] -- local execute: ulimit -a 
[2023-07-07 04:13:30.117] [DEBUG] -- exited code 0
[2023-07-07 04:13:30.118] [DEBUG] -- local execute: cat /proc/meminfo 
[2023-07-07 04:13:30.120] [DEBUG] -- exited code 0
[2023-07-07 04:13:30.121] [DEBUG] -- local execute: df --block-size=1024  
[2023-07-07 04:13:30.125] [DEBUG] -- exited code 0
[2023-07-07 04:13:30.126] [DEBUG] -- get disk info for path /, total: 251650445312 avail: 210873688064
[2023-07-07 04:13:30.126] [DEBUG] -- get disk info for path /dev, total: 67108864 avail: 67108864
[2023-07-07 04:13:30.126] [DEBUG] -- get disk info for path /dev/shm, total: 67108864 avail: 67108864
[2023-07-07 04:13:30.126] [DEBUG] -- get disk info for path /etc/hosts, total: 251650445312 avail: 210873688064
[2023-07-07 04:13:30.126] [DEBUG] -- get disk info for path /proc/acpi, total: 6280445952 avail: 6280445952
[2023-07-07 04:13:30.126] [DEBUG] -- get disk info for path /sys/firmware, total: 6280445952 avail: 6280445952
[2023-07-07 04:13:30.126] [DEBUG] -- local execute: df --block-size=1024 /root/ob/store/clog 
[2023-07-07 04:13:30.129] [DEBUG] -- exited code 0
[2023-07-07 04:13:30.129] [DEBUG] -- get disk info for path /, total: 251650445312 avail: 210873688064
[2023-07-07 04:13:30.129] [DEBUG] -- local execute: df --block-size=1024 /root/ob/store 
[2023-07-07 04:13:30.132] [DEBUG] -- exited code 0
[2023-07-07 04:13:30.132] [DEBUG] -- get disk info for path /, total: 251650445312 avail: 210873688064
[2023-07-07 04:13:30.132] [DEBUG] -- disk: {'/': {'total': 251650445312, 'avail': 210873688064, 'need': 0}, '/dev': {'total': 67108864, 'avail': 67108864, 'need': 0}, '/dev/shm': {'total': 67108864, 'avail': 67108864, 'need': 0}, '/etc/hosts': {'total': 251650445312, 'avail': 210873688064, 'need': 0}, '/proc/acpi': {'total': 6280445952, 'avail': 6280445952, 'need': 0}, '/sys/firmware': {'total': 6280445952, 'avail': 6280445952, 'need': 0}}
[2023-07-07 04:13:30.133] [WARNING] OBD-1012: (127.0.0.1) clog and data use the same disk (/)
[2023-07-07 04:13:30.133] [DEBUG] -- local execute: date +%s%N 
[2023-07-07 04:13:30.136] [DEBUG] -- exited code 0
[2023-07-07 04:13:30.136] [DEBUG] -- 127.0.0.1 time delta -0.441162109375
[2023-07-07 04:13:30.136] [INFO] [WARN] OBD-1012: (127.0.0.1) clog and data use the same disk (/)
[2023-07-07 04:13:30.136] [INFO] 
[2023-07-07 04:13:30.136] [DEBUG] - sub start_check ref count to 0
[2023-07-07 04:13:30.136] [DEBUG] - export start_check
[2023-07-07 04:13:30.137] [DEBUG] - Call obagent-py_script_start_check-1.3.0 for obagent-1.3.0-22.el7-d57fbb4962b2fbecb6282358c59295fdfba4d6ac
[2023-07-07 04:13:30.137] [DEBUG] - import start_check
[2023-07-07 04:13:30.139] [DEBUG] - add start_check ref count to 1
[2023-07-07 04:13:30.139] [INFO] Check before start obagent
[2023-07-07 04:13:30.141] [DEBUG] -- local execute: cat /root/obagent/run/ob_agentd.pid 
[2023-07-07 04:13:30.145] [DEBUG] -- exited code 1, error output:
[2023-07-07 04:13:30.145] [DEBUG] cat: /root/obagent/run/ob_agentd.pid: No such file or directory
[2023-07-07 04:13:30.145] [DEBUG] 
[2023-07-07 04:13:30.145] [DEBUG] -- 127.0.0.1 port check
[2023-07-07 04:13:30.145] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{print $2,$10}' | grep '00000000:1F99' | awk -F' ' '{print $2}' | uniq 
[2023-07-07 04:13:30.152] [DEBUG] -- exited code 0
[2023-07-07 04:13:30.152] [DEBUG] -- local execute: bash -c 'cat /proc/net/{tcp*,udp*}' | awk -F' ' '{print $2,$10}' | grep '00000000:1F98' | awk -F' ' '{print $2}' | uniq 
[2023-07-07 04:13:30.158] [DEBUG] -- exited code 0
[2023-07-07 04:13:30.159] [DEBUG] - sub start_check ref count to 0
[2023-07-07 04:13:30.159] [DEBUG] - export start_check
[2023-07-07 04:13:30.159] [DEBUG] - Call oceanbase-ce-py_script_start-4.0.0.0 for oceanbase-ce-4.1.0.0-100000192023032010.el7-8439ecf8db5e0649bd49671b41ea9e8c85756b63
[2023-07-07 04:13:30.159] [DEBUG] - import start
[2023-07-07 04:13:30.161] [DEBUG] - add start ref count to 1
[2023-07-07 04:13:30.162] [INFO] Start observer
[2023-07-07 04:13:30.162] [DEBUG] -- local execute: ls /root/ob/store/clog/tenant_1/ 
[2023-07-07 04:13:30.167] [DEBUG] -- exited code 2, error output:
[2023-07-07 04:13:30.167] [DEBUG] ls: cannot access /root/ob/store/clog/tenant_1/: No such file or directory
[2023-07-07 04:13:30.167] [DEBUG] 
[2023-07-07 04:13:30.167] [DEBUG] -- local execute: cat /root/ob/run/observer.pid 
[2023-07-07 04:13:30.171] [DEBUG] -- exited code 0
[2023-07-07 04:13:30.171] [DEBUG] -- local execute: ls /proc/221 
[2023-07-07 04:13:30.176] [DEBUG] -- exited code 2, error output:
[2023-07-07 04:13:30.176] [DEBUG] ls: cannot access /proc/221: No such file or directory
[2023-07-07 04:13:30.176] [DEBUG] 
[2023-07-07 04:13:30.176] [DEBUG] -- 127.0.0.1 start command construction
[2023-07-07 04:13:30.176] [DEBUG] -- starting 127.0.0.1 observer
[2023-07-07 04:13:30.177] [DEBUG] -- root@127.0.0.1 set env LD_LIBRARY_PATH to '/root/ob/lib:'
[2023-07-07 04:13:30.177] [DEBUG] -- local execute: cd /root/ob; /root/ob/bin/observer -r '127.0.0.1:2882:2881' -p 2881 -P 2882 -z 'zone1' -n 'obcluster' -c 1 -d '/root/ob/store' -i 'lo' -l 'INFO' -o __min_full_resource_pool_memory=2147483648,memory_limit='6G',system_memory='1G',datafile_size='5G',log_disk_size='15G',cpu_count=16,enable_syslog_wf=False,enable_syslog_recycle=True,max_syslog_file_count=4 
[2023-07-07 04:13:30.588] [DEBUG] -- exited code 0
[2023-07-07 04:13:30.589] [DEBUG] -- root@127.0.0.1 set env LD_LIBRARY_PATH to '/tmp/_MEIFrdMcB'
[2023-07-07 04:13:30.589] [INFO] observer program health check
[2023-07-07 04:13:33.592] [DEBUG] -- 127.0.0.1 program health check
[2023-07-07 04:13:33.592] [DEBUG] -- local execute: cat /root/ob/run/observer.pid 
[2023-07-07 04:13:33.596] [DEBUG] -- exited code 0
[2023-07-07 04:13:33.596] [DEBUG] -- local execute: ls /proc/56 
[2023-07-07 04:13:33.600] [DEBUG] -- exited code 2, error output:
[2023-07-07 04:13:33.600] [DEBUG] ls: cannot access /proc/56: No such file or directory
[2023-07-07 04:13:33.601] [DEBUG] 
[2023-07-07 04:13:33.601] [WARNING] OBD-2002: Failed to start 127.0.0.1 observer
[2023-07-07 04:13:33.601] [DEBUG] - sub start ref count to 0
[2023-07-07 04:13:33.601] [DEBUG] - export start
[2023-07-07 04:13:33.601] [ERROR] oceanbase-ce start failed
[2023-07-07 04:13:33.601] [INFO] See https://www.oceanbase.com/product/ob-deployer/error-codes .
[2023-07-07 04:13:33.601] [INFO] Trace ID: a0cbb15e-1c7c-11ee-acab-0242ac110002
[2023-07-07 04:13:33.602] [INFO] If you want to view detailed obd logs, please run: obd display-trace a0cbb15e-1c7c-11ee-acab-0242ac110002
[2023-07-07 04:13:33.602] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo release, count 1
[2023-07-07 04:13:33.602] [DEBUG] - share lock /root/.obd/lock/mirror_and_repo release, count 0
[2023-07-07 04:13:33.602] [DEBUG] - unlock /root/.obd/lock/mirror_and_repo
[2023-07-07 04:13:33.602] [DEBUG] - exclusive lock /root/.obd/lock/deploy_obcluster release, count 0
[2023-07-07 04:13:33.602] [DEBUG] - unlock /root/.obd/lock/deploy_obcluster
[2023-07-07 04:13:33.602] [DEBUG] - share lock /root/.obd/lock/global release, count 0
[2023-07-07 04:13:33.602] [DEBUG] - unlock /root/.obd/lock/global

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions