-
Notifications
You must be signed in to change notification settings - Fork 28
Open
Labels
bugSomething isn't workingSomething isn't working
Description
Bug Description
Running optimize while initial replication is running results in an exception.
After that, replication seems to continue running, so not sure if this exception is a problem
Steps to Reproduce
- Start initial replication (big source database that takes days to replicate)
- Wait for optimizer to kick in
Expected Behavior
No exception thrown
Actual Behavior
[dbopt None 2025-07-21 19:22:25,349 INFO] Optimizing table sirocco.mt
[runner 2025-07-21 19:22:29,029 INFO] running replication for sirocco (initial replication not finished - waiting)
[dbrepl sirocco 2025-07-21 19:22:30,327 INFO] launched db_replicator
[dbrepl sirocco 2025-07-21 19:22:30,357 INFO] running initial replication
[dbrepl sirocco 2025-07-21 19:22:30,361 INFO] running initial replication for table mt
[dbrepl sirocco 2025-07-21 19:22:30,362 INFO] continue from primary key [196174706, '0b60954f-da42-499b-9db0-ab1a01eefb3d']
[binlogrepl 2025-07-21 19:23:25,175 INFO] last transaction id: ('log-bin.000811', 379007410), processed events: 41300
[binlogrepl 2025-07-21 19:24:25,321 INFO] last transaction id: ('log-bin.000811', 380495281), processed events: 47148
[dbopt None 2025-07-21 19:24:25,408 WARNING] Unexpected Http Driver Exception
[dbopt None 2025-07-21 19:24:25,409 ERROR] error executing command OPTIMIZE TABLE `sirocco`.`mt` FINAL SETTINGS mutations_sync = 2: Error HTTPConnectionPool(host='localhost', port=8123): Read timed out. (read timeout=120) executing HTTP request attempt 1 (http://localhost:8123)
Traceback (most recent call last):
File "/opt/sirocco/mysql-clickhouse-replicator/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 534, in _make_request
response = conn.getresponse()
File "/opt/sirocco/mysql-clickhouse-replicator/venv/lib/python3.10/site-packages/urllib3/connection.py", line 565, in getresponse
httplib_response = super().getresponse()
File "/usr/lib/python3.10/http/client.py", line 1375, in getresponse
response.begin()
File "/usr/lib/python3.10/http/client.py", line 318, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.10/http/client.py", line 279, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.10/socket.py", line 705, in readinto
return self._sock.recv_into(b)
TimeoutError: timed out
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/sirocco/mysql-clickhouse-replicator/venv/lib/python3.10/site-packages/clickhouse_connect/driver/httpclient.py", line 456, in _raw_request
response = self.http.request(method, url, **kwargs)
File "/opt/sirocco/mysql-clickhouse-replicator/venv/lib/python3.10/site-packages/urllib3/_request_methods.py", line 143, in request
return self.request_encode_body(
File "/opt/sirocco/mysql-clickhouse-replicator/venv/lib/python3.10/site-packages/urllib3/_request_methods.py", line 278, in request_encode_body
return self.urlopen(method, url, **extra_kw)
File "/opt/sirocco/mysql-clickhouse-replicator/venv/lib/python3.10/site-packages/urllib3/poolmanager.py", line 459, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "/opt/sirocco/mysql-clickhouse-replicator/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 841, in urlopen
retries = retries.increment(
File "/opt/sirocco/mysql-clickhouse-replicator/venv/lib/python3.10/site-packages/urllib3/util/retry.py", line 474, in increment
raise reraise(type(error), error, _stacktrace)
File "/opt/sirocco/mysql-clickhouse-replicator/venv/lib/python3.10/site-packages/urllib3/util/util.py", line 39, in reraise
raise value
File "/opt/sirocco/mysql-clickhouse-replicator/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
response = self._make_request(
File "/opt/sirocco/mysql-clickhouse-replicator/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 536, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "/opt/sirocco/mysql-clickhouse-replicator/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 367, in _raise_timeout
raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='localhost', port=8123): Read timed out. (read timeout=120)
Environment
- mysql_ch_replicator version: 0.0.93
- Operating System: Ubuntu 22.04.05 LTS
- Python version: 3.10.12
Replicator Configuration
mysql:
host: '192.168.X.X'
port: 3306
user: 'crXXXXXX'
password: 'XXXXXXX'
clickhouse:
host: 'localhost'
port: 8123
user: 'default'
password: 'XXXXXXX'
log_level: info
binlog_replicator:
data_dir: '/var/lib/data/mysql-clickhouse-replicator/binlog'
records_per_file: 100000
databases: ['sirocco','sirocco_shorturl']
ignore_deletes: true
optimize_interval: 86400
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working