[BUG] MariaDB - Insertion cuts off after certain amount of records #209
Description
Bug Description
I'm running into a issue that I can't wrap my head around, it seems like a bug, but I couldn't find similar reported issues, so maybe it's related somehow to my setup.
Whenever I insert more than 250 records, mysql-ch-replicator inserts fewer records, I tried running different cases, it's a bit random: for 250 records it inserted only 224 records, 1K — 236, 10K — 389
Steps to Reproduce
- Create a simple table like this one
CREATE TABLE `tags` (
`id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
`project_id` int(11) NOT NULL,
`name` varchar(255) NOT NULL,
`created_at` timestamp NULL DEFAULT NULL,
`updated_at` timestamp NULL DEFAULT NULL,
PRIMARY KEY (`id`)
)-
Setup replication for this table
-
INSERT fake records
INSERT INTO tags (project_id, name, created_at, updated_at)
WITH RECURSIVE seq (n) AS (
SELECT 1
UNION ALL
SELECT n + 1 FROM seq WHERE n < 1000
)
SELECT
1111111 AS project_id,
CONCAT('Fake Tag ', n) AS name,
NOW() AS created_at,
NOW() AS updated_at
FROM seq;Expected Behavior
All inserted records would be replicated.
Actual Behavior
Upon inspection of mysql binlog, I can actually see all the INSERT records that must be replicated, and I can see multiple processing insert event that actually do match the number of records that must be inserted, but then inserting into tags, records: dict_values([...]) only outputs limited amount of records that are actually being inserted.
The final insert statements from query_log in ClickHouse contain only limited amount of rows. There are no errors in logs both in mysql-ch-replicator nor clickhouse
Environment
- mysql_ch_replicator version: 0.1.3
- Operating System: Docker over MacOS
- Python version: based on docker image
MySQL Configuration
[mysqld]
gtid_strict_mode = ON
gtid_domain_id = 0
server_id = 1
log_bin = /var/log/mysql/mysql-bin.log
binlog_expire_logs_seconds = 864000
max_binlog_size = 500M
binlog_format = ROWReplicator Configuration
mysql:
host: 'mysql'
port: 3306
user: 'xxxxxxx'
password: 'xxxxxxx'
clickhouse:
host: 'clickhouse'
port: 8123
user: 'xxxxxxx'
password: 'xxxxxxx'
connection_timeout: 30
send_receive_timeout: 300
binlog_replicator:
data_dir: '/app/data/binlog'
records_per_file: 100000
databases: 'default'
tables: ['tags']
log_level: 'debug'
types_mapping:
'tinyint(1)': 'Int8'
initial_replication_threads: 8