More tests and README update

- Tests for removal
- Tests for update
- README.md updated (added MySQL config and ClickHouse config settings)

Author: bakwc

README.md

@@ -42,7 +42,31 @@ You may need to also compile C++ components if they're not pre-built for your pl
 For realtime data sync from MySQL to ClickHouse:
 
 1. Prepare config file. Use `example_config.yaml` as an example.
-2. Start the replication:
+2. Configure MySQL and ClickHouse servers:
+ - MySQL server configuration file `my.cnf` should include following settings (required to write binary log in raw format, and enable password authentication):
+```ini
+[mysqld]
+# ... other settings ...
+gtid_mode = on
+enforce_gtid_consistency = 1
+default_authentication_plugin = mysql_native_password
+
+```
+
+ - ClickHouse server config `override.xml` should include following settings (it makes clickhouse apply final keyword automatically to handle updates correctly):
+```xml
+<clickhouse>
+    <!-- ... other settings ... -->
+    <profiles>
+        <default>
+            <!-- ... other settings ... -->
+            <final>1</final>
+        </default>
+    </profiles>
+</clickhouse>
+```
+
+3. Start the replication:
 
 ```bash
 mysql_ch_replicator --config config.yaml run_all

docker-compose-tests.yaml

@@ -12,6 +12,8 @@ services:
       - CLICKHOUSE_ADMIN_USER=default
       - CLICKHOUSE_HTTP_PORT=9123
     network_mode: host
+    volumes:
+      - ./tests_override.xml:/bitnami/clickhouse/etc/conf.d/override.xml:ro
 
   mysql_db:
     image: mysql/mysql-server:8.0.32

test_mysql_ch_replicator.py

@@ -312,6 +312,13 @@ def test_runner():
     assert_wait(lambda: len(ch.select(TEST_TABLE_NAME)) == 4)
     assert_wait(lambda: ch.select(TEST_TABLE_NAME, where="name='John'")[0]['rate'] == 12.5)
 
+    mysql.execute(f"DELETE FROM {TEST_TABLE_NAME} WHERE name='John';", commit=True)
+    assert_wait(lambda: len(ch.select(TEST_TABLE_NAME)) == 3)
+
+    mysql.execute(f"UPDATE {TEST_TABLE_NAME} SET age=66 WHERE name='Ivan'", commit=True)
+    time.sleep(4)
+    assert_wait(lambda: len(ch.select(TEST_TABLE_NAME)) == 3)
+
     run_all_runner.stop()
 
 

tests_override.xml

@@ -0,0 +1,7 @@
+<clickhouse>
+    <profiles>
+        <default>
+            <final>1</final>
+        </default>
+    </profiles>
+</clickhouse>