Add test for SST progress monitoring (effyis#390) #3911

PavelShilin89 · 2025-11-10T00:08:53Z

Summary

Adds comprehensive test for SST (State Snapshot Transfer) progress monitoring during replication.

Changes

New test: test/clt-tests/replication/test-sst-progress.rec
- Validates all 5 SST progress variables exist in SHOW STATUS
- Tests JOIN CLUSTER triggering SST transfer
- Verifies node synchronization
- Checks table replication across cluster nodes
Config fix: Adds watchdog = 0 to base config with idempotent check

SST Progress Variables Tested

cluster_<name>_node_state - Node state (donor/joiner/synced)
cluster_<name>_sst_total - Overall progress (0-100 or dash)
cluster_<name>_sst_stage - Current stage name or dash
cluster_<name>_sst_stage_total - Stage progress (0-100 or dash)
cluster_<name>_sst_tables - Total tables count or dash
cluster_<name>_sst_table - Current table name or dash

Related Issue

https://github.com/manticoresoftware/effyis/issues/390

This test validates that SST progress variables are properly exposed in SHOW STATUS during State Snapshot Transfer when a node joins a cluster. The test focuses on verifying that all SST progress variables exist: - cluster_<name>_node_state (donor/joiner/synced) - cluster_<name>_sst_total (0-100 or dash when complete) - cluster_<name>_sst_stage (stage name or dash) - cluster_<name>_sst_stage_total (0-100 or dash) - cluster_<name>_sst_tables (count or dash) - cluster_<name>_sst_table (name or dash) Test creates a 2-node cluster with a simple table, triggers SST via JOIN CLUSTER, and verifies that all progress variables are present in SHOW STATUS output regardless of SST completion timing. Also adds watchdog = 0 to base searchd config with idempotent check.

github-actions · 2025-11-10T00:29:32Z

clt

❌ CLT tests in test/clt-tests/test-configuration/
✅ OK: 7
❌ Failed: 1
⏳ Duration: 72s
👉 Check Action Results for commit 4e1a5b9

Failed tests:

🔧 Edit failed tests in UI:

Edit test/clt-tests/test-configuration/show-settings.rec

test/clt-tests/test-configuration/show-settings.rec

––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "SHOW SETTINGS;"
––– output –––
+----------------------------+------------------------------------------------------------+
| Setting_name               | Value                                                      |
+----------------------------+------------------------------------------------------------+
| configuration_file         | /.clt/test/clt-tests/base/searchd-with-flexible-ports.conf |
| worker_pid                 | %{NUMBER}                                                         |
| searchd.listen             | %{IPADDR}:%{NUMBER}:mysql41                                     |
| searchd.listen             | %{IPADDR}:%{NUMBER}                                             |
| searchd.listen             | %{IPADDR}:%{NUMBER}:http                                        |
| searchd.log                | /var/log/manticore-1/searchd.log                           |
| searchd.query_log          | /var/log/manticore-1/query.log                             |
| searchd.pid_file           | /var/log/manticore-1/searchd.pid                           |
| searchd.data_dir           | /var/log/manticore-1                                       |
| searchd.query_log_format   | sphinxql                                                   |
| searchd.query_log_commands | 1                                                          |
| searchd.server_id          | 1                                                          |
- | searchd.binlog_path        | /var/log/manticore-1/binlog                                |
+ | searchd.watchdog           | 0                                                          |
- | common.plugin_dir          | /usr/local/lib/manticore                                   |
+ | searchd.binlog_path        | /var/log/manticore-1/binlog                                |
- | common.lemmatizer_base     | /usr/share/manticore/                                      |
+ | common.plugin_dir          | /usr/local/lib/manticore                                   |
- +----------------------------+------------------------------------------------------------+
+ | common.lemmatizer_base     | /usr/share/manticore/                                      |
+ +----------------------------+------------------------------------------------------------+

github-actions · 2025-11-10T00:30:59Z

clt

❌ CLT tests in test/clt-tests/replication/
✅ OK: 6
❌ Failed: 1
⏳ Duration: 155s
👉 Check Action Results for commit 4e1a5b9

Failed tests:

🔧 Edit failed tests in UI:

Edit test/clt-tests/replication/test-sst-progress.rec

test/clt-tests/replication/test-sst-progress.rec

––– input –––
apt-get update -y > /dev/null; echo $?
––– output –––
OK
––– input –––
apt-get install -y iproute2 procps > /dev/null; echo $?
––– output –––
OK
––– input –––
grep -q "watchdog" test/clt-tests/base/searchd-with-flexible-ports.conf && echo "watchdog already exists" || sed -i '/^searchd {/,/^}/ s/^\([[:space:]]*\)}$/\1\twatchdog = 0\n\1}/' test/clt-tests/base/searchd-with-flexible-ports.conf; echo $?
––– output –––
#!/watchdog already exists|0/!#
+ 0
––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd --logreplication -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "CREATE CLUSTER sst_test"
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "SHOW STATUS LIKE 'cluster_sst_test_status'\G"
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "CREATE TABLE test_table (id bigint, title text)"
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "ALTER CLUSTER sst_test ADD test_table"
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "SHOW STATUS LIKE 'cluster_sst_test_indexes'\G"
––– output –––
OK
––– input –––
export INSTANCE=2
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd --logreplication -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
mysql -h0 -P2306 -e "JOIN CLUSTER sst_test AT '127.0.0.1:1312'"
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "SHOW STATUS LIKE 'cluster_sst_test_node_state'\G"
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "SHOW STATUS LIKE 'cluster_sst_test_sst_%'\G" | grep "Counter:" | wc -l
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "SHOW STATUS LIKE 'cluster_sst_test_sst_total'\G"
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "SHOW STATUS LIKE 'cluster_sst_test_sst_stage'\G"
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "SHOW STATUS LIKE 'cluster_sst_test_sst_stage_total'\G"
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "SHOW STATUS LIKE 'cluster_sst_test_sst_tables'\G"
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "SHOW STATUS LIKE 'cluster_sst_test_sst_table'\G"
––– output –––
OK
––– input –––
bash -c 'end=$((SECONDS+60)); while [ $SECONDS -lt $end ]; do all_synced=true; for port in 1306 2306; do mysql -h0 -P$port -e "SHOW STATUS LIKE '\''cluster_sst_test_status'\''\G" > /tmp/status_$port.log 2>/dev/null && grep -q "Value: primary" /tmp/status_$port.log || { all_synced=false; break; }; done; if $all_synced; then for port in 1306 2306; do echo "Port $port: Node synced"; done; exit 0; fi; sleep 1; done; echo "Timeout waiting for nodes to sync!"; exit 1'
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "SHOW STATUS LIKE 'cluster_sst_test_indexes'\G"
––– output –––
OK
––– input –––
mysql -h0 -P2306 -e "SHOW STATUS LIKE 'cluster_sst_test_indexes'\G"
––– output –––
OK

- Fix output regex: now matches 'watchdog already exists' or 'watchdog added' - Add cleanup step to remove watchdog from base config after test - Prevents affecting other tests that run after this one

github-actions · 2025-11-10T01:16:05Z

clt

❌ CLT tests in test/clt-tests/test-configuration/
✅ OK: 7
❌ Failed: 1
⏳ Duration: 73s
👉 Check Action Results for commit 8648952

Failed tests:

🔧 Edit failed tests in UI:

Edit test/clt-tests/test-configuration/show-settings.rec

test/clt-tests/test-configuration/show-settings.rec

––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "SHOW SETTINGS;"
––– output –––
+----------------------------+------------------------------------------------------------+
| Setting_name               | Value                                                      |
+----------------------------+------------------------------------------------------------+
| configuration_file         | /.clt/test/clt-tests/base/searchd-with-flexible-ports.conf |
| worker_pid                 | %{NUMBER}                                                         |
| searchd.listen             | %{IPADDR}:%{NUMBER}:mysql41                                     |
| searchd.listen             | %{IPADDR}:%{NUMBER}                                             |
| searchd.listen             | %{IPADDR}:%{NUMBER}:http                                        |
| searchd.log                | /var/log/manticore-1/searchd.log                           |
| searchd.query_log          | /var/log/manticore-1/query.log                             |
| searchd.pid_file           | /var/log/manticore-1/searchd.pid                           |
| searchd.data_dir           | /var/log/manticore-1                                       |
| searchd.query_log_format   | sphinxql                                                   |
| searchd.query_log_commands | 1                                                          |
| searchd.server_id          | 1                                                          |
- | searchd.binlog_path        | /var/log/manticore-1/binlog                                |
+ | searchd.watchdog           | 0                                                          |
- | common.plugin_dir          | /usr/local/lib/manticore                                   |
+ | searchd.binlog_path        | /var/log/manticore-1/binlog                                |
- | common.lemmatizer_base     | /usr/share/manticore/                                      |
+ | common.plugin_dir          | /usr/local/lib/manticore                                   |
- +----------------------------+------------------------------------------------------------+
+ | common.lemmatizer_base     | /usr/share/manticore/                                      |
+ +----------------------------+------------------------------------------------------------+

tomatolog

the test seems ok however I was against this kind of test as it checks the show status has counters that is useless

github-actions · 2025-11-19T11:48:49Z

clt

❌ CLT tests in test/clt-tests/replication/
✅ OK: 6
❌ Failed: 1
⏳ Duration: 271s
👉 Check Action Results for commit f0e4ad6

Failed tests:

🔧 Edit failed tests in UI:

Edit test/clt-tests/replication/test-sst-progress.rec

test/clt-tests/replication/test-sst-progress.rec

––– input –––
apt-get update -y > /dev/null; echo $?
––– output –––
OK
––– input –––
apt-get install -y iproute2 procps > /dev/null; echo $?
––– output –––
OK
––– input –––
grep -q "watchdog" test/clt-tests/base/searchd-with-flexible-ports.conf && echo "watchdog already exists" || { sed -i '/^searchd {/,/^}/ s/^\([[:space:]]*\)}$/\1\twatchdog = 0\n\1}/' test/clt-tests/base/searchd-with-flexible-ports.conf && echo "watchdog added"; }
––– output –––
OK
––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd --logreplication -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "CREATE CLUSTER sst_test"
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "SHOW STATUS LIKE 'cluster_sst_test_status'\G"
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "CREATE TABLE test_table (id bigint, title text, content text)"
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "ALTER CLUSTER sst_test ADD test_table"
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "SHOW STATUS LIKE 'cluster_sst_test_indexes'\G"
––– output –––
OK
––– input –––
export INSTANCE=2
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd --logreplication -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
manticore-load --host=127.0.0.1 --port=1306 --quiet --threads=4 --total=2000000 --load="INSERT INTO sst_test:test_table (id, title, content) VALUES (<increment>, '<text/10/50>', '<text/20/100>')" --batch-size=5000
––– output –––
#!/.*/!#
+ 4       ; 5000      ; 01:03       ; 2000000     ; 31574       ; 6           ; 10          ; 8           ; 4           ; 3           ; 630.5       ; 625.0       ; 905.0       ; 1250.0      ;
––– input –––
mysql -h0 -P1306 -NB -e "SELECT COUNT(*) FROM sst_test:test_table"
––– output –––
- #!/[0-9]+/!#
+ +---------+
+ | 2000000 |
+ +---------+
––– input –––
mysql -h0 -P2306 -e "JOIN CLUSTER sst_test AT '127.0.0.1:1312'" > /dev/null 2>&1 & mysql -h0 -P1306 -e "SHOW STATUS LIKE 'cluster_sst_test_node_state'\G"
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "SHOW STATUS LIKE 'cluster_sst_test_sst_%'\G" | grep "Counter:" | wc -l
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "SHOW STATUS LIKE 'cluster_sst_test_sst_total'\G"
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "SHOW STATUS LIKE 'cluster_sst_test_sst_stage'\G"
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "SHOW STATUS LIKE 'cluster_sst_test_sst_stage_total'\G"
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "SHOW STATUS LIKE 'cluster_sst_test_sst_tables'\G"
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "SHOW STATUS LIKE 'cluster_sst_test_sst_table'\G"
––– output –––
OK
––– input –––
bash -c 'end=$((SECONDS+60)); while [ $SECONDS -lt $end ]; do all_synced=true; for port in 1306 2306; do mysql -h0 -P$port -e "SHOW STATUS LIKE '\''cluster_sst_test_status'\''\G" > /tmp/status_$port.log 2>/dev/null && grep -q "Value: primary" /tmp/status_$port.log || { all_synced=false; break; }; done; if $all_synced; then for port in 1306 2306; do echo "Port $port: Node synced"; done; exit 0; fi; sleep 1; done; echo "Timeout waiting for nodes to sync!"; exit 1'
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "SHOW STATUS LIKE 'cluster_sst_test_indexes'\G"
––– output –––
OK
––– input –––
mysql -h0 -P2306 -e "SHOW STATUS LIKE 'cluster_sst_test_indexes'\G"
––– output –––
*************************** 1. row ***************************
Counter: cluster_sst_test_indexes
-   Value: test_table
+   Value:
––– input –––
sed -i '/watchdog = 0/d' test/clt-tests/base/searchd-with-flexible-ports.conf; echo $?
––– output –––
OK

…0K large documents and checking during active transfer

github-actions · 2025-11-20T12:48:05Z

clt

❌ CLT tests in test/clt-tests/bugs/
✅ OK: 11
❌ Failed: 1
⏳ Duration: 126s
👉 Check Action Results for commit 05df2bb

Failed tests:

🔧 Edit failed tests in UI:

Edit test/clt-tests/bugs/3847-conflict-handling-verification.rec

test/clt-tests/bugs/3847-conflict-handling-verification.rec

––– input –––
set -b +m
––– output –––
OK
––– input –––
grep -q 'threads = 4' test/clt-tests/base/searchd-with-flexible-ports.conf || sed -i '/searchd {/a\	threads = 4' test/clt-tests/base/searchd-with-flexible-ports.conf
––– output –––
OK
––– input –––
export INSTANCE=1
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=2
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
export INSTANCE=3
––– output –––
OK
––– input –––
mkdir -p /var/{run,lib,log}/manticore-${INSTANCE}
––– output –––
OK
––– input –––
stdbuf -oL searchd -c test/clt-tests/base/searchd-with-flexible-ports.conf > /dev/null
––– output –––
OK
––– input –––
if timeout 10 grep -qm1 '\[BUDDY\] started' <(tail -n 1000 -f /var/log/manticore-${INSTANCE}/searchd.log); then echo 'Buddy started!'; else echo 'Timeout or failed!'; cat /var/log/manticore-${INSTANCE}/searchd.log; fi
––– output –––
OK
––– input –––
wait_for_sync() { sleep 0.5; for i in {1..10}; do c1=$(mysql -h0 -P1306 -sN -e "SELECT COUNT(*) FROM test:tbl1" 2>/dev/null | grep -oE '[0-9]+' | head -1); c2=$(mysql -h0 -P2306 -sN -e "SELECT COUNT(*) FROM test:tbl1" 2>/dev/null | grep -oE '[0-9]+' | head -1); c3=$(mysql -h0 -P3306 -sN -e "SELECT COUNT(*) FROM test:tbl1" 2>/dev/null | grep -oE '[0-9]+' | head -1); if [ "$c1" = "$c2" ] && [ "$c2" = "$c3" ] && [ -n "$c1" ]; then return 0; fi; sleep 0.5; done; return 1; }
––– output –––
OK
––– input –––
mkdir /var/{lib,log}/manticore-{1,2,3}/test
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "CREATE CLUSTER test 'test' as path"; echo $?
––– output –––
OK
––– input –––
mysql -h0 -P2306 -e "JOIN CLUSTER test at '127.0.0.1:1312' 'test' as path"; echo $?
––– output –––
OK
––– input –––
mysql -h0 -P3306 -e "JOIN CLUSTER test at '127.0.0.1:1312' 'test' as path"; echo $?
––– output –––
OK
––– input –––
sleep 2
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "CREATE TABLE tbl1 (id bigint, attr1 int)"; echo $?
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "ALTER CLUSTER test ADD tbl1"; echo $?
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "INSERT INTO test:tbl1 (id, attr1) VALUES (1,1), (3,2), (10,3), (11,4), (12,5), (13,6), (14,7), (15,8), (20,9)"; echo $?
––– output –––
OK
––– input –––
wait_for_sync && echo "Cluster synchronized" || echo "Sync timeout"
––– output –––
OK
––– input –––
mysql -h0 -P1306 -NB -e "SELECT COUNT(*) FROM test:tbl1\G"
––– output –––
OK
––– input –––
mysql -h0 -P2306 -NB -e "SELECT COUNT(*) FROM test:tbl1\G"
––– output –––
OK
––– input –––
mysql -h0 -P3306 -NB -e "SELECT COUNT(*) FROM test:tbl1\G"
––– output –––
OK
––– input –––
manticore-load --host=127.0.0.1 --threads=4 --port=1306 --total=1000000 --query="REPLACE INTO test:tbl1 (id, attr1) VALUES (%RAND, %RAND)" --together --host=127.0.0.1 --threads=4 --port=2306 --total=1000000 --query="REPLACE INTO test:tbl1 (id, attr1) VALUES (%RAND, %RAND)" > /dev/null 2>&1 & LOAD_PID=$!; sleep 1; echo "Load started: $LOAD_PID"
––– output –––
OK
––– input –––
mysql -h0 -P2306 -e "UPDATE test:tbl1 SET attr1=1 WHERE id=13" & sleep 0.05; mysql -h0 -P1306 -e "REPLACE INTO test:tbl1 (id, attr1) VALUES (10, 999)" & wait
––– output –––
OK
––– input –––
mysql -h0 -P2306 -e "REPLACE INTO test:tbl1 (id, attr1) VALUES (11, 111)" & sleep 0.05; mysql -h0 -P1306 -e "REPLACE INTO test:tbl1 (id, attr1) VALUES (10, 101)" & wait
––– output –––
OK
––– input –––
mysql -h0 -P2306 -e "DELETE FROM test:tbl1 WHERE id=3" & sleep 0.05; mysql -h0 -P1306 -e "REPLACE INTO test:tbl1 (id, attr1) VALUES (10, 102)" & wait
––– output –––
OK
––– input –––
mysql -h0 -P2306 -e "INSERT INTO test:tbl1 (id, attr1) VALUES (100, 1)" & sleep 0.05; mysql -h0 -P1306 -e "INSERT INTO test:tbl1 (id, attr1) VALUES (200, 2)" & wait
––– output –––
OK
––– input –––
conflicts=0; for i in {1..30}; do result=$( (mysql -h0 -P2306 -e "UPDATE test:tbl1 SET attr1=1 WHERE id=13" 2>&1 & mysql -h0 -P1306 -e "REPLACE INTO test:tbl1 (id, attr1) VALUES (13, 999)" 2>&1 & wait) ); if echo "$result" | grep -q "error at PostRollback"; then ((conflicts++)); fi; done; echo "Conflicts: $conflicts/30"; test $conflicts -ge 1 && echo "PASS" || echo "FAIL"
––– output –––
OK
––– input –––
conflicts=0; for i in {1..30}; do result=$( (mysql -h0 -P2306 -e "UPDATE test:tbl1 SET attr1=1 WHERE id>13" 2>&1 & mysql -h0 -P1306 -e "REPLACE INTO test:tbl1 (id, attr1) VALUES (14, 888)" 2>&1 & wait) ); if echo "$result" | grep -q "error at PostRollback"; then ((conflicts++)); fi; done; echo "Conflicts: $conflicts/30"; test $conflicts -ge 1 && echo "PASS" || echo "FAIL"
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "REPLACE INTO test:tbl1 (id, attr1) VALUES (3, 2)" > /dev/null 2>&1; sleep 2
––– output –––
OK
––– input –––
conflicts=0; for i in {1..30}; do result=$( (mysql -h0 -P2306 -e "UPDATE test:tbl1 SET attr1=1 WHERE attr1=2" 2>&1 & mysql -h0 -P1306 -e "REPLACE INTO test:tbl1 (id, attr1) VALUES (3, 333)" 2>&1 & wait) ); if echo "$result" | grep -q "error at PostRollback"; then ((conflicts++)); fi; done; echo "Conflicts: $conflicts/30"; test $conflicts -ge 1 && echo "PASS" || echo "FAIL"
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "REPLACE INTO test:tbl1 (id, attr1) VALUES (3, 2)" > /dev/null 2>&1; sleep 2
––– output –––
OK
––– input –––
conflicts=0; for i in {1..30}; do result=$( (mysql -h0 -P2306 -e "UPDATE test:tbl1 SET attr1=1 WHERE attr1=2" 2>&1 & mysql -h0 -P1306 -e "DELETE FROM test:tbl1 WHERE id=3" 2>&1 & wait) ); if echo "$result" | grep -q "error at PostRollback"; then ((conflicts++)); fi; done; echo "Conflicts: $conflicts/30"; test $conflicts -ge 1 && echo "PASS" || echo "FAIL"
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "REPLACE INTO test:tbl1 (id, attr1) VALUES (3, 2)" > /dev/null 2>&1; sleep 2
––– output –––
OK
––– input –––
conflicts=0; for i in {1..30}; do result=$( (mysql -h0 -P2306 -e "DELETE FROM test:tbl1 WHERE id=3" 2>&1 & mysql -h0 -P1306 -e "REPLACE INTO test:tbl1 (id, attr1) VALUES (3, 303)" 2>&1 & wait) ); if echo "$result" | grep -q "error at PostRollback"; then ((conflicts++)); fi; done; echo "Conflicts: $conflicts/30"; test $conflicts -ge 1 && echo "PASS" || echo "FAIL"
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "REPLACE INTO test:tbl1 (id, attr1) VALUES (1, 1)" > /dev/null 2>&1; sleep 2
––– output –––
OK
––– input –––
conflicts=0; for i in {1..30}; do result=$( (mysql -h0 -P2306 -e "DELETE FROM test:tbl1 WHERE id=1" 2>&1 & mysql -h0 -P1306 -e "DELETE FROM test:tbl1 WHERE id=1" 2>&1 & wait) ); if echo "$result" | grep -q "error at PostRollback"; then ((conflicts++)); fi; done; echo "Conflicts: $conflicts/30"; test $conflicts -ge 1 && echo "PASS" || echo "FAIL"
––– output –––
OK
––– input –––
conflicts=0; for i in {1..30}; do result=$( (mysql -h0 -P2306 -e "UPDATE test:tbl1 SET attr1=111 WHERE id=15" 2>&1 & mysql -h0 -P1306 -e "UPDATE test:tbl1 SET attr1=222 WHERE id=15" 2>&1 & wait) ); if echo "$result" | grep -q "error at PostRollback"; then ((conflicts++)); fi; done; echo "Conflicts: $conflicts/30"; test $conflicts -ge 1 && echo "PASS" || echo "FAIL"
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "REPLACE INTO test:tbl1 (id, attr1) VALUES (1, 1001)" & sleep 0.05; mysql -h0 -P2306 -e "REPLACE INTO test:tbl1 (id, attr1) VALUES (10, 1010)" & sleep 0.05; mysql -h0 -P3306 -e "REPLACE INTO test:tbl1 (id, attr1) VALUES (20, 1020)" & wait
––– output –––
OK
––– input –––
conflicts=0; for i in {1..30}; do result=$( (mysql -h0 -P1306 -e "UPDATE test:tbl1 SET attr1=100 WHERE id=12" 2>&1 & mysql -h0 -P2306 -e "UPDATE test:tbl1 SET attr1=200 WHERE id=12" 2>&1 & mysql -h0 -P3306 -e "UPDATE test:tbl1 SET attr1=300 WHERE id=12" 2>&1 & wait) ); if echo "$result" | grep -q "error at PostRollback"; then ((conflicts++)); fi; done; echo "Conflicts: $conflicts/30"; test $conflicts -ge 1 && echo "PASS" || echo "FAIL"
––– output –––
OK
––– input –––
mysql -h0 -P1306 -e "REPLACE INTO test:tbl1 (id, attr1) VALUES (14, 14)" > /dev/null 2>&1; sleep 2
––– output –––
OK
––– input –––
conflicts=0; for i in {1..30}; do result=$( (mysql -h0 -P1306 -e "DELETE FROM test:tbl1 WHERE id=14" 2>&1 & mysql -h0 -P2306 -e "DELETE FROM test:tbl1 WHERE id=14" 2>&1 & mysql -h0 -P3306 -e "REPLACE INTO test:tbl1 (id, attr1) VALUES (15, 1500)" 2>&1 & wait) ); if echo "$result" | grep -q "error at PostRollback"; then ((conflicts++)); fi; done; echo "Conflicts: $conflicts/30"; test $conflicts -ge 1 && echo "PASS" || echo "FAIL"
––– output –––
Conflicts: %{NUMBER}/30
- PASS
+ FAIL
––– input –––
kill $LOAD_PID 2>/dev/null; wait $LOAD_PID 2>/dev/null; echo "Load stopped"
––– output –––
OK
––– input –––
wait_for_sync && echo "Final sync successful" || echo "Final sync failed"
––– output –––
OK
––– input –––
mysql -h0 -P1306 -NB -e "SELECT COUNT(*) FROM test:tbl1\G"
––– output –––
OK
––– input –––
mysql -h0 -P2306 -NB -e "SELECT COUNT(*) FROM test:tbl1\G"
––– output –––
OK
––– input –––
mysql -h0 -P3306 -NB -e "SELECT COUNT(*) FROM test:tbl1\G"
––– output –––
OK
––– input –––
c1=$(mysql -h0 -P1306 -sN -e "SELECT COUNT(*) FROM test:tbl1" | grep -oE '[0-9]+' | head -1); c2=$(mysql -h0 -P2306 -sN -e "SELECT COUNT(*) FROM test:tbl1" | grep -oE '[0-9]+' | head -1); c3=$(mysql -h0 -P3306 -sN -e "SELECT COUNT(*) FROM test:tbl1" | grep -oE '[0-9]+' | head -1); if [ "$c1" = "$c2" ] && [ "$c2" = "$c3" ]; then echo "All nodes synchronized ($c1 rows)"; else echo "Discrepancies: node1=$c1, node2=$c2, node3=$c3"; fi
––– output –––
OK
––– input –––
for i in 1 2 3; do grep -q 'FATAL:' /var/log/manticore-${i}/searchd.log && echo "Node #$i has FATAL" || echo "Node #$i OK"; done
––– output –––
OK

PavelShilin89 requested a review from tomatolog November 10, 2025 00:12

PavelShilin89 added the done label Nov 10, 2025

Fix watchdog cleanup and output formatting in SST progress test

200cd84

- Fix output regex: now matches 'watchdog already exists' or 'watchdog added' - Add cleanup step to remove watchdog from base config after test - Prevents affecting other tests that run after this one

test: remove watchdog=0 from base config (should only be in test scope)

587fd47

tomatolog approved these changes Nov 11, 2025

View reviewed changes

PavelShilin89 force-pushed the test/issue-effyis-390-sst-progress branch from b5210c3 to 587fd47 Compare November 18, 2025 19:23

Update test-sst-progress.rec

6cac6ba

PavelShilin89 force-pushed the test/issue-effyis-390-sst-progress branch from 0fcb9c0 to 6cac6ba Compare November 18, 2025 22:03

Enhance SST progress test to capture real-time metrics by inserting 3…

28146d3

…0K large documents and checking during active transfer

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Add test for SST progress monitoring (effyis#390) #3911

Add test for SST progress monitoring (effyis#390) #3911

PavelShilin89 commented Nov 10, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Nov 10, 2025

Uh oh!

github-actions bot commented Nov 10, 2025

Uh oh!

github-actions bot commented Nov 10, 2025

Uh oh!

tomatolog left a comment

Uh oh!

github-actions bot commented Nov 19, 2025

Uh oh!

github-actions bot commented Nov 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Add test for SST progress monitoring (effyis#390) #3911

Are you sure you want to change the base?

Add test for SST progress monitoring (effyis#390) #3911

Conversation

PavelShilin89 commented Nov 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

SST Progress Variables Tested

Related Issue

Uh oh!

github-actions bot commented Nov 10, 2025

clt

Failed tests:

Uh oh!

github-actions bot commented Nov 10, 2025

clt

Failed tests:

Uh oh!

github-actions bot commented Nov 10, 2025

clt

Failed tests:

Uh oh!

tomatolog left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Nov 19, 2025

clt

Failed tests:

Uh oh!

github-actions bot commented Nov 20, 2025

clt

Failed tests:

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

PavelShilin89 commented Nov 10, 2025 •

edited

Loading