validator_operator/validator_major_upgrades

[giscus[bot]]

validator_operator/validator_major_upgrades

https://docs.dev.sync.global/validator_operator/validator_major_upgrades.html

[branciard]

What does "fresh participant" means ?

fresh participant (a fresh participant needs to be deployed to upgrade across non-backwards-compatible changes to the Canton software)

What is the workflow when migration on major upgrade ?:

• check the "auto" dump well generated
• wait for new charts
• "fresh participant" ? use a new DB or drop content of existing ?
• start only once with migrating=true and new migration id. The initial "auto" dump is used to keep existing participants ? as said :

Upon (re-)initialization, the validator backend automatically consumes the migration dump and initializes the validator participant based on the contents of this dump.)

• check it is ok. how ?
• remove the migrating flag: migrating=false ?
• restart all ( migrating=false but with new migration id )

[julientinguely-da]

A fresh participant is a completely new participant node with a clean and empty database. It's required for major upgrades.

The migration workflow can be summarized in these steps:

• Create a dump file that exports all essential data from your current running participant (parties, contracts, etc.). Check for the Wrote domain migration dump log to ensure it has been created.
• Deploy the new Canton version using a fresh participant. Start it for the first time with a migrating=true flag but keep the incremented migration_id. This tells the new participant to load the state from your dump file into its new database.
• Remove the migrating=true flag but keep the new migration_id. It will now run normally with all your data restored on the new node.
• You can check that it worked successfully by checking through the Ingested transactions logs.

Hope that answers your questions.

[derrix060]

Note that if you can't find the Wrote domain migration dump logs, go inside the container where the validator-app is running and check if it does have a migration dump and the dump :

[2025-06-11 18:58:23] ~  $ kubectl exec -it validator-app-5574fb4677-v68zf -- bash
Defaulted container "validator-app" out of: validator-app, validator-init (init)
root@validator-app-5574fb4677-v68zf:/app# ls -lha /domain-upgrade-dump
total 31M
drwxr-xr-x 3 root root 4.0K Jun 11 13:01 .
drwxr-xr-x 1 root root 4.0K Jun  9 11:39 ..
-rw-r--r-- 1 root root  31M Jun 11 13:01 domain_migration_dump.json
drwx------ 2 root root  16K May 19 11:12 lost+found

Please take a note on the day that the migration_dump was created, in case you have migrated more than once before :)

[stas-sbi]

To ensure that the validator participant has caught up and the migration dump has been created as expected, operators can check the logs of the validator-app pod for Wrote domain migration dump messages.

To ensure that the validator app has caught up, operators can check the logs of the validator-app pod for the message Ingested transaction. If the latest such message is 10 or more minutes old, the validator app has very likely (with a large safety margin) caught up to the state on the participant, and hence to the state of the existing (paused) synchronizer.

Would a validator create a dump before it has caught up? If no then the check for "Ingested transaction" before migrating seems redundant and should probably be removed to reduce complexity of the procedure

[moritzkiefer-da]

Would a validator create a dump before it has caught up? If no then the check for "Ingested transaction" before migrating seems redundant and should probably be removed to reduce complexity of the procedure

There are two levels of caught up: Participant is caught up and validator is caught up. The validator waits for the participant before creating the dump but not for its own stores to catch up at this point.