validator_operator/validator_upgrades

[giscus[bot]]

validator_operator/validator_upgrades

https://docs.dev.sync.global/validator_operator/validator_upgrades.html

[waynecollier-da]

Reposting for Chad Clark [email protected]

from: https://daholdings.slack.com/archives/C08AP9QR7K4/p1743615372089649

We did an upgrade in DevNet from 0.3.15 to 0.3.18. The validator-app failed to start. The logs showed a problem with the ONBOARDING_SECRET. The secret in kubernetes was not updated and was the old secret from the previous install.

The operation 'request onboarding' failed with a non-retryable HTTP error, not retrying: 
org.lfdecentralizedtrust.splice.admin.api.client.commands.
HttpCommandException: HTTP 401 Unauthorized POST at '/api/sv/v0/onboard/validator' on sv.sv-1.dev.global.canton.network.sync.global. 
Command failed, message: Unknown secret.

We got a new secret and the upgrade proceeded fine. (Using https://sv.sv-1.dev.global.canton.network.sync.global/api/sv/v0/devnet/onboard/validator/prepare )

I thought previously the ONBOARDING_SECRET was not used during the upgrade process. Going forward, do we need to request a new secret before upgrading? What about MainNet? Thanks!

[isegall-da]

A new secret should not be required when upgrading. This sounds like you deleted your DBs during your upgrade, so your node tried to re-onboard. All DBs should be kept. If you're self-hosting the DB, make sure not to uninstall it.

We will make this explicit in the docs.

[Chad-Chata]

The databases were not removed. Oh! We updated the MIGRATION_ID from 0 to 1 but did not set migrating true in splice-node/examples/sv-helm/standalone-validator-values.yaml. That's probably the problem.

[waynecollier-da]

Remember to change the MIGRATION_ID only for major upgrades with downtime. For minor versions, leave this untouched.

[Brindrajsinh-Chauhan]

Slack thread: https://kaleidoio.slack.com/archives/C08AP9QR7K4/p1743624413074379

Is there a way to use the docker compose based super validator to mock a migration schedule to test out a few things and see it working in a controlled env?

[isegall-da]

Nice idea. I'll create an issue to write it down more thoroughly.
For now:
To trigger a pause, you'll need to expose the SV UI, log into it, and create a vote to pause the synchronizer. Since you're the only SV, the vote will immediately go through.
To actually complete a migration would be more involved, as you'll need to also migrate the SV correctly.