Build DB switch mechanism

[0xmovses]

Description

Before reading refer again to the L1 migration plan.

When we perform the migration, the current DB on movement branch (and all its data) need to be switched out to the latest upstream DB main. Aptos guarantees backward compatibility and this has been tested to some degree with a high success rate by @areshand.

We need to write the tooling / logic that will perform this switch, essentially upgrading the entire stateDB. This is the Phase 2 section of the L1 Migration Plan.

We then need to test it in three different environments

1. Locally (with some basic transaction data against a fresh local movement-full node main branch)
2. On a Testnet Mirror
3. On a Mainnet Mirror
4. On Real testnet

cc @radupopa369 & @nicholasflintwillow to support here on creating Mirror Environments.

Other steps should occur around before and after this DB upgrade, such as the framework upgrade. These steps are being done separately, if we have 🟢 for each of these steps we should be able to chain them into a final upgrade binary which will perform all the steps with one command. In the L1 migration plan, referred to as the meta-migrate binary. Another part of this is writing the necessary infrastructure code, to stop the old network, start the aptos-core full-node, run the meta-migrate binary / script. These infra and framework steps are not part of this issue.

1. Define the separate sub-tasks involved in creating this tool.
2. Implement the tool
3. Test the tool

Success Criteria

• Successful DB upgrade on the Mainnet Mirror
• Successful Replay of transactions post DB upgrade from blockheight 1 of movement network

[radupopa369]

Acknowledged @0xmovses . So as timeline we would like to have this by the end of next week.
As challenges for this task I see the need to stop all the services to take a snapshot of the data volume.
Also we need to see if we could get away with adding a new ec2 instance in the same aws environment (account) or we would want to create a new environment and a new aws account for each

[nicholasflintwillow]

Yeah we could definitely just add a new instance/instance type and deploy it. Just need to make sure it gets setup with new dns records.

[areshand]

To mirror the environment, just want to confirm if my understanding is correct:
1. We should have access to the latest Aptos binary, right?
2. When switching, I’ll stop the movement binary on the host, start the Aptos binary, and point it to the same DB folder that was used by the movement binary. During this time, are the transactions stored in the DA and will they be re-sent to the node once it comes back? cc: @musitdev

[musitdev]

The switch can be done in several way. Here you want run a full node and the Aptos node on the same host one after the other and share the same db files. So effectively, you can stop the fullnode and start the Aptos node. If the Aptos node can connect to the Da-sequencer, by getting from the fullnode da-db the last synced da height, you can get all missing blocks since the fullnode stopped.
So, to do that, you need to add to the Aptso node process:

• Read the da_db to get the last synced height
• Connect to the da-sequencer to get the new blocks and execute their Txs on the Aptos node. Perhaps, the easiest way is to submit the block Tx to the Aptos node that recreates a new block and executes it?

[l-monninger]

@areshand @0xmovses

You're duplicating work:

1. This "DB switch" is what https://github.com/movementlabsxyz/movement-migration is doing in abstract.
2. In situ, i.e., in the actual infrastructure this is `live movement` to `migrator movement` to `migrator movement-aptos` to `live movement-aptos` movement-migration#136 which I've been working on under the premise of K8s-based Migration movement-migration#146. I haven't quite figured out how to bring it all down into some kind of local environment, but the initial helm charts and IaC here:

• feat: mtma helm charts and minikube local development support movement-migration#148
• https://github.com/movement-infra/infrastructure-workspace/pull/134

[areshand]

@areshand @0xmovses

You're duplicating work:

1. This "DB switch" is what https://github.com/movementlabsxyz/movement-migration is doing in abstract.
2. In situ, i.e., in the actual infrastructure this is live movement to migrator movement to migrator movement-aptos to live movement-aptos movement-migration#136 which I've been working on under the premise of K8s-based Migration movement-migration#146. I haven't quite figured out how to bring it all down into some kind of local environment, but the initial helm charts and IaC here:

• feat: mtma helm charts and minikube local development support movement-migration#148
• https://github.com/movement-infra/infrastructure-workspace

In my mind, the switch should be one simple adhoc script. I am not sure why this involves k8s, the aptos binary and movement can live in the same pod. we can execute the script in the pod to do the switch? if there is already such a script I can reuse, please let me know.

[areshand]

Here are the things in my mind that we need to do for this task

Prepare

1. Turn-off the REMOVE_DETAILED_ERROR_FROM_HASH (48) flag on both mainnet and testnet
2. Modify the latest Aptos binary to have more permissive max gas limit and memory setting ([code](https://github.com/aptos-labs/aptos-core/blob/main/aptos-move/framework/table-natives/src/lib.rs#L454))

Switch

In the mirror the environment, assume we also have the latest Aptos binary.
we can have a simple script to

1. stop the existing movement binary
2. record the version and ledger info has as the waypoint https://github.com/aptos-labs/aptos-core/blob/main/types/src/waypoint.rs#L31
3. start the Aptos binary with config pointing to the db path
   1. TODO: figure out how not to lose any transaction during the switch

Validation

Comparing the db in the mirror environment with the mainnet DB. Since the gas used will be different, we cannot guarantee the balance is the same between the new aptos DB and maptos DB. But we should verify

1. only gas used shows difference between old (maptos db) and new db (aptos db) for all new transactions
   1. we can use the command [here](https://www.notion.so/Aptos-Binary-Migration-20f18675b2d780acae26fb82b4caefdb?pvs=21)
2. all state keys in old db exists in new db, all events matches, all transactions matches. refer to an example [here](https://github.com/aptos-labs/aptos-core/blob/main/storage/aptosdb/src/db_debugger/validation.rs#L149)

[l-monninger]

@0xmovses @areshand

Updated the comment above with a PR that I'm currently linking to submodule development: https://github.com/movement-infra/infrastructure-workspace/pull/134

[l-monninger]

@areshand @0xmovses

These steps are what https://github.com/movementlabsxyz/movement-migration is reasoning about. The idea is to handle this switch and validation within a checked-migration which is what the mtma CLI runs.

Prepare

1. Turn-off the REMOVE_DETAILED_ERROR_FROM_HASH (48) flag on both mainnet and testnet
2. Modify the latest Aptos binary to have more permissive max gas limit and memory setting ([code](https://github.com/aptos-labs/aptos-core/blob/main/aptos-move/framework/table-natives/src/lib.rs#L454))

1. Turning off the flag would land as an addendum to pre-l1-merge.
2. That would be an addendum to a node migration pass.

Switch

In the mirror the environment, assume we also have the latest Aptos binary. we can have a simple script to stop the existing movement binary record the version and ledger info has as the waypoint https://github.com/aptos-labs/aptos-core/blob/main/types/src/waypoint.rs#L31 start the Aptos binary with config pointing to the db path

I agree marking the waypoint at which the migration begins is essential to the live traffic switch. But, this script is effectively a node migration pass for which a modification to null would suffice

TODO: figure out how not to lose any transaction during the switch

This is where this does become a question of the surrounding infra IMO. That's why movementlabsxyz/movement-migration#136 is trying to clarify the assumptions that we have.

In the current solution I've begun under, the Cloudflare, IaC, and container placement, the traffic would be duplicated for a period sending both to the live (current) Movement network, i.e., live movement, and then also to the placement running mtma, i.e., migrator movement/movement-aptos . This is described in several places:

• https://github.com/movement-infra/infrastructure-workspace/pull/134
• https://github.com/movement-infra/cdktf-movement-aptos/tree/l-monninger/mirroring
• https://github.com/movement-infra/cdktf-movement-migration/tree/l-monninger/mirroring

Validation

Comparing the db in the mirror environment with the mainnet DB. Since the gas used will be different, we cannot guarantee the balance is the same between the new aptos DB and maptos DB. But we should verify only gas used shows difference between old (maptos db) and new db (aptos db) for all new transactions
we can use the command [here](https://www.notion.so/Aptos-Binary-Migration-20f18675b2d780acae26fb82b4caefdb?pvs=21)
all state keys in old db exists in new db, all events matches, all transactions matches. refer to an example [here](https://github.com/aptos-labs/aptos-core/blob/main/storage/aptosdb/src/db_debugger/validation.rs#L149)

Here are the things we are checking currently that run as an ~epilogue to the whatever migration in mtma: https://github.com/movementlabsxyz/movement-migration?tab=readme-ov-file#checks

mtma is intended to enable programmatic cutover--whether that's via some control plane, simply ending as a script that exit 0 or a K8s job , or something else.

It has additionally been suggested that we will want to track some criteria over time and that mtma can be a long-running program that validates. For example,

movementlabsxyz/movement-migration#135
More recently a concrete possibility of running a structured tracking validation period has been raised.

This more structured tracking validation period could potentially programmatically cut over traffic, if so desired.

[0xmovses]

Can you link to the line(s) of code in the movement-migration repo where a DB Switch occurs? (Not the tooling or capability that enables this, but the actual place where it happens) I asked the team this on standup and no one could show me where this code was. What we do have is a way to run two executors side by side. An Aptos Executor and a Movement Executor.

The CLI seems overly complex for what our goal ultimately is. That is a oneshot, one command script.

[l-monninger]

Can you link to the line(s) of code in the movement-migration repo where a DB Switch occurs? (Not the tooling or capability that enables this, but the actual place where it happens) I asked the team this on standup and no one could show me where this code was. What we do have is a way to run two executors side by side. An Aptos Executor and a Movement Executor.

The CLI seems overly complex for what our goal ultimately is. That is a oneshot, one command script.

@0xmovses

https://github.com/movementlabsxyz/movement-migration/blob/239d33ed12825b49752730f7672be0386786d216/migration/core/node/mtma-null/src/migrate/migrate.rs#L44 is where this happens in the migration.

You can get there by clicking this header in the main README.

Note

This uses a blanket implementation here to produce the Migrator struct in the Migratorstrategy.

There is also snapshot() for going to a Migrator of the same type introduced as part of the box environment API.

That is a oneshot, one command script.

At some point, I'm just not going to care what you guys do. But, the reason this is written has been explained numerous times from the start and is summarized in the video linked in "Getting Started".

I would and have also argued that "oneshot" is not a clear descriptor. You will likely have to run and debug this multiple times even within one environment.

Additionally, you may want to drop in and check a particular set of correctness criteria, whether as an internal or external user. So, why not structure the logic to make this easy?