Async thunks in transaction builder

[hayes-mysten]

What are we adding

Recently, we introduced "transaction thunks", which look like this:

tx.add((tx) => { /** do stuff with tx **/ })

This pattern allows sdks to add give users functions that add commands to transactions without having a reference to that transaction first.

Therefore, an sdk could build something like this:

// SDK
const mintNewThing = () => (tx) => { 
  return tx.moveCall({
    target: '0xabc::thing::mint',
    argument: [tx.object(WELL_KNOWN_OBJECT)]
  })
}

// User
tx.add(mintNewThing())

We've found this pattern to improve type safety, and we believe it makes it easier to understand how transactions ultimately get constructed.

We would like to extend this to allow "async transaction thunks", which would allow developers to add functionality to transactions like this:

// NEW: This returns an `async` function!
const mintNewThing = () => async (tx) => {
  const objectId = await lookupObjectId()
  return tx.moveCall({
    target: '0xabc::thing::mint',
    argument: [tx.object(objectId)]
  })
}

// User still SDK synchronously during construction
tx.add(mintNewThing())

Why

Some SDKs need async functionality to build transactions (for example, loading data from the chain to use as a transaction input), but we want to push SDKs towards preserving composability and compatibility with existing transaction building patterns.

The current approach to this is to have SDKs expose async methods that need to be awaited while building transactions:

// SDK
const mintNewThing = async () => {
  const objectId = await lookupObjectId()
  return (tx) => tx.moveCall({
    target: '0xabc::thing::mint',
    argument: [tx.object(WELL_KNOWN_OBJECT)]
  })
}

// User
tx.add(await mintNewThing())

This patterns works, but presents 2 problems:

1. Without extra orchestration on the users end, all async work becomes serial, causing slower transaction builds, because work can't happen in parallel.
2. Asynchronous transaction construction causes problems for web based wallets:
   • Web based wallets work by opening a popup window when the transaction the wallet is asked to sign a transaction
   • Browsers will block this popup if it is not in response to a users action (eg, a button click)
   • If we asynchronously construct a transaction after a user action, the popup window won't open until the transaction is constructed, and the browser might block the popup.

More generally, async functions are viral, so once one part of the call stack needs to be awaited, every parent function must also become async. Our goal with Transaction construction is to avoid any forced async logic in the critical transaction building flow.

Solution

To solve this, we can accept "async thunks" in the transactions add method. To make this usable, we will need synchronously return a result that can be used in subsequent transaction commands:

const newThing = tx.add(mintNewThing())
tx.transferObjects([newThing], address)

We previously added support for similar patterns through a feature called "transaction intents" that work by adding a placeholder command to the transaction, and later replacing that command through a transaction plugin that runs when the transaction is built.

This solves parts of this problem very well, but there is an additional complication: when you have multiple async tasks happening at the same time, execution become non-deterministic. This is a big problem for our transaction builder because it uses indexes as references to command results and indexes. If we just let our async functions execute, they will all race to add commands and inputs to the transaction, and we won't be able to consistently order those commands. The intuitive solution here would be to schedule the async tasks so that they run one at a time, but this has 2 problems: We want to do work in parallel so that things can build as fast as possible, and we don't actually know if a function is async until after we start executing it.

Implementations

We currently have 2 potential implementations for how this can work:

Transaction forking

This solution had a number of issues we discovered during testing and is unlikely to be used

We can pass these async thunks a fork/clone of the transaction. This allows the async function to add/mutate the forked transaction how ever it wants, and we can then deterministically merge the results of all async thunks after they finish executing.
This merging isn't trivial, and we will need to combine inputs, and update references throughout the transaction, but a lot of this was already implemented for transaction intents.
When adding async thunk to a transaction the flow is something like this:

1. We fork the current transaction creating a new copy of all the transaction state
2. We start executing the thunk that was added to the transaction
3. If it returns synchronously:
   1. We commit to the fork, and attach that forks data back to the original transaction
   2. Normal transaction building resumes
4. If it returns a promise
   1. We add a AsyncThunk intent to the transaction as a placeholder
   2. We create and return a placeholder result proxy
   3. Normal/synchronous transaction building resumes
5. When the promise resolves, we wait until all previous async thunks are fully resolved
   1. Then we merge the fork back into the main transaction
   2. pure inputs are just added as new inputs to the transaction
   3. object inputs are added via tx.object which will deduplicate them
   4. After all the inputs are added, we map over all the commands of the forked transaction, and update them to reference the new input indexes
   5. Then we find the newly added commands and replace the placeholder with the new commands
   6. this process will also update all the references to other commands, who's indexes might have changed
6. When we build a transaction, we await the last async thunk being resolved before proceeding with the normal build process

This solution has been implemented here: #195 but has some problems.

With this solution, if you mix inputs/results from one fork with another fork, correctly reconciling the proper order when merging becomes very difficult.

Transaction Sections

Rather than fully forking transactions and merging them back together, we can create proxies/copies of the transaction that mutate the same underlying data, but track which proxy commands and inputs were inserted with. This allows us to re-order the transaction before build to ensure that commands and inputs are added in a deterministic and expected order.

This would work something like:

1. Transactions have a new mechanism for tracking a current "section" of the transactions, and we track what the current section was for each command and input that was added.
2. When tx.add is called, we create a new sub-section and a proxy for the transaction. Anything added in the tx.add thunk will be added to this new section
3. Like the solution above, we insert a placeholder into the transaction so we have a useable result to use in subsequent calls before the async thunk returns
4. We also create a new section on the base transaction for anything added to transaction after tx.add was called.
5. During transaction construction, all inputs and commands are just appended to the transaction, and the order is non-deterministic
6. Before building the transaction, we wait for all tx.add thunks to resolve, and then we re-order the inputs and commands based what the current section was when those commands where added.

This solution is implemented here: #198