Where can I find an example with a database

[sh1man]

If you don't mind, can you write pseudo code for my understanding.

I saw this text in one of the examples, I'm curious about it.
https://github.com/hatchet-dev/hatchet-python-quickstart/blob/main/fast-api-react/backend/src/api/main.py

in a normal application you might use the message id to look up a workflowRunId
    for this simple case, we have no persistence and just use the message id as the workflowRunId

    you might also consider looking up the workflowRunId in a database and returning the results if the message has already been processed

[szvsw]

The solution described in that comment is a standard approach to managing asynchronous workloads which need to be persistent, rather than ephemeral.

Let's say that my asynchronous workflow is some sort of long running simulation- the user submits inputs, and ultimately wants to be able to get the outputs of that long running simulation. The users don't have access to the actual hatchet dashboard/database since this is a commercial application and this is hidden infrastructure.

First, consider the situation without a database.

1. User sends payload for processing.
2. API receives payload, and schedules a task in hatchet; hatchet creates a workflow run id.
3. API returns id to user.
4. User checks up on the run id periodically to see if it's finished.
5. Workflow run finishes
6. User ultimately gets the data out by using that run id.

However, what happens if, during step 4, the user's computer restarts? Or their browser crashes? Unless they wrote down that run id, or persistently stored it somehow on their own, they would never be able to re-locate that workflow run and recover its data (besides contacting me, the application maintainer, describing their inputs to me, and hoping I am supportive enough to go through the list of all runs and find ones that have identical inputs).

The comment in the code then is referencing the common requirement to persist an association between the workflow run id, and some other entity that might want to relocate that workflow run at some arbitrary point in the future. For instance, suppose there is a user account/auth system backed by a database; this database might include a table which has two columns - one for user id, and one for workflow run ids. Maybe the users also submit a name (possibly compound-unique on user id and workflow name) for each simulation so it's easy for them to identify it in the future. This table is used so that you can store and look up all the workflow runs associated with a particular user even if they exit the application/browser/logout, restart their computer, etc etc.

The sequence might then look something like this.

1. User submits payload (which maybe also includes a human readable name)
2. API authenticates user, then schedules workload with hatchet, getting a workflow run id
3. API writes to the database table which tracks user/workflow associations a new record tying this workflow run id to the current user, along with the name they gave it
4. API could return nothing whatsoever, or maybe a success message, or maybe a the workflow run id & the name, etc.
5. At some point in the future, the user can log in, and query for all their workflow run ids by sending a request to the API, which in turn authenticates them and queries the database for all workflows associated with that user.
6. The user can now individually check the status of those workflows, or maybe the api checks them automatically, or maybe there is a cron job which automatically collects results of finished tasks and pushes them to the main application database and associates them with the relevant user etc.

Obviously there are a million different ways to go with how you implement all of this according to what your needs are, but that should make the basic idea clear - if you don't persist your workflow run id, the IDs become ephemeral (in the sense that you could only really recover a specific workflow in the future by going through your hatchet workflow runs iteratively checking them one by one).

Hopefully that clarifies what those comments were alluding to!