Why can't `send` be executed from within steps?

[nate-getea]

The check at

dbos-transact-py/dbos/_core.py

Line 1240 in 876eafa

assert ctx.is_workflow(), "send() must be called from within a workflow"

seems to block the use of DBOS.send and its async variant from within workflow steps. As the method can be called from outside the DBOS application anyways using a client, what is the purpose of this limitation?

[apoliakov]

Hi @nate-getea

This is because a "send" itself is checkpointed in the database and, on workflow recovery, it is not repeated if already executed. It is also implemented as a database transaction that should not be re-attempted after success. In other words, send is "a special kind of step." Executing it inside another step would make the checkpointing behavior tricky.

Does that make sense?

[nate-getea]

Hi @nate-getea

This is because a "send" itself is checkpointed in the database and, on workflow recovery, it is not repeated if already executed. It is also implemented as a database transaction that should not be re-attempted after success. In other words, send is "a special kind of step." Executing it inside another step would make the checkpointing behavior tricky.

Does that make sense?

That makes sense yeah, though I would have liked to have it just work inside a step, perhaps with the underlying implementation using a client in that case.

[apoliakov]

Yeah I agree it would be convenient but notice that retryability would be tricky, even if we used the client.

Imagine this scenario:

@DBOS.step(retries_allowed=True, max_attemps=3)
def run_step():
   DBOS.send(x, "message")
   run_api_call_y()

Imagine our send succeeded (as a database transaction it usually does) but our run_api_call_y failed. Now we go to retry the step. But re-sending "message" to x again is certainly wrong.

[nate-getea]

Yeah I agree it would be convenient but notice that retryability would be tricky, even if we used the client.

Imagine this scenario:

@dbos.step(retries_allowed=True, max_attemps=3)
def run_step():
DBOS.send(x, "message")
run_api_call_y()
Imagine our send succeeded (as a database transaction it usually does) but our run_api_call_y failed. Now we go to retry the step. But re-sending "message" to x again is certainly wrong.

In the scenario you describe wouldn't it be enough to require the use of an idempotency key?

[apoliakov]

Theoretically - yes but we'd have to make room for one throughout the schema and make a number of code changes. We'd have to store the key in the system database next to each "send" and then add logic to check for it. What's trickier is we'd have to ask the user for the key for each send() call. If the user doesn't provide the key, we might need to auto-generate it reliably, but differentiating between step retries versus legitimate "multiple sends in a loop" would be tricky. Overall, it seems possible but not completely trivial.

Are you asking out of curiosity or is there a reason why it's difficult to call "send" in between steps for your case? If you can share an example, maybe we can offer some tips.