Skip to content

Make async service wind-down more robust #180

@mxgrey

Description

@mxgrey

The execution of async services is among the most complicated parts of this library. One issue which recently surfaced is that there is a small risk of a race condition in the reachability detection when a workflow relies on a stream coming out of an async service in order to reach the terminate operation. The race condition is this:

  1. If the async service sends a message out over its stream
  2. That streamed message is placed into the execution channel as an unresolved command
  3. The async service finishes and notifies its scope that it should test reachability
  4. Reachability is tested while the streamed message is still an unresolved command within the execution channel
  5. The reachability check fails because we don't realize that some unresolved command is going to eventually provide a new message to the scope

Currently we avoid this problem by only testing reachability after we've made sure the entire execution channel is drained of all commands. Theoretically this should work fine, but it's fragile. We may accidentally tweak the implementation of the flushing system without regard for this, or some other similar race condition may appear in the future that we haven't accounted for.

We should consider strategies to change how async services themselves are executed so that other parts of the code don't have to be sensitive to them. For example, we could track lifecycle of commands in the execution channel, and we only allow an async service to conclude after all its commands have been flushed from the channel and the channel itself is gone. This could also address #29 because we would be sure that the service is not seen as "finished" until the channel has dropped.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status

    Inbox

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions