Bug: Tester does not detect pending activity futures after workflow completion

[j-nowakowski]

Issue

We recently were trying to create a unit test to reproduce an issue in live code where a workflow was completed while a pending future was inflight, expecting the error workflow completed, but there are still pending futures. However we found that we could not reproduce the error when the future was an activity. This prevents us from writing tests which validate that we don't leave pending futures on accident. This occurs regardless of if the activity is mocked or not.

Workflows version v1.3.0.

Reproduce:

Create this file:

package main

import (
	"context"
	"testing"
	"time"

	"github.com/cschleiden/go-workflows/tester"
	wf "github.com/cschleiden/go-workflows/workflow"
	"github.com/stretchr/testify/require"
)

func TestFoo(t *testing.T) {
	wft := tester.NewWorkflowTester[any](WF)
	// wft.OnActivity(Act, mock.Anything).Return(nil)
	wft.Execute(t.Context())
	require.True(t, wft.WorkflowFinished())
	_, err := wft.WorkflowResult()
	require.ErrorContains(t, err, "workflow completed, but there are still pending futures")
}

func WF(ctx wf.Context) error {
	wf.ExecuteActivity[any](ctx, wf.DefaultActivityOptions, Act)
	return nil
}

func Act(context.Context) error {
	time.Sleep(10 * time.Second)
	return nil
}

Run command:

go test <file>

Expected result: the test is successful. Instead, the test reports no error returned from wft.WorkflowResult().

Thanks!

[cschleiden]

Thanks for reporting this! That's an interesting one.

One thing I never noticed before, scheduling an Activity internally uses a workflow.Go-routine for handling retries. The executor doesn't mark a workflow execution as completed as long as there is a single Go-routine still running, so we:

1. Start the workflow
2. Schedule the activity which internally
   a. Starts a Go-routine
   b. Which waits until we get any Activity completed/failed event
3. Workflow exits
4. We have one pending future
5. Go-routine from 2 is still running, executor doesn't mark the workflow as completed
6. Eventually activity completed/failed event comes in, pending future is removed
7. Go-routine exits
8. Executor marks the workflow as completed, there are no pending futures so no error

I do have a fix that marks a workflow that's being executed as completed as soon as it exits or panics and with that some of the existing tester tests are actually failing with a pending futures panic. Have to think about other implications a bit more