workflow.Go cubic performance degradation in workflow tester

[jacobnowakowski1]

Hi all, we encountered this issue when trying to spin up 256 "goroutines" in a workflow and saw significant performance degradation in our unit test.

Recreate

Using go-workflows version v0.19.0, create the following test file:

package foo

import (
	"context"
	"fmt"
	"testing"

	"github.com/cschleiden/go-workflows/tester"
	"github.com/cschleiden/go-workflows/workflow"
	"github.com/stretchr/testify/mock"
	"github.com/stretchr/testify/require"
)

func WF(ctx workflow.Context, n int) error {
	wg := workflow.NewWaitGroup()
	for range n {
		wg.Add(1)
		workflow.Go(ctx, func(ctx workflow.Context) {
			defer wg.Done()
			workflow.ExecuteActivity[any](ctx, workflow.DefaultActivityOptions, Act).Get(ctx)
		})
	}
	wg.Wait(ctx)
	return nil
}

func Act(ctx context.Context) error {
	return nil
}

func BenchmarkWorkflowGo(b *testing.B) {
	run := func(b *testing.B, n int) {
		b.Run("n="+fmt.Sprint(n), func(b *testing.B) {
			for i := 0; i < b.N; i++ {
				wft := tester.NewWorkflowTester[any](WF)
				wft.OnActivity(Act, mock.Anything).Return(nil)
				wft.Execute(context.Background(), n)
				require.True(b, wft.WorkflowFinished())
				_, err := wft.WorkflowResult()
				require.NoError(b, err)
			}
		})
	}
	run(b, 1)
	run(b, 31)
	run(b, 51)
	run(b, 71)
	run(b, 91)
	run(b, 111)
	run(b, 131)
	run(b, 151)
	run(b, 171)
	run(b, 191)
	run(b, 211)
}

Issue

I observe that the performance of this simple unit test degrades significantly as the number of workflow.Go calls get made, as the graph below shows:

Out of curiosity, I took the log/log of the data and found it formed a linear function of slope close to 3, indicating a runtime performance of O(n^3).

Is this expected? With this type of degradation, the workflow.Go functionality has limited use cases for us. Thank you!

[lyuboxa]

@jacobnowakowski1 Did you use a single worker or multiple? Thanks!

[jacobnowakowski1]

@lyuboxa This behavior was only observed in the context of the unit test, but your question makes me realize I should check if this is observed with a real backend + worker. Let me spin that up and collect results.

[jacobnowakowski1]

I tested with a local backend using the following code:

func BenchmarkWorkflowsGo(b *testing.B) {
	run := func(b *testing.B, n int) {
		b.Run("n="+fmt.Sprint(n), func(b *testing.B) {
			ctx, cancel := context.WithCancel(context.Background())

			be := sqlite.NewInMemoryBackend()
			var wg sync.WaitGroup
			b.Cleanup(func() {
				cancel()
				wg.Wait()
				require.NoError(b, be.Close())
			})
			wo := worker.DefaultOptions
			wo.MaxParallelActivityTasks = 256
			wo.MaxParallelWorkflowTasks = 256
			wo.ActivityWorkerOptions.MaxParallelActivityTasks = 256
			wo.WorkflowWorkerOptions.MaxParallelWorkflowTasks = 256
			wo.WorkflowExecutorCacheSize = n * 4
			wo.WorkflowExecutorCacheTTL = 10 * time.Minute
			numWorkers := 1
			for range numWorkers {
				wg.Add(1)
				go func() {
					defer wg.Done()
					w := worker.New(be, &wo)
					require.NoError(b, w.RegisterWorkflow(WF))
					require.NoError(b, w.RegisterActivity(Act))
					require.NoError(b, w.Start(ctx))
				}()
			}

			c := client.New(be)
			b.ResetTimer()
			for i := 0; i < b.N; i++ {
				wf, err := c.CreateWorkflowInstance(ctx, client.WorkflowInstanceOptions{
					InstanceID: "my-workflow",
				}, WF, n)
				require.NoError(b, err)
				require.NoError(b, c.WaitForWorkflowInstance(ctx, wf, 10*time.Minute))
				b.StopTimer()
				_, err = client.GetWorkflowResult[any](ctx, c, wf, 10*time.Minute)
				require.NoError(b, err)
				b.StartTimer()
			}
		})
	}

	for i := range 15 {
		run(b, int(math.Pow(2, float64(i))))
	}
}

The numWorkers var can be tweaked in the code. The results here are more aligned with expectation: The performance stays about constant till about 2000 goroutines. At this point we are getting close to our order of magnitude for maximum number of events per workflow of 50,000 anyway, so I'm not concerned about performance degradation past that. The degradation is likely due to the backend in any case, not due to the Workflow code using workflow.Go. Since I was running this locally, changing the number of workers didn't actually change the amount of compute, so adding more just added more overhead.

With these findings, I'm going to change the title of this issue to indicate that this is an issue when using the tester package. It took longer for the tester package to test a WF with 256 goroutines than a WF with a real backend!

[cschleiden]

Thanks for the great report and the analysis! I'll take a look