Recover from Redis restart during workflow exection

[enchobelezirev]

Description of the issue

During the execution of a workflow, when Redis instance is restarted, then the Activity cannot be completed successfully.
The error that is reported is related to NOSCRIPT error: "NOSCRIPT No matching script. Please use EVAL."

Error is returned from the worker/activity.go:49

The issue is happening when the Activity execution has been started and during the Activity execution(before completion), the Redis is restarted.

Tried with Redis Cluster and Redis Sentinel and in both situations, the error is the same.

Explanation of the setup

• Library version: 1.3.0
• Redis client used v9
• Backend: Redis
• Experimented with Redis Cluster and Redis Sentinel

Result

The library cannot proceed with the execution of the given workflow in which the error has occurred and continue to fail with the error: "NOSCRIPT No matching script. Please use EVAL."

If, however, the application that is holding the worker is restarted then the error is:

panic: interface conversion: interface {} is nil, not string

goroutine 30 [running]:
github.com/cschleiden/go-workflows/backend/redis.(*redisBackend).GetWorkflowTask(0xc000ebe390, {0x51a1fa0, 0xc000496700}, {0xc00074a060, 0x2, 0x2})
	/go/pkg/mod/github.com/cschleiden/go-workflows@v1.3.0/backend/redis/workflow.go:71 +0x9eb
github.com/cschleiden/go-workflows/internal/worker.(*WorkflowTaskWorker).Get(0x51a1f30?, {0x51a1fa0?, 0xc000496700?}, {0xc00074a060?, 0xc000cd4f50?, 0x11404a65d728?})
	/go/pkg/mod/github.com/cschleiden/go-workflows@v1.3.0/internal/worker/workflow.go:150 +0x2a
github.com/cschleiden/go-workflows/internal/worker.(*Worker[...]).poll(0x4, {0x51a1f30?, 0xc000622370?}, 0x7580ef?)
	/go/pkg/mod/github.com/cschleiden/go-workflows@v1.3.0/internal/worker/worker.go:237 +0xa6
github.com/cschleiden/go-workflows/internal/worker.(*Worker[...]).poller(0x51ca540, {0x51a1f30, 0xc000622370})
	/go/pkg/mod/github.com/cschleiden/go-workflows@v1.3.0/internal/worker/worker.go:124 +0x1a8
created by github.com/cschleiden/go-workflows/internal/worker.(*Worker[...]).Start in goroutine 1
	/go/pkg/mod/github.com/cschleiden/go-workflows@v1.3.0/internal/worker/worker.go:81 +0xc5

[enchobelezirev]

As far as I see, there is a known issue redis/go-redis#3228

Is there a way to checkout the problem here?
@cschleiden

[cschleiden]

Ah, that's an interesting scenario that is not handled today. All scripts are loaded eagerly when a Redis backend instance is started, but if Redis itself is restarted while the backend instance is active, that's not handled.

Could handle the NOSCRIPT error and eagerly load the scripts again 🤔 Have to look into this a bit.