Move to Redis lua scripts

[cschleiden]

This pull request refactors how workflow and activity events are persisted and queued in the Redis backend by replacing multi-step Go-side pipelining with new atomic Lua scripts. The changes improve consistency, reliability, and transactional safety when handling workflow events, activity completions, instance cancellations, and signals. The update also removes now-unnecessary helper methods and streamlines event marshaling.

Fixes: #440

[cschleiden]

!bench !large

[Copilot]

Pull Request Overview

This pull request refactors the Redis backend to use atomic Lua scripts for workflow and activity event operations, replacing Go-side pipelining with server-side execution. This improves transactional safety and consistency when persisting events, completing activities, canceling instances, and handling signals.

Key changes:

• Introduces four new Lua scripts for atomic operations: signal_workflow.lua, delete_instance.lua, complete_activity_task.lua, and cancel_workflow_instance.lua
• Removes helper methods addWorkflowInstanceEventP, addEventPayloadsP, and addEventToStreamP that are now replaced by Lua scripts
• Standardizes event marshaling by consistently using marshalEvent instead of splitting marshaling logic

Reviewed Changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
backend/redis/workflow.go	Updated CompleteWorkflowTask to use unified marshalEvent helper and removed deprecated addWorkflowInstanceEventP method
backend/redis/signal.go	Replaced pipelined operations with atomic signalWorkflowCmd Lua script execution
backend/redis/scripts/signal_workflow.lua	New Lua script for atomically adding signal events and queueing workflow tasks
backend/redis/scripts/delete_instance.lua	New Lua script for atomic instance deletion
backend/redis/scripts/complete_activity_task.lua	New Lua script combining activity completion, event addition, and workflow queueing
backend/redis/scripts/cancel_workflow_instance.lua	New Lua script for atomic workflow cancellation
backend/redis/redis.go	Updated script loading to include new Lua scripts and removed obsolete script preloading logic
backend/redis/instance.go	Refactored CreateWorkflowInstance and CancelWorkflowInstance to use marshalEvent and new Lua scripts
backend/redis/events.go	Removed deprecated helper methods replaced by Lua scripts
backend/redis/delete.go	Migrated from inline script to external deleteInstanceCmd Lua script
backend/redis/activity.go	Replaced pipelined activity completion with atomic completeActivityTaskCmd Lua script

[Copilot]

Inconsistent error handling: Line 24 uses redis.pcall (which suppresses errors) while lines 27, 30, and 32 use redis.call (which propagates errors). Since payload storage is critical for event processing, use redis.call here for consistent error propagation.

Suggested change

	redis.pcall("HSETNX", payloadHashKey, eventId, payload)
	redis.call("HSETNX", payloadHashKey, eventId, payload)

[Copilot]

Inconsistent error handling: Line 12 uses redis.pcall (which suppresses errors) while lines 15, 18, and 20 use redis.call (which propagates errors). Since payload storage is critical for event processing, use redis.call here for consistent error propagation.

Suggested change

	redis.pcall("HSETNX", payloadHashKey, eventId, payload)
	redis.call("HSETNX", payloadHashKey, eventId, payload)

[Copilot]

Inconsistent error handling: Line 33 uses redis.pcall (which suppresses errors) while other Redis operations use redis.call (which propagates errors). Since payload storage is critical for event processing, use redis.call here for consistent error propagation.

Suggested change

	redis.pcall("HSETNX", KEYS[4], ARGV[3], ARGV[5])
	redis.call("HSETNX", KEYS[4], ARGV[3], ARGV[5])

[github-actions]

Benchmark Results

SQLite run

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
sqlite-main	249.7 ± 27.1	220.2	310.8	1.06 ± 0.16
sqlite-pr	236.6 ± 25.5	200.8	284.7	1.00

Large SQLite payload run (1MB)

Command	Mean [s]	Min [s]	Max [s]	Relative
sqlite-main	5.358 ± 0.417	4.685	6.055	1.02 ± 0.10
sqlite-pr	5.230 ± 0.280	4.948	5.744	1.00

Redis run

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
redis-main	178.1 ± 28.2	144.7	238.8	1.07 ± 0.20
redis-pr	165.7 ± 17.3	144.2	199.7	1.00

Large Redis payload run (1MB)

Command	Mean [s]	Min [s]	Max [s]	Relative
redis-main	3.992 ± 0.424	3.357	4.672	1.00
redis-pr	4.156 ± 0.442	3.330	4.657	1.04 ± 0.16