DF crashes with --lua_auto_async=true

[heikomat]

Describe the bug
Dragonfly crashes when executing scripts with the --lua_auto_async=true flag active.
Without the flag everything runs fine.

To Reproduce

1. Start dragonfly with this command:

   dragonfly --cache_mode=false --lua_auto_async=true --table_growth_margin 0.1
   

2. optionally Load the script
3. execute the script with eval/evalsha

Expected behavior
I Expect the script to run just like without the lua_auto_async flag, but faster.

Environment (please complete the following information):

• OS: MacOS 26.2
• Kernel: Darwin MacBook-Pro-von-Heiko.local 25.2.0 Darwin Kernel Version 25.2.0: Tue Nov 18 21:07:05 PST 2025; root:xnu-12377.61.12~1/RELEASE_ARM64_T6020 arm64
• Containerized?: Docker Compose
• Dragonfly Version: tested on 1.35.1 and 1.36.0

Reproducible Code Snippet
Sorry, i have not yet been able to make it a simple reproducable snippet

Additional context
I tried manually replacing the couple of calls with acalls, but acall seems to have a different return value. I was unable to find documentation around acall and apcall. if i just replace call with apcall the script errors with this:

attempt to perform arithmetic on a function value

for this line:

actually_deleted_rules = actually_deleted_rules + redis.acall("HDEL", account_key, table.unpack(rules_to_delete))

---

by default there is no crash output from DF. When i add the flags --vmodule=main_service=1,set_family=2, i get this crash report:

*** SIGSEGV received at time=1769763295 on cpu 0 ***
PC: @     0xaaaac350f134  (unknown)  util::fb2::detail::Scheduler::Preempt()
    @     0xaaaac39436c4        464  absl::lts_20250512::AbslFailureSignalHandler()
    @     0xffffa12e37a0       4960  (unknown)
    @     0xaaaac2c90570        224  util::fb2::EventCount::await<>()
    @     0xaaaac2d5fcc8        176  dfly::ScriptMgr::UpdateScriptCaches()
    @     0xaaaac2d634dc        224  dfly::ScriptMgr::Insert[abi:cxx11]()
    @     0xaaaac2d18cbc        656  dfly::Service::Eval()
    @     0xaaaac333bff0        208  dfly::CommandId::Invoke()
    @     0xaaaac2d0fc8c         80  dfly::Service::InvokeCmd()
    @     0xaaaac2d10a94        512  dfly::Service::DispatchCommand()
    @     0xaaaac34bb150        416  facade::Connection::DispatchSingle()
    @     0xaaaac34bb61c        304  facade::Connection::ParseRedis()
    @     0xaaaac34bc26c       1632  facade::Connection::IoLoop()
    @     0xaaaac34bc8ec        272  facade::Connection::ConnectionFlow()
    @     0xaaaac34bdb64        368  facade::Connection::HandleRequests()
    @     0xaaaac3520bc4        560  util::ListenerInterface::RunSingleConnection()
    @     0xaaaac35211c0        192  boost::context::detail::fiber_entry<>()

---

I use ioredis with ioredis.defineCommand. This means it usually uses EVALSHA, except for when the script has not been loaded yet, then it uses EVAL.

---

This is the script i'm trying to execute. In high-load scenarios it is called with up to 6k accounts and 400-500 entities, resulting in about 2-3 million hash-entries to be created

---@diagnostic disable: undefined-global
-- KEYS[1..M] = per-account keys (BR<accountId1>, IBR<accountId1>, BR<accountId2>, IBR<accountId2>, ...)
--
-- ARGV[1] = invalidateInfoTTL (seconds, e.g. "60"; "0" to skip)
-- ARGV[2..] = BR-Cache keys invalidate, one string per account:
--   format: "field␟field␟field..."
--
-- Return: deletedCount

local invalidate_info_ttl = tonumber(ARGV[1])

local separator = string.char(0x1f) -- ChatGPT is very sure that this is the best unit separator ever
local account_count = #KEYS / 2

-- check if the correct number of arguments are provided
if #ARGV ~= 1 + account_count then
  return redis.error_reply("ARGV must be: invalidateInfoTtl, then exactly one blob per account")
end


-- Determine the timestamp that the rules where invalidated at (aka now)
local redis_time = redis.call("TIME")
local now_us = redis_time[1] * 1000000 + redis_time[2]

local actually_deleted_rules = 0

-- parses the string "field␟field␟field"
-- into the arary [field, field, field]
local function parse_fields(blob)
  local fields = {}
  local blen = #blob
  local pos = 1

  while pos <= blen do
    local next_sep = string.find(blob, separator, pos, true)
    local token_end = next_sep and (next_sep - 1) or blen
    fields[#fields + 1] = string.sub(blob, pos, token_end)
    pos = next_sep and (next_sep + 1) or (blen + 1)
  end

  return fields
end

for i = 1, account_count do
  -- Name of the hash that contians the cached results for the current account
  -- Format: BR<accountId>
  local account_key = KEYS[2*i - 1]
  
  -- Name of the hash that contians the timesstamps for recently invalidated
  -- rules for the current account. Format: IBR<accountId>
  local invalidated_account_key = KEYS[2*i]

  -- names of rules to delete for the current account.
  -- Format: "field␟field␟field..."
  local deleted_rules_for_account = ARGV[1 + i] 

  -- For each account: delete the rules and record when the deletion happened
  if deleted_rules_for_account and #deleted_rules_for_account > 0 then
    local rules_to_delete = parse_fields(deleted_rules_for_account)

    -- Fast path: one field
    if #rules_to_delete == 1 then
      actually_deleted_rules = actually_deleted_rules + redis.call("HDEL", account_key, rules_to_delete[1])
      redis.call("HSET", invalidated_account_key, rules_to_delete[1], now_us)
    else
      -- Bulk path: multiple fields
      actually_deleted_rules = actually_deleted_rules + redis.call("HDEL", account_key, table.unpack(rules_to_delete))

      local deletion_timestamps = {}
      for j = 1, #rules_to_delete do
        local field_index = (j - 1) * 2 + 1
        deletion_timestamps[field_index] = rules_to_delete[j]
        deletion_timestamps[field_index + 1] = now_us
      end

      redis.call("HSET", invalidated_account_key, table.unpack(deletion_timestamps))
    end

    -- The info about when rules were invalidated are only relevant temprorarily.
    -- Make sure they expire at some point.
    if invalidate_info_ttl and invalidate_info_ttl > 0 then
      redis.call("EXPIRE", invalidated_account_key, invalidate_info_ttl)
    end
  end
end

return actually_deleted_rules

[romange]

redis.acall does not return a value by definition as it is executed asynchronously. It can be used in contexts where you code is not dependent on return value. like:

redis.acall('lpush', ...)
redis.acall('lpush', ...)
redis.acall('lpush', ...)

or in your case redis.call("HSET", ... can be called asynchronously for example. any calls that ignore the return value can be be called asynchronously. Dragonfly will respect the order of invocation but lua script won't block on these commands execution.

[heikomat]

@romange makes sense. is atomicity still guaranteed if i use acall in my lua script? like, does dragonfly make sure all async calls are done before the EVAL/EVALSHA command returns?

When i

• replace all calls with acall manually (except for the redis.call('TIME'), i need that)
• not use any acall-return-values and
• not use the lua_auto_async=true flag

the script runs fine.

[dranikpg]

is atomicity still guaranteed if i use acall in my lua script? like, does dragonfly make sure all async calls are done before the EVAL/EVALSHA command returns?

Yes, as long as return values are dicarded, there should be no difference in behaviour.

The possible improvement is just related to implementation details and comes from the fact that multiple non-conflicting commands (like writing to different values) can be run simultaneously, multiple sequential writes to the same key can run as part of the same operation.

---

lua_auto_async=true tries to replace call() with acall() if the return value is descarded. It's a simple regex and it might have a bug for some corner case. I'll check it.

The best way, of course, is to use those calls manually if they have a positive performance impact

[heikomat]

lua_auto_async=true tries to replace call() with acall() if the return value is descarded

That's good to know!

The best way, of course, is to use those calls manually if they have a positive performance impact

I might have never run into the crash had i known this from the start, because i wouldn't have tried the lua_auto_async flag in that case. Manually replacing the correct calls now was an easy performance win, for my other lua redis scripts aswell.

I'd love to see a page in the docs about acall and apcall, what they do and when to use them. They appear in the docs exactly once where the lua_auto_async is explained, and at the time i had no clue what that even means.

It's a simple regex and it might have a bug for some corner case

So this is still definitely a bug that should be addressed, but in my opinion the even bigger win for the DF-users would be a couple lines in the docs about acall and apcall

I'll check it.

Thank you :)

[romange]

@abhijat when you have time please check if you can understand/reproduce the crash when running with
--vmodule=main_service=1,set_family=2 --lua_auto_async=true inside UpdateScriptCaches. To trigger this flow you will need to run a lua script with undeclared keys. I think vmodule just affects the timings but the bug exists in general.

[dranikpg]

actually_deleted_rules = actually_deleted_rules + redis.acall("HDEL", account_key, table.unpack(rules_to_delete))

This line shouldn't have acall in it. DetectPossibleAsyncCalls is marking it wrongly as an async call, I suppose that's the whole issue

[heikomat]

actually_deleted_rules = actually_deleted_rules + redis.acall("HDEL", account_key, table.unpack(rules_to_delete))

This line shouldn't have acall in it. DetectPossibleAsyncCalls is marking it wrongly as an async call, I suppose that's the whole issue

If you look at the full script i provided you can see, that that line doesn't exist there. This line is only in the "additional context" section because i tried that, and it obviously didn't work. But the original script, that didn't have that line, also didn't work if the lua_auto_async flag

[abhijat]

I have been trying to reproduce this today both locally (x86) and on an ARM graviton instance in AWS. I tried both the HEAD and v1.35.1, but I was not able to reproduce this crash with your script. I wrote a driver program based on your script

https://gist.github.com/abhijat/7f87233eca3b0b87120e9f6206e961d4

I tried 6000 accounts and 500 entries per account, but it doesn't seem to reproduce, with dragonfly running as:

./build-dbg/dragonfly --proactor_threads=16 --dir /tmp/dfly --vmodule=main_service=1,set_family=2 --lua_auto_async=true --alsologtostderr

Next steps:

• Does crash for you happen intermittently or is it very frequent (so I know whether I should run the program for hours or is it just not matching your setup)
• If you can share a minimal example it might be helpful, or maybe just an algorithm to follow in the driver for this script.
• Does it still crash on the latest build?
• Can you try a debug build with ASAN with your workload:
  • follow the steps in https://github.com/dragonflydb/dragonfly/blob/main/docs/build-from-source.md upto 3
  • in step 3, run ./helio/blaze.sh -DWITH_ASAN=ON
  • ninja -C build-dbg dragonfly
  • run dragonfly from the command line as below, when it crashes, share the output
  • ./build-dbg/dragonfly --proactor_threads=16 --dir /tmp/dfly --vmodule=main_service=1,set_family=2 --lua_auto_async=true --alsologtostderr

[abhijat]

Actually, I can see a very similar error after letting the test driver run for a while:

F20260212 11:47:25.393522 23157 scheduler.cc:306] Fiber _5] is already scheduled remotely
*** Check failure stack trace: ***
==23141==WARNING: ASan is ignoring requested __asan_handle_no_return: stack type: default top: 0xf324730398a0; bottom 0x52e00016b000; size: 0xa04472ece8a0 (176215846348960)
False positive error reports may follow
For details see https://github.com/google/sanitizers/issues/189
    @     0xb55d1ea808c8  google::LogMessage::Fail()
    @     0xb55d1ea805fc  google::LogMessage::SendToLog()
    @     0xb55d1ea7eeac  google::LogMessage::Flush()
    @     0xb55d1ea7e550  google::LogMessage::~LogMessage()
    @     0xb55d1e72588c  util::fb2::detail::Scheduler::ScheduleFromRemote()
    @     0xb55d1e73cb58  util::fb2::detail::FiberInterface::ActivateOther()
    @     0xb55d1e743d00  util::fb2::detail::WaitQueue::NotifyImpl()
    @     0xb55d1e743254  util::fb2::detail::WaitQueue::NotifyOne()
    @     0xb55d1e70bbc8  util::fb2::Mutex::unlock()
    @     0xb55d1c6cbf44  std::lock_guard<>::~lock_guard()
    @     0xb55d1c8db6ac  dfly::ScriptMgr::Insert()
    @     0xb55d1c62204c  dfly::Service::Eval()
    @     0xb55d1c6345d8  _ZZN4dfly7Service8RegisterEPNS_15CommandRegistryEENKUlN4absl12lts_202505124SpanIKSt17basic_string_viewIcSt11char_traitsIcEEEERKNS_14CommandContextEE0_clESB_SE_
    @     0xb55d1c6994a0  _ZN3fu27abi_4006detail10invocation6invokeIRKZN4dfly7Service8RegisterEPNS4_15CommandRegistryEEUlN4absl12lts_202505124SpanIKSt17basic_string_viewIcSt11char_traitsIcEEEERKNS4_14CommandContextEE0_JSG_SJ_EEEDTclcl7forwardIT_Efp_Espcl7forwardIT0_Efp0_EEEOSN_DpOSO_
    @     0xb55d1c689bec  _ZN3fu27abi_4006detail12type_erasure16invocation_table14function_traitIKFvN4absl12lts_202505124SpanIKSt17basic_string_viewIcSt11char_traitsIcEEEERKN4dfly14CommandContextEEE16internal_invokerINS2_3boxILb1EZNSE_7Service8RegisterEPNSE_15CommandRegistryEEUlSD_SH_E0_SaISP_EEELb1EE6invokeEPKNS2_13data_accessorEmSD_SH_
    @     0xb55d1dd36adc  _ZNK3fu27abi_4006detail12type_erasure6tables6vtableINS1_8propertyILb0ELb0EJKFvN4absl12lts_202505124SpanIKSt17basic_string_viewIcSt11char_traitsIcEEEERKN4dfly14CommandContextEEEEEE6invokeILm0EJPKNS2_13data_accessorERKmSE_SI_EEEDcDpOT0_
    @     0xb55d1dd36c4c  _ZN3fu27abi_4006detail12type_erasure7erasureILb1ENS1_6configILb1ELb1ENS_16capacity_defaultEEENS1_8propertyILb0ELb0EJKFvN4absl12lts_202505124SpanIKSt17basic_string_viewIcSt11char_traitsIcEEEERKN4dfly14CommandContextEEEEEE6invokeILm0ERKSN_JSG_SK_EEEDcOT0_DpOT1_
    @     0xb55d1dd36db4  fu2::abi_400::detail::type_erasure::invocation_table::operator_impl<>::operator()()
    @     0xb55d1dd2c900  dfly::CommandId::Invoke()
    @     0xb55d1c6171c0  dfly::Service::InvokeCmd()
    @     0xb55d1c614454  dfly::Service::DispatchCommand()
    @     0xb55d1e51d884  _ZZN6facade10Connection10ParseRedisEjENKUlvE_clEv
    @     0xb55d1e53a78c  _ZSt13__invoke_implIvRKZN6facade10Connection10ParseRedisEjEUlvE_JEET_St14__invoke_otherOT0_DpOT1_
    @     0xb55d1e538d04  _ZSt8__invokeIRKZN6facade10Connection10ParseRedisEjEUlvE_JEENSt15__invoke_resultIT_JDpT0_EE4typeEOS6_DpOS7_
    @     0xb55d1e537344  _ZSt6invokeIRKZN6facade10Connection10ParseRedisEjEUlvE_JEENSt13invoke_resultIT_JDpT0_EE4typeEOS6_DpOS7_
    @     0xb55d1e535778  _ZN4absl12lts_2025051219functional_internal12InvokeObjectIZN6facade10Connection10ParseRedisEjEUlvE_vJEEET0_NS1_7VoidPtrEDpNS1_8ForwardTIT1_E4typeE
    @     0xb55d1e5457bc  absl::lts_20250512::FunctionRef<>::operator()()
    @     0xb55d1e51d3d8  facade::Connection::DispatchSingle()
    @     0xb55d1e51e678  facade::Connection::ParseRedis()
    @     0xb55d1e522894  facade::Connection::IoLoop()
    @     0xb55d1e51a61c  facade::Connection::ConnectionFlow()
    @     0xb55d1e514244  facade::Connection::HandleRequests()
*** SIGABRT received at time=1770896845 on cpu 15 ***
PC: @     0xf324bae57608  (unknown)  (unknown)
    @     0xb55d1ebfcfac        176  absl::lts_20250512::WriteFailureInfo()
    @     0xb55d1ebfd5dc        240  absl::lts_20250512::AbslFailureSignalHandler()
    @     0xf324bc0b3968       4752  (unknown)
Aborted (core dumped)

This looks very similar to (if not the same) issue you get on release build, except because this is debug build the assert is hit earlier. We will continue to look into this.

If you have the logs still available, before the crash did you ever see error logs such as:

already scheduled remotely

[heikomat]

i'm sorry, i don't have the logs available anymore, and am currently short on time.
I do explicitly remember not seeing any logs at all without the flags -vmodule=main_service=1,set_family=2

[abhijat]

Actually, I can see a very similar error after letting the test driver run for a while:

I now believe this is ASAN related and not the actual crash originally discussed in this issue. Because this crash reproduces reliably (although after a long time) with ASAN turned on, and does not appear when ASAN is off.