Skip to content

[Invoker] Unconditional unreserve_slot calls inflate quota for vqueue invocations #4520

@tillrohrmann

Description

@tillrohrmann

Bug Description

Several handlers in crates/invoker-impl/src/lib.rs call self.quota.unreserve_slot() unconditionally, but vqueue invocations never call reserve_slot (they use the scheduler's permit system via a non-empty Permit). This causes the legacy quota's available_slots counter to drift upward over time when vqueue invocations terminate through these paths.

Correct Pattern

handle_invocation_task_closed (line 1082) and the yield branch in handle_invocation_task_should_yield (line 1284) correctly check ism._permit.is_empty() before unreserving:

if ism._permit.is_empty() {
    self.quota.unreserve_slot();
}

Affected Handlers (Missing Guard)

Handler Line Path
handle_invocation_task_suspended 1119 Invocation suspends (v1)
handle_invocation_task_suspended_v2 1182 Invocation suspends (v2)
handle_abort_invocation 1347 Single invocation abort
handle_abort_partition 1458 Partition-wide abort
handle_error_event / OnTaskError::Pause 1606 Error → pause
handle_error_event / OnTaskError::Kill 1670 Error → kill

Impact

  • For Quota::Unlimited (the default): unreserve_slot is a no-op, so there is no impact in the default configuration.
  • For Quota::Limited with vqueues enabled simultaneously: available_slots drifts upward beyond the configured limit, effectively defeating the concurrency limit.

Fix

Gate all unreserve_slot calls with if ism._permit.is_empty(), matching the pattern in handle_invocation_task_closed.

Found During

Code review of #4354 (invoker memory budget). This bug is pre-existing and unrelated to the memory budget changes (except for the yield handler which was already fixed).

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions