-
Notifications
You must be signed in to change notification settings - Fork 137
Description
Bug Description
Several handlers in crates/invoker-impl/src/lib.rs call self.quota.unreserve_slot() unconditionally, but vqueue invocations never call reserve_slot (they use the scheduler's permit system via a non-empty Permit). This causes the legacy quota's available_slots counter to drift upward over time when vqueue invocations terminate through these paths.
Correct Pattern
handle_invocation_task_closed (line 1082) and the yield branch in handle_invocation_task_should_yield (line 1284) correctly check ism._permit.is_empty() before unreserving:
if ism._permit.is_empty() {
self.quota.unreserve_slot();
}Affected Handlers (Missing Guard)
| Handler | Line | Path |
|---|---|---|
handle_invocation_task_suspended |
1119 | Invocation suspends (v1) |
handle_invocation_task_suspended_v2 |
1182 | Invocation suspends (v2) |
handle_abort_invocation |
1347 | Single invocation abort |
handle_abort_partition |
1458 | Partition-wide abort |
handle_error_event / OnTaskError::Pause |
1606 | Error → pause |
handle_error_event / OnTaskError::Kill |
1670 | Error → kill |
Impact
- For
Quota::Unlimited(the default):unreserve_slotis a no-op, so there is no impact in the default configuration. - For
Quota::Limitedwith vqueues enabled simultaneously:available_slotsdrifts upward beyond the configured limit, effectively defeating the concurrency limit.
Fix
Gate all unreserve_slot calls with if ism._permit.is_empty(), matching the pattern in handle_invocation_task_closed.
Found During
Code review of #4354 (invoker memory budget). This bug is pre-existing and unrelated to the memory budget changes (except for the yield handler which was already fixed).