Race condition in Global Concurrency Limits - recommended protection strategy?

[oleksandr-ieremchuk]

Environment

• Prefect version: 3.6.15 (self-hosted)
• Database: PostgreSQL 18.0
• Deployment: Kubernetes with Prefect Worker

Situation

We have multiple long-running workflow deployments for each tenant that must NOT run concurrently due to:

• Resource constraints (memory, CPU)
• External API rate limits
• Data consistency requirements

We use Global Concurrency Limits (GCL) with limit=1 to enforce this:

async with concurrency(
    ["tenant-xyz-global", "category-heavy-global"],
    occupy=1,
    timeout_seconds=3600,
    strict=False,
):
    # Workflow execution (~30-40 minutes)

Problem

We observed two flows simultaneously acquired slots from the same GCL despite limit=1.

Timeline from PostgreSQL flow_run table:

Flow A: Started 18:00:37, completed 18:35:56 (35 min) ✅
Flow B: Started 18:01:37 (waiting for slot), completed 18:39:23 (38 min)
Flow C: Started 18:01:46 (waiting for slot), completed 18:41:49 (40 min)

When Flow A released its slot at 18:35:56, both Flow B and Flow C (which had been waiting) captured the slot simultaneously, violating limit=1.

Database evidence:

SELECT name, active_slots, "limit", updated 
FROM concurrency_limit_v2 
WHERE name = 'tenant-xyz-global';

-- Result immediately after: active_slots=2, limit=1

Analysis

Looking at Prefect source code:

1. Orchestration rules (deployment/task concurrency) - PROTECTED:

• Use database transactions for atomic bulk_increment_active_slots
• Tests like test_concurrent_reacquisition_only_one_succeeds validate protection

2. HTTP API (/v2/concurrency_limits/increment-with-lease) - NOT PROTECTED:

• Multiple concurrent HTTP requests can read same active_slots=0 before any commits
• No SELECT FOR UPDATE or similar locking
• Race window: Request A reads → Request B reads → both increment → both commit

Questions

1. Is this expected behavior? Should Global Concurrency Limits be considered "best-effort" rather than strict guarantees?

2. What's the recommended approach for strict enforcement across multiple deployments for the same resource/tenant?

3. Should we use deployment-level concurrency instead? Our use case: multiple deployment types per tenant, need coordination between them.

4. Is atomic protection planned for HTTP API-based GCL, or is there architectural reason not to add it?