You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/queue-concurrency.mdx
+33-25Lines changed: 33 additions & 25 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -14,9 +14,10 @@ It's important to note that only actively executing runs count towards concurren
14
14
By default, all tasks have an unbounded concurrency limit, limited only by the overall concurrency limits of your environment. This means that each task could possibly "fill up" the entire
15
15
concurrency limit of your environment.
16
16
17
+
Each individual queue has a maximum concurrency limit equal to your environment's base concurrency limit. If you don't explicitly set a queue's concurrency limit, it will default to your environment's base concurrency limit.
18
+
17
19
<Note>
18
-
Your environment has a maximum concurrency limit which depends on your plan. If you're a paying
19
-
customer you can request a higher limit by [contacting us](https://www.trigger.dev/contact).
20
+
Your environment has a base concurrency limit and a burstable limit (default burst factor of 2.0x the base limit). Individual queues are limited by the base concurrency limit, not the burstable limit. For example, if your base limit is 10, your environment can burst up to 20 concurrent runs, but any single queue can have at most 10 concurrent runs. If you're a paying customer you can request higher limits by [contacting us](https://www.trigger.dev/contact).
With our [task checkpoint system](/how-it-works#the-checkpoint-resume-system), a parent task can trigger and wait for a subtask to complete. The way this system interacts with the concurrency system is a little complicated but important to understand. There are two main scenarios that we handle slightly differently:
186
+
With our [task checkpoint system](/how-it-works#the-checkpoint-resume-system), tasks can wait at various waitpoints (like waiting for subtasks to complete, delays, or external events). The way this system interacts with the concurrency system is important to understand.
187
+
188
+
Concurrency is only released when a run reaches a waitpoint and is checkpointed. When a run is checkpointed, it transitions to the `WAITING` state and releases its concurrency slot back to both the queue and the environment, allowing other runs to execute or resume.
186
189
187
-
- When a parent task waits for a subtask on a different queue.
188
-
- When a parent task waits for a subtask on the same queue.
190
+
This means that:
191
+
- Only actively executing runs count towards concurrency limits
192
+
- Runs in the `WAITING` state (checkpointed at waitpoints) do not consume concurrency slots
193
+
- You can have more runs in the `WAITING` state than your queue's concurrency limit
194
+
- When a waiting run resumes (e.g., when a subtask completes), it must re-acquire a concurrency slot
189
195
190
-
These scenarios are discussed in more detail below:
196
+
For example, if you have a queue with a `concurrencyLimit` of 1:
197
+
- You can only have exactly 1 run executing at a time
198
+
- You may have multiple runs in the `WAITING` state that belong to that queue
199
+
- When the executing run reaches a waitpoint and checkpoints, it releases its slot
200
+
- The next queued run can then begin execution
191
201
192
202
<Note>
193
203
We sometimes refer to the parent task as the "parent" and the subtask as the "child". Subtask and
@@ -196,7 +206,7 @@ These scenarios are discussed in more detail below:
196
206
197
207
### Waiting for a subtask on a different queue
198
208
199
-
During the time when a parent task is waiting on a subtask, the "concurrency" slot of the parent task is still considered occupied on the parent task queue, but is temporarily "released" to the environment. An example will help illustrate this:
209
+
When a parent task triggers and waits for a subtask on a different queue, the parent task will checkpoint and release its concurrency slot once it reaches the wait point. This prevents environment deadlocks where all concurrency slots would be occupied by waiting tasks.
For example purposes, let's say the environment concurrency limit is 1. When the parent task is triggered, it will occupy the only slot in the environment. When the parent task triggers the subtask, the subtask will be placed in the queue for the subtask. The parent task will then wait for the subtask to complete. During this time, the parent task slot is temporarily released to the environment, allowing another task to run. Once the subtask completes, the parent task slot is reoccupied.
222
-
223
-
This system prevents "stuck" tasks. If the parent task were to wait on the subtask and not release the slot, the environment would be stuck with only one task running.
224
-
225
-
And because only the environment slot is released, the parent task queue slot is still occupied. This means that if another task is triggered on the parent task queue, it will be placed in the queue and wait for the parent task to complete, respecting the concurrency limit.
233
+
When the parent task reaches the `triggerAndWait` call, it checkpoints and transitions to the `WAITING` state, releasing its concurrency slot back to both its queue and the environment. Once the subtask completes, the parent task will resume and re-acquire a concurrency slot.
226
234
227
235
### Waiting for a subtask on the same queue
228
236
229
-
Because tasks can trigger and wait recursively, or share the same queue, we've added special handling for when a parent task waits for a subtask on the same queue.
230
-
231
-
Recall above that when waiting for a subtask on a different queue, the parent task slot is temporarily released to the environment. When the parent task and the subtask share a queue, we also release the parent task slot to the queue. Again, an example will help illustrate this:
237
+
When a parent task and subtask share the same queue, the checkpointing behavior ensures that recursive task execution can proceed without deadlocks, up to the queue's concurrency limit.
In this example, the parent task and the subtask share the same queue with a concurrency limit of 1. When the parent task triggers the subtask, the parent task slot is released to the queue, giving the subtask the opportunity to run. Once the subtask completes, the parent task slot is reoccupied.
263
+
When the parent task checkpoints at the `triggerAndWait` call, it releases its concurrency slot back to the queue, allowing the subtask to execute. Once the subtask completes, the parent task will resume.
258
264
259
-
It's very important to note that we only release at-most X slots to the queue, where X is the concurrency limit of the queue. This means that you can only trigger and wait for X subtasks on the same queue. If you try to trigger and wait for more than X subtasks, you will receive a `RECURSIVE_WAIT_DEADLOCK` error. The following example will result in a deadlock:
265
+
However, you can only have recursive waits up to your queue's concurrency limit. If you exceed this limit, you will receive a `RECURSIVE_WAIT_DEADLOCK` error:
Now this will result in a `RECURSIVE_WAIT_DEADLOCK` error because the parent task is waiting for the subtask, and the subtask is waiting for the subsubtask, but there is no more concurrency available in the queue. It will look a bit like this in the logs:
298
+
This results in a `RECURSIVE_WAIT_DEADLOCK` error because the queue can only support one level of recursive waiting with a concurrency limit of 1:
If you are recursively triggering and waiting for tasks on the same queue, you can mitigate the risk of a deadlock by increasing the concurrency limit of the queue. This will allow you to trigger and wait for more subtasks.
304
+
To avoid recursive wait deadlocks when using shared queues:
305
+
306
+
1.**Increase the queue's concurrency limit** to allow more levels of recursive waiting
307
+
2.**Use different queues** for parent and child tasks to eliminate the possibility of deadlock
308
+
3.**Design task hierarchies** to minimize deep recursive waiting patterns
301
309
302
-
You can also use different queues for the parent task and the subtask. This will allow you to trigger and wait for more subtasks without the risk of a deadlock.
310
+
Remember that the number of recursive waits you can have on a shared queue is limited by that queue's concurrency limit.
0 commit comments