Design: Improving cluster reliability by preventing expensive queries #31277
Conversation
Signed-off-by: Moritz Hoffmann <[email protected]>
Signed-off-by: Moritz Hoffmann <[email protected]>
antiguru
changed the title
| We distinguish transient from permanent dataflows for the purposes of this design. | ||
| While we could apply the same mechanism to permanent dataflows, it is less clear how we would operate the system in the presence of a limit violation. | ||
| We cannot simply terminate a permanent dataflow, as it would affect all queries that depend on it, and our users expect dataflows to be always-on. | ||
| For this reason, we limit the scope of this design to transient dataflows, which are selects and subscribes. |
There was a problem hiding this comment.
Would it make sense to partition the system into a way of setting alerts based on resource usage (useful for transient and permanent dataflows) and actions taken on those alerts (more useful for transient dataflows, e.g., cancellation)?
|
|
||
| A solution needs to limit the heap size usage of a query to a configurable value, and enforce it with high probability. | ||
|
|
||
| To measure the heap size of a query, we hook into the arrangement heap size infrastructure. |
There was a problem hiding this comment.
Will this count consolidations in Unions or the cost of finishing? (I don't know if our sort implementation is in-place or allocating, for example.)
| Reducing the value yields the absolute heap size, which we can apply a threshold to detect limits exceeding their allocation. | ||
|
|
||
| We then forward the information to the controller, which can decide on how to handle dataflows with excessive resource utilization. | ||
| It can decide to terminate and fail a query, or let its execution continue. |
There was a problem hiding this comment.
Feels like we can surface this exact alert and have separately configurable actions?
| We cannot simply terminate a permanent dataflow, as it would affect all queries that depend on it, and our users expect dataflows to be always-on. | ||
| For this reason, we limit the scope of this design to transient dataflows, which are selects and subscribes. | ||
|
|
||
| We also do not address the problem how the user can express the limit. |
There was a problem hiding this comment.
While RBAC doesn't talk about limits, a given user could have quotas (just like UNIX limits/quotas). Would allow them to say that dbt gets to use arbitrary memory for its ad hoc work, but someone on the data team has limits about how they interact with the production cluster/replicas.
2 participants
Design doc on how to improve cluster reliability by preventing creating expensive objects or running expensive queries.
Checklist
$T ⇔ Proto$Tmapping (possibly in a backwards-incompatible way), then it is tagged with aT-protolabel.