Skip to content

Create API-enforced, configurable upper bounds on number of 'waiting' Tasks on a Transformation, number of Tasks ACTIONed on a TaskHub #134

@dotsdl

Description

@dotsdl

Currently, there exist no enforced limit on the number of Tasks a user can create for a given Transformation, meaning it is very easy to (accidentally or otherwise) create very large (thousands or more) Tasks on a single Transformation. There is also no enforced limit on the number of such Tasks that can have ACTIONSrelationships to a givenTaskHub`, which can choke the compute services due to the bottleneck of the claiming code (reduced by #121, but not able to scale infinitely).

To address this, we should create server-side configurable limits to:

  • the number of waiting Tasks allowed on a Transformation at a time
  • the number of ACTIONS relationships on a TaskHub at a time

User-client side, this will mean that when a user calls AlchemiscaleClient.create_tasks(...), they will get back ScopedKeys for Tasks created until no more can be created for that Transformation; if a Task can't be created due to the limit, this will be reflected in the list of ScopedKeys returned, with a None given in its place.

Likewise, when a user calls AlchemiscaleClient.action_tasks(...), the method will action as many of the Tasks as it can up till the limit, then a None will be given for those Tasks that were not actioned.

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Relationships

None yet

Development

No branches or pull requests

Issue actions