feat: adds the preemptible task hint

[claymcleod]

This PR proposes the formalization of the preemptible hint, which is already supported in both Sprocket and Cromwell (albiet as a runtime key).

Briefly, preemptible aims to indicate the number of times a preemptible instance may be tried before using a non-preemptible instance. Failures other than preemption are proposed to count against the max_retries requirement instead.

Before submitting this PR, please make sure:

• You have added a few sentences describing the PR here.
• You have added yourself or the appropriate individual as the assignee.
• You have updated the README.md or other documentation to account for these
  changes (when appropriate).
• You have updated the CHANGELOG.md describing the change and linking back to your pull request.
• You have read and agree to the CONTRIBUTING.md document.
• You have added or updated relevant example WDL tests to the specification.
  • See the guide for more details.

[claymcleod]

Note that this was already informally implemented in Sprocket, as I believe we just copied how Cromwell handles this key. Thus, there is no associated PR—you can just grab the latest published version of Sprocket and try it out. It should work for any TES server that implements pre-emptible instances at a minimum.

[claymcleod]

There is considerable debate about exactly how this should be implemented. Given this, I think it's best to exclude from the WDL 1.3 release. At this point, I'm not even positive I feel this feature should be stabilized in this exact form—the problem begs for a more formal specification IMO.

[vsmalladi]

I agree @claymcleod This should be moved to a 1.4 maybe

[claymcleod]

I've been thinking about this more, and I agree with Geoff's general direction. I also agree with Venkat that we want more flexibility in the future, but it's not a regression to introduce this approach first before a more complex solution—we need time to figure out what that should look like.

Given that, and the fact that we can formalize what both Sprocket and Cromwell do today, I think it's okay to move forward. A future execution_plan hint (or similar) could explicitly override these defaults when we're ready for that.

[claymcleod]

And, to be clear, I'm still in favor of just bumping this to 1.4.

[claymcleod]

If this isn't also clear from the answer above, I do think there is a world where we don't need a more complex way to outline an execution plan. We'll have to find that out as we go through the design process.

[patmagee]

I feel like we should be clear here what a "preemptible" instance is.. at one point it went without saying since the clouds defined that behaviour for us, but now I think it would be worthwhile specifying it.

[claymcleod]

@patmagee @geoffjentry @vsmalladi I have redesigned this proposal for WDL 1.4 by separating the preferred number of preemptible attempts from the requirement to use preemptible instances exclusively.

The new Boolean preemptible_only requirement defaults to false. When it is true, the engine must honor a positive preemptible value and must never run the task on a dedicated instance. Using preemptible_only: true without a positive preemptible value is a validation error. An engine that cannot provide a preemptible instance must fail the task.

Preemptions now count only against preemptible, while retries caused by all other failures count only against max_retries. When preemptible_only is false, starting a dedicated attempt after exhausting the preemptible attempts consumes one retry, so max_retries: 0 does not permit that transition. When preemptible_only is true, exhausting the preemptible attempts fails the task regardless of the remaining max_retries.

I chose two flat attributes to express the full range of options without introducing new type machinery. WDL cannot cleanly express preemptible: { max_attempts: 3, only: true }: a bare { ... } is a map literal whose values must be homogeneous, while object { ... } uses the deprecated Object type and a named struct would require reserving a new type.

The spec also defines “preemptible instance” and explicitly states that absence of preemptible is equivalent to 0. I would appreciate another review of this design.

[vsmalladi]

@claymcleod My major concern is the split between preemptible_only and preemptible in requirments and hints

If the specification states, "When preemptible_only is true, the engine must honor this hint and must not run the task on a non-preemptible instance," then classifying preemptible_only as a hint breaks the fundamental definition of what a "hint" is supposed to be.

A hint can safely ignore by an engine. But the way its written now. if an engine is forced to fail the task because a setting is true, that setting is no longer a hint. It becomes a hard requirement.

I would suggest move preemtible to the requirement section to be cleanest.

[claymcleod]

preemptible_only is already a requirement. “This hint” refers to preemptible, but I agree that requiring the engine to honor it contradicts the general semantics of hints.

The intended distinction is that preemptible_only: true independently prohibits non-preemptible execution, while preemptible remains an advisory attempt count that the engine may safely ignore. I will remove the language and validation rule that make preemptible_only depend on a positive preemptible value. Moving preemptible into requirements would remove the intentionally optional behavior.