Formalize the `preemptible` runtime key

[claymcleod]

We should have a vendor neutral way of specifying (a) if a task should be considered preemptible and (b) how many tries the task should do before we move to a non-preemptible submission.

[patmagee]

"preemptible" is a bit of a concept that is on the out anyways with google, azure and aws all now moving to "spot" instances instead of preemptible instances...

[claymcleod]

Interesting, are the concepts identical, just different naming? Notably, TES uses the preemptible field to denote this concept—I tend to think this phrase is actually more descriptive of what is happening, even if the clouds change the brand name to "spot".

[geoffjentry]

@claymcleod re your TES comment, there's some history to take into account here. But tl;dr I agree with you and raised the same argument several years ago. Though I wouldn't be opposed to changing the name to match common parlance, but leaving it as a boolean flag.

WDL has preemptible for the obvious reason that early Cromwell development had heavy involvement with Google, and it was the only cloud provider supported

But you pointed out the nomenclature in TES. When TES was getting started these types of topics were a large debate: should they be encoded in the TES standard or left as an implementation detail. When the topic of these types of instances came up we strongly supported the pro side as at the time WDL had preemptible but other groups involved with TES had weaker stories - this seemed like an advantage for us.

In terms of nomenclature, also at the time AWS spot was more powerful but had many more knobs that needed turning. While people gravitated towards spot or something like that, the discussion turned to how to configure it at the TES level. We pushed for preemptible citing the argument that you raised, and that one language (WDL) already supported it out of the box with one cloud (Google) - it'd be a flag to use spot instances, but the rest was left as an implementation detail. Also less spoken were the bits that this made our lives easier as the field was now passthrough for WDL, and also we felt the more tied TES & WDL were the better off things would be. (in retrospect, this was all a lot of maneuvering for not much gain!)

[vsmalladi]

Im okay leaving it as preemptible just with a more nuanced definition that included instances that can be taken away vs reserved