Abort k8s job using CLI

[clumsy]

Description

We only have cancel runner API which is implemented using delete_namespaced_custom_object which will effectively remove any traces of the jobs from the cluster.

Alternatively we want to abort a job, e.g. using vcjob["status"]["state"]["phase"] = "Aborted" + replace_namespaced_custom_object_status such that the pod spec remains available to (1) inspect the contents, (2) clone via apply command.

Motivation/Background

The users are forced to implement this manually and avoid using torchx CLI for k8s scehduler

Detailed Proposal

Either change the behavior of cancel (abort seems closer to cancel semantics than delete), or add a new runner API.

Alternatives

Don't use torchx CLI for k8s and use a custom script to abort the job

Additional context/links

https://kubernetes-asyncio.readthedocs.io/en/latest/kubernetes_asyncio.client.api.custom_objects_api.html#kubernetes_asyncio.client.api.custom_objects_api.CustomObjectsApi.replace_namespaced_custom_object_status

[clumsy]

Please let me know what you think, @kiukchung @d4l3k @andywag @tonykao8080
I can provide a PR shortly we everyone's aligned.

[kiukchung]

thanks for the write up!

Could you shine more light into the two motivating reasons: (1) introspect the contents (which contents? the CRD?) (2) clone via apply command, may be we need a clone() API in torchx.

It makes sense to change the behavior of kubernetes_scheuduler.cancel() to abort (rather than deleting the namespace) given that:

1. Re-submission of the same AppDef works. (e.g. run -> cancel -> run of the same job works)
2. (less important but should document if not the case) There is a auto-cleaning/archiving mechanism in Volcano/Kubernetes such that vc CRDs left behind do not cause the data-plane to run out of memory/disk space (eventually requiring the maintainer of the cluster to go clean things up)

I've seen schedulers where if the previous job's job-definition is not removed from the scheduler's data-plane subsequent jobs will fail to submit with a "Job Already Exists" error.

[clumsy]

Sure, @kiukchung

1. This works since we add unique suffix for the job using TorchX, so run (suffix#1) -> cancel -> run (suffix#2) works
2. Yes, but this is also true if the user is not using torchx cancel or deleting the CRDs manually. I'm in favor of providing an additional command or a flag for cancel to be honest - we do have one for log after all

Regarding (1) it's just to be able to inspect the job spec after the job was stopped (e.g. there was an error), when deleting CRD - we completely lose the original spec. For (2) - I don't think we need clone to be honest, that one is actually where you can get Job Already Exists if you haven't changed the suffix, createTimestamp or UUIDs, the users can do this separately, e.g. for some automation.

[clumsy]

Do you have any remaining concerns we still need to address for this PR, @kiukchung?