Proposal: Convert Kubernetes Transport to Kubernetes Operator

[mnonnenmacher]

Problem

The configuration of transports is currently very verbose because for most components multiple transport configurations are required.
For example, the workers all require separate sender and receiver configurations, and the orchestrator requires one receiver and seven sender configurations (one for each worker).

However, the only reason to configure different transport types within the application is that the orchestrator can use the Kubernetes transport to create Kubernetes jobs instead of actually sending a message. Apart from that it would not make sense to use different transports, for example, RabbitMQ for one queue and ActiveMQ for another.

The Kubernetes transport also introduces complexity into the configuration because it has many configuration parameters that are not related to message exchange but to creating Kubernetes jobs, like imagePullPolicy or memoryRequest.

Proposal

Replace the Kubernetes transport with a Kubernetes operator that receives the messages to start jobs and creates Kubernetes jobs for them. This component could be merged with the kubernetes-job-monitor, to not have another component that needs to be deployed, and to concentrate Kubernetes related functionality in one component.

Benefit

Only one transport configuration is required per component, and infrastructure related configuration like memoryRequest can be removed from the transport configuration options.

Challenges

The workers currently use the Kubernetes transport to "receive" the job ID to process. This would have to be changed so that they can either receive the job ID via a message or via a startup parameter.

The separate transport configurations are currently also used to configure custom queue names, however, it is questionable if this flexibility is required.

[oheger-bosch]

I am not sure whether I fully understand this proposal. Do you mean to drop the transport abstraction completely and always use a messaging system - which in case of Kubernetes would somehow be integrated using this new operator implementation? Could you maybe elaborate a bit more how such a solution would look like?

I agree that the current configuration of senders and receivers is rather verbose, but maybe there are other ways to better organize and simplify it. For instance, it could be tried to separate the protocol configuration from the endpoint configuration (which could then be much simpler) or something like that (not yet fully thought through).

[mnonnenmacher]

Currently when a new run is created, the message flow is like this:

flowchart LR
  core --> | message bus | orchestrator
  orchestrator --> | Kubernetes transport | worker

Loading

The last step is implemented using the transport API, but it does not actually use a message bus but the Kubernetes API to create a job and then reads the job information from environment variables.

The proposal is to change it to this:

flowchart LR
  kubeop["Kubernetes operator"]
  core --> | message bus | orchestrator
  orchestrator --> | message bus | kubeop
  kubeop --> | Kubernetes API | worker

Loading

So yes, the proposal is to drop the Kubernetes transport implementation and only have actual messaging systems implement the transport API.