Proposal: Integrate Config Worker with Core Component

[mnonnenmacher]

Problem

The config worker has a very simple task which is to transform the CreateOrtRun object received via the API by running a custom parameter validation Kotlin script which needs to be loaded from the configuration repository. The actual processing of the script does not require many resources and is very fast, unlike the other workers which often need a lot of resources and run for a long time. This means that reasons to scale other workers independently do not apply to the config worker.

Also, there is no database table to store information about config jobs which causes some issues:

• The orchestrator needs to handle config jobs different than all other jobs.
• The API cannot return any information about the status of config jobs (and the UI can also not show it). This is especially inconvenient when the creation of the Kubernetes pod takes a bit longer and it is unclear why the analyzer worker does not start.

Proposal

Instead of running the parameter validation script in a separate worker, run it directly in the core module when a new ORT run is triggered.

Benefits

• The orchestrator logic can be simplified.
• The setup becomes simpler because one less component needs to be configured.
• The processing should become faster overall, especially because in Kubernetes it does not require creating a new pod anymore.

Challenges

• Because the parameter validation script and potentially other configuration files need to be loaded from the configuration repository, the processing of the CreateOrtRun object needs to happen asynchronously and should not block the API response.
• Downloaded files must be managed (deleted/reused/...).

[nnobelis]

I have no strong feeling on this change.

Because the parameter validation script and potentially other configuration files need to be loaded from the configuration repository, the processing of the CreateOrtRun object needs to happen asynchronously and should not block the API response.
All downloaded files must be deleted after processing.

However this sound highly optimized. If the validation script and other configuration files are now downloaded into a persistent container, why the overhead of downloading and deleting the files for each ORT run ?
Shouldn't we reuse these files if possible ? (even if it raise the complexity for updating them due to concurrent accesses).

[mnonnenmacher]

However this sound highly optimized. If the validation script and other configuration files are now downloaded into a persistent container, why the overhead of downloading and deleting the files for each ORT run ? Shouldn't we reuse these files if possible ? (even if it raise the complexity for updating them due to concurrent accesses).

I have changed the wording a bit to not make the assumption that deleting them is the only option. If any optimizations are required can only be decided once it is known if downloading the files for each run is actually an issue.

[nnobelis]

Yes, after thinking about it, introducing straightaway an extra complexity by trying to reuse the configuration is exactly our previous behavior (implement everything on the first try) that lead to the current complexity in ORT Server.
Let's do it when needed©️

[oheger-bosch]

Well, I don't think that the functionality implemented by the Config worker fits into the core module. It is part of the ORT run and should therefore be in the responsibility of Orchestrator. Whether it is implemented as a separate pod in Kubernetes or not is a different issue; this is merely an implementation detail and should be transparent to end users and not affect the UI.
The core module should focus on request processing and not do too much. BTW, I still think that core is the wrong name; it should be named api or something like that.

[mnonnenmacher]

It is part of the ORT run [...] should be transparent to end users [...].

I find that contradictory. And what do you actually mean with "is part of the ORT run"?

The core module should focus on request processing and not do too much.

But why? IMO splitting out components into separate deployments should be based on actual requirements. For example, because components need to be scaled independently, or because they would negatively impact each other when deployed together. And the whole point of this proposal is to challenge if the execution of the parameter validation scripts consumes so many resources that it needs to be scaled independently.

For the workers we made that decision because we already knew that the ORT tools can take a long time, use a lot of CPU, or cause out of memory issues. For the config worker or the orchestrator we have not seen such issues and I would also not expect them. So actually, I would merge them all together into a modular monolith (where also the name core makes sense) and only split it when actually required. And my assumption is that it will never be required.

[oheger-bosch]

From my PoV, ORT Server currently deals with two domains, which are of course coupled to a degree: one is ORT runs and their execution, the other is the admin part, e.g. managing the hierarchy, secrets, etc. This is what I mean with "part of ORT run".

While the independent deployment may not be needed performance-wise currently (I tend to be careful with statements like "will never be required"), there are other benefits of having a separation. Having the two bounded contexts isolated and loosely coupled via a messaging queue is probably a good thing. Another benefit I see is that Orchestrator could still run while the API pod gets undeployed. This could be useful to set the server in a maintenance mode in which no new runs can be started, but runs in progress are still processed.

One rather practical question to the proposed rework of the Config worker: So far, this worker uses some functionality from the workers:common module, e.g. the worker context. How would you deal with that, or where should this common code live in future?

[mnonnenmacher]

While the independent deployment may not be needed performance-wise currently (I tend to be careful with statements like "will never be required")

If the post-processing of the CreateOrtRun JSON object ever becomes so expensive that it needs to be scaled independently there is a serious failure in the design. After all it's just taking the JSON object and changing some values here and there.

Having the two bounded contexts isolated and loosely coupled via a messaging queue is probably a good thing

I agree, but this does not require to deploy them separately. It's common to use messaging also within monolithic applications.

One rather practical question to the proposed rework of the Config worker: So far, this worker uses some functionality from the workers:common module, e.g. the worker context. How would you deal with that, or where should this common code live in future?

I haven't thought about such details yet, but I guess some refactorings will be required.