Process combination to support complex UDP logic.

[jdries]

Use case: UDP's with complex parameter construction logic

Users that want to expose their complex EO services via openEO UDP are increasingly asking to include complex decision logic. I provide these concrete examples:

• The correct date range for a load_collection call depends on seasons, which are different depending on AOI. So given an input year, the UDP wants to query a seasonality database.
• Depending on a selected set of desired outputs, the UDP wants to make significant changes to the actual process graph being used.
• load_collection property filters need dynamic adjustment based on availability of data in the catalog. For instance, if sufficient sentinel-1 data is available for a single orbit direction, it is possible to filter on that, otherwise we require both ascending and descending directions.

This complex logic is often encoded in client side logic. This however presents a problem when trying to expose the service to generic web UI's that are not aware of this complex logic. The openEO web editor wizard is one example.

Solution 1: UDF to generate complex parameters

A call to run_udf can determine complex parameters.
The advantage of this solution is that custom code usage can be limited, and the majority of the process graph remains fully transparant.

Limitation

• Top level run_udf is a problem for 'validation' call.
• Backend needs to support run_udf
• May not satisfy the most complex use cases

Solution 2: UDF to generate full process graph

A call to run_udf can generate the full process graph.
The 'array_apply' process is capable of invoking arbitrary process graphs.

The advantage of this solution is that it allows quite complex process graph construction.

Limitation

• Top level run_udf is a problem for 'validation' call.
• Backend needs to support run_udf
• The UDP will appear less transparent, because the majority of the actual process graph is only constructed at runtime.
• Job options will have to be defined for all UDP variants generated by the UDF

Related issue: Open-EO/openeo-geopyspark-driver#1064

Solution 3: External API or UDP to perform process graph construction

This is similar to solution 2, but the custom PG is constructed by a call to an external service or even an openEO UDP that consists of a UDF.

The advantage of this solution is that it allows quite complex process graph construction.
It can also allow to work with backends that do not support run_udf.

Limitation

• The service usage will be a 2-step process: generating the PG and then executing it. Generic UI's will also not 'know' how this works, unless if we manage to standardize it.

Solution 4: use application package

In this solution, the UDP author would encapsulate the complex client side logic in an application package. This AP could be deployed as an openEO process or using OGC API processes part 2.

Limitations

• The AP would be making openEO API calls, but how would it be able to authenticate?
• If the jobs are long running, the calling AP would also have to be a long-running process that actively monitors the batch jobs it depends on.

[jdries]

Detail: logged this issue here for the sake of transparency, even though it is about combining existing functionality.
Not sure if there's a better place for it.

[soxofaan]

About using run_udf as entrypoint of a UDF/service (solution 1 and 2):
openEO process run_udf just has a generic data argument (type any), which means that there is no standardized way (yet) to define what the UDF expects in terms of required/optional inputs, data types, etc. Having some kind of standard/convention is important to build tooling around this feature, especially as this is about sharing/exposing services to other users.

And related, from solution 3:

Generic UI's will also not 'know' how this works"

This is very comparable to the lack of a standard to define the UDF data interface.

Solution 4 with application packages has an advantage here as it comes with standardization on this front

[soxofaan]

On dynamically generating a full process graphs at the level of a openEO backend (Solution 2 and 4):
this undermines openEO functionality that assumes a static, well defined process graph: validation, cost estimation/prediction, accounting/permission/security middelware that inspects the process graph, federation features like cross-backend processing or graph splitting in general, ...

[soxofaan]

On

May not satisfy the most complex use cases"

I don't think that is a problem, there will always be a more complex requirement that you have to handle at a higher level

[soxofaan]

Solution 1 is already possible I think without any change required to openeo-api or openeo-processes.

Solution 2 might also be already technically possible, but I guess it's a bit against the spirit of openEO's original design

Dynamically generating a process graph is somewhat related to these other (old) discussions:

• Add run_process #307
• Run process graphs inside process graphs openeo-api#413

[jdries]

Having some kind of standard/convention is important to build tooling around this feature, especially as this is about sharing/exposing services to other users.

Note that I was thinking of using the UDP system for this: the UDF would be wrapped in a UDP, allowing us to specify input parameters in a standardized way.

I would certainly agree that solution 2 has downsides if we compare it to the best possible approach, but alternatives like wrapping openEO calls in another 'black box' webservice have similar issues.

[m-mohr]

Variant 1 seems to be the most "openEOish" way. Although run_udf is not recommended on the top-level, that's primarily for parallelization purposes and large scale processing. I'd assume that this doesn't apply for computing such parameters. As such there's no problem on the specification side and I think it's doable to solve this without any change.

PS: Some less complex usecases can probably also be solved just with native openEO processes. Users are not always aware that they can actually pass in other process results into load_collection, e.g. date computations into temporal_extent.