Propagate data export endpoints to webserver

[bisgaard-itis]

What do these changes do?

• Connects rpc endpoints in storage for exporting data with rest endpoints in webserver
• Introduces "async job" terminology for jobs which are handled by the storage worker.
• Refactors implementation of async jobs so they will be easier to ruse in the future.

Related issue/s

• closes Support zipping in webserver (for frontend) #7197

How to test

cd services/web
make install dev
cd tests/unit
python -m pytest with_dbs/03/test_storage_rpc.py

Dev-ops checklist

• No ENV changes or I properly updated ENV (read the instruction)

[sonarqubecloud]

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[bisgaard-itis]

The listing and start of a job could also be made generic. I think that this would be ideal in our situation.

This way we don't need to write each time a different RPC interface. We just provide a task name and its parameters.

It if cannot be started an error will be raised by the worker (which is fine).

That's a good point. I have moved the list_jobs endpoint to the generic implementation of the async jobs. Then I introduced filtering á la Docker (https://docs.docker.com/reference/api/engine/version/v1.47/#tag/Container/operation/ContainerList) which should be generic enough that we can have a single endpoint for different types of tasks. Concerning the endpoint for triggering a task, I am hesitant to move it to a generic implementation. The reasons are the exception types. I think on the client side (meaning the webserver here) it is useful that exceptions specific to what kind of error is going on can be propagated. That is quite specific to different types of tasks. So if OK with you I will leave it as is for now. We can always move later if you prefer.

There is one challenge which is still not entirely clear to me how to handle: Permission and access rights. From talking with Sylvain I think this should be handled in storage, but for that we might need to propagate more info from the webserver (e.g. project_ids)

[GitHK]

one more thing and it should be all set

[GitHK]

thanks a lot!

[sonarqubecloud]

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud