✨ api and web-server: introduce job-project mapping via projects_to_jobs table and RPC integration

[pcrespov]

What do these changes do?

The api-server microservice exposes a resource-oriented REST API, defining resources such as solvers, studies, and (soon) programs. These resources can spawn jobs, whose resource names follow the pattern:

• /solvers/{solver_id}/version/{version_id}/jobs/{job_id}
• /studies/{study_id}/jobs/{job_id}

Internally, these jobs are implemented as projects in the web-server. To distinguish job-based projects from front-end-created ones, we introduce a new table: projects_to_jobs. Managed by the web-server, it stores the resource parent of each job-project mapping. This enables more effective handling of API-originated job requests.

To avoid modifying the existing and already cluttered create_project REST endpoint (POST /projects), we take a conservative approach:

• This endpoint is mainly used by the front-end, which does not require job semantics.
• API-server to web-server communication is expected to use RPC going forward.

Instead, we introduce a new RPC controller: projects_rpc. It exposes a mark_project_as_job method, which is called after a job is successfully created and its UUID is assigned. This RPC marks the project as a job by registering it in projects_to_jobs.

Also note that this PR migrates existing job-related projects into the new projects_to_jobs table. Going forward, all newly created job-projects will be recorded in this table via the mark_project_as_job RPC method.

Next Steps

Future PRs will leverage this mechanism to:

• Query all jobs for a user (e.g. via metadata filtering).
• Retrieve jobs associated with a specific parent resource (e.g. a study or a solver).

Details

Database Schema Changes:

• Added a new table projects_to_jobs with columns id, project_uuid, and job_parent_resource_name to map projects used as jobs in the public API. This change includes creating the table, adding constraints, and populating it with initial data. [1] [2]
• Added tests to ensure the correct population of the projects_to_jobs table during migration.

Error Handling Improvements:

• Introduced a new RPCInterfaceError class for handling RPC interface exceptions and provided a method to transform domain exceptions into interface exceptions. [1] [2]
• Updated existing test cases to use the new RPCInterfaceError and validate the correct transformation of domain errors.

Webserver RPC Server Updates:

• Added a new method mark_project_as_job in the WebserverRpcSideEffects class to mark a project as a job using the RabbitMQ RPC client.
• Updated the create_job function in the API routes to use the new mark_project_as_job method via the RPC client. [1] [2]

Related issue/s

• Resolves Add projects_to_jobs table in webserver #7386

How to test

• Migration is tested with

cd packages/posgres-database
make install-dev
pytest -vv tests/test_models_projects_to_jobs.py

• existing tests in api-server and web-server covers the rest

Dev-ops

None

[codecov]

Codecov Report

Attention: Patch coverage is 87.58621% with 18 lines in your changes missing coverage. Please review.

Project coverage is 88.02%. Comparing base (c0e0037) to head (14eb2c8).
Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7435      +/-   ##
==========================================
+ Coverage   87.40%   88.02%   +0.62%     
==========================================
  Files        1729     1727       -2     
  Lines       66864    66141     -723     
  Branches     1133     1171      +38     
==========================================
- Hits        58441    58221     -220     
+ Misses       8102     7587     -515     
- Partials      321      333      +12     

Flag	Coverage Δ
integrationtests	65.20% <80.24%> (+0.01%)	⬆️
unittests	86.50% <87.58%> (-0.08%)	⬇️

Components	Coverage Δ
api	∅ <ø> (∅)
pkg_aws_library	93.90% <ø> (ø)
pkg_dask_task_models_library	97.09% <ø> (ø)
pkg_models_library	92.05% <ø> (ø)
pkg_notifications_library	85.00% <ø> (ø)
pkg_postgres_database	88.17% <100.00%> (+0.03%)	⬆️
pkg_service_integration	70.03% <ø> (ø)
pkg_service_library	72.22% <40.00%> (-0.21%)	⬇️
pkg_settings_library	90.78% <ø> (ø)
pkg_simcore_sdk	85.46% <ø> (ø)
agent	96.46% <ø> (ø)
api_server	90.83% <100.00%> (+0.09%)	⬆️
autoscaling	96.08% <ø> (ø)
catalog	91.82% <ø> (ø)
clusters_keeper	99.24% <ø> (ø)
dask_sidecar	91.25% <ø> (ø)
datcore_adapter	98.11% <ø> (ø)
director	76.87% <ø> (+0.09%)	⬆️
director_v2	91.30% <ø> (ø)
dynamic_scheduler	97.35% <ø> (ø)
dynamic_sidecar	90.16% <ø> (+0.04%)	⬆️
efs_guardian	89.79% <ø> (ø)
invitations	93.28% <ø> (ø)
payments	92.66% <ø> (ø)
resource_usage_tracker	89.12% <ø> (ø)
storage	86.80% <ø> (-0.04%)	⬇️
webclient	∅ <ø> (∅)
webserver	88.23% <100.00%> (+2.38%)	⬆️

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c0e0037...14eb2c8. Read the comment docs.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[matusdrobuliak66]

Thanks!

[GitHK]

please consider my DB change suggestion

[sanderegg]

very nice thanks for the explanations!

[GitHK]

Thanks a lot for the change!

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud