✨ Add Support for Filtering Solver Jobs by Custom Metadata via metadata.any Query Parameter #7678

pcrespov · 2025-05-14T13:18:55Z

What do these changes do?

This PR adds support for filtering solver jobs by custom metadata. The filter works by matching any specified metadata key-value pairs using the metadata.any query parameter.

Example usage:

GET /solvers/-/releases/-/jobs?metadata.any=key1:val*&metadata.any=key2:exactval

This will return jobs where any metadata key-value pair matches the given patterns (e.g., wildcard support like val*).
NOTE that this entrypoint is still not released so it will not be shown in the openapi specs. When released, it will display as follows:

Request Flow Overview

This is the request path from the public API to the PostgreSQL database and back, via api-server, RabbitMQ, and web-server.

Client request:
- GET /solvers/-/releases/-/jobs?metadata.any=...
simcore_service_api_server:
- list_all_solvers_jobs(filter_job_metadata_params: JobMetadataFilter | None)
  - 🧐 Depends(get_job_metadata_filter) parses and validates the query and builds a JobMetadataFilter instance
- SolverService.list_jobs(filter_any_custom_metadata=...)
- JobService.list_jobs(filter_any_custom_metadata=...)
- WbApiRpcClient.list_projects_marked_as_jobs(filter_any_custom_metadata=...)
- projects_rpc_client.list_projects_marked_as_jobs(filters=ListProjectsMarkedAsJobRpcFilters(any_custom_metadata=...))
RabbitMQ: transports the request to simcore_service_webserver.
simcore_service_webserver:
- projects domain:
  - _controller.projects_rpc.list_projects_marked_as_jobs(filters=...)
  - _jobs_service.list_my_projects_marked_as_jobs(filter_any_custom_metadata=...)
  - _jobs_repository.ProjectJobsRepository.list_projects_marked_as_jobs(filter_any_custom_metadata=...)
  - _apply_custom_metadata_filter() applies the filter with .where(...) in the SQL query
PostgreSQL: executes the filtered query.
Response: returned via the same path in reverse.

Related issue/s

Follows up ✨ api-server: listing of all solver jobs #7566 and adds filters
Resolves 1. Support listing all solver jobs from a given user. This means implementing GET /v0/solvers/-/releases/-/jobs/page #7399

How to test

services/web/server/tests/unit/with_dbs/02/test_projects__jobs_service.py: tests filter in project's service-layer
services/web/server/tests/unit/with_dbs/02/test_projects_rpc.py: test filter from web-server RPC client all the way to postgres and back
services/api-server/tests/unit/api_solvers/test_api_routers_solvers_jobs_read.py tests filter from api-server REST client all the way to a mocked web-server RPC client.
services/api-server/tests/unit/test_api_dependencies.py test api-server dependency injection that parses query and gets filter

Dev-ops

None

GitHK

reask when done

codecov · 2025-05-14T17:35:12Z

Codecov Report

Attention: Patch coverage is 98.50746% with 1 line in your changes missing coverage. Please review.

Project coverage is 87.59%. Comparing base (b422df3) to head (8df7f01).
Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7678      +/-   ##
==========================================
+ Coverage   87.55%   87.59%   +0.03%     
==========================================
  Files        1802     1797       -5     
  Lines       70122    69991     -131     
  Branches     1137     1137              
==========================================
- Hits        61393    61306      -87     
+ Misses       8419     8375      -44     
  Partials      310      310

Flag	Coverage Δ
integrationtests	`64.43% <21.42%> (-0.06%)`	⬇️
unittests	`86.81% <98.50%> (+0.04%)`	⬆️

Components	Coverage Δ
api	`∅ <ø> (∅)`
pkg_aws_library	`93.92% <ø> (ø)`
pkg_dask_task_models_library	`98.48% <ø> (ø)`
pkg_models_library	`93.08% <100.00%> (+0.01%)`	⬆️
pkg_notifications_library	`85.26% <ø> (ø)`
pkg_postgres_database	`88.58% <ø> (ø)`
pkg_service_integration	`69.92% <ø> (ø)`
pkg_service_library	`72.02% <0.00%> (ø)`
pkg_settings_library	`90.90% <ø> (ø)`
pkg_simcore_sdk	`85.72% <ø> (+0.23%)`	⬆️
agent	`96.46% <ø> (ø)`
api_server	`91.60% <100.00%> (+0.08%)`	⬆️
autoscaling	`96.08% <ø> (ø)`
catalog	`92.64% <ø> (ø)`
clusters_keeper	`99.25% <ø> (ø)`
dask_sidecar	`89.86% <ø> (ø)`
datcore_adapter	`98.12% <ø> (ø)`
director	`76.80% <ø> (ø)`
director_v2	`91.13% <ø> (+0.01%)`	⬆️
dynamic_scheduler	`96.76% <ø> (ø)`
dynamic_sidecar	`90.15% <ø> (ø)`
efs_guardian	`89.79% <ø> (ø)`
invitations	`93.28% <ø> (ø)`
payments	`92.63% <ø> (ø)`
resource_usage_tracker	`89.02% <ø> (-0.17%)`	⬇️
storage	`87.56% <ø> (ø)`
webclient	`∅ <ø> (∅)`
webserver	`85.82% <100.00%> (-0.01%)`	⬇️

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b422df3...8df7f01. Read the comment docs.

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

sanderegg

wip?

Copilot

Pull Request Overview

This PR adds support for filtering solver jobs by custom metadata via the metadata.any query parameter. Key changes include introducing a new RPC filters model and updating multiple services, repositories, and tests to support filtering by both job parent resource name and custom metadata.

Reviewed Changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
services/web/server/tests/unit/with_dbs/02/test_projects_rpc.py	Updated tests to use the new filters model for job parent resource and metadata filtering.
services/web/server/tests/unit/with_dbs/02/test_projects__jobs_service.py	Modified test parameters and added tests for metadata filtering in projects marked as jobs.
services/web/server/src/simcore_service_webserver/projects/_jobs_service.py	Renamed pagination and filter parameters to improve clarity.
services/web/server/src/simcore_service_webserver/projects/_jobs_repository.py	Introduced functions to apply job parent resource name and custom metadata filters in SQL queries.
services/api-server/*	Updated API endpoints, dependencies, and RPC client calls to use the new filters model.
packages/models-library/src/models_library/rpc/webserver/projects.py	Updated RPC filters model and examples for listing projects marked as jobs.

Comments suppressed due to low confidence (1)

packages/models-library/src/models_library/rpc/webserver/projects.py:39

[nitpick] The JSON schema example uses 'any_of_metadata' instead of the actual field name 'any_custom_metadata'. For consistency, update the example key to 'any_custom_metadata'.

"any_of_metadata": [

services/web/server/src/simcore_service_webserver/projects/_jobs_repository.py

GitHK

👍

...ces/api-server/src/simcore_service_api_server/api/dependencies/models_schemas_job_filters.py

sanderegg

thanks!

packages/models-library/src/models_library/rpc/webserver/projects.py

services/api-server/src/simcore_service_api_server/api/routes/solvers_jobs_getters.py

services/api-server/src/simcore_service_api_server/services_rpc/wb_api_server.py

Copilot

Pull Request Overview

Adds end-to-end support for filtering solver (and study) jobs by custom metadata via the metadata.any query parameter.

Introduces a new metadata.any Pydantic schema and FastAPI dependency to parse name-pattern filters.
Propagates metadata filters through API-server → RPC → web-server → repository, converting glob-style patterns to SQL ILIKE on a JSONB column.
Updates unit tests across both services to cover exact, wildcard, multi-field (OR), and combined resource-prefix + metadata filters.

Reviewed Changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
services/web/server/tests/unit/with_dbs/02/test_projects_rpc.py	Added RPC client tests for metadata filters
services/web/server/tests/unit/with_dbs/02/test_projects__jobs_service.py	Added service-layer tests for project metadata filtering
services/web/server/src/simcore_service_webserver/projects/_jobs_service.py	Extended `list_my_projects_marked_as_jobs` to accept metadata filters
services/web/server/src/simcore_service_webserver/projects/_jobs_repository.py	Implemented `_apply_custom_metadata_filter` on JSONB metadata
services/web/server/src/simcore_service_webserver/projects/_controller/projects_rpc.py	Updated RPC controller to unpack metadata filters
services/api-server/tests/unit/test_api_dependencies.py	Added tests for `metadata.any` query dependency
services/api-server/src/simcore_service_api_server/api/dependencies/models_schemas_job_filters.py	Implemented `get_job_metadata_filter` dependency
services/api-server/src/simcore_service_api_server/models/schemas/jobs_filters.py	Defined `MetadataFilterItem` & `JobMetadataFilter` schemas
services/api-server/src/simcore_service_api_server/services_rpc/wb_api_server.py	Converts dict filters into RPC-model filters
services/api-server/src/simcore_service_api_server/api/routes/solvers_jobs_getters.py	Added metadata filter dependency to solver-jobs endpoint
services/api-server/src/simcore_service_api_server/_service_solvers.py	Propagates metadata filters through solver service
services/api-server/src/simcore_service_api_server/_service_jobs.py	Propagates metadata filters through job service
packages/service-library/src/servicelib/rabbitmq/rpc_interfaces/webserver/projects.py	Updated RPC interface to accept filter model
packages/models-library/src/models_library/rpc/webserver/projects.py	Added `ListProjectsMarkedAsJobRpcFilters` & `MetadataFilterItem` models

services/web/server/src/simcore_service_webserver/projects/_jobs_repository.py

services/api-server/src/simcore_service_api_server/models/schemas/jobs_filters.py

packages/models-library/src/models_library/rpc/webserver/projects.py

mguidon · 2025-05-15T10:19:59Z

No sure what the status is regarding this https://github.com/ITISFoundation/private-issues/issues/49

I need to be able to get a list of solver-key:solver-version by filtering on the display_version. For instance:

Ideally, I could ask for filter: { "key": "isolve*", "display-version": "9.0.0*"} and I get back all of these:

isolve-cpu:9.0.0
isolve-gpu:9.0.0-rc.2
isolve-mpi:9.0.0-rc.1
isolve-mpi:9.0.0-beta.2
isolve-cpu:9.0.0-beta
isolve-gpu:9.0.0-alpha

services/web/server/src/simcore_service_webserver/projects/_jobs_repository.py

pcrespov · 2025-05-15T17:05:42Z

No sure what the status is regarding this ITISFoundation/private-issues#49

I need to be able to get a list of solver-key:solver-version by filtering on the display_version. For instance:

Ideally, I could ask for filter: { "key": "isolve*", "display-version": "9.0.0*"} and I get back all of these:
isolve-cpu:9.0.0
isolve-gpu:9.0.0-rc.2
isolve-mpi:9.0.0-rc.1
isolve-mpi:9.0.0-beta.2
isolve-cpu:9.0.0-beta
isolve-gpu:9.0.0-alpha

@mguidon

ITISFoundation/private-issues#49 is completed but I only see that version_display was added to Program and NOT to Solvers. I think there was a miscommunication here (we can fix that tomorrow though)
For starters you can inject these values as part of your custom metadata as you create the job and use the filters in this PR to list them
- 💡 we could automatically inject it upon creation as well
- GET /solvers/-/releases/-/jobs?metadata.any=solver_name:isolve*9.0.0*
Tomorrow I can also start a new PR with dedicated filters and even provide the version display in the Solvers (in case you need it!)
- 💡 GET /solvers/-/releases/-/jobs?solver_name.any=isolve*&version_display.any=1.2*&version_display.any=2.*

pcrespov · 2025-05-15T17:05:57Z

@mergify queue

mergify · 2025-05-15T17:06:04Z

mguidon · 2025-05-15T17:43:22Z

For starters you can inject these values as part of your custom metadata as you create the job and use the filters in this PR to list them

💡 we could automatically inject it upon creation as well

GET /solvers/-/releases/-/jobs?metadata.any=solver_name:isolve*9.0.0*

We really need to work on our communication... I need to get the solver version back based on the display version to create a job with the correct solver in the first place.

mergify · 2025-05-15T18:55:21Z

This pull request has been removed from the queue for the following reason: checks failed.

The merge conditions cannot be satisfied due to failing checks:

You may have to fix your CI before adding the pull request to the queue again.
If you update this pull request, to fix the CI, it will automatically be requeued once the queue conditions match again.
If you think this was a flaky issue instead, you can requeue the pull request, without updating it, by posting a @mergifyio requeue comment.

sonarqubecloud · 2025-05-16T12:14:59Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

pcrespov · 2025-05-16T12:49:39Z

@mergify queue

mergify · 2025-05-16T12:49:48Z

pcrespov added this to the Bazinga! milestone May 14, 2025

pcrespov self-assigned this May 14, 2025

pcrespov requested review from GitHK, matusdrobuliak66 and sanderegg as code owners May 14, 2025 13:18

pcrespov added a:webserver webserver's codebase. Assigning the area is particularly useful for bugs a:api framework api, data schemas, a:apiserver api-server service labels May 14, 2025

pcrespov marked this pull request as draft May 14, 2025 13:37

GitHK reviewed May 14, 2025

View reviewed changes

pcrespov force-pushed the is7092/job-project-filter branch from 4987755 to 0dd4c6b Compare May 14, 2025 17:32

pcrespov force-pushed the is7092/job-project-filter branch from 2ced33f to 0f538c8 Compare May 14, 2025 19:14

sanderegg reviewed May 15, 2025

View reviewed changes

pcrespov changed the title ~~WIP: is7092/job-project-filter~~ WIP: ✨ Add Support for Filtering Solver Jobs by Custom Metadata via metadata.any Query Parameter May 15, 2025

pcrespov changed the title ~~WIP: ✨ Add Support for Filtering Solver Jobs by Custom Metadata via metadata.any Query Parameter~~ ✨ Add Support for Filtering Solver Jobs by Custom Metadata via metadata.any Query Parameter May 15, 2025

pcrespov marked this pull request as ready for review May 15, 2025 08:16

pcrespov requested a review from giancarloromeo as a code owner May 15, 2025 08:16

pcrespov requested review from GitHK, bisgaard-itis, Copilot, mguidon and sanderegg May 15, 2025 08:16

Copilot AI reviewed May 15, 2025

View reviewed changes

services/web/server/src/simcore_service_webserver/projects/_jobs_repository.py Outdated Show resolved Hide resolved

GitHK approved these changes May 15, 2025

View reviewed changes

...ces/api-server/src/simcore_service_api_server/api/dependencies/models_schemas_job_filters.py Show resolved Hide resolved

sanderegg approved these changes May 15, 2025

View reviewed changes

pcrespov requested a review from Copilot May 15, 2025 09:13

Copilot AI reviewed May 15, 2025

View reviewed changes

matusdrobuliak66 reviewed May 15, 2025

View reviewed changes

services/web/server/src/simcore_service_webserver/projects/_jobs_repository.py Show resolved Hide resolved

pcrespov added 11 commits May 15, 2025 17:38

adds model examples

d12c321

adds filters in service layer

de011ba

adds filters to the public rest API

e8ced39

renames

b96a36d

@sanderegg review:relative imports

ea8928b

cleanup

853a9bb

fixes texamples

7e3141e

doc

b5d5817

cleanup after rebase

7243b1f

extends tests

66c9060

refactor: update custom metadata handling to use NameValueTuple

0ba6902

pcrespov force-pushed the is7092/job-project-filter branch from 46864a6 to 0ba6902 Compare May 15, 2025 16:54

pcrespov added the 🤖-automerge marks PR as ready to be merged for Mergify label May 15, 2025

Merge branch 'master' into is7092/job-project-filter

5c5d78d

giancarloromeo approved these changes May 15, 2025

View reviewed changes

Merge branch 'master' into is7092/job-project-filter

8df7f01

pcrespov enabled auto-merge (squash) May 16, 2025 12:12

pcrespov merged commit b86f633 into ITISFoundation:master May 16, 2025
94 of 95 checks passed

pcrespov deleted the is7092/job-project-filter branch May 16, 2025 13:40

pcrespov mentioned this pull request May 16, 2025

Solver listing filters by key and version_display patterns #7692

Closed

3 tasks

matusdrobuliak66 mentioned this pull request Jun 6, 2025

🚀 Release v1.83.0 #7682

Closed

92 tasks

matusdrobuliak66 mentioned this pull request Aug 5, 2025

🚀 Release v1.84.0 #7947

Closed

88 tasks

✨ Add Support for Filtering Solver Jobs by Custom Metadata via metadata.any Query Parameter #7678

✨ Add Support for Filtering Solver Jobs by Custom Metadata via metadata.any Query Parameter #7678

Uh oh!

Conversation

pcrespov commented May 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What do these changes do?

Request Flow Overview

Related issue/s

How to test

Dev-ops

Uh oh!

GitHK left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov bot commented May 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

sanderegg left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

GitHK left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sanderegg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mguidon commented May 15, 2025

Uh oh!

Uh oh!

pcrespov commented May 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pcrespov commented May 15, 2025

Uh oh!

mergify bot commented May 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🛑 The pull request has been removed from the queue default

Uh oh!

mguidon commented May 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mergify bot commented May 15, 2025

The merge conditions cannot be satisfied due to failing checks:

Uh oh!

sonarqubecloud bot commented May 16, 2025

Quality Gate passed

Uh oh!

pcrespov commented May 16, 2025

Uh oh!

mergify bot commented May 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟠 Waiting for conditions to match

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

pcrespov commented May 14, 2025 •

edited

Loading

GitHK left a comment •

edited

Loading

codecov bot commented May 14, 2025 •

edited

Loading

pcrespov commented May 15, 2025 •

edited

Loading

mergify bot commented May 15, 2025 •

edited

Loading

🛑 The pull request has been removed from the queue `default`

mguidon commented May 15, 2025 •

edited

Loading

mergify bot commented May 16, 2025 •

edited

Loading