♻️ [Maintenance] Refactor Studies-dispatcher Domain to Use asyncpg Instead of aiopg

[pcrespov]

What do these changes do?

This PR migrates database access from aiopg to asyncpg in the following components:

• The studies_dispatcher domain within the webserver
• Replace aiopg exceptions with sqlalchemy.exc
• Adds warnings in deprecated modules so we do not forget to remove them

In detail summary (AI generated)

This pull request introduces several important changes as part of the migration from legacy aiopg/psycopg2-based database exception handling to SQLAlchemy's async support and exception classes. The main focus is on deprecating old database error handling, updating exception usage throughout the codebase, and refactoring service catalog access patterns to use SQLAlchemy's async engine and connection. Additionally, deprecation warnings are added to legacy modules to guide future maintenance.

Database Exception Handling Migration:

• Replaced usage of aiopg_errors and psycopg2 exceptions with SQLAlchemy's IntegrityError and DatabaseError across multiple repositories and handlers, ensuring consistent exception handling and simplifying future maintenance. [1] [2] [3] [4] [5]
• Updated exception handler registrations to use DBAPIError from SQLAlchemy instead of PostgresError from asyncpg, improving compatibility with the new async database layer.

Deprecation Notices for Legacy Modules:

• Added explicit DeprecationWarning messages to aiopg_errors.py and db/_aiopg.py modules, informing developers of their deprecated status and pointing to migration resources. [1] [2]
• Updated comments and code in db/plugin.py to mark legacy keys and engine usage as deprecated.

Service Catalog Refactoring:

• Refactored service catalog and validation functions to accept and use SQLAlchemy's AsyncEngine and AsyncConnection, removing dependency on legacy aiopg engine and connection types. This includes updating query execution and result handling to async patterns. [1] [2] [3] [4]
• Updated controller endpoints to use the new async engine and settings injection patterns for service queries and validation. [1] [2]

Test Adjustments:

• Modified test stubs to raise SQLAlchemy's DatabaseError with appropriate parameters, aligning test exception simulation with the new database layer.

Minor Codebase Cleanups:

• Removed unused imports and legacy connection handling code in several modules, further streamlining the migration to SQLAlchemy async. [1] [2] [3]

These changes collectively advance the migration to SQLAlchemy's async support, improve exception handling consistency, and lay the groundwork for deprecating legacy database modules.

Related issue/s

• part of migrate code from aiopg to asyncpg #4529

How to test

Driving test

cd services/web/server
make install-dev
pytest -vv tests/unit/with_dbs/04/studies_dispatcher

Dev-ops

None

[codecov]

Codecov Report

❌ Patch coverage is 93.54839% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.62%. Comparing base (77faa41) to head (20c4202).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #8467      +/-   ##
==========================================
- Coverage   87.64%   87.62%   -0.03%     
==========================================
  Files        1983     1983              
  Lines       77279    77284       +5     
  Branches     1333     1333              
==========================================
- Hits        67732    67719      -13     
- Misses       9148     9166      +18     
  Partials      399      399              

Flag	Coverage Δ
integrationtests	64.13% <50.00%> (+<0.01%)	⬆️
unittests	86.31% <93.54%> (-0.02%)	⬇️

Components	Coverage Δ
pkg_aws_library	93.59% <ø> (ø)
pkg_celery_library	83.76% <ø> (ø)
pkg_dask_task_models_library	79.33% <ø> (ø)
pkg_models_library	93.07% <ø> (ø)
pkg_notifications_library	85.20% <ø> (ø)
pkg_postgres_database	87.95% <100.00%> (+0.01%)	⬆️
pkg_service_integration	70.17% <ø> (ø)
pkg_service_library	70.92% <ø> (ø)
pkg_settings_library	90.19% <ø> (ø)
pkg_simcore_sdk	84.95% <ø> (ø)
agent	93.53% <ø> (ø)
api_server	91.78% <ø> (ø)
autoscaling	95.71% <ø> (ø)
catalog	92.36% <ø> (ø)
clusters_keeper	99.13% <ø> (ø)
dask_sidecar	91.78% <ø> (-0.57%)	⬇️
datcore_adapter	97.94% <ø> (ø)
director	75.81% <ø> (ø)
director_v2	90.96% <ø> (+0.09%)	⬆️
dynamic_scheduler	96.71% <ø> (ø)
dynamic_sidecar	90.43% <ø> (ø)
efs_guardian	89.74% <ø> (ø)
invitations	91.41% <ø> (ø)
payments	92.71% <50.00%> (ø)
resource_usage_tracker	92.15% <ø> (-0.06%)	⬇️
storage	86.45% <100.00%> (-0.21%)	⬇️
webclient	∅ <ø> (∅)
webserver	87.37% <100.00%> (-0.06%)	⬇️

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 77faa41...20c4202. Read the comment docs.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[GitHK]

👍

[mergify]

🧪 CI Insights

Here's what we observed from your CI run for 20c4202.

✅ Passed Jobs With Interesting Signals

Pipeline	Job	Signal	Health on master	Retries	🔍 CI Insights	📄 Logs
CI	integration-tests	Base branch is healthy, but retries were needed. Could be early signs of flakiness 👀		1	View	View
	system-tests	Base branch is broken, but the job passed. Looks like this might be a real fix 💪		0	View	View

[wvangeit]

Thanks

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud