Skip to content

Conversation

@sanderegg
Copy link
Member

@sanderegg sanderegg commented Mar 21, 2025

What do these changes do?

Going towards sqlalchemy 2.0

  • replaced aiopg with asyncpg in simcore-sdk package
  • added asyncpg connection in api-server to access simcore-sdk modules
  • use of asyncpg connection in director-v2 to access simcore-sdk modules
  • small modifications in dynamic-sidecar
  • postgres-database unit test somehow take longer to run

Related issue/s

How to test

Dev-ops checklist

@sanderegg sanderegg added this to the The Awakening milestone Mar 21, 2025
@sanderegg sanderegg self-assigned this Mar 21, 2025
@codecov
Copy link

codecov bot commented Mar 21, 2025

Codecov Report

Attention: Patch coverage is 87.50000% with 13 lines in your changes missing coverage. Please review.

Project coverage is 88.58%. Comparing base (ac9aa7c) to head (b6cda45).
Report is 1 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #7404      +/-   ##
==========================================
+ Coverage   87.60%   88.58%   +0.97%     
==========================================
  Files        1759     1649     -110     
  Lines       68177    64053    -4124     
  Branches     1124      949     -175     
==========================================
- Hits        59726    56740    -2986     
+ Misses       8142     7047    -1095     
+ Partials      309      266      -43     
Flag Coverage Δ
integrationtests 65.19% <87.03%> (+0.05%) ⬆️
unittests 87.02% <70.19%> (+0.22%) ⬆️
Components Coverage Δ
api ∅ <ø> (∅)
pkg_aws_library ∅ <ø> (∅)
pkg_dask_task_models_library ∅ <ø> (∅)
pkg_models_library 92.70% <ø> (ø)
pkg_notifications_library 85.26% <ø> (ø)
pkg_postgres_database 88.18% <100.00%> (-0.01%) ⬇️
pkg_service_integration ∅ <ø> (∅)
pkg_service_library 72.92% <33.33%> (+0.22%) ⬆️
pkg_settings_library ∅ <ø> (∅)
pkg_simcore_sdk 85.67% <89.28%> (+0.26%) ⬆️
agent 96.46% <ø> (ø)
api_server 91.28% <100.00%> (+0.04%) ⬆️
autoscaling 96.08% <ø> (ø)
catalog 92.65% <ø> (ø)
clusters_keeper 99.25% <ø> (ø)
dask_sidecar 91.29% <ø> (ø)
datcore_adapter 98.12% <ø> (ø)
director 76.78% <ø> (-0.10%) ⬇️
director_v2 91.39% <100.00%> (+0.01%) ⬆️
dynamic_scheduler 97.40% <ø> (ø)
dynamic_sidecar 90.14% <100.00%> (+0.02%) ⬆️
efs_guardian 89.79% <ø> (ø)
invitations 93.28% <ø> (ø)
payments 92.66% <ø> (ø)
resource_usage_tracker 89.12% <ø> (ø)
storage 87.75% <100.00%> (+0.17%) ⬆️
webclient ∅ <ø> (∅)
webserver 88.34% <100.00%> (+2.30%) ⬆️

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ac9aa7c...b6cda45. Read the comment docs.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@sanderegg sanderegg force-pushed the maintenance/simcore-sdk-asyncpg-migration branch 2 times, most recently from 35e031c to 667b71e Compare March 27, 2025 17:47
@sanderegg sanderegg force-pushed the maintenance/simcore-sdk-asyncpg-migration branch 4 times, most recently from 4b3d6e4 to 96a2f1c Compare April 4, 2025 15:23
@sanderegg sanderegg modified the milestones: The Awakening, Pauwel Kwak Apr 8, 2025
@sanderegg sanderegg force-pushed the maintenance/simcore-sdk-asyncpg-migration branch 2 times, most recently from 5f12257 to ed9b0b7 Compare April 8, 2025 07:23
@sanderegg sanderegg requested a review from Copilot April 8, 2025 07:25
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot reviewed 24 out of 26 changed files in this pull request and generated no comments.

Files not reviewed (2)
  • packages/simcore-sdk/requirements/_base.in: Language not supported
  • packages/simcore-sdk/requirements/_base.txt: Language not supported
Comments suppressed due to low confidence (4)

services/director-v2/tests/unit/_helpers.py:38

  • [nitpick] Verify that using engine.begin() provides the appropriate transactional behavior compared to the previous aiopg.acquire() usage.
async with sqlalchemy_async_engine.begin() as conn:

services/director-v2/src/simcore_service_director_v2/modules/comp_scheduler/_scheduler_dask.py:307

  • Ensure that self.asyncpg_db_engine is correctly initialized and configured to meet the connection requirements for clean_task_output_and_log_files_if_invalid.
await clean_task_output_and_log_files_if_invalid(self.asyncpg_db_engine, user_id, project_id, node_id)

packages/simcore-sdk/src/simcore_sdk/node_ports_common/dbmanager.py:90

  • Verify that using 'engine.begin()' in the DBContextManager context is appropriate here and provides the expected connection behavior compared to the previous acquire() approach.
async with (DBContextManager(self._db_engine) as engine, engine.begin() as connection):

packages/postgres-database/tests/conftest.py:329

  • [nitpick] Verify that the fixture's cleanup logic after yield correctly deletes all created fake products to prevent side effects in subsequent tests.
async def create_fake_product(asyncpg_engine: AsyncEngine) -> AsyncIterator[Callable[[str], Awaitable[Row]]]:

@sanderegg sanderegg force-pushed the maintenance/simcore-sdk-asyncpg-migration branch from ed9b0b7 to 712ec10 Compare April 8, 2025 21:02
@sanderegg sanderegg requested a review from Copilot April 8, 2025 21:04
@sanderegg sanderegg marked this pull request as ready for review April 8, 2025 21:04
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot reviewed 25 out of 27 changed files in this pull request and generated no comments.

Files not reviewed (2)
  • packages/simcore-sdk/requirements/_base.in: Language not supported
  • packages/simcore-sdk/requirements/_base.txt: Language not supported
Comments suppressed due to low confidence (2)

services/director-v2/tests/integration/02/test_dynamic_sidecar_nodeports_integration.py:296

  • Consider renaming the parameter 'sqlalchemy_async_engine' to 'asyncpg_engine' for consistency with the rest of the codebase.
async def db_manager(sqlalchemy_async_engine: AsyncEngine) -> DBManager:

packages/simcore-sdk/src/simcore_sdk/node_ports_common/dbmanager.py:90

  • [nitpick] Consider splitting the combined async context managers into nested 'async with' statements for improved clarity of transaction and connection handling.
async with (DBContextManager(self._db_engine) as engine, engine.begin() as connection):

Copy link
Member

@pcrespov pcrespov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thx

@sanderegg sanderegg force-pushed the maintenance/simcore-sdk-asyncpg-migration branch from 49e6283 to caf5e4d Compare April 9, 2025 20:43
@sanderegg sanderegg force-pushed the maintenance/simcore-sdk-asyncpg-migration branch from 31e2dc4 to b6cda45 Compare April 23, 2025 16:39
@sonarqubecloud
Copy link

@sanderegg sanderegg removed the request for review from bisgaard-itis April 24, 2025 05:58
@sanderegg sanderegg merged commit 3880aff into ITISFoundation:master Apr 24, 2025
93 of 94 checks passed
@sanderegg sanderegg deleted the maintenance/simcore-sdk-asyncpg-migration branch April 24, 2025 05:59
@matusdrobuliak66 matusdrobuliak66 mentioned this pull request May 8, 2025
34 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants