Skip to content

WIP: Mai/db base repo #6281

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 50 commits into from
Closed

WIP: Mai/db base repo #6281

wants to merge 50 commits into from

Conversation

@pcrespov
Copy link
Member

@pcrespov pcrespov commented Aug 30, 2024 •
edited
Loading

What do these changes do?

This PR moves towards transitioning the database engine from aiopg to asyncpg (related to #4529).

As an initial step, the TagsRepo is now implemented using asyncpg, along with various helpers like transaction contexts and the setup/teardown of an asyncio engine in the webserver.

  • ♻️ Splits the db_asyncio_engine implementation for servicelib.aiohttp and servicelib.fastapi.
  • 🎨 The web-server's db plugin now also sets up an additional asyncpg engine.
    • NOTE: that now we have up to 3 client engines initialized app, namely (see test_all_pg_engines_in_app)
        1. aiopg engine (deprecated)
        1. asyncpg engine via sqlalchemy.ext.asyncio (new)
        1. low-level asyncpg Pool (deprecated)

Related issue/s

How to test

cd packages/postgres-database
make install-dev
make tests

Dev-ops checklist

NOne

@pcrespov pcrespov self-assigned this Aug 30, 2024
@pcrespov pcrespov added this to the Eisbock milestone Aug 30, 2024
@codecov
Copy link

codecov bot commented Aug 30, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 54.61165% with 187 lines in your changes missing coverage. Please review.

Project coverage is 79.6%. Comparing base (cafbf96) to head (ae33ea5).
Report is 536 commits behind head on master.

Files with missing lines Patch % Lines
...ibrary/src/servicelib/aiohttp/db_asyncpg_engine.py 0.0% 33 Missing ⚠️
...service-library/src/servicelib/db_asyncpg_utils.py 0.0% 29 Missing ⚠️
...ibrary/src/servicelib/fastapi/db_asyncpg_engine.py 0.0% 16 Missing ⚠️
.../server/src/simcore_service_webserver/tags/_api.py 42.3% 15 Missing ⚠️
.../server/src/simcore_service_webserver/db/_aiopg.py 81.6% 9 Missing ⚠️
...er/src/simcore_service_webserver/tags/_handlers.py 10.0% 9 Missing ⚠️
...server/src/simcore_service_webserver/groups/api.py 58.8% 7 Missing ⚠️
.../simcore_service_webserver/payments/_methods_db.py 14.2% 6 Missing ⚠️
...c/simcore_service_webserver/folders/_folders_db.py 37.5% 5 Missing ⚠️
...ver/src/simcore_service_webserver/users/_tokens.py 16.6% 5 Missing ⚠️
... and 26 more
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff            @@
##           master   #6281      +/-   ##
=========================================
- Coverage    84.5%   79.6%    -5.0%     
=========================================
  Files          10    1472    +1462     
  Lines         214   61551   +61337     
  Branches       25    1987    +1962     
=========================================
+ Hits          181   49004   +48823     
- Misses         23   12241   +12218     
- Partials       10     306     +296
Flag Coverage Δ
integrationtests 72.0% <ø> (?)
unittests 79.1% <54.6%> (-5.5%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...library/src/servicelib/aiohttp/application_keys.py 100.0% <100.0%> (ø)
...catalog/src/simcore_service_catalog/core/events.py 100.0% <100.0%> (ø)
.../src/simcore_service_payments/services/postgres.py 100.0% <100.0%> (ø)
...vice_resource_usage_tracker/modules/db/__init__.py 100.0% <100.0%> (ø)
...s/storage/src/simcore_service_storage/constants.py 100.0% <100.0%> (ø)
services/storage/src/simcore_service_storage/db.py 97.5% <100.0%> (ø)
...s/storage/src/simcore_service_storage/db_tokens.py 100.0% <100.0%> (ø)
...rage/src/simcore_service_storage/simcore_s3_dsm.py 89.8% <ø> (ø)
...server/src/simcore_service_webserver/_constants.py 100.0% <ø> (ø)
...erver/src/simcore_service_webserver/db/_asyncpg.py 100.0% <100.0%> (ø)
... and 46 more

... and 1366 files with indirect coverage changes

@pcrespov pcrespov force-pushed the mai/db-base-repo branch 2 times, most recently from 5213a06 to 4926381 Compare September 5, 2024 12:42
@pcrespov pcrespov added a:database associated to postgres service and postgres-database package t:maintenance Some planned maintenance work labels Sep 5, 2024
@pcrespov pcrespov force-pushed the mai/db-base-repo branch 3 times, most recently from 8c82e6f to b852167 Compare September 9, 2024 13:26
@pcrespov pcrespov mentioned this pull request Sep 11, 2024
Open
3 tasks
@pcrespov pcrespov force-pushed the mai/db-base-repo branch 3 times, most recently from f453521 to ae33ea5 Compare September 13, 2024 09:04
@pcrespov pcrespov modified the milestones: Eisbock, MartinKippenberger Sep 18, 2024
@sonarqubecloud
Copy link

sonarqubecloud bot commented Sep 27, 2024

Quality Gate Passed Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

@pcrespov pcrespov mentioned this pull request Oct 1, 2024
Merged
@pcrespov
Copy link
Member Author

pcrespov commented Oct 1, 2024

Moved remaining features from this PR to #6479

@pcrespov pcrespov closed this Oct 1, 2024
@pcrespov pcrespov deleted the mai/db-base-repo branch November 22, 2024 13:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sanderegg sanderegg Awaiting requested review from sanderegg sanderegg will be requested when the pull request is marked ready for review sanderegg is a code owner

@GitHK GitHK Awaiting requested review from GitHK GitHK will be requested when the pull request is marked ready for review GitHK is a code owner

@matusdrobuliak66 matusdrobuliak66 Awaiting requested review from matusdrobuliak66 matusdrobuliak66 will be requested when the pull request is marked ready for review matusdrobuliak66 is a code owner

@giancarloromeo giancarloromeo Awaiting requested review from giancarloromeo giancarloromeo will be requested when the pull request is marked ready for review giancarloromeo is a code owner

Assignees

@pcrespov pcrespov

Labels

a:database associated to postgres service and postgres-database package t:maintenance Some planned maintenance work

Projects

None yet

Milestone

MartinKippenberger

Development

Successfully merging this pull request may close these issues.

1 participant

@pcrespov