Skip to content

Conversation

@qcdyx
Copy link
Contributor

@qcdyx qcdyx commented Nov 27, 2024

Summary:

Closes #826

  • Updated the database connection setup to use psycopg2 for connection pooling.
  • Refactored the code to use a global SQLAlchemy engine.
  • Implemented a context manager for managing database sessions.
  • Ensured session isolation for each request to maintain data integrity and prevent conflicts.

Expected behavior:

  • database connections are reused, reducing the overhead of creating new connections and improving performance.
  • the context manager ensures that sessions are properly opened, committed, rolled back, and closed, providing robust transaction management.
  • ensured that each request gets its own isolated session.

Load test shows that ramping to 5000 users at a rate of 50.00 per second uses 22 active connections - we have max 100 connections.
image

I'll work on a knowledge share sub-issue and explain all the changes in details.

Testing tips:

Provide tips, procedures and sample files on how to test the feature.
Testers are invited to follow the tips AND to try anything they deem relevant outside the bounds of the testing tips.

Please make sure these boxes are checked before submitting your pull request - thanks!

  • Run the unit tests with ./scripts/api-tests.sh to make sure you didn't break anything
  • Add or update any needed documentation to the repo
  • Format the title like "feat: [new feature short description]". Title must follow the Conventional Commit Specification(https://www.conventionalcommits.org/en/v1.0.0/).
  • Linked all relevant issues
  • Include screenshot(s) showing how this pull request works and fixes the issue(s)

@qcdyx qcdyx changed the title 293 psycopg feat: Use psycopg2 for Connection Pooling and Implement Global Engine with Context Manager for Session Management Nov 27, 2024
@qcdyx qcdyx requested review from cka-y and davidgamez December 2, 2024 22:03
Copy link
Member

@davidgamez davidgamez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Amazing improvement!

@qcdyx qcdyx merged commit 2aec616 into main Dec 17, 2024
3 checks passed
@qcdyx qcdyx deleted the 293-Psycopg branch December 17, 2024 21:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Use a global SQLAlchemy engine for db connection pooling with psycopg2

4 participants