Skip to content

Add support for PostgreSQL as store engine#255

Merged
git-hulk merged 4 commits intoapache:unstablefrom
borismartinovic01:postgres-store-engine
Feb 3, 2025
Merged

Add support for PostgreSQL as store engine#255
git-hulk merged 4 commits intoapache:unstablefrom
borismartinovic01:postgres-store-engine

Conversation

@borismartinovic01
Copy link
Contributor

Description

This PR adds PostgreSQL as a store engine. The implementation is kept as similar as possible to other supported engines.
Two tables are created: kv and locks tables. The kv table stores data regarding get, set, list and delete operations. The locks table is used for leader election. Design is based on publish/subscribe so there is no polling mechanism. This is achieved by utilizing the trigger and LISTEN/NOTIFY combination.

In other engines, we would assign a session to the lock so it expires after a defined period, but here we don't have this functionality so we use the pg_cron extension to define a job that will be executed every 6 seconds (session TTL for other engines) and deletes created lock entry. Job is scheduled by INSERT trigger created on the locks table. DELETE trigger created on the locks table removes this cronjob. If the cronjob is scheduled only once to run every 6 seconds, this time and TTL of lock entry will diverge. By creating and removing a cronjob at the same time when lock entry is created and removed we ensure TTL of 6 seconds is correct.

Testing

Tested the implementation by running Postgres as Docker container locally and running UTs

@codecov-commenter
Copy link

codecov-commenter commented Jan 13, 2025

Codecov Report

Attention: Patch coverage is 63.68715% with 65 lines in your changes missing coverage. Please review.

Project coverage is 46.85%. Comparing base (6c56470) to head (0ca14a2).
Report is 38 commits behind head on unstable.

Files with missing lines Patch % Lines
store/engine/postgresql/postgresql.go 63.68% 51 Missing and 14 partials ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##           unstable     #255      +/-   ##
============================================
+ Coverage     43.38%   46.85%   +3.47%     
============================================
  Files            37       45       +8     
  Lines          2971     4217    +1246     
============================================
+ Hits           1289     1976     +687     
- Misses         1544     2033     +489     
- Partials        138      208      +70     
Flag Coverage Δ
unittests 46.85% <63.68%> (+3.47%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@borismartinovic01
Copy link
Contributor Author

Hey folks, I see that this PR hasn't been approved yet. If you have any feature requirements which need to be written from scratch (like the one I did for Consul) for either controller or kvrocks core, please send them to me, I'll implement whatever you need.
cc: @PragmaTwice @git-hulk

@git-hulk
Copy link
Member

git-hulk commented Jan 28, 2025

@borismartinovic01 Thanks for your contributions! Will take a look in those two days.

@PragmaTwice
Copy link
Member

PragmaTwice commented Jan 28, 2025

Thank you for your contribution! I didn't give a review since I'm not familiar with postgres. I'm fine if it looks good to other reviewers.

And, some git conflicts need to be resolved now.

@git-hulk git-hulk merged commit fb1863c into apache:unstable Feb 3, 2025
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants