Skip to content

PostgresKeyValueStorage and NamespaceRoutedKeyValueStorage #2959

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kmatasfp merged 13 commits into from
Mar 13, 2026
Merged

PostgresKeyValueStorage and NamespaceRoutedKeyValueStorage #2959

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
a2c6512
feat: KeyValueStorage implementation for Postgres
kmatasfp Mar 10, 2026
c80437b
feat: impromve multi queries
kmatasfp Mar 10, 2026
4e70ee6
Merge branch 'main' into 2957-keyvaluestorage-for-postgres
kmatasfp Mar 11, 2026
8d4d0b4
Merge branch 'main' into 2957-keyvaluestorage-for-postgres
kmatasfp Mar 11, 2026
757d549
feat: remove hash for set_storage, does not add value
kmatasfp Mar 11, 2026
f607ac4
chore: formatting
kmatasfp Mar 11, 2026
65df04c
Merge branch 'main' into 2957-keyvaluestorage-for-postgres
kmatasfp Mar 11, 2026
8545f8c
feat: clearer purpose for the consts
kmatasfp Mar 11, 2026
bd3d2b5
feat: add RoutedKeyValueStorage
kmatasfp Mar 11, 2026
d64590c
Merge branch 'main' into 2957-keyvaluestorage-for-postgres
kmatasfp Mar 12, 2026
bbe2435
feat: make RoutedKeyValueStorage more generic
kmatasfp Mar 13, 2026
4387bd0
fix: formatting
kmatasfp Mar 13, 2026
97e8f3f
Merge branch 'main' into 2957-keyvaluestorage-for-postgres
kmatasfp Mar 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
0 ...ecutor/db/migration/postgres/001_init.sql → ...on/postgres/indexed/postgres/001_init.sql
View file
Open in desktop
File renamed without changes.
26 changes: 26 additions & 0 deletions golem-worker-executor/db/migration/postgres/keyvalue/postgres/001_init.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
CREATE TABLE kv_storage (
namespace TEXT NOT NULL,
key TEXT NOT NULL,
value BYTEA NOT NULL,
PRIMARY KEY (namespace, key)
);

CREATE TABLE set_storage (
namespace TEXT NOT NULL,
key TEXT NOT NULL,
value_hash BYTEA NOT NULL,
value BYTEA NOT NULL,
PRIMARY KEY (namespace, key, value_hash)
Copy link
Contributor Author

@kmatasfp kmatasfp Mar 10, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI PK on (namespace, key, value BYTEA) can become expensive, so we use blake3 hash of the value here instead

Copy link
Contributor Author

@kmatasfp kmatasfp Mar 10, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assumption is we store values larger than 32 bytes in the set, if not then hash optimization does not make any sense

Copy link
Contributor

@vigoo vigoo Mar 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We only use the "set" subset of the trait for tracking running agents per executor, and the value is always a serialized OwnedAgentId. We can change the trait API to use strings for this, if that helps

Copy link
Contributor Author

@kmatasfp kmatasfp Mar 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah i think using strings as a value here would be more efficient and I take for sorted_set the value is equally small in reality @vigoo ?

Copy link
Contributor Author

@kmatasfp kmatasfp Mar 11, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and from my investigation we seem to write more than just a OwnedAgentId to the set, we write also this guy as a value

golem/golem-common/src/model/mod.rs

Line 677 in e00a1dc

pub struct AgentStatusRecord {

we do it here:

golem/golem-worker-executor/src/services/worker.rs

Line 309 in e00a1dc

status_value,

Copy link
Contributor Author

@kmatasfp kmatasfp Mar 11, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To answer my own question, sorted_set values seems be all over the place, from small to largish, its serialized value of:

golem/golem-common/src/model/mod.rs

Line 302 in e00a1dc

pub enum ScheduledAction {

Copy link
Contributor

@vigoo vigoo Mar 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The AgentStatusRecord is a typical "KV cache" scenario. There the value is big indeed, the whole serialized record. But that's the "get/set" API.

What I was referring to is the "set-like" API which is only used to to store a set of currently running agents.

The sorted set "subset" of the kv store trait is only used for the scheduler.

(We could split the KV Store trait to 3 sub-traits actually, but let's not do it now)

Copy link
Contributor Author

@kmatasfp kmatasfp Mar 11, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes you are right as always xD. Confused myself, early morning here xD

Copy link
Contributor Author

@kmatasfp kmatasfp Mar 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok removed value hash for set_storage kept it for sorted_set_storage as ScheduledAction::Invoke can be largish

);

CREATE TABLE sorted_set_storage (
namespace TEXT NOT NULL,
key TEXT NOT NULL,
value_hash BYTEA NOT NULL,
value BYTEA NOT NULL,
score DOUBLE PRECISION NOT NULL,
PRIMARY KEY (namespace, key, value_hash)
);

CREATE INDEX idx_sorted_set_storage_namespace_key_score
ON sorted_set_storage (namespace, key, score);
9 changes: 9 additions & 0 deletions golem-worker-executor/src/lib.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ use crate::storage::indexed::sqlite::SqliteIndexedStorage;
use crate::storage::indexed::IndexedStorage;
use crate::storage::keyvalue::memory::InMemoryKeyValueStorage;
use crate::storage::keyvalue::multi_sqlite::MultiSqliteKeyValueStorage;
use crate::storage::keyvalue::postgres::PostgresKeyValueStorage;
use crate::storage::keyvalue::redis::RedisKeyValueStorage;
use crate::storage::keyvalue::KeyValueStorage;
use crate::workerctx::WorkerCtx;
Expand Down Expand Up @@ -356,6 +357,14 @@ pub async fn create_worker_executor_impl<Ctx: WorkerCtx, A: Bootstrap<Ctx> + ?Si
Arc::new(RedisKeyValueStorage::new(pool.clone()));
(Some(pool), None, key_value_storage)
}
KeyValueStorageConfig::Postgres(postgres) => {
let key_value_storage: Arc<dyn KeyValueStorage + Send + Sync> = Arc::new(
PostgresKeyValueStorage::configured(postgres)
.await
.map_err(|err| anyhow!(err))?,
);
(None, None, key_value_storage)
}
KeyValueStorageConfig::InMemory(_) => {
(None, None, Arc::new(InMemoryKeyValueStorage::new()))
}
Expand Down
17 changes: 17 additions & 0 deletions golem-worker-executor/src/services/golem_config.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -546,6 +546,7 @@ impl SafeDisplay for OplogConfig {
#[serde(tag = "type", content = "config")]
pub enum KeyValueStorageConfig {
Redis(RedisConfig),
Postgres(KeyValueStoragePostgresConfig),
Sqlite(DbSqliteConfig),
MultiSqlite(KeyValueStorageMultiSqliteConfig),
InMemory(KeyValueStorageInMemoryConfig),
Expand All @@ -559,6 +560,10 @@ impl SafeDisplay for KeyValueStorageConfig {
let _ = writeln!(&mut result, "redis:");
let _ = writeln!(&mut result, "{}", inner.to_safe_string_indented());
}
KeyValueStorageConfig::Postgres(inner) => {
let _ = writeln!(&mut result, "postgres:");
let _ = writeln!(&mut result, "{}", inner.to_safe_string_indented());
}
KeyValueStorageConfig::Sqlite(inner) => {
let _ = writeln!(&mut result, "sqlite:");
let _ = writeln!(&mut result, "{}", inner.to_safe_string_indented());
Expand All @@ -576,6 +581,18 @@ impl SafeDisplay for KeyValueStorageConfig {
}
}

#[derive(Clone, Debug, Serialize, Deserialize)]
pub struct KeyValueStoragePostgresConfig {
#[serde(flatten)]
pub postgres: DbPostgresConfig,
}

impl SafeDisplay for KeyValueStoragePostgresConfig {
fn to_safe_string(&self) -> String {
self.postgres.to_safe_string()
}
}

#[derive(Clone, Debug, Serialize, Deserialize)]
pub struct KeyValueStorageMultiSqliteConfig {
pub root_dir: PathBuf,
Expand Down
3 changes: 2 additions & 1 deletion golem-worker-executor/src/storage/indexed/postgres.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,8 @@ use include_dir::include_dir;
use sqlx::{Postgres, QueryBuilder};
use std::time::Duration;

static DB_MIGRATIONS: include_dir::Dir = include_dir!("$CARGO_MANIFEST_DIR/db/migration");
static DB_MIGRATIONS: include_dir::Dir =
include_dir!("$CARGO_MANIFEST_DIR/db/migration/postgres/indexed");

#[derive(Debug, Clone)]
pub struct PostgresIndexedStorage {
Expand Down
1 change: 1 addition & 0 deletions golem-worker-executor/src/storage/keyvalue/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@

pub mod memory;
pub mod multi_sqlite;
pub mod postgres;
pub mod redis;
pub mod sqlite;

Expand Down
Loading
Loading