IndexStorage implementation for postgres

[kmatasfp]

Resolves: #2951

Postgre IndexStorage implementation for worker executor can be enabled with:

GOLEM__INDEXED_STORAGE__TYPE="Postgres"
GOLEM__INDEXED_STORAGE__CONFIG__HOST="your-postgres-endpoint"
GOLEM__INDEXED_STORAGE__CONFIG__PORT=5432
GOLEM__INDEXED_STORAGE__CONFIG__DATABASE="golem_worker_executor"
GOLEM__INDEXED_STORAGE__CONFIG__USERNAME="postgres"
GOLEM__INDEXED_STORAGE__CONFIG__PASSWORD="***"
GOLEM__INDEXED_STORAGE__CONFIG__MAX_CONNECTIONS=10

[kmatasfp]

fyi all the config here is to make drop_prefix less painful

[kmatasfp]

prob still need to tune these based on how it behaves in the wild, but should be good starting point

[noise64]

looks reasonable, but i would like to add some periodic montiroing on the stats, vacumming and CPU (as a separate issue, later), so we can see how these perform (and this is just a generic reminder, we also should monitor our other postgres instances)

[noise64]

looks reasonable, but i would like to add some periodic montiroing on the stats, vacumming and CPU (as a separate issue, later), so we can see how these perform (and this is just a generic reminder, we also should monitor our other postgres instances)

[noise64]

One thing we might want to consider is partioning by HASH(namespace, key), which is probably easy to add now, and can help us buying more time until we want to use a more scaleable backend. I will think a bit more about this, and we can discuss / add it as a seperate card, if we wan to, until release.

[noise64]

This is a generic comment, not for this issue and backend: after discussing with @vigoo a bit, we might want to make the ScanCursor string based (instead of integer), that ideally could allow us making scanning perform better, e.g. with progress, we could skip the offset part, and use the index directly.

this is also something, that i would like to think a bit more, and discuss / solve as a separate issue, if needed.