[bug]: LND crashed on postgres "out of shared memory" error

[IanPasteur]

Background

I run LND with PostgreSQL backend on a virtual server on server hardware. There is 4 GB RAM dedicated to LND and PostgreSQL only (bitcoind runs on another virtual server). I have about 8 channels opened and there is not much routing nor load at all.

Since update to 18.5-beta day before yesterday (not sure if it is related), LND crashed twice with this error:

[CRT] SQLB: Caught unhandled error: ERROR: out of shared memory (SQLSTATE 53200)
[INF] SQLB: Sending request for shutdown
[ERR] DISC: Unable to determine if node **** is advertised: unknown postgres error: ERROR: out of shared memory (SQLSTATE 53200)

This "out of memory" error is strange as there should be enough memory available (there is only 25 % of 4 GB memory used in average). Postgres memory configuration follows:

shared_buffers = 512MB
max_connections = 64  # RTL does not work well if set to lower value
work_mem = 8MB

maintenance_work_mem = 64MB  # should not be relevant for this issue
effective_cache_size = 2GB  # only an estimation for query planner

The whole database size is currently 270 MB.

There could be some issue with the db locks, as I found these errors in the postgres log:

ERROR:  out of shared memory
HINT:  You might need to increase "max_pred_locks_per_transaction".
STATEMENT:  SELECT id FROM channeldb_kv WHERE parent_id IS NULL AND key=$1 AND value IS NULL

I also found another error in postgres log and I am not sure if it is related:

ERROR:  could not serialize access due to read/write dependencies among transactions
DETAIL:  Reason code: Canceled on identification as a pivot, during write.
HINT:  The transaction might succeed if retried.
STATEMENT:  INSERT INTO channeldb_kv (key, value, parent_id) VALUES($1, $2, $3) ON CONFLICT (key, parent_id) WHERE parent_id IS NOT NULL DO UPDATE SET value=$2 WHERE channeldb_kv.value IS NOT NULL

Similar errors were mentioned in #8049 and it should be fixed already, so I am confused.

Your environment

• lnd Version: 0.18.5-beta
• Linux 6.1.0-31-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.128-1 (2025-02-07) x86_64 GNU/Linux
• Bitcoin Core version v28.1.0
• Dedicated virtual server for LND and PostgreSQL with 4 GB RAM and 4 vCPU. Server hardware (Xeon CPU, ECC RAM, SSD drives with RAID, ...)

Steps to reproduce

I will provide more details if I would be able to discover some steps to reproduce.

Expected behaviour

LND with postgres does not consume nor demand so much RAM with such low load.

Actual behaviour

Postgres returns "out of shared memory" and LND crashes.

[Roasbeef]

Is it able to run consistently with this extra section in the config?

max_locks_per_transaction = 128
max_pred_locks_per_transaction = 1024

The shared memory error here is actually complaining about there not being enough locks in a given transaction.

[DerEwige]

Is it able to run consistently with this extra section in the config?

max_locks_per_transaction = 128
max_pred_locks_per_transaction = 1024

The shared memory error here is actually complaining about there not being enough locks in a given transaction.

afaik it is the other way around

it is out of shared memory, because lnd uses 50 connections to the DB per default

lnd/sample-lnd.conf

Line 1573 in 27440e8

; db.postgres.maxconnections=50

so especially during bootstraping of the graph there are a lot of parallel transactions running and

max_locks_per_transaction = 128
max_pred_locks_per_transaction = 1024

will only increase the memory usage

postgress's best practice is for one client not to use more than 2x[number of cores] connections
so if your postgress sql has 4 cores, lnd should not use more than 8 connections

futhermore lnd uses
Serializable isolation

Which is the savest, but causes the most conflicting transactions, so a minimum amount of parallel transactions should be used

[IanPasteur]

Thank you both for the reply 🙌

I will try to reduce db.postgres.maxconnections to 8, then?

Is there some documentation about recommended lnd and postgresql parameters? I could't find any on github or in the docs

[djkazic]

@IanPasteur the error is not because of the connection count. It is because of the locks configuration, please raise both and restart postgres to see if the issue is resolved. The maxconnections value of 50 is just an upper limit and is typically not hit with KVDB access patterns.

Even if you reduce the connections count you will still encounter that shared memory error eventually with a burst of DB activity.

The serialize errors are expected and benign -- they are an artifact of the KVDB schema we use to store data in postgres and will be phased out once SQL schema migration paths are available.

[IanPasteur]

Thanks for clearing this out.

I encountered higher connection count while I am browsing in RTL web interface. So I downscale db.postgres.maxconnections to 16 for lnd.

PostgreSQL is configured currently this way:

max_connections = 24
shared_buffers = 512MB
work_mem = 8MB
max_locks_per_transaction = 128
max_pred_locks_per_transaction = 1024

I will let you know how it works in few days.

PS: Are these tuning parameters recommended somewhere? Should I open a PR to LND docs?

[saubyk]

Are these tuning parameters recommended somewhere?

here's a gist authored by @djkazic
https://gist.github.com/djkazic/526fa3e032aea9578997f88b45b91fb9

[saubyk]

moving this thread to a discussion instead. Can open an issue, if there is something to fix