Skip to content

Commit cf72b83

Browse files
rustyrussellShahanaFarooqui
authored andcommitted
wallet: fix migration speed for postgres.
Testing a large db shows Postgres slowing down exponentially as it inserts the channel_events. Rather than updating the index in the db every time, do it at the end, for spectacular speedup: ``` lightningd-1 2025-10-08T05:39:44.333Z INFO lightningd: Creating database lightningd-1 2025-10-08T05:39:47.581Z DEBUG lightningd: Transferring 6166 chain_events lightningd-1 2025-10-08T05:39:48.455Z DEBUG lightningd: Transferring 1660043 channel_events lightningd-1 2025-10-08T05:39:54.390Z INFO lightningd: Inserted 103100/1660043 channel_events lightningd-1 2025-10-08T05:40:04.390Z INFO lightningd: Inserted 283280/1660043 channel_events lightningd-1 2025-10-08T05:40:14.390Z INFO lightningd: Inserted 464065/1660043 channel_events lightningd-1 2025-10-08T05:40:24.390Z INFO lightningd: Inserted 629559/1660043 channel_events lightningd-1 2025-10-08T05:40:34.390Z INFO lightningd: Inserted 800659/1660043 channel_events lightningd-1 2025-10-08T05:40:44.390Z INFO lightningd: Inserted 975433/1660043 channel_events lightningd-1 2025-10-08T05:40:54.390Z INFO lightningd: Inserted 1134719/1660043 channel_events lightningd-1 2025-10-08T05:41:04.390Z INFO lightningd: Inserted 1290549/1660043 channel_events lightningd-1 2025-10-08T05:41:14.390Z INFO lightningd: Inserted 1443304/1660043 channel_events lightningd-1 2025-10-08T05:41:24.390Z INFO lightningd: Inserted 1590013/1660043 channel_events lightningd-1 2025-10-08T05:41:29.148Z INFO lightningd: bookkeeper migration complete: migrated 6166 chainmoves, 1660043 channelmoves, 132481 descriptions ``` Now we complete the entire migration in 1 minute 45 seconds. Thanks to @Michael1101 for reporting this. Signed-off-by: Rusty Russell <[email protected]> Changelog-Fixed: db: migration from v25.09 on a reasonable size account database could take almost infinite time.
1 parent 3c137fa commit cf72b83

File tree

1 file changed

+11
-3
lines changed

1 file changed

+11
-3
lines changed

wallet/account_migration.c

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -366,6 +366,7 @@ void migrate_from_account_db(struct lightningd *ld, struct db *db)
366366
struct db_stmt *stmt;
367367
int version;
368368
struct timemono prev;
369+
u64 num_channel_events;
369370

370371
/* Initialize wait indices: we're going to use it to generate ids. */
371372
load_indexes(db, ld->indexes);
@@ -507,11 +508,13 @@ void migrate_from_account_db(struct lightningd *ld, struct db *db)
507508

508509
log_debug(ld->log, "Transferring %zu channel_events",
509510
tal_count(channel_events));
511+
512+
/* There can be lots of these, so do a single update at the end */
513+
num_channel_events = 0;
510514
for (size_t i = 0; i < tal_count(channel_events); i++) {
511515
const struct channel_event *ev = channel_events[i];
512516
struct mvt_account_id *account = tal(ev, struct mvt_account_id);
513517
enum mvt_tag tag;
514-
u64 id;
515518

516519
/* We removed currency support, because the only way you could
517520
* use it was to inject your own events, and nobody did that
@@ -536,8 +539,7 @@ void migrate_from_account_db(struct lightningd *ld, struct db *db)
536539
" payment_group_id,"
537540
" fees) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?);"));
538541
set_mvt_account_id(account, NULL, ev->acct_name);
539-
id = channel_mvt_index_created(ld, db, account, ev->credit, ev->debit);
540-
db_bind_u64(stmt, id);
542+
db_bind_u64(stmt, ++num_channel_events);
541543
db_bind_mvt_account_id(stmt, db, account);
542544
db_bind_credit_debit(stmt, ev->credit, ev->debit);
543545
if (!mvt_tag_parse(ev->tag, strlen(ev->tag), &tag))
@@ -569,6 +571,12 @@ void migrate_from_account_db(struct lightningd *ld, struct db *db)
569571
log_info(ld->log, "Inserted %zu/%zu channel_events", i, tal_count(channel_events));
570572
}
571573

574+
wait_index_increase(ld, db,
575+
WAIT_SUBSYSTEM_CHANNELMOVES,
576+
WAIT_INDEX_CREATED,
577+
num_channel_events,
578+
NULL);
579+
572580
log_info(ld->log, "bookkeeper migration complete: migrated %zu chainmoves, %zu channelmoves, %zu descriptions",
573581
tal_count(chain_events),
574582
tal_count(channel_events),

0 commit comments

Comments
 (0)