rm_wal_segments_pending

Remove upload.pending files from WAL segment directories in parallel, unblocking WalPurgeJob so it can reap old WAL segments.

Background

When repointing a QuestDB instance to a new empty object store, the replication startup path (process_startup_table) iterates every WAL segment directory to delete upload.pending files. On large tables with millions of accumulated WAL segments, this single-threaded filesystem walk can block server startup indefinitely.

This tool performs the same cleanup in parallel while the database is running in standalone (non-replicated) mode, allowing WalPurgeJob to reap old segments before the next replicated startup.

Building

Requires a Rust toolchain (1.85+).

cargo build --release

The binary will be at target/release/rm_wal_segments_pending.

Usage

rm_wal_segments_pending <db_root> [--dry-run] [--threads=N]

Arguments

• <db_root> — Path to the QuestDB database root directory (e.g. /var/lib/questdb/db).
• --dry-run — Count upload.pending files without deleting them.
• --threads=N — Number of parallel threads (default: 12).

Examples

Count pending files across all tables:

./rm_wal_segments_pending /var/lib/questdb/db --dry-run

Remove pending files with 24 threads:

./rm_wal_segments_pending /var/lib/questdb/db --threads=24

Prerequisites

1. The database must be running in standalone (non-replicated) mode. The replication uploader must not be active — otherwise there is a race between this tool deleting upload.pending files and the uploader creating new ones.

2. After running this tool, wait for WalPurgeJob to reap the old WAL segments before restarting with replication enabled.

How it works

The tool walks the database directory structure:

db_root/
  my_table/
    wal1/           # WAL directory (walN where N is a u32)
      0/            # Segment directory (u32 name)
        .pending/
          upload.pending   <-- this file is deleted
      1/
        .pending/
          upload.pending   <-- this file is deleted
    wal2/
      ...
  other_table/
    ...

1. Iterates all table directories under the database root.
2. For each table, enumerates walN directories (tables without WAL directories are skipped silently).
3. Uses a rayon thread pool to process WAL directories in parallel.
4. Within each WAL directory, iterates segment directories and deletes .pending/upload.pending if present.
5. Logs progress every second per table (files removed and rate).

The tool is safe to run against a live database in standalone mode:

• WalPurgeJob may concurrently remove directories that this tool is scanning. NotFound errors from such races are silently ignored.
• New WAL and segment directories created by ongoing writes will not have upload.pending files (the uploader is not running), so they are harmless.

Exit codes

• 0 — Success (or no WAL directories found).
• 1 — Completed with errors.
• 2 — Invalid usage (bad arguments, unknown flags).