feat(api): public compactPad API + bin/compactPad CLI over existing Cleanup

[JohnMcLear]

Summary

Addresses #6194. Develop already ships a working revision-cleanup path under src/node/utils/Cleanup.ts — deleteAllRevisions(padId) collapses full history via copyPadWithoutHistory, and deleteRevisions(padId, keepRevisions) keeps the last N. Both are wired into the admin-settings endpoint but neither is accessible through the public API, and there's no CLI for operators who want to run compaction from a terminal. That's the gap this PR fills.

(Earlier drafts of this PR re-implemented a compact helper on Pad; once the Cleanup module landed on develop that became redundant. Refactored to strictly wrap what already exists.)

What's added

Surface	Behavior
API.compactPad(padID, keepRevisions?)	keepRevisions null/omitted → Cleanup.deleteAllRevisions (full collapse). Positive integer N → Cleanup.deleteRevisions(padId, N) (keep last N). Returns {ok, mode: 'all'|'keepLast', keepRevisions?}. Validates non-negative integer.
APIHandler 1.3.1	Registers compactPad: ['padID', 'keepRevisions'], bumps latestApiVersion to 1.3.1.
bin/compactPad.ts	CLI: node bin/compactPad.js <padID> (collapse all) or node bin/compactPad.js <padID> --keep N. Prints before/after revision counts so operators see concrete savings.
Backend tests	Verify: full-collapse mode, keep-last mode, negative-keep rejection, non-numeric-keep rejection, text preservation in both paths.

Why wrap over re-implement

• Cleanup.deleteAllRevisions / deleteRevisions are tested in place and already used in production via the admin UI.
• A second implementation of the same primitive (what my earlier iteration did) risks diverging behavior between the admin-UI path and the CLI/API path.
• Operators who want finer control already have --keep N; hitting the full admin UI isn't a realistic workflow for scripted maintenance.

Test plan

• pnpm run ts-check clean locally after rebase onto develop
• Backend tests: collapse mode, keep-last mode, validation errors
• CI green
• Manual: run node bin/compactPad.js <padID> against a pad with many revisions; observe the before/after count

Closes #6194

🤖 Generated with Claude Code

[qodo-free-for-open-source-projects]

Review Summary by Qodo

Add compactHistory() and compactPad CLI for database space reclamation

✨ Enhancement

Walkthroughs

Description

• Add compactHistory() method to collapse pad revision history into single base revision
• Implement compactPad API endpoint and admin CLI for in-place database space reclamation
• Preserve pad text, attributes, and chat history while clearing saved-revision bookmarks
• Include comprehensive backend tests covering empty pads, text preservation, and post-compact edits

Diagram

flowchart LR
  A["Long-lived Pad<br/>with Heavy History"] -->|compactHistory| B["Single Base Revision<br/>head=0"]
  B -->|preserves| C["Text & Attributes<br/>Chat History"]
  B -->|clears| D["Saved Revisions<br/>Old Changesets"]
  E["Admin CLI<br/>compactPad.ts"] -->|calls API| F["API.compactPad<br/>v1.3.1"]
  F -->|invokes| B

Loading

File Changes

1. bin/compactPad.ts ✨ Enhancement +66/-0

Admin CLI for pad history compaction

• New admin CLI tool for compacting pad revision history
• Reports pre-flight revision count via getRevisionsCount API
• Calls compactPad HTTP endpoint and displays savings
• Includes usage documentation and unhandled rejection handler

bin/compactPad.ts

---

2. src/node/db/API.ts ✨ Enhancement +23/-0

Public API wrapper for pad compaction

• Add compactPad(padID, authorId) public API function
• Wraps Pad.compactHistory() and returns removed revision count
• Includes JSDoc with example return values and parameter documentation

src/node/db/API.ts

---

3. src/node/db/Pad.ts ✨ Enhancement +59/-0

Core pad history compaction implementation

• Implement compactHistory(authorId) method to collapse all revisions into single base
• Builds changeset from current atext using SmartOpAssembler and opsFromAText
• Deletes all existing revision records and clears saved-revision bookmarks
• Resets pad state and appends new base revision, returns count of removed revisions

src/node/db/Pad.ts

---

View more (2)

4. src/node/handler/APIHandler.ts ⚙️ Configuration changes +6/-1

Register compactPad in API v1.3.1

• Create new API version 1.3.1 extending 1.3.0
• Register compactPad endpoint with parameters padID and authorId
• Update latestApiVersion from 1.3.0 to 1.3.1

src/node/handler/APIHandler.ts

---

5. src/tests/backend/specs/compactPad.ts 🧪 Tests +80/-0

Backend tests for pad compaction feature

• Add four backend test specs for Pad.compactHistory() and API.compactPad()
• Test empty pad no-op, text preservation with head reset to 0, saved-revision cleanup
• Verify subsequent edits append cleanly on top of collapsed base revision

src/tests/backend/specs/compactPad.ts

---

[qodo-free-for-open-source-projects]

Code Review by Qodo

🐞 Bugs (2) 📘 Rule violations (2) 📎 Requirement gaps (0)

1. compactPad missing feature flag 📘 Rule violation ☼ Reliability

Description

The new destructive compactPad feature (API + CLI) is enabled unconditionally with no
feature-flag/disable mechanism, violating the requirement that new features be disabled by default.
This can expose a new admin-capability surface area without an explicit opt-in toggle.

Code

src/node/handler/APIHandler.ts[R145-152]

+version['1.3.1'] = {
+  ...version['1.3.0'],
+  compactPad: ['padID', 'authorId'],
+};
+
// set the latest available API version here
-exports.latestApiVersion = '1.3.0';
+exports.latestApiVersion = '1.3.1';

Evidence

PR Compliance ID 5 requires new features to be behind a feature flag and disabled by default. This
PR registers the new HTTP API function compactPad in the latest API version and bumps
latestApiVersion with no conditional gating, making the feature available immediately.

src/node/handler/APIHandler.ts[145-152]
src/node/db/API.ts[638-659]
bin/compactPad.ts[57-58]
Best Practice: Repository guidelines

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The new `compactPad` API/CLI feature is enabled by default with no feature flag.
## Issue Context
Compliance requires new features to be behind a feature flag and disabled by default, with no behavior/path changes when the flag is off.
## Fix Focus Areas
- src/node/handler/APIHandler.ts[145-152]
- src/node/db/API.ts[638-659]
- bin/compactPad.ts[57-58]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. HTTP API docs not updated 📘 Rule violation ⚙ Maintainability

Description

The documentation still states the latest HTTP API version is 1.3.0, but the code updates it to
1.3.1 and adds the new compactPad endpoint without updating the docs. This creates an
API/documentation mismatch for integrators and administrators.

Code

src/node/handler/APIHandler.ts[R145-152]

+version['1.3.1'] = {
+  ...version['1.3.0'],
+  compactPad: ['padID', 'authorId'],
+};
+
// set the latest available API version here
-exports.latestApiVersion = '1.3.0';
+exports.latestApiVersion = '1.3.1';

Evidence

PR Compliance ID 6 requires updating doc/ documentation when APIs change. The PR bumps
latestApiVersion to 1.3.1, but the HTTP API docs still declare 1.3.0 as the latest version,
indicating the documentation was not updated alongside the API change.

src/node/handler/APIHandler.ts[145-152]
doc/api/http_api.md[100-103]
doc/api/http_api.adoc[65-70]
Best Practice: Repository guidelines

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The HTTP API documentation is out of date: it still lists `1.3.0` as the latest API version even though the code now sets `1.3.1`.
## Issue Context
This PR introduces a new API version (`1.3.1`) and a new endpoint, so `doc/api/http_api.*` must be updated in the same change set.
## Fix Focus Areas
- src/node/handler/APIHandler.ts[145-152]
- doc/api/http_api.md[100-103]
- doc/api/http_api.adoc[65-70]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. Changeset base length mismatch ☑ 🐞 Bug ≡ Correctness

Description

Pad.compactHistory() packs a changeset with oldLen=2 but resets the pad's base atext to "\n" (length
1) before applying it, causing appendRevision() to throw on the mismatched apply assertion. This
makes compactHistory() fail at runtime for any pad with head > 0.

Code

src/node/db/Pad.ts[R593-612]

+    const oldLength = 2;
+    const newLength = assem.getLengthChange();
+    const newText = oldAText.text;
+    const baseChangeset = pack(oldLength, newLength, assem.toString(), newText);
+
+    // Drop every existing revision + saved-revision pointer and reset the
+    // pad's in-memory state to pre-any-revisions.
+    const deletions: Promise<void>[] = [];
+    for (let r = 0; r <= originalHead; r++) {
+      // @ts-ignore
+      deletions.push(this.db.remove(`pad:${this.id}:revs:${r}`));
+    }
+    await Promise.all(deletions);
+    this.savedRevisions = [];
+    this.head = -1;
+    this.atext = makeAText('\n');
+    // pool is retained — attributes from the composed text will reuse it,
+    // and we do not know which other pads may hold references to pool ids.
+
+    await this.appendRevision(baseChangeset, authorId);

Evidence

compactHistory() hard-codes oldLength=2, then resets this.atext to makeAText('\n') and calls
appendRevision(), which applies the changeset to the current this.atext. Changeset.applyToText()
asserts that the base string length equals the changeset oldLen, so applying an oldLen=2 changeset
to a 1-character string will throw.

src/node/db/Pad.ts[581-613]
src/node/db/Pad.ts[150-156]
src/static/js/Changeset.ts[404-407]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`Pad.compactHistory()` creates `baseChangeset` with `oldLength = 2`, but resets `this.atext` to `makeAText('\n')` (length 1) before calling `appendRevision(baseChangeset, ...)`. `appendRevision()` applies the changeset to `this.atext`, and `Changeset.applyToText()` asserts `str.length === oldLen`, so this will throw.
### Issue Context
The code comment indicates the intent is to apply the changeset on top of a freshly-initialized pad that has text `"\n\n"` (length 2).
### Fix Focus Areas
- src/node/db/Pad.ts[581-613]
### Fix approach
Update the reset state so the base text length matches the packed changeset, for example:
- Reset `this.atext` to `makeAText('\n\n')` (and keep `oldLength = 2`), **or**
- Change the packed `oldLength` (and corresponding changeset construction) to match the actual reset base text.
After the change, ensure `compactHistory()` can successfully call `appendRevision()` without triggering the mismatched-apply assertion.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

View more (2)

4. Unbounded revision deletions 🐞 Bug ➹ Performance

Description

Pad.compactHistory() creates one Promise per revision and awaits Promise.all(), which can exhaust
memory and overwhelm the DB for large pads (the target use case). This can degrade or crash the
Etherpad process during compaction.

Code

src/node/db/Pad.ts[R600-605]

+    const deletions: Promise<void>[] = [];
+    for (let r = 0; r <= originalHead; r++) {
+      // @ts-ignore
+      deletions.push(this.db.remove(`pad:${this.id}:revs:${r}`));
+    }
+    await Promise.all(deletions);

Evidence

compactHistory() pushes an unbounded number of removal promises into an array and runs them all at
once. The same file uses a bounded-concurrency pattern (timesLimit(..., 500, ...)) for deleting
revisions/chats during pad removal, indicating this codepath should also be concurrency-limited.

src/node/db/Pad.ts[600-605]
src/node/db/Pad.ts[706-714]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`compactHistory()` deletes all revision keys by building a `deletions` array and calling `Promise.all(deletions)`. For large `head` values this can allocate huge memory and issue a massive number of concurrent DB operations.
### Issue Context
`Pad.remove()` already uses bounded concurrency via `timesLimit(..., 500, ...)` for the same style of per-revision deletions.
### Fix Focus Areas
- src/node/db/Pad.ts[600-605]
- src/node/db/Pad.ts[706-714]
### Fix approach
Replace the unbounded `Promise.all(deletions)` pattern with a bounded-concurrency loop, e.g. using `timesLimit(originalHead + 1, 500, async (r) => { await this.db.remove(`pad:${this.id}:revs:${r}`, null); })` (mirroring `Pad.remove()` semantics).
This avoids O(N) Promise allocation and prevents DB overload.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

5. No session eviction/lock 🐞 Bug ☼ Reliability

Description

compactHistory() performs destructive revision deletion and resets head/atext without kicking
connected users, so concurrent edits can race with the compaction and lead to lost updates or
inconsistent client state. Etherpad already kicks sessions before destructive pad removal, but
compactHistory() does not.

Code

src/node/db/Pad.ts[R581-613]

+  async compactHistory(authorId = '') {
+    const originalHead = this.head;
+    if (originalHead <= 0) return 0;
+
+    // Build a single changeset that produces the current atext on top of a
+    // freshly-initialized pad ("\n\n" per copyPadWithoutHistory comment).
+    // This mirrors the existing copyPadWithoutHistory path exactly so we
+    // inherit its tested correctness.
+    const oldAText = this.atext;
+    const assem = new SmartOpAssembler();
+    for (const op of opsFromAText(oldAText)) assem.append(op);
+    assem.endDocument();
+    const oldLength = 2;
+    const newLength = assem.getLengthChange();
+    const newText = oldAText.text;
+    const baseChangeset = pack(oldLength, newLength, assem.toString(), newText);
+
+    // Drop every existing revision + saved-revision pointer and reset the
+    // pad's in-memory state to pre-any-revisions.
+    const deletions: Promise<void>[] = [];
+    for (let r = 0; r <= originalHead; r++) {
+      // @ts-ignore
+      deletions.push(this.db.remove(`pad:${this.id}:revs:${r}`));
+    }
+    await Promise.all(deletions);
+    this.savedRevisions = [];
+    this.head = -1;
+    this.atext = makeAText('\n');
+    // pool is retained — attributes from the composed text will reuse it,
+    // and we do not know which other pads may hold references to pool ids.
+
+    await this.appendRevision(baseChangeset, authorId);
+    return originalHead;

Evidence

Pad.remove() explicitly kicks all sessions from the pad before deleting revisions. There is a
dedicated kickSessionsFromPad() implementation, but compactHistory() never calls it despite deleting
every revision and rewriting the pad state.

src/node/db/Pad.ts[673-679]
src/node/handler/PadMessageHandler.ts[177-189]
src/node/db/Pad.ts[581-613]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`compactHistory()` deletes all revisions and resets in-memory pad state while users might still be connected and editing the pad. This can race with `handleUserChanges()` and cause lost updates or client errors.
### Issue Context
`Pad.remove()` calls `padMessageHandler.kickSessionsFromPad(padID)` before destructive operations.
### Fix Focus Areas
- src/node/db/Pad.ts[581-613]
- src/node/db/Pad.ts[673-679]
- src/node/handler/PadMessageHandler.ts[177-189]
### Fix approach
Before deleting revisions/resetting state, kick active sessions from the pad (and ideally ensure no further queued edits are processed) similarly to `Pad.remove()`.
At minimum:
- Call `padMessageHandler.kickSessionsFromPad(this.id)` at the start of `compactHistory()` when `originalHead > 0`.
Optionally, also add safeguards to prevent concurrent mutations during compaction (e.g., a per-pad mutex or equivalent queuing).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

6. padCreate hook emitted ☑ 🐞 Bug ⚙ Maintainability

Description

compactHistory() sets head to -1 then calls appendRevision(), which increments head to 0 and
triggers the padCreate hook, even though the pad already existed. Plugins listening for padCreate
may perform incorrect initialization or logging for an existing pad being compacted.

Code

src/node/db/Pad.ts[R606-613]

+    this.savedRevisions = [];
+    this.head = -1;
+    this.atext = makeAText('\n');
+    // pool is retained — attributes from the composed text will reuse it,
+    // and we do not know which other pads may hold references to pool ids.
+
+    await this.appendRevision(baseChangeset, authorId);
+    return originalHead;

Evidence

compactHistory() forces head=-1 and then uses appendRevision() to write the new base revision, and
appendRevision() selects the hook based on whether head becomes 0. This guarantees padCreate will
fire during compaction.

src/node/db/Pad.ts[606-613]
src/node/db/Pad.ts[158-165]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Compaction is not a pad creation event, but current logic resets `head=-1` and then calls `appendRevision()`, which will trigger the `padCreate` hook when `head` becomes 0.
### Issue Context
`appendRevision()` chooses `hook = this.head === 0 ? 'padCreate' : 'padUpdate'`. With `head=-1` before the call, compaction will always emit `padCreate`.
### Fix Focus Areas
- src/node/db/Pad.ts[606-613]
- src/node/db/Pad.ts[158-165]
### Fix approach
Refactor compaction to avoid emitting `padCreate` for an existing pad. Options:
- Write revision 0 directly (compute the new atext, set `head=0`, `atext=...`, then `db.set(pad:${id}:revs:0, ...)` and `saveToDatabase()`), and emit a more appropriate hook (`padUpdate` or a new `padCompact`).
- Or enhance `appendRevision()` (or add a new helper) to allow specifying which hook to emit for special operations like compaction.
Keep plugin semantics consistent: existing pad compacting should not look like a new pad creation.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

ⓘ The new review experience is currently in Beta. Learn more

[qodo-free-for-open-source-projects]

1. compactpad missing feature flag 📘 Rule violation ☼ Reliability

The new destructive compactPad feature (API + CLI) is enabled unconditionally with no
feature-flag/disable mechanism, violating the requirement that new features be disabled by default.
This can expose a new admin-capability surface area without an explicit opt-in toggle.

Agent Prompt

## Issue description
The new `compactPad` API/CLI feature is enabled by default with no feature flag.

## Issue Context
Compliance requires new features to be behind a feature flag and disabled by default, with no behavior/path changes when the flag is off.

## Fix Focus Areas
- src/node/handler/APIHandler.ts[145-152]
- src/node/db/API.ts[638-659]
- bin/compactPad.ts[57-58]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

[qodo-free-for-open-source-projects]

2. Http api docs not updated 📘 Rule violation ⚙ Maintainability

The documentation still states the latest HTTP API version is 1.3.0, but the code updates it to
1.3.1 and adds the new compactPad endpoint without updating the docs. This creates an
API/documentation mismatch for integrators and administrators.

Agent Prompt

## Issue description
The HTTP API documentation is out of date: it still lists `1.3.0` as the latest API version even though the code now sets `1.3.1`.

## Issue Context
This PR introduces a new API version (`1.3.1`) and a new endpoint, so `doc/api/http_api.*` must be updated in the same change set.

## Fix Focus Areas
- src/node/handler/APIHandler.ts[145-152]
- doc/api/http_api.md[100-103]
- doc/api/http_api.adoc[65-70]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

[qodo-free-for-open-source-projects]

4. Unbounded revision deletions 🐞 Bug ➹ Performance

Pad.compactHistory() creates one Promise per revision and awaits Promise.all(), which can exhaust
memory and overwhelm the DB for large pads (the target use case). This can degrade or crash the
Etherpad process during compaction.

Agent Prompt

### Issue description
`compactHistory()` deletes all revision keys by building a `deletions` array and calling `Promise.all(deletions)`. For large `head` values this can allocate huge memory and issue a massive number of concurrent DB operations.

### Issue Context
`Pad.remove()` already uses bounded concurrency via `timesLimit(..., 500, ...)` for the same style of per-revision deletions.

### Fix Focus Areas
- src/node/db/Pad.ts[600-605]
- src/node/db/Pad.ts[706-714]

### Fix approach
Replace the unbounded `Promise.all(deletions)` pattern with a bounded-concurrency loop, e.g. using `timesLimit(originalHead + 1, 500, async (r) => { await this.db.remove(`pad:${this.id}:revs:${r}`, null); })` (mirroring `Pad.remove()` semantics).

This avoids O(N) Promise allocation and prevents DB overload.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

[qodo-free-for-open-source-projects]

5. No session eviction/lock 🐞 Bug ☼ Reliability

compactHistory() performs destructive revision deletion and resets head/atext without kicking
connected users, so concurrent edits can race with the compaction and lead to lost updates or
inconsistent client state. Etherpad already kicks sessions before destructive pad removal, but
compactHistory() does not.

Agent Prompt

### Issue description
`compactHistory()` deletes all revisions and resets in-memory pad state while users might still be connected and editing the pad. This can race with `handleUserChanges()` and cause lost updates or client errors.

### Issue Context
`Pad.remove()` calls `padMessageHandler.kickSessionsFromPad(padID)` before destructive operations.

### Fix Focus Areas
- src/node/db/Pad.ts[581-613]
- src/node/db/Pad.ts[673-679]
- src/node/handler/PadMessageHandler.ts[177-189]

### Fix approach
Before deleting revisions/resetting state, kick active sessions from the pad (and ideally ensure no further queued edits are processed) similarly to `Pad.remove()`.

At minimum:
- Call `padMessageHandler.kickSessionsFromPad(this.id)` at the start of `compactHistory()` when `originalHead > 0`.

Optionally, also add safeguards to prevent concurrent mutations during compaction (e.g., a per-pad mutex or equivalent queuing).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools