`application.db` growth / pruning

[tcharchian]

Discussed in #817

^{Originally posted by tcharchian February 27, 2026}

Task: application.db is growing unexpectedly; pruning does not appear to be effective. This impacts storage and performance.

Reports from multiple node operators indicate that block pruning is not reclaiming space as expected, which can result in continued growth of application.db and increased storage pressure. This issue tracks investigation, evidence gathering, and community-proposed mitigations.

Sustained application.db growth can lead to:

• disk exhaustion and downtime
• performance degradation
• more operational overhead during maintenance and upgrades

Observations

• Pruning may be configured, yet application.db continues to grow on some setups.
• Similar symptoms have been reported by multiple independent hosts
• The current state suggests a need to confirm whether this is:
  • a configuration issue,
  • expected retention behavior,
  • or a bug / unintended interaction

Goals

• Identify the most likely cause(s) and validate them with reproducible evidence.
• Produce a short guide with:
  • recommended pruning settings (where applicable),
  • safe maintenance steps (e.g., compaction guidance),
  • monitoring suggestions (growth rate, alert thresholds).
• If a code-level change is justified, track it via linked PRs and governance-appropriate process.

[Mayveskii]

I’d like to work on this issue, will start by reproducing application.db growth and investigating pruning behavior

[AlexeySamosadov]

PR: #846

[Mayveskii]

Root cause: application.db growth — code review + live data
TL;DR: Not a config problem. The pruning system silently deletes nothing on nodes that missed upgrade v0.2.4 (InferencePruningMax = 0 in DefaultEpochParams), and 9+ large collections were never pruned at all.
Live data (epochs 158–188, gonka.gg):
2.3M inferences over 31 epochs, peak 241K/epoch
Estimated application.db growth: ~7 GB (with payloads) / ~1.2 GB (metadata only)
Root causes in code:
InferencePruningMax = 0 by default (inference-chain/x/inference/types/params.go, DefaultEpochParams). On nodes without upgrade v0.2.4, pruning deletes zero records per block. The upgrade handler (app/upgrades/v0_2_4/upgrades.go) sets it to 5000 — but only for nodes that actually ran it.
Deprecated payload fields still stored on-chain. inference.proto still carries prompt_payload, response_payload, original_prompt inside every Inference written to application.db via SetInference.
9 large epoch-keyed collections never pruned. EpochPerformanceSummaries, ConfirmationPoCEvents, PoCValidationsV2, MLNodeWeightDistributions and others in keeper.go have zero pruning logic and grow unboundedly regardless of config.

[Mayveskii]

PR: #846

were exactly working on it....

[AlexeySamosadov]

Summary of what was done in PR #846:

Root cause identified: 9 epoch-keyed collections were never pruned, only 3 were cleaned up (Inferences, PoCBatches, PoCValidations v1). The unpruned collections:

1. PoCValidationsV2
2. PoCV2StoreCommits
3. MLNodeWeightDistributions
4. EpochGroupData
5. EpochGroupValidations
6. EpochPerformanceSummaries
7. ConfirmationPoCEvents
8. RandomSeeds
9. PoCValidationSnapshots

Additionally, InferencePruningMax and PocPruningMax defaulted to 0 in the proto definition, meaning pruning was silently disabled on any node that didn't receive the v0.2.4 upgrade handler.

Fix: extended Prune() to clean all 9 collections, added fallback default (5000) when configured max is zero, set sane defaults in DefaultEpochParams(). Tests added and passing.