Recoverability test: ensure realistic SP outages are recoverable "quick enough"

[BigLep]

Done Criteria

After the 202601-2 wave of Curio hardening/performance improvements, take an SP out of service for ~5 minutes (e.g., simulating a power outage) and measure how long it takes to recover from the backlog.

Why Important

Power outages will occur and we want to make sure that these events don't tank an SP.

Notes

1. A real power outage happened on 2026-01-21 affecting calib.ezpdpz.net and calib2.ezpdpz.ne. It lasted 5 minutes, but took ~12 hours to catch up (slack thread). Note that this happened in calibration which has a 12x more frequent proving period, but we believe this is a simulation of what will happen if a mainnet node has 12x the number of datasets as these nodes had.

[BigLep]

@TippyFlitsUK: can you please add any other data about what you observed on 2026-01-21 so we have it to compare against for when we simulate a power outage again with the various 202601-2 performance improvements?

It would be good to know: how many datasets, how much total data store, how long it took to recover (in Slack I think you said 12 hours).

[TippyFlitsUK]

Thanks, Steve.

Approximately 12 hours is the recovery period I observed

A few additional details:

PDP Node Summary - calib.ezpdpz.net

Metric	Value
Total Datasets	1,624
Active Datasets	1,187
Terminated Datasets	437
Total Pieces	20,061
Total Data Stored	665 GB

Breakdown by Status

Status	Datasets	Pieces	Data Size
Active	1,187	12,014	519 GB
Terminated	437	8,047	146 GB

Task analysis - calib.ezpdpz.net

Period: Last 48 hours (Jan 20-22, 2026)
Generated: 2026-01-22 14:19 UTC

Retry Configuration

Task	Source File	MaxFailures	RetryWait
PDPv0_ProvPeriod	task_next_pp.go:271	3	Exponential (5s × 2^n)
PDPv0_Prove	task_prove.go:790	5	None (immediate)

Task Success Rates

Task	Unique Tasks	Succeeded	Failed	Success Rate
PDPv0_ProvPeriod	29,330	28,324	1,006	96.57%
PDPv0_Prove	28,672	24,913	3,759	86.89%

Retry Distribution

PDPv0_ProvPeriod (max 3 attempts)

Attempts	Tasks	Succeeded	Failed
1	27,331	27,331	0
2	598	598	0
3 (max)	1,405	399	1,006

PDPv0_Prove (max 5 attempts)

Attempts	Tasks	Succeeded	Failed
1	24,176	24,176	0
2	242	242	0
3	186	186	0
4	201	201	0
5 (max)	3,868	109	3,759

Key Observations

• Tasks only record failure after exhausting ALL retries
• PDPv0_Prove: 3,759 tasks (13.1%) hit max retries and failed
• PDPv0_ProvPeriod: 1,006 tasks (3.4%) hit max retries and failed

[beck-8]

I just tried taking my node down for a few hours. A large number of tasks are also generated during startup and occupy a large amount of memory. However, it has not been found to affect the upload of new data.

---

Waiting for several hours

---

The scheduling system also seems to have some issues.

{"level":"info","ts":"2026-01-23T22:19:21.501+0800","logger":"harmonytask","caller":"harmonytask/task_type_handler.go:195","msg":"Beginning work on Task","id":7387708,"from":"poller","name":"PDPv0_Prove","sector":null}
{"level":"debug","ts":"2026-01-23T22:19:21.612+0800","logger":"harmonytask","caller":"harmonytask/harmonytask.go:419","msg":"skipped scheduling UpdateStore type tasks on due to Did not accept UpdateStore task: out of RAM: required 536870912 available 325115904)"}
{"level":"debug","ts":"2026-01-23T22:19:21.618+0800","logger":"harmonytask","caller":"harmonytask/harmonytask.go:419","msg":"skipped scheduling ScrubCommDCheck type tasks on due to Did not accept ScrubCommDCheck task: out of RAM: required 369098752 available 325115904)"}
{"level":"debug","ts":"2026-01-23T22:19:21.639+0800","logger":"harmonytask","caller":"harmonytask/harmonytask.go:419","msg":"skipped scheduling PDPv0_Prove type tasks on due to Did not accept PDPv0_Prove task: out of RAM: required 2147483648 available 325115904)"}
{"level":"debug","ts":"2026-01-23T22:19:21.648+0800","logger":"harmonytask","caller":"harmonytask/harmonytask.go:419","msg":"skipped scheduling PDPv0_IPNI type tasks on due to Did not accept PDPv0_IPNI task: out of RAM: required 1073741824 available 325115904)"}
{"level":"debug","ts":"2026-01-23T22:19:21.650+0800","logger":"harmonytask","caller":"harmonytask/harmonytask.go:419","msg":"skipped scheduling IPNI type tasks on due to Did not accept IPNI task: out of RAM: required 1073741824 available 325115904)"}
^C
➜  ~ free -h
               total        used        free      shared  buff/cache   available
Mem:            31Gi       8.6Gi       7.6Gi       544Mi        14Gi        21Gi
Swap:             0B          0B          0B