[SEV] F3 Instance: 183760 not reaching a decision

[rjan90]

Current Status

2025-07-22T05:59:14Z: 🔄 In Progress

• Overall chain advancing as expected. All metrics, including block production and chain quality, indicate healthy chain progression.
• F3 is finalizing but is more choppy and further behind than pre-event levels.
• We're still working to identify the cause of the drop in participation that is causing F3 choppy progress as outlined by the "Stop the Bleeding" steps.
• Current hypothesis is that some SP's haven't set the F3 initial power table CID, they no longer possess the full state for epoch 4918680, and then they drop out of participation when they restart with clean state. This at least explains the slowish decline we have seen over the last 30 days, but doesn't explain the large ~5% drop that triggered thisevent. We made a push to get this initial power table CID set (blog entry) but there was no forcing function for SPs to do this. We expect a network upgrade will help here. A network upgrade will also help get nodes updates to the latest SP software which has F3 resiliency improvements based on the two mainnet participation events (plus other security improvements).

What Happened / What Was Observed

The chain is advancing with Expected Consensus (EC) as expected. All metrics, including block production and chain quality, indicate healthy chain progression.

F3 started experiencing a lack of decision due to what we thought was a similar failure mode as previously encountered in #13094 (comment).

Root Cause Analysis

The failure mode causing F3 to lack progression this time around appears to be:

• Aider process failure: The Aider process that we had set up to help propagate messages failed to kick in due to disk space issues / disk rotation problems on the observer node that gathers messages in the network.
  • Message propagation disruption: The Aider process could therefor not help propagate/broadcast the messages to the network because it was just receiving zero-byte files due to lack of disk space on the observer.

Upon further investigation, it looks like the lack of progress was actually triggered by a drop in participation:

Without the Aider effectively running with enough saved stated in the Observer, it wasn't able to help offset this decline in participation.

Stop The Bleeding

• ✅ The Aider and the Observer node are now back up and running again, and are helping propagating messages around in the network
  • ⏰ Round timing: Due to exponential backoff function in F3, rounds keep doubling and the CONVERGE step is timeout based to give all nodes the timeout amount to receive messages.

Round	CONVERGE timeout	CONVERGE ending datetime
10	7h	~2025-07-11 23:30 UTC
11	14h	~2025-07-12 13:30 UTC
12	28h	~2025-07-13 17:30 UTC

• (in progress) See if anyone on #fil-fast-finality is aware of the ~5% participation drop that was observed around ~2025-07-11 2:20 UTC. (slack thread)

  • As of 2025-07-21, no reason has been determined

• (in progress) Give visibility to minerIds not participating in F3 at #global-storage-provider-community so hopefully figure out any fixes or improvements that need to be made to F3 software.

  • Post: https://filecoinproject.slack.com/archives/C02GQUMFQVA/p1752704795614989
  • 2025-07-21: this hasn't lead to any determinations of SP behavior or code/config bugs that could have triggered this event.

Action Items after Stopping the Bleeding

Note: The longer-term fix for the mentioned propagation issue #13094 have landed (e.g., filecoin-project/go-f3#1024). They would have been helpful in this case. That said, they haven't been bubbled up for certain Lotus/Venus/Forest versions and not yet widely adopted by enough participants in the network just yet for the network to not need the help of the Aider-process. We can count on this more resilient software being deployed after the nv27 network upgrade.

• Identify Node versions that the community should ideally upgrade to, ideally before the nv27 network upgrade
  • Lotus: Lotus Node v1.33.1 Release #13132
  • Forest: next release after 0.27.0 since fix: forest-cli f3 pt get 0 not working ChainSafe/forest#5785 has been merged
  • Venus: Update f3 to v0.8.7 venus#6480
• Get better monitoring in place to catch if Aider isn't actually "aiding" (e.g., disk space monitoring/alarming, monitoring/alarming on the amount of broadcasting the process is doing)
  • Better operationalize the FilOz-run f3-observer and f3-aider go-f3#1041

Note: these exponentially increasing CONVERGE timeouts are painful and we'd like to put an upper limit, but this isn't a quick/easy change. We are living with this for now and are not committing to a change currently.

Additional resources

The internal operational docs for F3 are in https://filoznotebook.notion.site/F3-Operational-Excellence-5cdce4f1aa6e4c398b475f6e690c47fe . This also references a public F3 dashboard: https://grafana.f3.eng.filoz.org/public-dashboards/e9d8fe95ae9a4341ba2e730f1a4c86be

[BigLep]

Per the update to the issue description, the Aider process is restored. We'll be updating on the progress of instance 183760 through the day.

We're also working right now on updating the issue description with current state of action items. The flow will be to make updates there in the description and to leave a comment below stating that there was an update.

[BigLep]

New status posted in the issue description:

• 2025-07-11T16:53:00Z:
  • Overall chain: still advancing as expected with Expected Consensus. All metrics, including block production and chain quality, indicate healthy chain progression.
  • F3 finalization: we aren't catching up yet and are still investigating why.
  • We will keep updating this issue as we learn more.
  • Action items haven't been fully fleshed out yet as we are focused on investigating why f3 isn't making progress.

The key thing we're focused on right now is identifying why F3 still isn't making progress.

[BigLep]

New status posted in the issue description:

2025-07-11T21:09:53Z: 🔄 In Progress

• Overall chain: still advancing as expected with Expected Consensus. All metrics, including block production and chain quality, indicate healthy chain progression.
• F3 finalization: we aren't catching up yet. Participation is just below threshold and the restored Aider hasn't been running long enough to coincide with the long timeouts that we're hitting in round 10.
• We will update this issue after the round 10 CONVERGE at ~2025-07-11 23:30 UTC so see if we start making progress. If we don't, there is still hope for round 11 CONVERGE, which will be at ~2025-07-12 13:30 UTC.
• The action items have been updated in the issue description.

[rjan90]

New status posted in the issue description:

2025-07-13T14:59:01Z: 🔄 In Progress

• Overall chain still advancing as expected with Expected Consensus. All metrics, including block production and chain quality, indicate healthy chain progression.
• F3 finalization: We're now catching up to the head of the chain 🚀. After instance 183760 reached a decision around 2025-07-13T02:33:00Z, F3 was approximately 4928 epochs behind. As of this update, F3 is now just 1400 epochs behind. At the current rate, we anticipate F3 will be fully caught up by end of day.
• We’ll continue closely monitoring progress. Starting 2025-07-14, we'll begin work on the action items described in the issue, once F3 has fully caught up to the head of the chain.

[rjan90]

New status posted in the issue description:

2025-07-14T06:58:42Z: 🔄 In Progress

• Overall chain advancing as expected. F3 finalization has caught up to the head of the chain, and we are now finalising with F3 again. All metrics, including block production and chain quality, indicate healthy chain progression.
• We'll begin work on the action items described in the issue now that F3 has caught up. F3 Progress is still choppy at times, with larger drawdown then expected due to the low participation rate:
  

[BigLep]

2025-07-16T03:18:22Z: 🔄 In Progress

• Overall chain advancing as expected. F3 finalization has caught up to the head of the chain, and we are now finalising with F3 again. All metrics, including block production and chain quality, indicate healthy chain progression.
• We're still working to identify the cause of the drop in participation that is causing F3 choppy progress. The steps we're taking have been added to the "Stop The Bleeding" section.

An action was added to "Give visibility to minerIds not participating in F3 at #global-storage-provider-community" that we'll execute on 2025-07-16.

[TippyFlitsUK]

Hey @shizhigu 👋

Can you please confirm if you are responsible for these SPs? Many thanks 🙏

f01889668
f03055005
f01315096
f03055018
f01518369
f01106668
f02131881
f02131801
f02131855