State Cleanup Ideas

[Stebalien]

See #771 for gas improvements, this discussion focuses on cleaning up dead state.

Delete miner state when unenrolling from cron

We unenroll from cron when the locked funds and pledge all go to zero, indicating that there are no live sectors and no remaining vesting funds:

https://github.com/filecoin-project/builtin-actors/blob/8ca9c6c667fd069705ca186a7e4fd2346d6b4cd1/actors/miner/src/lib.rs#L4400-L4410

When that happens, we have an opportunity to remove miner state.

Ideally we'd just send all funds to the beneficiary and delete the entire actor, but some users may object to that. Instead, we can just delete all the sector infos, deadline information, and market information:

1. Zero out all miner fields except:
   1. info (miner info).
   2. fee_debt (could still be non-zero).
   3. allocated_sectors (we still don't want to reuse sector IDs)
   4. proving_period_start
2. Remove the miner's claims from the verifreg. They should all be expired? Can we do this?
3. Remove the miner from the market's provider_sectors table. The deal states will get cleaned up automatically over time.

This will still leave some dead state, but nothing that won't be cleaned up automatically (unless I'm missing something).

Forced Sector Cleanup

The above only handles "dead" miner actors. If we want to cleanup sectors from live miner actors, we likely need to force it.

At the moment, any SP can run CompactPartitions to cleanup/compact their partitions, deleting dead sector infos. Unfortunately, SPs rarely do this as the cost of keeping around dead state is pretty low.

One option is to have forced garbage collection by:

1. Tracking the ratio of live to dead sectors in a partition/deadline.
2. Once we reach some ratio (e.g., 1:1), the SP is incentivize to compact their partitions. E.g., by:
   1. Charging additional ramping gas as the ratio of live/dead sectors grows.
   2. Preventing them from withdrawing funds.
   3. Disabling vesting.

The idea is to provide a strong incentive to "fix" the problem quickly, without causing an emergency.

Forced Claim Cleanup

Forcing the cleanup of expired claims would require a little bit of extra state, but it's possible. Basically, we'd:

1. Break the "max claim" time into "cleanup periods".
2. Record an array (ring buffer) counting the number of claims we expect to expire within each claim period.
3. When we reach the end of any given claim period (plus some buffer?), incentivize the SP to cleanup the number of claims recorded in that period.

Given that the term size is 5 years, we can keep keep track of this with a 60 element ring-buffer (maybe a bit longer to take term starts into account?).

In terms of incentives, increasing gas fees as "cleanup debt" builds seems like a decent approach. We could also have a hard cutoff, but I'm not a huge fan of those.

[arajasek]

I'm in favour of this general workstream, so I'm glad to see some ideas about how we can improve this -- the size of the current state tree is an invisible tax that saps energy from the Filecoin ecosystem (node operator time in downloading a snapshot, barrier to new folks joining the network, processing time of anyone performing analytics over the entire state, developer and protocol evolution time when we perform state migrations).

Having said that, I'm not especially inclined towards the ideas listed here. I don't really want to add more work into cron (even deleting state), and Forced Cleanups, while agreeable, are ultimately just more protocol logic that I'm disinclined to add.

What I would personally be more favourable towards is one-off cleanups that occur at network upgrades. This keeps the protocol simpler (no new potentially buggy code), doesn't add anything into cron (which would only affect things moving forward anyway), and gives the network much more control over the cleanup: we can all observe and agree exactly what would get cleaned up at the migration time. I don't want to derail this discussion, so if there's support for that idea I'm happy to write up an alternative proposal.

[Stebalien]

I also don't want to add too much work to cron, but I'm willing to add a bit of work if:

1. The implementation is reasonably simple.
2. The overhead is minimal (e.g., if we're already updating something, we do a bit of O(1) cleanup).

Also, I don't think this is high priority. But these random cleanups are potentially a good way for someone new to get into protocol development.

[dd45e640b42e6da7da96faee3996ef7c]

much needed discussion.

literally years ago we discussed somewhere to allow active miner termination, delete the actor, under certain conditions in exchange for unlocking the vested rewards immediately. would give an extra ~6 month less state compared to waiting for all funds to release from reward vesting.

In terms of incentives, increasing gas fees as "cleanup debt" builds seems like a decent approach.

i don't like this approach at all. its literally a punishment we try to sell as an incentive here.

In terms of punishment, increasing gas fees as "cleanup debt" builds seems like a decent approach.

this should not be the beginning of the discussion on how to design incentives to "force" positive behavior. most of the screws you wanna turn can easily be turned in the opposite direction to have folks benefit from doing this. we as a community agreed that those states are needed and being kept in the way they are. asking a specific group to come up for the clean up cost by threatening a punishment should be the last resort. between cutting the 180 days vesting time down to 0 and Disable vesting. (or increasing/discounting gas fees as "cleanup debt/balance" builds) are plenty of stages of escalation.

in the end i agree with @arajasek that this is best solved via network upgrades. it's community debt, not debt of specific actors on chain.

[Stebalien]

literally years ago we discussed somewhere to allow active miner termination, delete the actor, under certain conditions in exchange for unlocking the vested rewards immediately. would give an extra ~6 month less state compared to waiting for all funds to release from reward vesting.

I'm strongly against this as it adds additional incentive to terminate sectors (so one can cash out by deleting their SP). However, we can:

1. Make vesting "lazy" (i.e., vest on-demand when the SP withdraws funds instead of eagerly in cron).
2. Cleanup all state except the vesting schedule and the root miner actor object once the miner becomes "inactive".

That would effectively turn "zombie" miner actors into vesting multisigs, with little impact on chain state/cron.

In terms of incentives, increasing gas fees as "cleanup debt" builds seems like a decent approach.

i don't like this approach at all. its literally a punishment we try to sell as an incentive here.

It's paying for unnecessary chain state. The idea is that SPs should never hit those fees unless they fail to cleanup dead state.

However, thinking about this, I think the best way to handle this is to just trigger an automatic cleanup the next time the SP tries to claim an allocation. This will be expensive, but the SP can avoid that cost by triggering a targeted cleanup before it becomes an issue.

In terms of punishment, increasing gas fees as "cleanup debt" builds seems like a decent approach.

this should not be the beginning of the discussion on how to design incentives to "force" positive behavior

This is how blockchains work. You align incentives by rewarding good behavior (usually through block rewards) and by punishing bad behavior (slashing, fines, etc.).

We could (and likely should) have some upgrades to "cleanup" state, but that requires ongoing work by teams that could otherwise be focused on improving the protocol in lasting ways.

[dd45e640b42e6da7da96faee3996ef7c]

i browsed over it for some time yesterday after my comment and the scientific studies (i have for sure not read nearly enough to come to a final conclusion) seem to point in the direction of:

• if you want people to do something: you reward
• if you do not want people to do something: you punish

https://affectivebrain.com/wp-content/uploads/2014/09/Action-controls-dopaminergic-enhancement-of-reward-representations.pdf for example. or a bit more practical https://academic.oup.com/cid/article/54/1/1/368989

not being punished is not a reward just because there is a monetary gain connected to it.

I'm strongly against this as it adds additional incentive to terminate sectors (so one can cash out by deleting their SP).

i am not sure if this statement works out as termination fees would consume most of the vested rewards. but i am not sure if thats accurate and if it is as close to 100% of vesting rewards as i expect it to be.

terminating everything to cash out should roughly mean to give up vesting rewards to get the pledge back - right now.

[Stebalien]

This is a "pay your taxes or get fined" situation. We're not talking about teaching children how to behave.

[Stebalien]

To be clear, I'd love to be able to rework this to rely on rewards instead, but designing effective and difficult to game reward systems is quite difficult. On the other hand, we already have such a system (the block rewards). Any penalties here are effectively just deductions from said block reward.

Also note: I'm pretty sure we can do all this with minimal penalties, if any. E.g., prevent withdrawal of funds, automatically trigger cleanup if the SP doesn't, etc. But there will likely be some cases where we have no better option.

[steven004]

Ideally we'd just send all funds to the beneficiary and delete the entire actor, but some users may object to that.

I think we have the opportunity to implement this by allowing the miner owner to terminate a "zombie" miner and delete all related state data.

FIP-0077 proposes adding an opportunity cost for creating new miners. By integrating this idea, we could treat the fee for new miner creation as collateral, which the network would retain until the miner owner decides to terminate the miner actor. This approach could simplify both the implementation of adding costs and the removal of miner states, as they would become more independent.

For the cleanup of other forced sectors or claims, I would prefer handling this during a network upgrade, as suggested by @arajasek .

[rvagg]

I think a problem with doing this as collateral is that it doesn't really help address the "spam" issue in terms of state churn, which has its own cost. It might be true that it should lead to less miner-related state information at any point in time, but it doesn't provide a disincentive to spam the chain creating new miners, as per FIP-0077, leading to state churn. This churn has significant cost on archival nodes and snapshot sizes when spread over time. There's an awkward balance between instantaneous state size, and over-time state churn, with the costs often being worn by different parties. Just having fewer unused miner IDs created overall would be good for both.

[steven004]

Just having fewer unused miner IDs created overall would be good for both.

Agreed.
A more aggressive approach would be to charge a creation fee directly when creating a miner, which is purely a cost, not collateral.
This would effectively prevent the creation of unused miner IDs. At the same time, it would not really increase the cost of creating a miner that much, as owners of miner IDs can offset most of the costs by directly selling the miner on the trading market (e.g. SPex) when it is unused for her.

This approach would not only prevent the creation of unused miner IDs, but could also deter the creation of new miner, since the cost of creating a new miner might exceed the cost of purchasing one from the market.

[ZenGround0]

If we require running compact partitions to release IP after expiration this would solve 95% of future state issues. And it would serve the extra benefit of removing more work from cron. If we ever need/get to safe cron and move handle proving deadline out of cron we could just do compact partitions inline.

There is the additional problem of 30-40G of dead sector infos belonging to SPs who may be long gone from the network. There's a lot of things we could do here but the simplest thing is probably just a migration to clean these up.

[rvagg]

If we require running compact partitions to release IP after expiration this would solve 95% of future state issues.

@ZenGround0 can you explain this a little more? Starting from what "IP" stands for in this context. How would this solve future state issues?

[ZenGround0]

Absolutely

By IP I mean "Initial Pledge" which is a maybe deprecated term for the FIL collateral that SPs post for security along with their sector.

Right now the vast bulk of the garbage state and ~ 25-40% (guessing cause I haven't measured in a year) of the total bytes of a snapshot are the sector infos (dominated by CommR hashes) of sectors which have expired or terminated.

Right now SPs get their pledge back when sectors expire and terminate. However neither expiry nor termination clean up the sector infos. There is an expensive essentially-never-called operation called CompactPartitions to do that. The only incentives SPs have to do that are to slightly reduce their future gas costs on operations by having smaller data structures and to be good network citizens.

If we returned pledge only after compacting partitions then there would be an incentive to do this and we'd have a cleaner state tree moving forward. Of course it only works for future sectors leaving the network. Any that have already left no longer have a pledge to give back. Its maybe a bit better than that because we could force cleanup per deadline to claim any pledge and so already dead sectors in live deadlines would get cleaned up. But this doesn't address the common case of miner actors that are fully dead with no live pledge left to incentivize cleanup.