2021-12-11: Filecoin Data Storage Drop Analysis

[jennijuju]

TL;DR: We believe 1+ Filecoin SPs storing replicas of data for the Slingshot competition terminated a number of sectors yesterday. Slingshot clients involved will need to make new redundant deals with different storage providers over the next few days to refresh their target data replication rate.

An event where an individual storage provider has a failure and loses data due to force majeure, is a normal occurrence for all storage networks. For example, in global storage networks, individual machines and even full data centers may have extended outages - which are mitigated by having many globally redundant replicas.

The Filecoin network similarly helps ensure resiliency despite localized failures by:

• Creating many copies of data across many different geographic locations (the Slingshot competition, for example, rewards up to 10 independent copies of data to increase resiliency)
• Replicating data stored in the network across many independent storage providers (clients usually store deals with 4-6 independent providers)
• Repairing deals in light of failures by creating new replicas with new storage providers when an SP goes offline, maintaining a high storage redundancy rate over time.

On Dec 11, 2021, 1+ Filecoin Storage Providers (SPs) terminated around 465k sectors containing ~420K deals. We think this is related to a SP that has claimed all their sector data and original raw data were lost due to a fire (Slack thread source). The full list of the storage providers that are involved in this event can be found here. As a consequence of terminating their sectors, these storage providers burned 56k FIL as penalties taken from their available balance, locked rewards and collaterals.

Based on the initial investigation, we believe these SPs are participating in Slingshot and the terminated sectors mostly contain replicas of data stored for the Slingshot competition. According to Slingshot’s rules, this data must be stored redundantly in 10 independent replicas across 4+ miners (rules), so the impact of a single copy going offline is minimal.

The Slingshot team is currently working on gathering a list of deal IDs and piece CIDs (initial draft here) that were in the terminated sectors to initiate normal repair actions and maintain data replication rates. The Slingshot clients involved will need to make new redundant deals with different storage providers over the next few days to refresh their target data replication rate. In the future, with the addition of a programmability layer like FVM, this process can be smoothly automated via “repair” contracts and won’t need client engagement.

This is a great reminder for all Filecoin network participants to ensure healthy replication across many independent storage providers to ensure resiliency to faults and smooth repair!

[jennijuju]

x-posting from #slingshot-announcements

Dear Slingshot community,

As you saw from the above post, over this weekend, several collocated storage providers terminated many PiBs of sectors containing Slingshot data. Storage outages happen frequently in storage networks/datacenters/etc., requiring repair and recovery as a result. It was expected that this would happen on Filecoin too (hence, the idea of repair mining in the original Filecoin whitepaper). We are initiating an effort to Repair and Restore the data that was contained in these terminated sectors.

Repair is the process of creating new replicas by retrieving a surviving replica and making new deals with one or more storage providers. Restore is a process in which network participants make new deals by getting the data from the source of the public Slingshot dataset source. We will publish more information about the Repair and Restore program later this afternoon.

In the meantime, the Slingshot Admin Team has been doing an additional investigation into the incident:

• There were ~1677TiB of unique data (~13PiB of total data) that were included in the terminated sectors
• ~1623TiB (97%) of the data had all replicas lost from the network simultaneously
• Not all the data was from Slingshot, but the VAST majority was.

A few implications:

1. Several projects starting in Slingshot phase 2.1 likely had been cheating the rules and storing all 4-10 piece replicas with the same/correlated storage provider cluster(s). For example, one of the involved participants shared that all of the SPs they worked with are located in the same IDC that caught on fire which resulted in data storage loss (source). This type of scenario is exactly what the Slingshot rules were designed to prevent, but some participants successfully cheated the system, resulting in major impact to the network.
2. As a result, only ~54TiB of unique data still has at least 1 replica in an active deal in the Filecoin network. We will start a Slingshot Repair effort to focus on retrieving these pieces from the network, and make new repair deals with new SPs to preserve redundancy. More details coming later today.
3. As mentioned above, there is still ~1623TiB of unique data that we cannot repair in the same way that we normally would as all replicas are gone from the network. Fortunately, since the majority of this data is Slingshot data, i.e. data from public sources, these data can be downloaded from source, processed, and re-stored on Filecoin. We are launching Slingshot Restore later today to work with the Filecoin community and restore these datasets on Filecoin.

Slingshot Phase 2 has largely been a huge success, but we know there is a need to adjust the program as we enter the new year and take stock of the goals that matter most for Filecoin data onboarding in 2022. We also need to account for the actions that led to this data loss. We are taking the following actions immediately:

• Slingshot 2.7 will continue as planned, but this will be the last phase of Slingshot 2 (as mentioned before). We will revisit in 2022 how we want to continue the Slingshot program with new initiatives.
• If a Slingshot project had all of their replicas for any individual data piece in the sectors that were recently terminated:
• That team will be disqualified from earning any rewards in 2.7.
• All ongoing vesting rewards from earlier phases stop immediately. These rewards will instead be allocated towards the Repair/Restore efforts.
• We will publish this list of teams that are affected as soon as possible.
• Starting in 2.7, Slingshot rewards will vest over 12 months instead of 6 months (to align with required min deal duration).
• For those participating in 2.7, remember to select reliable storage providers to store your data. While a reliable SP is expected to have 1:N replications of the data to prevent single data source of failure that might lead to client’s data loss, it is highly recommended that the client should also ensure having healthy replication by making multiple deals for the same data with more than one independent storage providers, ideally geographically distributed to ensure resilient to faults and smooth repair! This is a requirement for the Slingshot competition and it is the client’s responsibility.

We are truly looking forward to building repair functionality into Filecoin, and demonstrating the resilience of the Filecoin network in the face of these kinds of events. We’ll be sending out more information over the next couple days as we begin the Repair and Restore effort to recover several PBs of public good data on the Filecoin network.

Thanks,
The Slingshot Team

[jennijuju]

As mentioned in the earlier announcement, we are launching a program called Slingshot Restore to restore ~12.5PiB of Slingshot data on Filecoin as quickly as possible. You can read about the full program details here.
Please note that we are accepting applications from clients and storage providers that want to participate in Slingshot Restore. → Please submit your application by December 16, 2021 at 2300 UTC // 1600 PT. ← There are some pretty big rewards associated with participating in this program and achieving massive success with the first Filecoin large-scale repair effort .
Looking forward to seeing your applications

[jennijuju]

we are launching a Slingshot Repair program that aims to restore replication levels of the Slingshot data that still has at least one active deal on Filecoin. You can read about the full program here.
Repair will be done using Estuary - so one way you can help the effort is by signing up to become one of Estuary Storage providers. The deals will be assigned algorithmically so there is no guarantee that deals will be allocated to any specific Storage provider.
We are setting up a dashboard where you can track the progress of the program - please stay tuned for further announcement!