No Buffer PoRep: Removing 10 layers storage between PreCommit and ProveCommit

[lucaniz]

@Kubuxu @lucaniz @irenegia @nicola @rosariogennaro

Problem

The current interactive PoRep protocol (PreCommit + ProveCommit) requires to seal and keep a buffer of 12 layers (11 SDR layers + 1 data layer) between PreCommit and ProveCommit. This is a cost that we are targeting to reduce.

We present the solution space looking for feedbacks on the more impactful solution for the network.

Solution Space

We have two different approaches, resulting in three possible solutions:

1. Make PoRep non-interactive: NI-PoRep would remove the need to submit PreCommit and would allow to have a unique message for proving new sectors (only ProveCommit would be needed). Challenge generation happens locally and thus there is no need of getting into PreCommit step. Notably, the removal of PreCommit would also remove the need for keeping around 10 layers for the time from PreCommit to ProveCommit (note that during sealing the 10 layers are still produced and are still required to produce the proof)

   In the context of Non Interactive PoRep we identify two different paths forward

   1. NI-PoRep with Groth16 + SnarkPack: Making PoRep Non Interactive now using the current proof system, at the cost of a substantial proving overhead (see later the numbers);
   2. NI-PoRep with Testudo: Making PoRep Non Interactive in the future (~1y from now), betting on using a new proof system result of an on ongoing research effort;

2. Make PoRep Synthetic: Achieve reduction in used up space by reducing the set of challenges that might be chosen during the interactive Commit step from all possible challenges to some predetermined number that is feasible to precompute.

Summary

	Buffer	Proving overhead (wrt current PoRep)	Research Needed	Time Horizon
NI-PoRep + SnarkPack	Produced and immediately consumed	~8.5x	No	Ready to be implemented
NI-PoRep + Testudo	Produced and immediately consumed	TBD (hopefully minor)	Yes (research heavy)	Not earlier than 1y
Synthetic PoRep	Produced, reduced to 6% and consumed with 150 epochs delay	Low	No	Ready to be implemented

In what follows we give more details for each possibility

Approach 1: Non Interactive PoRep

The real change with respect to now is that challenges are created locally without on-chain interaction.

Due to this, on one hand we can merge PreCommit and ProveCommit in a unique step, but on the other hand this flexibility would allow a malicious prover to grind challenges. To mitigate and ensure this it is sufficient to increase the number of challenges (see below)

High Level Protocol Flow

Using the Fiat-Shamir heuristic, the protocol can be summarized as follows:

1. SP computes a Replica R, using CommD.
2. SP uses CommR to locally generate a seed via an hash function seed = H(CommR,...,) .
3. SP uses the seed to compute challenges over which he has to produce a proof that is posted onchain.

Fiat-Shamir Heuristic needs 80 bits of security (instead of the current 10), which means that we need 8x current number of challenges compared with today. The increased number of challenges brings over an additional proving overhead, which can be different in the two scenarios

(1.i) NI-PoRep with Groth16 + SnarkPack

We can use current proof system getting NI-PoRep with 8x challenges and at 8x proving cost (ie, we need 80 snark proofs instead of the current 10). In order to reduce this overhead (on chain footprint and verification), we can use SnarkPack. From the paper, we estimated that SnarkPack proof size amortization will kick in when sealing at least 2 sectors but terms of the verification even one NI-PoRep already benefits from SnarkPack (note that the more sector you seal at a time, the better the amortization becomes).
In this case, the cost overview will be:
Let’s recap the new costs:

• Proving cost:

  • 8x snark cost compared with today (80 snarks)
  • SnarkPack Aggregation cost: this cost is logarithmic, so log(80) ~ 6.7

  ⇒ Total proving overhead: 8.7x

• Verification cost: it is the SnarkPack cost (counting each NI-PoRep sector as 8 sectors with the current proof system).

• Proof size: comparable to today.

  • See the graph (from the paper): breakeven at ~130 proofs (23Kb for SnarkPack proof Vs 24kb for 130 Groth 16 proofs). It follows that this approach starts being meaningful regarding proof size when sealing 2 sectors together (Single Groth 16 proof is ~0.2Kb)

Notes

• NI-PoRep with Groth16 + SnarkPack does not require a new trusted setup

(1.ii) NI-PoRep with Testudo

We could have proving cost comparable to today (or even better) upon the launch of [Testudo ](https://www.notion.so/Testudo-1201ca34e72d44c899e8242bb6202a25), which is in early research stage and is not expected to ship in 2023.

Approach 2: Synthetic PoRep

This part is taken from a former discussion (See #245)

We propose a Synthetic PoRep protocol where

• A miner can complete the challenge generation and vanilla proof computation before performing PreCommit on-chain thus removing layers data before the sector is pre-committed on-chain.
• The GPU cost for SNARK generation during Commit is not significantly increased

High Level Protocol Flow

Differences between currently deployed PoRep and Synthetic PoRep are limited to challenge generation and additional capabilities for the miner.

1. Starting point
   1. We assume there is a sector S for which the miner completed PreCommit1 and PreCommit2 computations.
   2. This sector is not listed on-chain.
   3. The miner possesses knowledge or CommR and CommD of that sector (acquired in PreCommit2 step) and the layers needed to generate CommR
2. Miner generates “synthetic” challenges from CommR

   1. Based on the CommR, the miner generates a list of SynthChallengesNumber (currently ~270k) challenges Ch.

      all

   2. Knowing all the challenges, the miner computes responses for them which take the form of SynthChallengesNumber vanilla proofs and saves them for future use.

3. Miner can remove layers data
   1. As the miner knows responses to all possible challenges that will be asked in the interactive step, he can remove the layers data which is needed to respond to challenges.
4. Miner publishes “PreCommitsSector”
   1. Using the same flow as today miner submits the sector for PreCommit.
   2. This establishes when in the future the randomness for interactive response will be known (PreCommitChallengeDelay)
5. Miner generates and publishes ProveCommitSectors proof

   1. Miner waits PreCommitChallengeDelay (~150 epochs).

   2. The randomness revealed at PreCommitEpoch+PreCommitChallengeDelay selects VerifiedChallengesNumber (176) challenges to be verified on-chain from Ch.

      all

   3. Miner takes the VerifiedChallengesNumber vanilla proofs which were generated earlier corresponding to selected challenges and computes SNARK proofs of these challenges.

   4. Miner publishes the ProveCommit either in individual or aggregated form.

6. Chain verifies proof
   1. Using interactive randomness as a seed chain generates VerifiedChallengesNumber challenges by selecting VerifiedChallengesNumber indices out of SynthChallengesNumber and computing them.
   2. Generated challenges are fed into proof verification.

Parameter Selection

Based on the current analysis following parameters are under consideration:

• SynthChallengesNumber = ~271,000 with VerifiedChallengesNumber= 176 - verification cost similar to today
• SynthChallengesNumber = ~84,000 with VerifiedChallengesNumber=198 - verification and GPU cost increased by 12.5%

[luckyparadise]

Hi @lucaniz thanks for pushing this out. I am a Project Manager with the Governance team at Filecoin Foundation. Are you hoping to land this as a FIP anytime soon?

[lucaniz]

@luckyparadise at the current stage we are looking for feedbacks and comments on which solution would be the most impactful for the network

[anorth]

Dear readers who are storage providers: please tell us which of these you think would be most beneficial for your operation, and why.

• NI PoRep: No pre-commit, no pre-commit deposit, no intermediate data storage, just one-shot proving of replicas, but with 8x SNARK computation cost
• Synthetic PoRep: still pre-commit, but intermediate storage is small. Same SNARKs.

[deltazxm]

NI PoRep,Because SNARK is cheap now.

[Fatman13]

Would ASIC be a concern for NI PoRep?

[hail100]

I want know what hardware needs？

[nai2283]

NI-Porep+snarkpack sounds like a bad direction. Precommit and Commit require very different hardware configuration, in order to maintain the same level of sealing capacity, 8.5x proving effort means 8.5 extra investment on commit rigs, imo the clearance precoomit rigs just can't compensate that extra costs.

[NSC-FIL]

As an SP, I'm for keeping the current PoRep process to keep our network stable while the FVM and L2 is being developed, so we can isolate issues and bugs, as these are rolled out. I'm for more research and testing at scale on NI-PoRep with Testudo for this reason.

[lucaniz]

Thank for your comment. current PoRep and "new" one would co-exist (like having 2 options in place instead of one). Would this change your point of view? How and why?

[SBudo]

I would be more inclined to vote for the synthetic PoRep.
Immediate gain in space savings, slight increase in GPU, but fast enterprise NVMe storage is more expensive than GPUs these days.