feat: get piece leaf count #249

SgtPooki · 2025-09-26T18:15:28Z

🎯 Add Piece Details with Leaf Count Information

Overview

This PR adds the ability to retrieve detailed piece information including leaf counts and calculated raw sizes for pieces stored in data sets. This provides users with more granular insights into their stored data, particularly useful for understanding storage costs and data organization.

New Features

1. Enhanced Piece Data Interface

Added DataSetPieceDataWithLeafCount interface that extends existing DataSetPieceData
Includes leafCount (number of leaves in piece's merkle tree) and rawSize (calculated as leafCount * 32 bytes)

2. New API Methods

StorageManager:

getDataSetPieceDataWithLeafCount(dataSetId, pieceId) - Get leaf count for individual pieces

StorageContext:

getDataSetPiecesWithDetails() - Get all pieces with leaf count and raw size information

WarmStorageService:

getPieceLeafCount(dataSetId, pieceId) - Core method to fetch leaf count from PDPVerifier contract

3. PDPVerifier Integration

Added getPieceLeafCount(dataSetId, pieceId) method to interact with the PDPVerifier contract
Retrieves leaf count information directly from the blockchain

Key Benefits

Storage Cost Analysis: Users can now calculate exact storage costs using leaf counts
Data Insights: Better understanding of data organization and piece sizes

Usage Examples

// Get all pieces with details
const context = await synapse.storage.createContext({ dataSetId, providerId })
const piecesWithDetails = await context.getDataSetPiecesWithDetails()

piecesWithDetails.forEach(piece => {
  console.log(`Piece ${piece.pieceId}:`)
  console.log(`  CID: ${piece.pieceCid}`)
  console.log(`  Leaf Count: ${piece.leafCount}`)
  console.log(`  Raw Size: ${piece.rawSize} bytes`)
})

// Get individual piece leaf count
const leafCount = await synapse.storage.getDataSetPieceDataWithLeafCount(dataSetId, pieceId)
const rawSize = leafCount * 32

Files Changed

Core Implementation:

src/types.ts - Added DataSetPieceDataWithLeafCount interface
src/pdp/verifier.ts - Added getPieceLeafCount() method
src/warm-storage/service.ts - Added getPieceLeafCount() wrapper method
src/storage/manager.ts - Added public API method
src/storage/context.ts - Added getDataSetPiecesWithDetails() method

Testing:

src/test/warm-storage-service.test.ts - Tests for WarmStorageService methods
src/test/storage.test.ts - Tests for StorageManager and StorageContext methods

Documentation:

utils/example-piece-details.js - Working example script demonstrating the new functionality

Testing

✅ All existing tests pass
✅ New functionality has tests
✅ Example script successfully demonstrates the feature with real data

Example Output

📊 Data Set Summary:
  PDP Verifier Data Set ID: 158
  Piece Count: 4

✅ Retrieved 4 pieces with details:
  Piece 1: Leaf Count: 319044, Raw Size: 10209408 bytes (9970.13 KB)
  Piece 2: Leaf Count: 319044, Raw Size: 10209408 bytes (9970.13 KB)
  Piece 3: Leaf Count: 319044, Raw Size: 10209408 bytes (9970.13 KB)
  Piece 4: Leaf Count: 319044, Raw Size: 10209408 bytes (9970.13 KB)

📈 Total: 1276176 leaves, 40837632 bytes (39880.50 KB)

Related filecoin-project/filecoin-pin#50

src/storage/context.ts

rvagg · 2025-09-26T18:56:19Z

Leaf count is nice, and useful for total data set if we can get it, but at the piece level we should have it in the Piece CID because we're using Piece CID v2. The only catch is that the way we currently get the piece list for a data set is a little broken and will be returning the wrong size (but the right multihash).

There's two separate issues to resolve here:

That Curio bug in our pdpv0 branch needs to be fixed regardless so it returns the Piece CID as it should be (I had a brief look this week and it wasn't obvious where the counting was wrong, I'm slightly worried it's recording the wrong size on the way in to the database rather than coming out. (For now I'd just pretend they are correct and use them with the understanding that we're going to fix this).
We really shouldn't be asking Curio for the piece list for our data set in the first place. PDPVerifier has the piece list, we just need an accessor for it, it'll probably have to be paginated to account for very large data sets but this is the best version here so we go to the chain, which we can trust, not Curio, which we shouldn't. Then we have the v2 CIDs and we can decode them to get the size (there should be some code in Synapse to help with this in piece.ts, or at least it should point to how to do it if someone wants to give it a go).

Total leaf count for a whole data set though would be useful too if we can get that off the chain.

BigLep · 2025-09-29T17:45:38Z

My understanding of the situation is that the ball is in @SgtPooki court to:

Get leaf count from the PieceCIDv2 (just assume it's correct indepenendent of Curio bug)
Update the PDPVerifier wrapper in synapse-sdk to expose getActivePieces from the PDPVerifier contract. For now, synapse-sdk would expose a getAllActivePieces which then walks the PDPVerifier.getActivePieces until hasMore=false.

SgtPooki

self review.. some comments pointing reviewers to places where i'm not sure about things

SgtPooki · 2025-09-30T18:18:31Z

src/pdp/verifier.ts

+   * @returns The number of leaves for this piece
+   */
+  async getPieceLeafCount(dataSetId: number, pieceId: number): Promise<number> {
+    // TODO: DO we need to call the contract for leaf count?


callout here.. not sure if piece.ts leafCount calculation is enough?

this is ok, but I don't think it's all that useful here; let's remove it for now and ignore leaf counts -- they mostly shouldn't be a concern to the user other than their fairly close relationship to size

SgtPooki · 2025-09-30T18:20:17Z

src/storage/context.ts

+      // TODO: should we call the contract for leaf count? i.e. pdpVerifier.getPieceLeafCount(this._dataSetId, piece.pieceId)
+      const leafCount = getLeafCount(piece.pieceCid) ?? 0
+      // TODO: is there a better way to get the raw size?
+      const rawSize = getRawSize(piece.pieceCid) ?? 0


this is the only place where we're getting setting rawSize and leafCount now.. should we be calling the contract instead of these helper methods?

SgtPooki · 2025-09-30T18:20:56Z

src/storage/context.ts

+        // Parse the piece data as a PieceCID
+        // The contract stores the full PieceCID multihash digest (including height and padding)
+        // The data comes as a hex string from ethers, we need to decode it as bytes then as a CID
+        const pieceDataHex = result.pieces[i].data
+        const pieceDataBytes = ethers.getBytes(pieceDataHex)
+
+        const cid = CID.decode(pieceDataBytes)
+        const pieceCid = asPieceCID(cid)
+        if (!pieceCid) {
+          throw createError(
+            'StorageContext',
+            'getAllActivePiecesGenerator',
+            `Invalid PieceCID returned from contract for piece ${result.pieceIds[i]}`
+          )
+        }
+
+        yield {
+          pieceId: result.pieceIds[i],
+          pieceCid,
+          subPieceCid: pieceCid,
+          subPieceOffset: 0, // TODO: figure out how to get the sub piece offset
+        } satisfies DataSetPieceData


Note: a core contributor should really eyeball this to make sure i'm doing things properly

SgtPooki · 2025-09-30T18:21:30Z

src/test/piecelink.test.ts

+        // Expected leaf count is 2^height where height is calculated from size
+        const expectedHeight = Size.Unpadded.toHeight(BigInt(size))
+        const expectedLeafCount = 2 ** expectedHeight
+
+        assert.isNotNull(leafCount)
+        assert.strictEqual(leafCount, expectedLeafCount)


is this accurate?

SgtPooki · 2025-09-30T18:21:43Z

src/test/piecelink.test.ts

+        // Expected raw size is leaf count * 32
+        const expectedHeight = Size.Unpadded.toHeight(BigInt(size))
+        const expectedLeafCount = 2 ** expectedHeight
+        const expectedRawSize = expectedLeafCount * 32


is this accurate?

SgtPooki · 2025-09-30T18:22:15Z

src/test/storage.test.ts

    })
  })
+
+  describe('getAllActivePieces', () => {


lots of tests and mocking setup in here. I tried following existing patterns, but some mock helpers would be nice. especially for contract calls. Something like this would be amazing:

const mockContractContext = createMockContractContext() mockContractContext.mock(key, singleCallResponse) // key maps to transaction data prefix?

synapse.test.ts has a new mock system that we are trying to move towards

SgtPooki · 2025-09-30T18:24:52Z

src/storage/context.ts

+        rawSize,
+        leafCount,
+        subPieceCid: piece.pieceCid,
+        subPieceOffset: 0, // TODO: figure out how to get the sub piece offset


how do I fulfill this value accurately?

hugomrdias · 2025-10-01T14:25:03Z

Couple of comments here:

i dont think we should have any get all method in the sdk that hammers a contract, developer should be able to paginated to their liking and not be forced into the full list of pieces.
im not sure if we need to have this in the core sdk seems very filecoin-pin specific

Anyway i dont feel like i can fully review this properly, we probably need rod to answers some of the question you added

SgtPooki · 2025-10-06T13:14:52Z

i dont think we should have any get all method in the sdk that hammers a contract, developer should be able to paginated to their liking and not be forced into the full list of pieces.

That makes sense. Can remove.

im not sure if we need to have this in the core sdk seems very filecoin-pin specific

I'm guessing you're talking about the getAllActivePiecesGenerator, getPiecesWithDetails, and getAllActivePieces methods?

We still need some methods exposed from synapse-sdk that aren't currently. I'm all ears for what we want to handle in the sdk and can handle other in filecoin-pin..

I think we should at least export these methods:

pdp/verifier.ts: getActivePieces, getPieceLeafCount
piece/piece.ts: getLeafCount, getRawSize

Will wait for update from @rvagg before doing anything else

rvagg · 2025-10-07T05:43:08Z

src/piece/piece.ts

+ * @param pieceCid - The PieceCID to extract raw size from
+ * @returns The raw size in bytes or null if invalid
+ */
+export function getRawSize(pieceCid: PieceCID | CID | string): number | null {


Not quite right because height only gets us the padded size, we need to unpad using the padding that's also encoded. See #283

rvagg · 2025-10-07T05:43:27Z

src/storage/context.ts

+      // TODO: should we call the contract for leaf count? i.e. pdpVerifier.getPieceLeafCount(this._dataSetId, piece.pieceId)
+      const leafCount = getLeafCount(piece.pieceCid) ?? 0


let's ditch this

rvagg · 2025-10-07T06:11:29Z

OK, here's my current thoughts about this:

I really like that you're fetching the pieces from the contract, I've been wanting to get rid of the query to the SP to get the pieces (I thought I had an issue for this somewhere?). It's not trustworthy, we should be getting it off the chain like you're doing here.
Leaves are a problem and we should ignore them wherever possible I think. Leaves are just the raw size rounded up to the nearest 32 because that's the proving unit. But our actual sizes don't have to be a multiple of 32. Now that I'm looking at PDPVerifier I can see that the rawSizes it returns is the leaf count multiplied by 32, i.e. actual raw size rounded up to the nearest 32, which is wrong. I've opened an issue about this @ Fix & clarify all uses of "rawSize" pdp#217, but we can just ignore it entirely for our purposes.
I think that we could just encourage use of feat: getSizeFromPieceCID(cid) to extract size from PieceCIDv2 #283 to devs to work this out themselves from a CID and not bother augmenting here with sizes.

Which leaves us with just "get pieces from contract". So I think we should pivot here slightly, so here's my suggestion:

Leave getActivePieces as the heart of this in pdp/verifier.ts, but don't return rawSizes from there, it's not helpful
Rename getAllActivePiecesGenerator to just getPieces, it can return an async generator, that's just how you list them, it's the new getDataSetPieces, but just return AsyncGenerator<PieceCID> - i.e. no need to do anything else but query the contract, yield CIDs and keep going as long as you're asked to.
Let's change getDataSetPieces - the name is bad, so let's @deprecate that method and replace the guts of it with what you have for getAllActivePieces, leaving the API stable (no additional args, just use defaults), so it uses getAllActivePiecesGenerator to do a collect-all. Eventually we'll remove this and let the user do collection according to their needs.
Remove everything else

Then we just consume PieceCIDs in Filecoin Pin, use https://github.com/FilOzone/synapse-sdk/pull/283 over there to get the sizes we want and encourage devs to use that pattern. We probably should document all of this too.

rvagg · 2025-10-08T05:19:15Z

I've just realised that we also need pieceId along with the CIDs, so our getPieces should return an async generator of [PieceCID, number] or { cid: PieceCid, id: number }. We'll just need to document that the number is important for certain other operations.

rvagg · 2025-10-14T10:40:14Z

getSizeFromPieceCID now available as a util in pieces.ts

SgtPooki added 2 commits September 26, 2025 13:53

feat: allow consumers to get leafcount and size of pieces

cf89ea3

docs: provide example script for using new method

96d6a5a

github-project-automation bot added this to FS Sep 26, 2025

github-project-automation bot moved this to 📌 Triage in FS Sep 26, 2025

chore: fix lint

4c5e0a6

rvagg reviewed Sep 26, 2025

View reviewed changes

src/storage/context.ts Outdated Show resolved Hide resolved

rjan90 assigned SgtPooki Sep 29, 2025

rjan90 moved this from 📌 Triage to 🔎 Awaiting review in FS Sep 29, 2025

BigLep mentioned this pull request Sep 29, 2025

Add data-set subcommand with --ls filecoin-project/filecoin-pin#7

Open

SgtPooki added 3 commits September 30, 2025 13:09

refactor: migrate to getAllActivePieces

2018a6e

fix: bring back getPiecesWithDetails

aaccbe1

chore: remove some old code, add comments for reviewers

5972649

SgtPooki commented Sep 30, 2025

View reviewed changes

rvagg reviewed Oct 7, 2025

View reviewed changes

rvagg mentioned this pull request Oct 8, 2025

feat: add piece deletion to storage context #285

Merged

rvagg mentioned this pull request Oct 9, 2025

Bug: pieceStatus confuses piece ownership #296

Open

This was referenced Oct 10, 2025

fix: do not call to synapse-sdk directly filecoin-project/filecoin-pin-website#31

Open

feat: provide src/core method for getting dataset pieces filecoin-project/filecoin-pin#97

Open

rjan90 mentioned this pull request Oct 14, 2025

Change provider resolution algorithm for new data set #302

Closed

		// TODO: should we call the contract for leaf count? i.e. pdpVerifier.getPieceLeafCount(this._dataSetId, piece.pieceId)
		const leafCount = getLeafCount(piece.pieceCid) ?? 0

feat: get piece leaf count #249

Are you sure you want to change the base?

feat: get piece leaf count #249

Uh oh!

Conversation

SgtPooki commented Sep 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🎯 Add Piece Details with Leaf Count Information

Overview

New Features

1. Enhanced Piece Data Interface

2. New API Methods

3. PDPVerifier Integration

Key Benefits

Usage Examples

Files Changed

Testing

Example Output

Uh oh!

Uh oh!

rvagg commented Sep 26, 2025

Uh oh!

BigLep commented Sep 29, 2025

Uh oh!

SgtPooki left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hugomrdias commented Oct 1, 2025

Uh oh!

SgtPooki commented Oct 6, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rvagg commented Oct 7, 2025

Uh oh!

rvagg commented Oct 8, 2025

Uh oh!

rvagg commented Oct 14, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

SgtPooki commented Sep 26, 2025 •

edited

Loading