PieceCID Generation Error on Duplicate File Uploads (PDP)

[nijoe1]

Description

When uploading the same file multiple times to SP (calibration network), the system generates incorrect PieceCIDs for subsequent uploads. The issue appears to incorrectly calculate file sizes based on the upload count.

Steps to Reproduce

1. Upload a file (183.54 MB) to SP - generates correct PieceCID
2. Upload the same file again - generates wrong PieceCID with doubled size (367.09 MB)
3. Upload the same file a third time - generates wrong PieceCID with tripled size (550.64 MB)

Expected Behavior

Uploading the same file multiple times should consistently generate the same PieceCID.

Actual Behavior

Each duplicate upload produces a different PieceCID with the file size apparently multiplied by the upload count (size × number of uploads).

Example Data

First upload (correct):

• Piece 4
• CID: bafkzcibfrhupqiyxoqhzdc3nh3qcyc3srqbbr4osamnmydobnyz5eajw4aoviqc6kmfq
• File size: 183.54 MB

Second upload (incorrect):

• Piece 5
• CID: bafkzcibfslipcryyoqhzdc3nh3qcyc3srqbbr4osamnmydobnyz5eajw4aoviqc6kmfq
• File size: 367.0925 MB

Third upload (incorrect):

• Piece 7
• CID: bafkzcibgto4ov2qbdf2a7emlnu7oalalokgaeghr2ibrvtanyfxdhuqbg3qb2vcalzjqw
• File size: 550.6388 MB

Additional Information

• Environment: Calibration network (https://calib.ezpdpz.net)
• The SDK validates the first CID correctly but reports mismatch errors on subsequent uploads
• Issue is reproducible
• Test file available at: https://0x9c5e3cac8166ed93f76bc0469b8bf3ca715ba6b7.calibration.filcdn.io/bafkzcibfrhupqiyxoqhzdc3nh3qcyc3srqbbr4osamnmydobnyz5eajw4aoviqc6kmfq

Dataset Response Example

The PDP server returns piece data showing the incorrect pieceCid values while maintaining correct subPieceCid values for pieces 4, 5, and 7.

Potential Areas to Investigate

• PieceCIDv2 calculation logic in the SP
• SDK and SP agreement on PieceCID generation
• Size calculation during duplicate uploads
• Multihash vs full CID validation

[nijoe1]

cc: @rvagg @TippyFlitsUK

[rvagg]

@nijoe1 the most helpful bit of information here would be the HTTP call that we need to be focusing on where the bad data is coming from. Is this GET /pdp/data-sets/{set-id}? I think that's where were we send the list back to the user:

curio/pdp/handlers.go

Line 451 in 1030f21

func (p *PDPService) handleGetDataSet(w http.ResponseWriter, r *http.Request) {

There's an awkward compound SQL in there to get piece information, and then a summation of subpiece sizes to get the piece size, so the error is probably going to be in there (assuming I'm looking at the right call).