Difference in CID Generation between IPFS and Singularity

[SethDocherty]

Problem

CIDs generated with the Kubo CLI via the ipfs add command and Singularity were different when files exceeded 174 MB. The reason for that was the inability to modify the maximum number of node links with Kubo, as it was hardcoded to 174. I provide more details in this post on the IPFS forums.

That was the case until the 0.35.0 release, which introduced new DAG-shaping options, including the ability to set the maximum links. After running some preliminary tests, I'm seeing Singularity generating different CIDs for files larger than 1024 MB. I made note of this difference in the Should we profile CIDs? post in the IPFS forums.

Could there be a need to tweak how content is chunked with Singularity? Or it could be an edge case that wasn't identified when the new DAG-shaping options were implemented with the Kubo CLI. In either case, I'm not sure if this is a Singularity or Kubo issue, but I've mentioned this inconsistency in ipfs/specs#499 (comment).

Why is this important?

Aligning Singularity's default DAG construction parameters with Kubo's would ensure a consistent and predictable user experience, reinforcing the core value propositions of the decentralized networks. Even being more transparent with Singularity's default DAG construction is a good step forward, since it provides end users with vital metadata details in the preparation of datasets. When different tools produce different CIDs for identical data, it becomes difficult to verify the authenticity of a file. This creates confusion and erodes trust in the data's integrity.

[2color]

The reason for difference is explained here: https://discuss.ipfs.tech/t/should-we-profile-cids/18507/44

The difference is the following:

(IPFS) bafybeignp2eaklnbejnlcrxaldpiuoc63tk63vdsokleegajxpvczzxiau:
Intermediary node1 → 1024 links
Intermediary node2 → 1 link (last block)
Singularity: bafybeia2jsxebrhwuehoptuhpmhmlxhot74nalyihzud2uufosptoakjyu
Intermediary node 1 → 1024 links
last block
Both export to the same file (by chance or not).

Our “balanced” dag builder documentation says:

// Package balanced provides methods to build balanced DAGs, which are generalistic
// DAGs in which all leaves (nodes representing chunks of data) are at the same
// distance from the root. Nodes can have only a maximum number of children; to be
// able to store more leaf data nodes balanced DAGs are extended by increasing its
// depth (and having more intermediary nodes).
In Singularity’s DAG, the last leaf node is not as the same distance from the root as the others.