IPIP 0499: CID Profiles #499
+134
−0
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,134 @@ | ||||||||||
| --- | ||||||||||
| title: 'IPIP-0499: CID Profiles' | ||||||||||
| date: 2025-11-14 | ||||||||||
| ipip: proposal | ||||||||||
| editors: | ||||||||||
| - name: Michelle Lee | ||||||||||
| github: mishmosh | ||||||||||
| affiliation: | ||||||||||
| name: IPFS Foundation | ||||||||||
| url: https://ipfsfoundation.org | ||||||||||
| - name: Daniel Norman | ||||||||||
| github: 2color | ||||||||||
| affiliation: | ||||||||||
| name: Independent | ||||||||||
| url: https://norman.life | ||||||||||
| relatedIssues: | ||||||||||
| - https://discuss.ipfs.tech/t/should-we-profile-cids/18507 | ||||||||||
| order: 0499 | ||||||||||
| tags: ['ipips'] | ||||||||||
| --- | ||||||||||
|
|
||||||||||
| ## Summary | ||||||||||
|
|
||||||||||
| This proposal introduces **configuration profiles** for CIDs that represent files and directories using [UnixFS](https://specs.ipfs.tech/unixfs/). | ||||||||||
|
|
||||||||||
| ## Motivation | ||||||||||
|
|
||||||||||
| UnixFS CIDs are currently non-deterministic. The same file or directory can produce different CIDs across implementations, because parameters like chunk size, DAG width, and layout vary between implementations. Often, these parameters are not even configurable by users. | ||||||||||
|
|
||||||||||
| This creates three problems: | ||||||||||
|
|
||||||||||
| - **Verification difficulty:** The same content produces different CIDs across tools, making content verification unreliable. | ||||||||||
| - **Additional overhead:** Users must store and transfer UnixFS merkle proofs to verify CIDs, adding storage overhead, network bandwidth, and complexity. | ||||||||||
| - **Broken expectations:** Unlike standard hash functions where identical input produces identical output, UnixFS CIDs behave unpredictably. | ||||||||||
|
|
||||||||||
| Configuration profiles solve this by explicitly defining all parameters that affect CID generation. This preserves UnixFS flexibility (users can still choose parameters) while enabling deterministic results. | ||||||||||
|
|
||||||||||
| ## Detailed design | ||||||||||
|
|
||||||||||
| We introduce a set of **named configuration profiles**, each specifying the complete set of parameters for generating UnixFS CIDs. When implementations use these profiles, they guarantee that the same input, processed with the same profile, will yield the same CID across different tools and implementations. | ||||||||||
|
|
||||||||||
| ### UnixFS parameters | ||||||||||
|
|
||||||||||
| Here is the complete set of UnixFS parameters that affect the resulting string encoding of the CID: | ||||||||||
|
|
||||||||||
| 1. CID version, e.g. CIDv0 or CIDv1 | ||||||||||
| 1. Multibase encoding for the CID, e.g. base32 | ||||||||||
| 1. Hash function used for all nodes in the DAG, e.g. sha2-256 | ||||||||||
| 1. UnixFS file chunking algorithm | ||||||||||
| 1. UnixFS file chunk size or target (if required by the chunking algorithm) | ||||||||||
| 1. UnixFS DAG layout (e.g. balanced, trickle etc...) | ||||||||||
| 1. UnixFS DAG width (max number of links per `File` node) | ||||||||||
| 1. `HAMTDirectory` fanout, i.e. the number of bits determines the fanout of the `HAMTDirectory` (default bitwidth is 8 == 256 leaves). | ||||||||||
| 1. `HAMTDirectory` threshold (max `Directory` size before switching to `HAMTDirectory`): based on an estimate of the block size by counting the size of PNNode.Links | ||||||||||
|
Contributor
There was a problem hiding this comment.
Suggested change
Member
There was a problem hiding this comment. Bit odd to discourage, when both most popular implementations in GO and JS use size-based heurstic - #499 (comment) Unsure how to handle this. Perhaps clarify the heuristic is implementation-specific, and when deterministic behavior is expected, a specific heuristic should be used?
Member
There was a problem hiding this comment. I don't think we should be estimating the block size as it's trivial to calculate it exactly. Can we not just define this (and punt to the spec for the details) to make it less hand-wavey?
Suggested change
|
||||||||||
| 1. Leaf Envelope: either `dag-pb` or `raw` | ||||||||||
| 1. Whether empty directories are included in the DAG. Some implementations may apply filtering. | ||||||||||
| 1. Whether hidden entities (including dot files) are included in the DAG. Some implementations may apply filtering. | ||||||||||
| 1. Directory wrapping for single files: in order to retain the name of a single file, some implementations have the option to wrap the file in a `Directory` with link to the file. | ||||||||||
| 1. Presence and accurate setting of `Tsize`. | ||||||||||
|
|
||||||||||
mishmosh marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||||||
| The handling of symlinks and symlink follows is defined by the [UnixFS](https://specs.ipfs.tech/unixfs/) spec. | ||||||||||
|
|
||||||||||
| ## CID profiles | ||||||||||
|
|
||||||||||
| To enable consistent CID generation, we define a series of named profiles that specify complete UnixFS parameter sets. Profile names may have any prefix, but must end in `YYYY-MM`. | ||||||||||
|
|
||||||||||
| The initial profile in the series, **`unixfs-2025`**, captures the baseline default parameters used by multiple implementations as of November 2025. | ||||||||||
|
|
||||||||||
| | Parameter | `unixfs-2025` | | ||||||||||
| | ----------------------------- | ------------------------------------------------------- | | ||||||||||
| | CID version | CIDv1 | | ||||||||||
| | Hash function | sha2-256 | | ||||||||||
| | Max chunk size | 1MiB | | ||||||||||
| | DAG layout | balanced | | ||||||||||
| | DAG width (children per node) | 1024 | | ||||||||||
| | `HAMTDirectory` fanout | 256 blocks | | ||||||||||
| | `HAMTDirectory` threshold | 256KiB (estimated by counting the size of PBNode.links) | | ||||||||||
| | Leaves | raw | | ||||||||||
| | Empty directories | TODO | | ||||||||||
mishmosh marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||||||
| | Hidden entities | TODO | | ||||||||||
|
|
||||||||||
| ## Legacy profiles | ||||||||||
|
|
||||||||||
| We also define a series of **legacy profiles**, used by various implementations as of November 2025: | ||||||||||
|
|
||||||||||
| | | `kubo-legacy-2015` (kubo default) | `helia-2025` | `storacha-2025` | `kubo-2025` | `kubo-wide-2025` | `dasl-2025` | | ||||||||||
| | ----------------------------- | ------------------------------ | --------------- | ------------------ | ------------------ | ----------------------- | ------------- | | ||||||||||
| | CID version | CIDv0 | CIDv1 | CIDv1 | CIDv1 | CIDv1 | CIDv1 | | ||||||||||
| | Hash function | sha2-256 | sha2-256 | sha2-256 | sha2-256 | sha2-256 | sha2-256 | | ||||||||||
| | Max chunk size | 256KiB | 1MiB | 1MiB | 1MiB | 1MiB | not specified | | ||||||||||
| | DAG layout | balanced | balanced | balanced | balanced | balanced | not specified | | ||||||||||
| | DAG width (children per node) | 174 | 1024 | 1024 | 174 | **1024** | not specified | | ||||||||||
| | `HAMTDirectory` fanout | 256 blocks | 256 blocks | 256 blocks | 256 blocks | **1024** | not specified | | ||||||||||
| | `HAMTDirectory` threshold | 256KiB (est:links[name+cid]) | 256KiB (est) | 1000 **links** | 256KiB | **1MiB** | not specified | | ||||||||||
| | Leaves | raw | raw | raw | raw | raw | not specified | | ||||||||||
| | Empty directories | Included | Included | Ignored | Included | Included | not specified | | ||||||||||
|
|
||||||||||
| See related discussion at https://discuss.ipfs.tech/t/should-we-profile-cids/18507/ | ||||||||||
|
|
||||||||||
| ### User benefit | ||||||||||
|
|
||||||||||
| Profiles provide 3 key advantages for working with content-addressed data: | ||||||||||
|
|
||||||||||
| 1. **Predictable, deterministic behavior:** Profiles restore the expected property of content addressing: identical input data always produces identical CIDs, regardless of which implementation generates them. | ||||||||||
|
|
||||||||||
| 2. **Lightweight verification:** Users can verify content without needing to rely on additional merkle proofs or CAR files. | ||||||||||
|
|
||||||||||
| 3. **Simplified workflow:** Users can select a profile and automatically get consistent CIDs across all implementations, without needing to configure or understand the underlying parameters. | ||||||||||
|
|
||||||||||
| ### Compatibility | ||||||||||
|
|
||||||||||
| UnixFS data encoded with the CID profiles defined in this IPIP remains fully compatible with existing implementations, since it conforms to the [https://specs.ipfs.tech/unixfs/](specification). | ||||||||||
|
|
||||||||||
| To generate CIDs in compliance with this IPIP, implementations must support the parameters defined in the profiles and support the set of named profiles. They MAY also support legacy profiles. | ||||||||||
|
|
||||||||||
| * Adding new functionality to support parameters and/or profiles | ||||||||||
| * Exposing configuration options for profiles | ||||||||||
|
|
||||||||||
| ### Alternatives | ||||||||||
|
|
||||||||||
| As an alternative to profiles, users can store and transfer CAR files of UnixFS content, which include the merkle DAG nodes needed to verify the CID. | ||||||||||
|
|
||||||||||
| ## Test fixtures | ||||||||||
|
Member
There was a problem hiding this comment. Just noting this is (imo) a blocker. We did not merge UnixFS spec until we had sensible set of fixtures that people could use as reference. The spec may be incomplete, but a fixture will let people reverse-engineer any details, and then PR improvement to spec. Without fixtures for each UnixFS node type, we risk unknown unknown silently impacting final CID (e.g. because we did not know that someone may decide to place leaves one level sooner as "optimization" and someone else always at bottom, as "formal consistency")
Member
There was a problem hiding this comment. Thanks!
|
||||||||||
|
|
||||||||||
| TODO | ||||||||||
|
|
||||||||||
| List relevant CIDs. Describe how implementations can use them to determine | ||||||||||
| specification compliance. This section can be skipped if IPIP does not deal | ||||||||||
| with the way IPFS handles content-addressed data, or the modified specification | ||||||||||
| file already includes this information. | ||||||||||
|
|
||||||||||
| ### Copyright | ||||||||||
|
|
||||||||||
| Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/). | ||||||||||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If this number is dynamic based on the lengths of the actual link entries in the dag, we will need to specify what algorithm that estimation follows. I would put such things in a special "ipfs legacy" profile to be honest, along with cidv0, non-raw leaves etc. We probably should heavily discourage coming up with profiles that do weird things, like dynamically setting params or not using raw-leaves for things.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So, each layout would have its own set of layout-params:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, that's exactly what we're doing by defining this profile.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
wait is kubo dynamically assigning HAMT Directory threshold, currently? i was assuming this was a static number!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The current spec mentions fanout but not threshold, so i'm a little confused what current implementations are doing and if it's even worth fitting into the profile system or just giving up and letting a significant portion of HAMT-shared legacy data just but unprofiled/not-recreatable using the profiles...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@lidel Is this written down in any of the specs? Or is it just in the code at this point?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@lidel @hsanjuan Trying to understand/resolve this thread. Can you confirm if this is current kubo behavior?
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
AFAK decision when to do HAMTDirectory is an implementation-specific behavior. So far the rule of thumb is to keep blocks under 1-2MiB and usually good idea to match chunk size defined (default or defined by user).
Implementation-wise both GO (Boxo/Kubo) and JS (Helia) have size-based heuristic that makes decision when to switch from normal Directory to HAMTDirectory:
iirc (from 2 year old memory, something to check/confirm) is that the size estimation details may/are likely different between GO and JS. They both estimate the serialized DAGNode size by calculating the aggregate byte length of directory entries (link names + CIDs), though the JavaScript implementation appears to include additional metadata in its calculation:
estimatedSize = sum(len(link.Name) + len(link.Cid.Bytes()) for each link)If true, the slight differences in calculation methods might result in directories sharding at marginally different sizes.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you want to be exact you have to take into account any non-zero value fields in the serialized root UnixFS metadata since these affect the block size.
It's quite possible that Kubo will produce a HAMT block that's too big with a certain combination of directory entry names if someone has also changed the encoded directory's default mtime or whatever, probably because the "should-I-shard" feature pre-dates Kubo's ability to add UnixFSv1.5 metadata to things.
Really there's no need to estimate anything - it's trivial to count the actual bytes that a block will take up and then shard if necessary.