IPIP-0524: Remove IPLD translation from IPFS HTTP Gateway #524
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The Accept-header driven translation between various IPLD formats is lossy, complicated, and often has us jumping through weird hoops to massage data into an acceptable format. Maybe we can do without it? Simplifying the spec will make creating server implementations easier and the translation can still be done by clients if they require it. Refs: ipfs/gateway-conformance#200
🚀 Build Preview on IPFS ready
|
This was referenced Dec 1, 2025
lidel
added a commit
to ipfs/gateway-conformance
that referenced
this pull request
Dec 3, 2025
Update tests to reflect IPIP-0524 which removes codec conversions from the gateway spec. Gateways now return 406 Not Acceptable when the requested format doesn't match the block's codec. - Replace TestDagPbConversion with TestCodecMismatchReturns406 - Remove codec conversion tests (dag-pb→dag-json, dag-cbor→dag-json) - Fix TestPathing to request matching codec format - Add body verification for DAG-CBOR traversal test - Clarify that HTML rendering remains in spec (unlike conversions) Implementations supporting optional codec conversions for backward compatibility are free to skip the 406 tests. Ref: ipfs/specs#524
lidel
added a commit
to ipfs/boxo
that referenced
this pull request
Dec 3, 2025
Add AllowCodecConversion to gateway.Config to control codec conversion behavior per IPIP-0524. When false (default), the gateway returns 406 Not Acceptable if the requested format doesn't match the block's codec. When true, conversions between codecs are performed for backward compatibility. Codec conversion tests moved here from gateway-conformance since conversions are now an optional implementation feature, not a spec requirement. Gateway-conformance now tests for 406 responses. Ref: ipfs/specs#524 Ref: ipfs/gateway-conformance#254
lidel
added a commit
to ipfs/boxo
that referenced
this pull request
Dec 3, 2025
Add AllowCodecConversion to gateway.Config to control codec conversion behavior per IPIP-0524. When false (default), the gateway returns 406 Not Acceptable if the requested format doesn't match the block's codec. When true, conversions between codecs are performed for backward compatibility. Codec conversion tests moved here from gateway-conformance since conversions are now an optional implementation feature, not a spec requirement. Gateway-conformance now tests for 406 responses. Ref: ipfs/specs#524 Ref: ipfs/gateway-conformance#254
lidel
added a commit
to ipfs/boxo
that referenced
this pull request
Dec 3, 2025
Add AllowCodecConversion to gateway.Config to control codec conversion behavior per IPIP-0524. When false (default), the gateway returns 406 Not Acceptable if the requested format doesn't match the block's codec. When true, conversions between codecs are performed for backward compatibility. Codec conversion tests moved here from gateway-conformance since conversions are now an optional implementation feature, not a spec requirement. Gateway-conformance now tests for 406 responses. Ref: ipfs/specs#524 Ref: ipfs/gateway-conformance#254
lidel
added a commit
to ipfs/boxo
that referenced
this pull request
Dec 3, 2025
Add AllowCodecConversion to gateway.Config to control codec conversion behavior per IPIP-0524. When false (default), the gateway returns 406 Not Acceptable if the requested format doesn't match the block's codec. When true, conversions between codecs are performed for backward compatibility. Codec conversion tests moved here from gateway-conformance since conversions are now an optional implementation feature, not a spec requirement. Gateway-conformance now tests for 406 responses. Ref: ipfs/specs#524 Ref: ipfs/gateway-conformance#254
lidel
added a commit
to ipfs/boxo
that referenced
this pull request
Dec 3, 2025
Add AllowCodecConversion to gateway.Config to control codec conversion behavior per IPIP-0524. When false (default), the gateway returns 406 Not Acceptable if the requested format doesn't match the block's codec. When true, conversions between codecs are performed for backward compatibility. Codec conversion tests moved here from gateway-conformance since conversions are now an optional implementation feature, not a spec requirement. Gateway-conformance now tests for 406 responses. Ref: ipfs/specs#524 Ref: ipfs/gateway-conformance#254
lidel
added a commit
to ipfs/kubo
that referenced
this pull request
Dec 3, 2025
Wire up boxo's AllowCodecConversion config to control codec conversion behavior per IPIP-0524. When false (default), the gateway returns 406 Not Acceptable if the requested format doesn't match the block's codec. Clients should fetch raw blocks (`?format=raw`) and convert client-side. Ref: ipfs/specs#524 Ref: ipfs/boxo#1077 Ref: ipfs/gateway-conformance#254
2 tasks
lidel
added a commit
to ipfs/kubo
that referenced
this pull request
Dec 3, 2025
Wire up boxo's AllowCodecConversion config to control codec conversion behavior per IPIP-0524. When false (default), the gateway returns 406 Not Acceptable if the requested format doesn't match the block's codec. Clients should fetch raw blocks (`?format=raw`) and convert client-side. Ref: ipfs/specs#524 Ref: ipfs/boxo#1077 Ref: ipfs/gateway-conformance#254
lidel
added a commit
to ipfs/kubo
that referenced
this pull request
Dec 3, 2025
Wire up boxo's AllowCodecConversion config to control codec conversion behavior per IPIP-0524. When false (default), the gateway returns 406 Not Acceptable if the requested format doesn't match the block's codec. Clients should fetch raw blocks (`?format=raw`) and convert client-side. Ref: ipfs/specs#524 Ref: ipfs/boxo#1077 Ref: ipfs/gateway-conformance#254
lidel
reviewed
Dec 3, 2025
Member
There was a problem hiding this comment.
Small clarification question inline, but other than that looks sensible, removes complexity related to a niche feature which can be implemented in userland thanks to ?format=raw.
Cleaned up conformance in:
GO implementation in Boxo/Kubo allows opt-in to previous behavior, just in case anyone depended on this niche feature from IPLD era:
Comment on lines
+164
to
+165
| - [application/json](https://www.iana.org/assignments/media-types/application/json) – Validates block data as JSON before returning it as-is. Invalid JSON produces 406 Not Acceptable. | ||
| - [application/cbor](https://www.iana.org/assignments/media-types/application/cbor) – Validates block data as CBOR before returning it as-is. Invalid CBOR produces 406 Not Acceptable. |
Member
There was a problem hiding this comment.
@achingbrain how strong you feel about doing actual JSON or CBOR parsing here, when plain version is requested?
The main purpose here, in my mind, is not data validation, but to maximize interop ensure correct Content-Type in response, as there are tools which will make request but wont parse response if its not application/json etc.
Could we simplify spec here noting that if requested CID codec is matching [dag-]json or [dag-]cbor, we dont parse payload, but return the same response as raw but with specific vanilla content type.
It makes implementation simpler (no need to deal with CBOR/JSON parsing edge cases), reduces cost of running gateway (no JSON/CBOR parsing), but we still have the interop benefit of returning vanilla JSON/CBOR when necessary by a third-party system.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The Accept-header driven translation between various IPLD formats is lossy, complicated, and often has us jumping through weird hoops to massage data into an acceptable format. Maybe we can do without it?
Simplifying the spec will make creating server implementations easier and the translation can still be done by clients if they require it.