Subdomain support for CIDs longer than 63

[lidel]

I hoped to punt this until we need to switch away from sha256 in CIDs, but we may need to solve this problem sooner than expected due to ED25519 keys being new default soon (#6916)

Problem: DNS label limit of 63

RFC 1034: "each node has a label, which is zero to 63 octets in length"

The default CIDv1 Base32 with multihash of sha256 and RSA libp2p-key fits:

• Default CID: http://bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi.ipfs.dweb.link
• RSA: https://bafzbeih7uocwo6vmbjusf4wzw6h5cruhw3gf4jxhevmxssnz5vkgryk2za.ipns.dweb.link

but if we use ED25519 libp2p-key then we are 2 characters over the limit:

• ED25519 libp2p-key: https://bafzaajaiaejca4syrpdu6gdx4wsdnokxkprgzxf4wrstuc34gxw5k5jrag2so5gk.ipns.dweb.link
• CID created with --hash sha2-512 will be even longer: https://bafkrgqe3ohjcjplc6n4f3fwunlj6upltggn7xqujbsvnvyw764srszz4u4rshq6ztos4chl4plgg4ffyyxnayrtdi5oc4xb2332g645433aeg.ipfs.dweb.link

Label longer than 63 characters means the hostname can't resolve:

$ ping bafzaajaiaejca4syrpdu6gdx4wsdnokxkprgzxf4wrstuc34gxw5k5jrag2so5gk.ipns.dweb.link
ping: bafzaajaiaejca4syrpdu6gdx4wsdnokxkprgzxf4wrstuc34gxw5k5jrag2so5gk.ipns.dweb.link: Name or service not known

And links are not picked up by tools like Slack:

Note: I used ED25519 as an example, but not limited to that single type of CID. Even if we find a way to fit ED25519 in a single label, the problem remains for CIDs with a multihash created with longer hash functions.

Solved: IPNS-specific fix for ED25519 keys

In parallel to the generic fix, we could represent ED25519 keys in a way that fits under 63 characters, solving the UX issue for IPNS websites loaded from public gateways.

Done: #7441 – we support {cidv1base36}.ipns.dweb.link which perfectly fits

Open Problem: generic solution for long CIDs

I am happy to open PR with a fix, but unsure if I have the best fix in mind, would love to gather feedback first.

❓ (A) support split CIDs (but have broken TLS)

The first idea I have is to split the label when the max is reached.
To maximize entropy for Origin isolation, the remainder should be on the left side:

• https://ba.fzaajaiaejca4syrpdu6gdx4wsdnokxkprgzxf4wrstuc34gxw5k5jrag2so5gk.ipns.dweb.link

Pros:

• 👍 each long CID gets own Origin – we keep isolation
• 👍 path redirect provided by subdomain gateway can take care of splitting
• 👍 future-proof solution for longer hashes such as sha2-512
  • the next limit is pretty far away: the maximum length of full domain name: 253 characters, including dots
  • sha2-512 on dweb.link is 121 characters

Cons:

• 💢 decreased entropy in security guarantees provided by origin isolation
• 💢 wildcard TLS certificate does not pass validation for more than a single level of labels
  • this will produce annoying UX on public gateways such as dweb.link: TLS warning when opening IPFS website on IPNS. we get the same problem as ENS gateway at *.eth.link (https://blog.almonit.eth.link vs https://almonit.eth.link
• 💢 copying & pasting CID as-is no longer works on public gateways (user needs to put . in the middle etc)
  • Note: to make it easier UX-wise, we should allow . anywhere inside of CID, but internally merge labels, and return a redirect to canonical version that splits at deterministic position (enforcing maximum label for Origin).

❓ (B) redirect long CIDs to an "insecure" subdomain

This would make it possible for content to load, but longer CIDs would not get Origin isolation per CID.

To make this bit more clear and idiomatic, we could present this as "cross origin resource sharing" endpoint that allows both CORS requests + supports loading everything from a single origin + has paths locked down in browsers like noted in ipfs/in-web-browsers#157.

Think in terms of

• https://dweb.link/ipfs/superlongcid redirecting to https://cors.dweb.link/superlongcid

Pros:

• 👍 does not break TLS wildcard certs (easy setup for gateway operators)
• 👍 useful outside this problem: provides idiomatic way for exposing path gateway on subdomain gateways (for use when origin isolation is not needed)

Cons:

• 💢 long CIDs don't get Origin isolation

❓ (C) swap DAG root with CID that uses shorter hash function

Pros:

• 👍 "just works"

Cons:

• 💢 decreased entropy
• 💢 newly created root blocks need to be persisted somehow: if I bookmark the page loaded via shortened CID and then the root block gets garbage-collected, the address is dead.
  • potential fix: we could always create redundant sha256 root block for every DAG that uses longer hash function for interop

❓ (D) leverage HTTP proxy mode (on localhost)

When Gateway port is used as HTTP proxy, local client does not perform DNS lookup, but original URL is sent in HTTP request to the proxy for processing.

Because HTTP proxy IS go-ipfs node in that scenario, it does not do DNS lookup, but extract original (long) CID and resolves it, without involvement of DNS.

As long user agents are not overzealous in validating URLs, this would allow for long (>63) CIDs on subdomains.

This is important, because it enables localhost gateway (used by Brave) to resolve long CIDs correctly without any additional hacks.

UX details tbd. This could be the solution for localhost gateway, but for public ones we still need something else.

Other ideas?

Would love to find a better way to work around this

cc @aschmahmann @Stebalien ipfs/in-web-browsers#89

[ribasushi]

It's a bit unfortunate that keys are so overly verbose: https://cid.ipfs.io/#bafzaajaiaejca4syrpdu6gdx4wsdnokxkprgzxf4wrstuc34gxw5k5jrag2so5gk

It looks like we have an actual protobuf construct inside the raw bytes. Is this... something we need to do?

If we shave off 2 bytes, nothing extra needs to be done...

[lidel]

I am afraid even if we find a hacky workaround for libp2p-keys in ED25519, the problem remains for CID that use longer hash functions than sha256.

[Stebalien]

For context, we're trying to encode 40 bytes into 62 characters (with one character for the multibase prefix).

I believe base36 would work, if that's an option. That should give us exactly 63 characters.

We could change how we encode these peer IDs in text and use an ed25519 specific codec (<cidv1>-<ed25519>-<multihash>). That would still be a reasonable encoding of an ed25519 CID but I'd prefer to avoid it.

[Stebalien]

But I agree we should support longer keys regardless. But will this be a problem for TLS certs? Can we get a double-star cert?

[lidel]

• I am not aware of any CA that provides double wildcard certs.
  That is why ENS gateway still has the TLS warning (example: https://blog.almonit.eth.link).

• Switching the default text representation of PeerID to Base36 would introduce work across ecosystem to bubble up support (missing from multibase.csv atm) and its not as popular as RFC version of Base32. Not sure what's lesser evil, that, or a new codec.

[Stebalien]

@aschmahmann and I discussed this and it is possible to shrink ed25519 pids, but it's painful and requires coordination with all libp2p implementations.

To shrink ed25519 keys, we need to:

1. Encode them as <cidv1>-<ed25519>-<multihash> in text. This will reduce the id size to 36 bytes (from 40).
2. Ideally, migrate to CIDs on the wire in libp2p. That would save us 10% on the wire for ed25519 keys and make it easier to interoperate with other p2p networks (because we could use their native key formats instead of wrapping them in protobufs before hashing).

Unfortunately, if we want to get 1 in the near future, we'd make it significantly harder to get 2. Basically, if we start using the new ed25519 pid encoding now, we'd have to convert back to the normal pid binary format (raw multihash) when decoding. However, if/when we decide to use CIDs as the binary pid format, we'd have trouble round-tripping.

That is, in the ideal world, if we encounter a text-based PID as a CID:

• If it uses the libp2p-key multicodec, it's a legacy peer ID. Encode it as a multihash on the wire.
• If it uses any other multicodec, it's a new peer ID. Encode it as a CID on the wire.

However, if we implement 1 before 2, we'd have to encode legacy keys in this new CID format. When converting back, we'd end up with the wrong "on the wire" format.