Delegated router/ IPNI indexer for "filecoin-pin"

[BigLep]

Context

This issue pertains to the work of IP Shipyard standing up a publicly accessible delegated router / IPNI indexer for the "filecoin community" for IPFS CIDs found within Filecoin PDP (warm storage) Pieces. This is a temporary measure while cid.contact is stabilized. It is being prompted by work from the work happening in #fil-fic on FIL Slack. It is funded at least in part by FilOz.

Done criteria

1. A publicly accessible IPNI indexer has been stood up.
2. It can receive advertisements related to PDP / filecoin-pin. It only needs to accept advertisements for SPs in the "ServiceRegistry". These Curio SPs configure this my specifying DirectAnnounceURLs.
3. It can fetch the advertisement entries.
4. cid.contact forwards requests to this new endpoint while cid.contact is in "stabalization mode".

Design

• Where the infra is running: "us-east"
• Who owns the infra: IP Shipyard
• Who is advertising to its: a subset of Filecoin SPs running a special Curio branch are the only publishers. If all the code worked today (discussed below) this would currently be less than 20. The focus right now isn't getting a lot of SPs, but rather SPs that provide a good quality of service that we can interact with.
• How SPs are announcing to it: this will happen the week of 2025-10-06 from the Curio branch that SPs participating in Filecoin Onchain Cloud are running. This is happening as part of Backport index and ipni pdp tasks to PDPv0 curio#670, where effectively cid.contact and filecoinpin.contact will both be DirectAnnounceURLs.
• Introspect-ability: This is provided by https://filecoinpin.contact/providers which will be populated as FilecoinPin SPs are added (pending Backport index and ipni pdp tasks to PDPv0 curio#670 and rollout). The IPNI specific /cid endpoint is of course also available and IPFS Delegated Routing /routing/v1 is also available upon request.
• Expected impact of adding this to the fallback on cid.contract response times: we believe this will be minimal because it's another database in the requested regions of NA or EU (same as the other already approved hard coded endpoints). We also believe the majority of downstream consumers of cid.contact rely on streaming responses anyhow.

[BigLep]

filecoinpin.contact has been registered

[BigLep]

We're now on the step of getting cid.contact to forward requests to filecoinpin.contact. These are the open steps we're aware of to accomplish:

1. filcoinpin.contact Design Doc: A small descriptive github issue on where the infra is running, how SPs are announcing to it, etc so the IPNI community can understand the impact of adding this to the fallback on response times and infra impact

2. Introspect-ability: the filecoinpin.contact should list its providers and make it easy to query all the cids in its index (same as any other IPNI index), so that IPNI maintainers can do robust testing to evaluate cid.contact catch up vs the filecoinpin.contact endpoint

3. Demonstrate efficiently reachable infra: the provider index (filecoinpin.contact) needs to be running on infra in the US or Europe so as not to cost an additional 300ms delay for all fallback queries to finish (aka the participating SPs need to announce to this provider ahead of time and hold the index on provider infra, not re-query the SPs manually in real time for any/all queries and wait for them to respond). Currently all the providers on the fallback are running in AWS US West (so this is preferred), but any efficiently reachable server in the US or Europe would work.

---

Lets update the issue description with these answers.

[BigLep]

This was added to the issue description:

• Where the infra is running: "us-east"
• Who owns the infra: IP Shipyard
• Who is advertising to its: a subset of Filecoin SPs running a special Curio branch are the only publishers. If all the code worked today (discussed below) this would currently be less than 20. The focus right now isn't getting a lot of SPs, but rather SPs that provide a good quality of service that we can interact with.
• How SPs are announcing to it: this will happen the week of 2025-10-06 from the Curio branch that SPs participating in Filecoin Onchain Cloud are running. This is happening as part of Backport index and ipni pdp tasks to PDPv0 curio#670, where effectively cid.contact and filecoinpin.contact will both be DirectAnnounceURLs.
• Introspect-ability: This is provided by https://filecoinpin.contact/providers which will be populated as FilecoinPin SPs are added (pending Backport index and ipni pdp tasks to PDPv0 curio#670 and rollout). The IPNI specific /cid endpoint is of course also available and IPFS Delegated Routing /routing/v1 is also available upon request.
• Expected impact of adding this to the fallback on cid.contract response times: we believe this will be minimal because it's another database in the requested regions of NA or EU (same as the other already approved hard coded endpoints). We also believe the majority of downstream consumers of cid.contact rely on streaming responses anyhow.

[BigLep]

@willscott: any question on the above before adding filecoinpin.contact as a temporary fallback for cid.contact?

[willscott]

This seems like a reasonable path.

My initial concern was largely reacting to https://filecoinproject.slack.com/archives/C02T827T9N0/p1759305900940809?thread_ts=1757016661.178479&cid=C02T827T9N0

There is no main instance, it's effectively an aggregator for a number of index providers

Having the design and knowing that it is serving locally and not routing back to SPs makes me more comfortable.

Two notes:

• The scatter/gather list that the 'fallbacks' represent ( https://github.com/ipni/indexstar/blob/main/main.go#L39-L42 ) does not use streaming request/response afaik
• One thing that seems useful is that we don't end up with inverted priorities - that the IPNI team is in a place of a high priority of debugging/removing this fallback, while 'since it works enough' it is a low priority for your team to help with debugging. that's why we are asking for the introspect-ability up front. the '/cid' and delegated routing you list don't really help with this, as we can't use existing verification tools without knowledge of what the 'ground truth' of CIDs are. We can (and i guess now plan to) monitor by comparing the underlying providers exposed, but this creates additional special casing in the verification path by the ipni team.

[momack2]

Thanks Will! Glad we're moving forward. Is this sufficient to get filecoinpin.contact added to the fallback now we're confident it won't add additional overhead and latency? Would love to be able to test as soon as the above PRs land.

Re your last point - are you saying you'd prefer the IPFS Delegated Routing /routing/v1 to be available upfront? Or what do you believe is missing here that other endpoints are doing?

[BigLep]

What are the next steps @willscott for cid.contact forwarding requests to filecoinpin.contact?

On the topic of inspectability, what are other groups (e.g., Storacha) providing you that is helpful? We're happy to do what is needed to get the fallback removed, but need to know what would be useful.