Provider Selection UX/DX Design Discussion: Beyond M2

[rjan90]

Background

Following the implementation of basic provider connectivity validation in #116, #118, #119, and #123, we now have a functional foundation for provider selection with ping validation and fallback mechanisms. However, as noted by @rvagg in #116, we need to address the broader question: "How should developers go about selecting a provider? What tools should we give them?"

Current State (M2 Implementation)

✅ Implemented:

• Basic ping validation during provider selection
• Smart provider selection based on existing proof sets
• Random provider selection with connectivity fallback
• Provider registration and approval workflow
• Basic provider information APIs (getApprovedProvider, getAllApprovedProviders)
• HTML tools for provider management and registration

Problem Statement & Questions for Builders

The current implementation provides basic connectivity validation but lacks sophisticated tooling for developers to make informed decisions about provider selection. We need to understand what builders actually need.

Some potential key questions:

1. Discovery & Information Needs

• What information do you need to evaluate whether a provider is suitable for your use case?
• What would make you confident in choosing one provider over another?

2. Selection Criteria

• What factors matter most when selecting a storage provider? (cost, latency, reliability, geographic location, etc.)
• Do you need different selection criteria for different types of data or applications?

3. Operational Needs

• Do you need tools to migrate between providers?
• Once you've selected a provider, what ongoing information do you need about their performance?

4. Cost & Economics

• How do you prefer to compare costs across providers?
• How important is cost optimization vs. performance/reliability?

Call for Feedback

We want to hear from builders using the SDK:

1. Share your use cases - What are you building and how do you think about provider selection?
2. Describe your pain points - What's hard about choosing providers today?
3. Suggest solutions - What tools or features would solve your problems?
4. Share workflows - How does provider selection fit into your development process?

Please comment with your experiences, needs, and ideas. This feedback will drive the design of post-M2 provider selection enhancements.

[rvagg]

Misc thoughts on architectural problems that need to be resolved before introducing more flexibility. See #135 for an expansion with a proposal about this. This isn't directly relevant to this issue but will help evolve into the API requirements.

Expand

This seems like a good place to note a general, related, codesmell that I'm not happy about and is increasing as we add more functionality. I don't yet have a good proposal for dealing with this but I'd like to come up with something that's simple and feels natural / idiomatic, but it's tricky.

The core problem is that many storage operations are specific to an SP. So our createStorage() is essentially a setupRelationshipWithStorageProvider() where we either set up a new relationship and make a proof set with an SP, or we find an existing one we've got a relationship with and an existing proof set on. Then we what's returned from that is a SynapseStorage object, which is essentially a StorageProviderAndProofSetManager. It's a thing that's tied to one SP and a proof set we have with that SP. This is fine most of the things we need to do with storage - upload, download, inspect a proof set or root's details, even look at last proven status.

But then there are cross-cutting concerns that don't fit this model, including:

• Download a CommP from any SP that has it, or even just FilCDN
• Query proof set or root status when I don't know which SP manages it
• Perform any other isolated proof set operation that doesn't require me to talk to the SP, i.e. I could just ask the chain, why do I need to care about the relationship with the SP?

I added a SynapseStorage#download(commp), but why should I have to know which SP has it? So there's also a Synapse#download(commp) which makes more sense from a usability perspective. Now we have the same problem with SynapseStorage#pieceStatus(commp) added in #127; why should I have to set up my SynapseStorage just to query that? But I don't want a proliferation of the same functionality on Synapse and SynapseStorage, that indicates that we don't have the right API here.

I think we need to rethink the units of concern here: SP, proof set, and roots. And somehow deal with them independently but also require coupling where that's necessary. I just don't have a proposal yet that doesn't make for complicated DX.