Capability to skip deals for Indexing

[honghaoq]

We have launched indexer to production now, and recently hearing plenty of SPs reporting the need to skip deals for indexing. For context, today Indexer is enabled by default with Lotus 1.15.1, with an optional config that allows SP to turn off indexing entirely. Some SPs reported that they need a more granular level control to not index certain deals, since some deals have private data that should not be made public to the network, while indexer considers all indexed content public (all indexed data are retrievable as well).

We believe this is a market issue instead of an indexing issue, because the content needs to be made not available for retrieval at the same time that it is not published to the index. (If a content not indexed but available for retrieval, then we are misrepresenting its state to providers.)

Opening this thread to discuss the requirement as suggested by @jennijuju from Lotus team. Perhaps we need to align with Boost / Venus as well for a unified solution.

[honghaoq]

Hi @jennijuju could you help provide input on this? curious what the next step should be from market side, thanks!

[anorth]

If a content not indexed but available for retrieval, then we are misrepresenting its state to providers

FWIW I would consider this to be a perfectly reasonable state of affairs. It's quite analogous to a robots.txt file instructing web crawlers not to index some content, while it's still available to those who can reach it via hyperlinks.

Configuration of retrievability and indexing probably need to be distinct (tho indexed but not retrievable would be an unusual state).