Improve extensibility of Admin UI search of collection list

[DanRibbens]

Prompted from this discussion on Discord.

The problem is that under certain conditions the built-in search in Payload doesn't run optimally. One option is to have direct extensibility to allow 3rd party search tool handle it.

Another detail to explore is allowing text search indexes that could improve the performance without integrating with another search service.

I looked into improving the MongoBB search functionality but the search index type and $search aggregation isn't supported in all such as with DocumentDB and Cosmos.

In addition to extending the above, some improvements to built-in search:

• Cancel outstanding stale search queries on change
• Provide loading feedback when querying in the UI
• Investigate built-in search performance issues of large collections (100k+)

[bcksl]

search and $search

For those using a supported implementation, this would be a great addition. Have you considering doing this by adding a search clause to Payload Queries? If so, that functionality could be enabled, and used by the search query builder in the admin panel with a config toggle. Would also suggest having a searchable: boolean or flag on Field that allows users to instruct Payload to add a search index for that field, in the same vein as unique and index.

Hooks

To support a variety of different backends, search hooks should support:

1. A toggle for whether Payload needs to fetch the returned document ids from the database ( see Partial document support).
2. A toggle for whether or not Payload needs to apply access control logic to the list or if it has already been considered.
3. A hook to reindex a document when it has been modified (see Reindexing).

The hooks could also interface with the searchable toggle on Field to enable search.

If it's desirable to support multiple search backends, searchable could optionally take an object something like the following, where indexOptions and searchOptions are objects that get passed to their respective hooks whenever used for that field.

{
  strategy: string,
  indexOptions: any,
  searchOptions: any,
}

Implementing multiple backends may not be considered worthwhile, because it brings a whole host of problems, like combining search results from disparate search providers. It would be simplest to mandate a single search provider, and have the search API be something like:

// `fields` is the record containing the field names (flattened paths for nested fields)
// and their respective `searchOptions` from above
type SearchHook = ({
  req: PayloadRequest,
  fields: Record<string, any>,
  query: string
}) => Results | Promise<Results>

The search hook must be parameterized over req to enable search implementations to provide their own access control, for example using faceted indexes.

Partial document support

Many search providers are also functionally document databases, meaning it would be performant to allow a search to directly return documents from the search engine. Consider adding a flag to the search query which would override the hook default preference to either force Payload to fetch complete documents from the db from the returned ids, or return the documents as fetched by the search query. This relates to item (1) from the beginning of Hooks, as the result of any search query can be generally specified to be a list of partial documents which must at least contain an id.

As these documents will most likely retain the shape of the original, if not all the fields, access control may still be applied to the result.

Reindexing

It would be very useful to be able to instruct Payload to rebuild the search index for an entire collection, either by calling the index hook again for all documents, or optionally providing an additional hook for bulk (re-)indexing of documents. This makes it a lot easier to add an index for a field after the field has been in use, among other things. Consider adding a subcommand to the CLI like payload search:reindex, HTTP routes/GraphQL mutations/Local APIs, and components to the admin panel to accomplish same, with associated access control specifiers. Providing informative progress indication for the admin UI is reasonably straightforward, since even bulk reindexing can be chunked.

[geminigeek]

hi,

i suggest expose a custom endpoint for search and let the custom endpoint do search or whatever is required , show error in UI if the result is not in required format , also kindly look into exposing a debouche for search as you type, or disable it and use button to initiate search, i suggested a similar use case few months back #1030