Milo search

[drewr]

Summary

Even though we built a richer API layer with GraphQL in datum-cloud/milo#440, we will still need an ergonomic way for operators and customers to find resources, including users, assets, docs, pages, etc. That means unstructured search across everything in our platform. We take it for granted that every (good) site has a search bar at the top, and especially take for granted the ones that actually find information. Search is ubiquitous but good search is actually pretty involved.

Scope for this issue is only the Milo portal. Once we have infra we're happy with we can design an experience for the user portal of Datum Cloud.

(This issue replaces earlier research enhancement #351.)

Important

Refer to the search project for a detailed breakdown of progress on this enhancement.

Stuff we need

• A search engine. For our purposes, "search engine" means an inverted index exposed over a network-enabled API. We could use a hosted solution like Elastic Cloud or Algolia, but I'd like us to own this knowledge internally. Rather than survey the entire landscape, I'd like for us to start with Meilisearch and see how far we get.
• An indexing path. @scotwells drew up a design for how the Activity API can work, but buried in there (according to @joshlreese) is the way to hook into k8s events and generate webhooks to fire them out to our indexing process.
• Auto-complete functionality in the search bar. This doesn't come for free. Good search engines like Elasticsearch optimize this by indexing both n-grams and creating FSTs, but we don't have to worry about prematurely optimizing for the a few hundred thousand or million records we'll need to access from the web console.
• Reindexing functionality. Search is a process of continually refining your index. This has to be in place as part of development.
• Focus only a few resources to start:
  • users
  • domain names - Moved to milestone 1.3
  • zones - Moved to milestone 1.3
  • org
  • personal contact details - Moved to milestone 1.3
  • key resource names/IDs

Things we don't need (at least not yet)

• Ubiquitous search. We already have internal observability internally that works well. We're not looking to store billions of logs. This is purely for finding resources related to our users and products.
• Cloud portal search. That will be easy enough once Milo has the functionality.

Related issues

• Unified Resource Search: indexed fuzzy/multi-field search for portals and APIs #351: General research in the space.
• Global Search for Cloud and Staff Portal #523: Broad vision for the cloud portal.

[drewr]

@scotwells Assigning you for it to get organized and staffed accordingly