Consolidate and cleanup Obsearchability AI Assistant page

[leemthompo]

closes https://github.com/elastic/search-docs-team/issues/237

👁️ > 👁️ URL preview

Summary

• updated "Observability AI Assistant" to "Elastic AI Assistant for Observability and Search", added variable
• improved page structure with clear sections and better hierarchy
• consolidated privacy information, warnings, and notes into prose sections where possible to avoid admonition fatigue
• created provider table for clearer setup options
• reorganized knowledge base section with recommended approaches first
• reorganized embedding methods to prioritize simpler approach
• added clear navigation paths throughout document
• reorganized functions as subcategory of chat rather than separate feature
• improved section organization to match user workflow
• rephrased LLM limitations warning for clarity

Follow up ideas

• add quickstart guide at top for basic setup
• breakout information onto separate pages

[eedugon]

I think the refinement and improvements are good and beneficial here @leemthompo , thanks!

My comments for this or a next PR are:

• Separating the info in multiple pages and its own TOC structure is probably a good plan considering how long and complex the document is.

• I don't understand the doc that is now in /solutions/search/ai-assistant.md. It feels really weird. I think we should improve the flow of that document, and clarify why it is so different than the doc in the Observability solution.

  • If the main doc is 100% valid for observability and search we can use a snippet and provide the same content in both places.
  • If the introduction list that the search AI assistant has is considered useful then it should also be in the observability side.

That's the only thing that looks a bit weird in my opinion. The long doc itself looks good.

Final comment: What if we try to put AI Assistant as a Level 1 citizen on the Solutions and Use Cases documentation? So having it at the same level than the solutions, with its own structure.

[eedugon]

LGTM! Added some comments as the short doc in search feels a bit weird and not correctly presented or scoped.

[leemthompo]

Thanks for review @eedugon!

I don't understand the doc that is now in /solutions/search/ai-assistant.md. It feels really weird.

💯 it's basically a soft redirect for visibility of AI Assistant functionality in search spaces, I can trim it down fully right now and just make it a link to the consolidated doc.

If the main doc is 100% valid for observability and search we can use a snippet and provide the same content in both places.

Yup but it's too big to snippet and way too big to duplicate for SEO reasons. This first cut is really just trying to simplify the big existing doc.

Otherwise, I think it's now much easier to reason about splitting that page into multiple subsections 👍

Final comment: What if we try to put AI Assistant as a Level 1 citizen on the Solutions and Use Cases documentation? So having it at the same level than the solutions, with its own structure.

Yeah that might be an idea, I'm still not sure how feasible it will be to factor out shared stuff for Security + Obs/Search assistants, because security seems to be meaningfully different.

It's basically a confusing subject, because on top of this a lot of the core functionality is changing rapidly too, hence why the original Obs doc has gotten so long 😄

[leemthompo]

I'll create a follow up ticket for splitting the big page up

[szabosteve]

Thanks, LGTM!