vLLM custom connector setup guide

[benironside]

Resolves #3474 by creating a tutorial for how to connect a custom LLM running in vLLM to Elastic.

Technical reviewers, I left a few questions for you in comments. Also:

• Has this been tested with the Obs/Search Assistant, or are these instructions security-only.
• Is this supported in v9.0+?
• @dhru42 I could use some insight into how the use-case for this guide differs from the existing self-managed LLM guide

[github-actions]

🔍 Preview links for changed docs

• solutions/security/ai/connect-to-vLLM.md
• solutions/security/ai/connect-to-own-local-llm.md
• solutions/security/ai/set-up-connectors-for-large-language-models-llm.md

[benironside]

Is it just Elastic Cloud that this works with? Not other deployment types?

[benironside]

Note to self: Following https://github.com/elastic/sdh-security-team/issues/1417 to confirm if this system prompt fix works

[patrykkopycinski]

since 9.1.7 it seems it is not needed anymore, but we can keep it until we change the recommended model,
more important in this case is to make sure they add

feature_flags.overrides:
  securitySolution.inferenceChatModelDisabled: true

to config/kibana.yml otherwise Mistral is not going to work with Security Assistant (more details in linked SDH above)

[dhru42]

@dhru42 I could use some insight into how the use-case for this guide differs from the existing self-managed LLM guide

can we make the existing page generic then link to two methods:

1. Connect to your own local LLM with LM Studio (exists already)
2. Connect to your own local LLM with vLLM (the google doc i shared)

[benironside]

@dhru42 I could use some insight into how the use-case for this guide differs from the existing self-managed LLM guide

can we make the existing page generic then link to two methods:

1. Connect to your own local LLM with LM Studio (exists already)

2. Connect to your own local LLM with vLLM ([the google doc i shared](https://docs.google.com/document/d/1pGKBECl6T4LdFhctAWURRZJrdEC8qKRVWz0bWnylN4s/edit?usp=sharing))

Yeah, that makes sense. @dhru42 I'm still curious to better understand the different use-cases that each option addresses. How should users pick which to set up?

After docs on-week (this week), I'll work on it and sync up with Patryk and/or Garrett to discuss details.

[dhru42]

@benironside there's a formatting issue. could you ensure that all the steps are reflected as shown in the docs, otherwise it LGTM.

[patrykkopycinski]

is is something we will be able to update shortly? I mean we should avoid recommending Mistral-Small-3.2-24B as it has a lot of issues with Security Assistant tool calling

[benironside]

We can update this any time. For now, since this model isn't recommended, I replaced it with [YOUR_MODEL_ID]. Make sense to you?

[patrykkopycinski]

I think it's going to be less confusing if we stay to the previous version and just update it with a new model, because the list of params depends on the model id

[patrykkopycinski]

since 9.1.7 it seems it is not needed anymore, but we can keep it until we change the recommended model,
more important in this case is to make sure they add

feature_flags.overrides:
  securitySolution.inferenceChatModelDisabled: true

to config/kibana.yml otherwise Mistral is not going to work with Security Assistant (more details in linked SDH above)

[github-actions]

Vale Linting Results

Summary: 3 suggestions found

💡 Suggestions (3)

File	Line	Rule	Message
solutions/security/ai/connect-to-own-local-llm.md	14	Elastic.Capitalization	'Connect to your own local LLM using LM Studio' should use sentence-style capitalization.
solutions/security/ai/connect-to-vLLM.md	21	Elastic.Capitalization	'Connect vLLM to' should use sentence-style capitalization.
solutions/security/ai/connect-to-vLLM.md	93	Elastic.WordChoice	Consider using 'can, might' instead of 'may', unless the term is in the UI.

[bmorelli25]

This is a good start, but right now the page feels like it's trying to be a guide and an example. If you pick a single type of content, it'll be more useful and easier to follow. I think you should structure this page as a guide, similar to these:

• https://www.elastic.co/docs/solutions/search/get-started/semantic-search
• https://www.elastic.co/docs/manage-data/data-store/data-streams/quickstart-tsds

The example content, like the Server info is probably still useful, but could be folded up into the relevant step.

Thoughts?

[bmorelli25]

This looks great! Just two more small comments for your consideration.

🚢 🚢