Way to provide own LLM interfaces as constructor to the FastAgent

[lahcim]

Currently FastAgent takes only the model, which must match any of the hardcoded models (anthophic, openai, deepseek etc).
I do want to use groq, which is not supported yet in the code.

I am ok writing my own wrapper class for this, but it seems there is no way to provide this to the FastAgent, as it only uses internal factory class, which also uses internal hardcoded set of LLMs.

So everything seems hardcoded inside fastagent, and cannot easily add new interfaces.

Can we expose adding own models for people who want to provide more? and not realying only on hardcoded ones?

[evalstate]

Hi @lahcim - I've added support for generic models in version 0.2.3 - not sure if that meets your needs?

There's a generic provider, which uses ollama defaults (but they can be overriden arbitrarily in the config). That will then pass the model name straight through the OpenAI endpoint, so should cover a lot of cases.

I do want to keep a distinction between those models fully tested, added to the automated test suite where the various modalities/tool calls are known to work vs. generic support? Does that make sense?

Anyway - let me know if the above helps or you need assistance pulling together a PR.

[lahcim]

Do you have some example of this one? Like how exactly provide this in config? Can this be also provided as class constructor argument or something?

[evalstate]

For ollama running locally the below is all you need (nothing extra in the config file, the defaults should work)

import asyncio
from mcp_agent.core.fastagent import FastAgent

# Create the application
fast = FastAgent("FastAgent Example")


# Define the agent
@fast.agent(name="llama", instruction="be kind", model="generic.llama3.2:latest")
async def main():
    # use the --model command line switch or agent arguments to change model
    async with fast.run() as agent:
        await agent()


if __name__ == "__main__":
    asyncio.run(main())

or --model=generic.qwen2.5:latest from the command line (NB agent definition takes prefence)

For groq (untested by me) I'd add this to your secrets file:

generic:
   base_url: "https://api.groq.com/openai/v1"
   api_key: very_secret_key

And use a model name of e.g. generic.llama-3.3-70b-versatile. LMK if that works.

[evalstate]

Oh, and if you prefer an environment variable for the API Key, use GENERIC_API_KEY.

[claydegruchy]

For anyone running into issues getting this going with ollama locally; the example python given works but in the fastagent.config.yaml you need to use the ollama openai compaditbility endpoint (which is usually served from http://localhost:11434/v1/). You can find more info on this endpoint here:
https://github.com/ollama/ollama/blob/main/docs/openai.md

[zba]

Hi, I did checked it and it seems need to allow more logic userspace, becase opensource models very different, I had to intercept openai.chat create function to inject correct tool instructions and message sequence for google gemma model.

Better if there would be official handlers for sorting messages before llm and for handling llm answer.