How to configure custom openai compatible LLM

[mekkiamiri]

hello everyone,
i started using embabel java example.
i am stuck to configure it to use my own LLM which is open ai compatible: my goal is to configure wiht model, base url and api key.
can you please help ?
I need the spring boot application.yml configuration and java code that works with.
thanks

[alexheifetz]

Which LLM are you trying to add. Please share more details.

[mekkiamiri]

@alexheifetz i use gemini and mistral that are deployed in my company. My company offer access with openai compatible api that is configured with:

• base-url: a custom url from my company
• an api key: that is custom also
• and model: with a custom name

[alexheifetz]

sample code on how OpenAI models are currently integrated with the embabel-agent framework:

https://github.com/embabel/embabel-agent/blob/main/embabel-agent-api/src/main/kotlin/com/embabel/agent/config/models/OpenAiModels.kt

We are in the process of modularizing LLM(s) currently part of embabel-agent-api into dedicated Spring Boot Starters. Reference code will be available in the few days.

[alexheifetz]

are you using spring-ai-vertex-ai-gemini and spring-ai-mistral-ai for integration?

[mekkiamiri]

@alexheifetz no, i think i did not need.

[alexheifetz]

from @mekkiamiri

<@1278523834307317833> this is the applicationlyml and the configuration java class:
embabel:
custom-llm:
base-url: "https://api.example.dev/proxyllm-s3/v1/"
model: Custom-Gemini-2.5-Pro
api-key: "******"
models:
defaultLlm: Custom-Gemini-2.5-Pro

public CustomOpenAiModels(@value("${embabel.custom-llm.base-url}") String baseUrl,
@value("${embabel.custom-llm.api-key}") String apiKey,
ObservationRegistry observationRegistry) {
super(baseUrl, apiKey, null, null, observationRegistry);
}

@Bean
public Llm CustomOpenAiLLM() {
    // Call the superclass method to create the Llm bean
    return openAiCompatibleLlm(
            "Custom-Gemini-2.5-Pro",
            null, // pricingModel - can be added if known
            "CustomOpenAiCompatible", // A custom provider name
            null, // knowledgeCutoffDate - can be added if known
            null, // retryTemplate - can be configured if needed
            null  // optionsConverter - null uses the default
    );
}

}
and i used @action
Story craftStory(UserInput userInput, OperationContext context) {
return context.ai()
.withLlm("Custom-Gemini-2.5-Pro")

but i have the same error Default LLM 'gpt-4.1-mini' not found in available models: []

[alexheifetz]

how are you running: shell? please share log if you have it.

[mekkiamiri]

@alexheifetz my code you have shared works. it was an issue with my IDE.
i add that i followed the document https://docs.embabel.com/embabel-agent/guide/0.1.3-SNAPSHOT/index.html#reference.customizing
thank you and thanks to for help

[mekkiamiri]

@alexheifetz i am sorry to reopen the discussion again: when i run the application and test with : x " tell story about car " for example, i get error :
, role=USER, name=null, toolCallId=null, toolCalls=null, refusal=null, audioOutput=null, annotations=null]], model=Custom-Gemini-2.5-Pro, store=null, metadata=null, frequencyPenalty=null, logitBias=null, logprobs=null, topLogprobs=null, maxTokens=2000000, maxCompletionTokens=null, n=null, outputModalities=null, audioParameters=null, presencePenalty=null, responseFormat=null, seed=null, serviceTier=null, stop=null, stream=false, streamOptions=null, temperature=null, topP=null, tools=null, toolChoice=null, parallelToolCalls=null, user=null, reasoningEffort=null, webSearchOptions=null]] as "application/json" with org.springframework.http.converter.json.MappingJackson2HttpMessageConverter 00:20:58.813 [main] INFO LlmDataBindingProperties - LLM invocation rank-Agents: Retry attempt 10 of 10 due to: 404 - {"detail":"Not Found"} 00:20:58.813 [main] INFO RetryProperties - Operation ranker: Retry error. Retry count: 1
for more information, you can see the attached file.
log.txt

i did not find how to debug this issue. It seems that connection error when call LLM.
could you please help ?
thanks

[alexheifetz]

Details of the error have been omitted. You can use the stacktrace command to print the full stacktrace.
in the shell (after the failure) please type stacktrace to get full stack trace. Tnx

[alexheifetz]

please share complete log ( you can sanitize sensitive information if there is any ). Tnx!

[alexheifetz]

attach as file.

[mekkiamiri]

@alexheifetz i edited my last message with the attached log.txt file

[mekkiamiri]

@alexheifetz i debug the request to LLM but i did not find the solution:

• the error : Caused by: org.springframework.web.client.HttpClientErrorException$BadRequest: 400 Bad Request: "{"error":{"message":"{'error': '/chat/completions: Invalid model name passed in model=None. Call /v1/models to view available models for your key.'}","type":"None","param":"None","code":"400"}}"
  at org.springframework.web.client.HttpClientErrorException.create(HttpClientErrorException.java:103)
  at org.springframework.web.client.StatusHandler.lambda$defaultHandler$3(StatusHandler.java:86)
  at org.springframework.web.client.StatusHandler.handle(StatusHandler.java:146)
  at org.springframework.web.client.DefaultRestClient$DefaultResponseSpec.applyStatusHandlers(DefaultRestClient.java:838)
  at org.springframework.web.client.DefaultRestClient$DefaultResponseSpec.lambda$readBody$4(DefaultRestClient.java:827)
  at org.springframework.web.client.DefaultRestClient.readWithMessageConverters(DefaultRestClient.java:216)
  at org.springframework.web.client.DefaultRestClient$DefaultResponseSpec.readBody(DefaultRestClient.java:826)
  at org.springframework.web.client.DefaultRestClient$DefaultResponseSpec.lambda$toEntityInternal$2(DefaultRestClient.java:782)

• and the request body is:
  `ChatCompletionRequest[messages=[ChatCompletionMessage[rawContent=Your job is rank objects of type Agent (agent) based on user input.
  Given the user input, choose the name that best reflects the user's intent.

            User input: <hello story>
            
            Available choices, in format <name>: <description>:
            - WriteAndReviewAgent: Generate a story based on user input and review it
            
            Return the name of the chosen Agent and the confidence score from 0-1.
            IMPORTANT: The fully qualified name must be exactly the same as in the list.
            
            Example:
            {"rankings":[{"name":"cillum dolore eu fugiat","confidence":5.274506588170524},{"name":"Ut enim ad minim veniam","confidence":54.81819851204135},{"name":"consequat Duis aute","confidence":84.68060839264511}]}
            
            Your response should be in JSON format.
            Do not include any explanations, only provide a RFC8259 compliant JSON response following this format without deviation.
            Do not include markdown code blocks in your response.
            Remove the ```json markdown from the output.
            Here is the JSON Schema instance your output must adhere to:
            ```{
              "$schema" : "https://json-schema.org/draft/2020-12/schema",
              "type" : "object",
              "properties" : {
                "rankings" : {
                  "type" : "array",
                  "items" : {
                    "type" : "object",
                    "properties" : {
                      "confidence" : {
                        "type" : "number",
                        "description" : "confidence score from 0-1"
                      },
                      "name" : {
                        "type" : "string",
                        "description" : "name of what we're ranking"
                      }
                    },
                    "additionalProperties" : false
                  }
                }
              },
              "description" : "List of ranked choices",
              "additionalProperties" : false
            }```
            
            , role=USER, name=null, toolCallId=null, toolCalls=null, refusal=null, audioOutput=null, annotations=null]], model=Custom-Gemini-2.5-Pro, store=null, metadata=null, frequencyPenalty=null, logitBias=null, logprobs=null, topLogprobs=null, maxTokens=2000000, maxCompletionTokens=null, n=null, outputModalities=null, audioParameters=null, presencePenalty=null, responseFormat=null, seed=null, serviceTier=null, stop=null, stream=false, streamOptions=null, temperature=null, topP=null, tools=null, toolChoice=null, parallelToolCalls=null, user=null, reasoningEffort=null, webSearchOptions=null]
             """`
  

[mekkiamiri]

@alexheifetz i find the issue. the maxTokens should be less than 65537. But in my test it was 2000000.
thanks again for your help

[alexheifetz]

glad to hear you found the root cause.

[alexheifetz]

have you looked at https://github.com/embabel/embabel-agent-examples
it has number of examples to demonstrate workflows.

[mekkiamiri]

@alexheifetz thanks for the answer.
in the examples, i see only 1 agent with multiple actions.
i ask if i have for example 2 agents and every agent has his actions, how to create a workflow for the 2 agents.
thanks