[Enhancement]: Customize Parameters for Custom Endpoints

[danny-avila]

What features would you like to see added?

Custom parameters for custom endpoints via librechat.yaml configuration, using existing parameters from other providers or brand new, dynamic ones

More details

See #6503, #7179, #7180

Code of Conduct

• I agree to follow this project's Code of Conduct

[mawburn]

@danny-avila is this meant to be suggested default settings or overrides?

The linked PR changes the way model specs work and don't seem to be related?

[nhtruong]

I'd like to take a crack at this.

[danny-avila]

@nhtruong thanks!

@mawburn there's a long discussion in the linked PR and this is the relevant comment: #6503 (review)

This is not related to default settings or overrides, this would change what parameters appear and the user can choose from for custom endpoints:

[mawburn]

@danny-avila

This is not related to default settings or overrides, this would change what parameters appear and the user can choose from for custom endpoints:

Ok, but I don't think those are the same thing that @nhtruong created in his tickets, we are on the same team and were discussing this on Slack. We are looking for default parameters to be set per model or just globally for the project. Mostly things like Temp, because 1.0 temp is not a good setting to be default.

I can see if this is rolled into what parameters are seen too, but they seem like separate concerns to me because default parameters should also apply to OpenAI endpoints.

[nhtruong]

For clarification, my original issue attempts to add something like this in the librechat.yaml for custom endpoints:

    - name: 'Cohere'
      ...
      addParams:
        temperature:
          - min_range: 0.2
          - max_range: 0.7
          - default: 0.6
          - step: 0.1
        top_p: 0.5

That setting will enforce the range for temperature when the user updates this parameter for Cohere or Cohere agents. The goal is allowing the admin to limit the range of this value but still give the user some wiggle room instead of a single overriden value.

[danny-avila]

I moved this to discussions since some clarification is needed. I think we're saying the same thing more or less, since customization of the parameters can limit the parameters and set defaults.

I understand this as "parameter customization" or "dynamic parameters." All the mentioned refs fall into this one way or another: #6503, #7179, #7180

I guess we can make some distinctions though:
A. Customize/set-defaults for existing parameters
B. introduce new, dynamic parameters unknown to LC
C. include parameters from other providers

I think the thing you want to tackle is A and then I can move on to B, C, since they would involve similar changes.

I have a branch where I started on all 3, including something very similar to what @nhtruong is suggesting, but not via addParams. I think we should leave that alone although it's functionally similar. It's more of a "parameter guard" before the API call than a way to influence the full stack.

I will update that branch and push it up soon.

[nhtruong]

Thanks @danny-avila
Should I still work on this feature to add custom range for existing parameters? (i.e. part A) since your new branch is gonna cover that as well? I'd rather not step on your foot if you're already working on it but I'd love to take that off of your plate if you haven't really started yet.

[danny-avila]

No worries @nhtruong, I just realized I had already merged most of the changes I made. Please feel free to take over. I'm going to share some code that is hopefully valuable to get you started.

---

Basically we can customize any of the parameters, both the UI and the backend, by making use of SettingDefinition I already made:

LibreChat/packages/data-provider/src/generate.ts

Lines 52 to 82 in 5efad8f

	export interface SettingDefinition {
	key: string;
	description?: string;
	type: 'number' | 'boolean' | 'string' | 'enum' | 'array';
	default?: number | boolean | string | string[];
	showLabel?: boolean;
	showDefault?: boolean;
	options?: string[];
	range?: SettingRange;
	enumMappings?: Record<string, number | boolean | string>;
	component: ComponentType;
	optionType?: OptionType;
	columnSpan?: number;
	columns?: number;
	label?: string;
	placeholder?: string;
	labelCode?: boolean;
	placeholderCode?: boolean;
	descriptionCode?: boolean;
	minText?: number;
	maxText?: number;
	minTags?: number; // Specific to tags component
	maxTags?: number; // Specific to tags component
	includeInput?: boolean; // Specific to slider component
	descriptionSide?: 'top' | 'right' | 'bottom' | 'left';
	items?: OptionWithIcon[]; // Specific to combobox component
	searchPlaceholder?: string; // Specific to combobox component
	selectPlaceholder?: string; // Specific to combobox component
	searchPlaceholderCode?: boolean; // Specific to combobox component
	selectPlaceholderCode?: boolean; // Specific to combobox component
	}

This structure is already being used for what we see in the UI now:

LibreChat/client/src/components/SidePanel/Parameters/settings.ts

Lines 10 to 13 in 5efad8f

	import type { SettingsConfiguration, SettingDefinition } from 'librechat-data-provider';
	// Base definitions
	const baseDefinitions: Record<string, SettingDefinition> = {

Our custom config would have to be transmitted to the client to make use of it in:
https://github.com/danny-avila/LibreChat/blob/main/client/src/components/SidePanel/Parameters/Panel.tsx

There are 2 helper functions I already made, feel free to improve/adjust as needed.

For validating the "SettingDefinition":

LibreChat/packages/data-provider/src/generate.ts

Line 214 in 5efad8f

export function validateSettingDefinitions(settings: SettingsConfiguration): void {

For dynamic zod schema generation:

LibreChat/packages/data-provider/src/generate.ts

Line 104 in 5efad8f

export function generateDynamicSchema(settings: SettingsConfiguration) {

The zod schemas are used to parse model parameters throughout the app. I tried to centralize this through the following functions:

LibreChat/packages/data-provider/src/parsers.ts

Line 163 in 5efad8f

export const parseConvo = ({

LibreChat/packages/data-provider/src/parsers.ts

Line 331 in 5efad8f

export const parseCompactConvo = ({

There may be a few places where the "endpoint schemas" are used individually, but it would just be a matter of cross-referencing the following:

LibreChat/packages/data-provider/src/parsers.ts

Lines 24 to 46 in 5efad8f

	type EndpointSchema =
	| typeof openAISchema
	| typeof googleSchema
	| typeof anthropicSchema
	| typeof gptPluginsSchema
	| typeof assistantSchema
	| typeof compactAgentsSchema
	| typeof bedrockInputSchema;
	export type EndpointSchemaKey = Exclude<EModelEndpoint, EModelEndpoint.chatGPTBrowser>;
	const endpointSchemas: Record<EndpointSchemaKey, EndpointSchema> = {
	[EModelEndpoint.openAI]: openAISchema,
	[EModelEndpoint.azureOpenAI]: openAISchema,
	[EModelEndpoint.custom]: openAISchema,
	[EModelEndpoint.google]: googleSchema,
	[EModelEndpoint.anthropic]: anthropicSchema,
	[EModelEndpoint.gptPlugins]: gptPluginsSchema,
	[EModelEndpoint.assistants]: assistantSchema,
	[EModelEndpoint.azureAssistants]: assistantSchema,
	[EModelEndpoint.agents]: compactAgentsSchema,
	[EModelEndpoint.bedrock]: bedrockInputSchema,
	};

[nhtruong]

Great and thanks for the tips.

[jameslamine]

Thanks Danny and @nhtruong for tackling this! Really looking forward to this, especially B and C

[nhtruong]

What I'm working on will handle B/C too. Aiming to have some draft PR sometime next week.

[dlouzan]

I've just stumbled upon this discussion while trying to add configuration for the following use case:

• We have a custom enterprise endpoint, over which we offer multiple different models such as deepseek, mistral, qwq, etc
• I was trying to setup "startup defaults" per each of these models while at the same time allowing users to modify them at their leisure after the fact
• The idea is that a user selects a dropdown model, and gets by default the admin-selected values
• For instance, deepseek suggests using a default 0.5 to 0.7 temperature, perhaps we wanted to set specific max tokens for some models, others recommend not using a system prompt, and so on.

As I understand it, with the existing configuration options, and even with the approach currently being discussed here, we'd need to create individual duplicated endpoint entries for each model default parameters that I wanted to configure right? That's probably not optimal but I guess it'd work, with several hiccups, e.g. we'd get menu entries for each duplicate endpoint, perhaps request tracking, api keys are not correctly tracked/duplicated in request tracking, etc.

Thanks for any feedback!

[nhtruong]

Here's the progress update:

If you add the following entry to endpoints.custom in librechat.yaml:

    - name: 'Google my Google'
      apiKey: ...
      baseURL: ...
      models:
        default:
          [
            'googlevertexai-us-central1:gemini-2.5-pro-preview-03-25',
            'googlevertexai-us-central1:gemini-2.5-pro-exp-03-25',
            'googlevertexai-us-central1:gemini-2.5-flash-preview-04-17',
          ]
        fetch: false
      titleConvo: true
      titleModel: 'gpt-3.5-turbo'
      summarize: false
      modelDisplayLabel: 'Gemini'
      customParams:
        includeDefaultParams: [temperature]
        paramDefinitions:
          - key: atmospheric_pressure
            type: number
            component: slider
            range:
              min: 0
              max: 0.7
              step: 0.1
            optionType: custom
            default: 0.5
            columnSpan: 4
          - key: agent_name
            type: string
            component: input
            default: ''
            optionType: custom
            columnSpan: 2
          - key: agent_code
            type: string
            component: input
            default: ''
            optionType: custom
            columnSpan: 2

the following parameters will be populated when editing an agent:

This also validates includeDefaultParams (boolean or an array of default param keys) and the properties of paramDefinitions.

There's still a lot of work left:

• Persisting custom param values for agents
• Retrieving and applying these values when sending a prompt to an endpoint.
• Apply all of this to the Parameters section (as opposed to Agent Parameters)

Here's the WIP Draft: #7225

[nhtruong]

I've traced the agent.model_parameters to graphConfig in

  return Run.create({
    runId,
    graphConfig,
    customHandlers,
  });

however, Run is part of @librechat/agents npm package. The source code of the package is stored in https://github.com/danny-avila/agents and it's a private repo.

[nhtruong]

I'll focus on existing parameters for now as the ability to set your own min/max and default value for these parameters is very valuable.

[danny-avila]

@nhtruong can you give an example of what you're seeing vs. what you're expecting?

[nhtruong]

Let's say I hook Librechat up to a homegrown LLM model that accepts temperature and pressure as parameters to finetune the the answers given by the model. So, I add these parameters to endpoints.custom.[].customParams.paramDefinitions of librechat.yaml. When I create an agent, I'll set the temperature at 0.5 and pressure at 3.0 (pressure is just a hypothetical new parameter), and the agent.model_parameters will be saved as { temperature: 0.5, pressure: 3.0 }.

Then I ask this agent a question, those parameters will go through several transformations and in agents/run.js, these parameters will be included in graphConfig, which has the following value:

{"signal":{},"llmConfig":{"provider":"openAI","streaming":true,"streamUsage":false,
"temperature":0.5,"pressure":3.0,
"user":"--redacted--","openAIApiKey":"--redacted--",
"configuration":{"baseURL":"--readacted--","defaultHeaders":{}},
"model":"gemini-x2","usage":true},"instructions":""}

It appears to me that only known parameters like temperature will be extracted from that JSON object and fed into to the LLM endpoints. So, my goal is updating the bit of code that sends API requests to the LLM endpoint to also include new parameters.

[danny-avila]

Understood, yes "unknown" params will get discarded. I need to add this functionality as described here: #7251 (comment)

[nhtruong]

I'm almost done with part A, which is Customize/set-defaults for existing parameters. I just need to add more tests and tidy up my code. While working on this, I also added a small feature that lets user of custom endpoints pick the appropriate set of params for their endpoints.

As of right now if you the following custom endpoints defined:

# librechat.yaml

endpoints:
  custom:
    - name: 'Αnthropic'
      # ...
    - name: 'Google Gemini'
      # ...

The set of parameters available to both of these endpoints are identical. It inherits from OpenAI params, which doesn't make sense for neither Anthropic nor Google:

---

---

With this new feature you can update the librechat.yaml file like so:

endpoints:
  custom:
    - name: 'Αnthropic'
      # ...
      customParams:
        defaultParamsEndpoint: 'anthropic'

    - name: 'Google my Google'
      # ...
      customParams:
        defaultParamsEndpoint: 'google'
        paramDefinitions:
          - key: temperature
            range:
              min: 0
              max: 0.7
              step: 0.1
            default: 0.5

Now Anthropic custom endpoint will use parameters for anthropic:

And Google Gemini custom endpoint will use parameters for google. On top of that, we get to pick our own range and default value for temperature parameter:

---

I also added a lot of validation checkpoints to assure that values put into customParams are legit.

[nhtruong]

PR for customizing existing parameters is ready: #7342