Skip to content

Commit a9f7f4c

Browse files
taras-yemetstaras-yemets
committed
wip
1 parent c7c9073 commit a9f7f4c

File tree

1 file changed

+144
-22
lines changed

1 file changed

+144
-22
lines changed

‎docs/cody/model-configuration/examples.mdx‎

Lines changed: 144 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,8 @@ Sourcegraph-supplied models come with preconfigured providers, identified by the
2020

2121
### Override provider config for all models in the namespace
2222

23-
When Sourcegraph-supplied models are used and a provider override for a Sourcegraph-supported provider (same ID) is specified, the override applies to all Sourcegraph-supplied models within that provider.
23+
When Sourcegraph-supplied models are used and a provider override for a Sourcegraph-supported provider (same ID) is specified,
24+
the override applies to all Sourcegraph-supplied models within that provider.
2425
For example, if you specify an override for a provider with ID `"anthropic"`, it will apply to all models from the `"anthropic"` provider.
2526

2627
Example configuration:
@@ -55,66 +56,187 @@ In the configuration above, we:
5556
- Route requests for Anthropic models directly to the Anthropic API (via the provider override specified for "anthropic").
5657
- Route requests for other models (such as the Fireworks model for "autocomplete") through Cody Gateway.
5758

58-
### Override provider configur for some models and use the Sourcegraph-configured provider config for the rest
59+
### Override provider config for some models in the namespace and use the Sourcegraph-configured provider config for the rest
5960

60-
It's possible to route requests directly to the LLM provider (bypassing the Cody Gateway) for some models while using the Sourcegraph-configured provider config for the rest.
61+
It's possible to route requests directly to the LLM provider (bypassing the Cody Gateway) for some models while using the
62+
Sourcegraph-configured provider config for the rest.
6163

6264
Example configuration:
6365

66+
In the configuration above, we:
67+
68+
- Enable Sourcegraph-supplied models (the `sourcegraph` field is not empty or `null`).
69+
- Define a new provider with the ID `"anthropic-byok"` and configure it to use the Anthropic API.
70+
- Since this provider is unknown to Sourcegraph, no Sourcegraph-supplied models are available for it.
71+
Therefore, we add a custom model in the `"modelOverrides"` section.
72+
- Use the custom model configured in the previous step (`"anthropic-byok::2024-10-22::claude-3.5-sonnet"`) for `"chat"`.
73+
Requests are sent directly to the Anthropic API as set in the provider override.
74+
- For `"fastChat"` and `"autocomplete"`, we use Sourcegraph-supplied models via Cody Gateway.
75+
76+
## Config examples for various LLM providers
77+
78+
Below are configuration examples for setting up various LLM providers using BYOK.
79+
These examples are applicable whether or not you are using Sourcegraph-supported models.
80+
81+
**Note:**
82+
83+
- In this section, all configuration examples have Sourcegraph-supplied models disabled. To use a combination of
84+
Sourcegraph-supplied models and BYOK, please refer to the previous section.
85+
- Ensure that at least one model is available for each Cody feature ("chat", "edit", "autocomplete"), regardless of
86+
the provider and model overrides configured. To verify this, [view the configuration](/cody/model-configuration#view-configuration)
87+
and confirm that appropriate models are listed in the `"defaultModels"` section.
88+
89+
<Accordion title="Anthropic">
90+
6491
```json
6592
{
6693
"cody.enabled": true,
6794
"modelConfiguration": {
68-
"sourcegraph": {},
95+
"sourcegraph": null,
6996
"providerOverrides": [
7097
{
71-
"id": "anthropic-byok",
72-
"displayName": "Anthropic BYOK",
98+
"id": "anthropic",
99+
"displayName": "Anthropic",
73100
"serverSideConfig": {
74101
"type": "anthropic",
75102
"accessToken": "sk-ant-token",
76-
"endpoint": "https://api.anthropic.com/v1/messages"
103+
"endpoint": "https://api.anthropic.com/v1/messages"
77104
}
78105
}
79106
],
80107
"modelOverrides": [
81108
{
82-
"modelRef": "anthropic-byok::2024-10-22::claude-3.5-sonnet",
109+
"modelRef": "anthropic::2024-10-22::claude-3.5-sonnet",
83110
"displayName": "Claude 3.5 Sonnet",
84111
"modelName": "claude-3-5-sonnet-latest",
85-
"capabilities": ["edit", "chat", "vision"],
112+
"capabilities": ["edit", "chat"],
86113
"category": "accuracy",
87114
"status": "stable",
88115
"tier": "free",
89116
"contextWindow": {
90117
"maxInputTokens": 45000,
91118
"maxOutputTokens": 4000
92119
}
120+
},
121+
{
122+
"modelRef": "anthropic::2023-06-01::claude-3-haiku",
123+
"displayName": "Claude 3 Haiku",
124+
"modelName": "claude-3-haiku-20240307",
125+
"capabilities": ["edit", "chat"],
126+
"category": "speed",
127+
"status": "stable",
128+
"tier": "free",
129+
"contextWindow": {
130+
"maxInputTokens": 7000,
131+
"maxOutputTokens": 4000
132+
}
133+
},
134+
{
135+
"modelRef": "anthropic::2023-01-01::claude-instant-1.2",
136+
"displayName": "Claude Instant",
137+
"modelName": "claude-instant-1.2",
138+
"capabilities": ["autocomplete", "edit", "chat"],
139+
"category": "other",
140+
"status": "deprecated",
141+
"tier": "free",
142+
"contextWindow": {
143+
"maxInputTokens": 7000,
144+
"maxOutputTokens": 4000
145+
}
93146
}
94147
],
95148
"defaultModels": {
96-
"chat": "anthropic-byok::2024-10-22::claude-3.5-sonnet",
149+
"chat": "anthropic::2024-10-22::claude-3.5-sonnet",
97150
"fastChat": "anthropic::2023-06-01::claude-3-haiku",
98-
"autocomplete": "fireworks::v1::deepseek-coder-v2-lite-base"
151+
"autocomplete": "anthropic::2023-01-01::claude-instant-1.2"
99152
}
100153
}
101154
```
102155

103156
In the configuration above, we:
104157

105-
- Enable Sourcegraph-supplied models (the `sourcegraph` field is not empty).
106-
- Define a new provider with the ID `"anthropic-byok"` and configure it to use the Anthropic API.
107-
- Since this provider is unknown to Sourcegraph, no Sourcegraph-supplied models are available for it. Therefore,
108-
we add a custom model in the `"modelOverrides"` section.
109-
- Use the custom model configured in the previous step (`"anthropic-byok::2024-10-22::claude-3.5-sonnet"`) for `"chat"`.
110-
Requests are sent directly to the Anthropic API as set in the provider override.
111-
- For `"fastChat"` and `"autocomplete"`, we use Sourcegraph-supplied models via Cody Gateway.
158+
- Set up a provider override for Anthropic, routing requests for this provider directly to the specified Anthropic endpoint (bypassing Cody Gateway).
159+
- Add three Anthropic models:
160+
- Two models with chat capabilities (`"anthropic::2024-10-22::claude-3.5-sonnet"` and `"anthropic::2023-06-01::claude-3-haiku"`),
161+
providing options for chat users.
162+
- One model with autocomplete capability (`"anthropic::2023-01-01::claude-instant-1.2"`).
163+
- Set the configured models as default models for Cody features in the `"defaultModels"` field.
112164

113-
## Config examples for various LLM providers
165+
</Accordion>
114166

115-
Below are configuration examples for setting up various LLM providers using BYOK.
116-
These examples are applicable whether or not you are using Sourcegraph-supported models.
167+
<Accordion title="Fireworks">
168+
```json
169+
"cody.enabled": true,
170+
"modelConfiguration": {
171+
"sourcegraph": null,
172+
"providerOverrides": [
173+
{
174+
"id": "fireworks",
175+
"displayName": "Fireworks",
176+
"serverSideConfig": {
177+
"type": "fireworks",
178+
"accessToken": "token",
179+
"endpoint": "https://api.fireworks.ai/inference/v1/completions"
180+
}
181+
}
182+
],
183+
"modelOverrides": [
184+
{
185+
"modelRef": "fireworks::v1::mixtral-8x22b-instruct",
186+
"displayName": "Mixtral 8x22B",
187+
"modelName": "accounts/fireworks/models/mixtral-8x22b-instruct",
188+
"capabilities": ["edit", "chat"],
189+
"category": "other",
190+
"status": "stable",
191+
"tier": "free",
192+
"contextWindow": {
193+
"maxInputTokens": 7000,
194+
"maxOutputTokens": 4000
195+
}
196+
},
197+
{
198+
"modelRef": "fireworks::v1::starcoder-16b",
199+
"modelName": "accounts/fireworks/models/starcoder-16b",
200+
"displayName": "(Fireworks) Starcoder 16B",
201+
"contextWindow": {
202+
"maxInputTokens": 8192,
203+
"maxOutputTokens": 4096
204+
},
205+
"capabilities": ["autocomplete"],
206+
"category": "balanced",
207+
"status": "stable"
208+
}
209+
],
210+
"defaultModels": {
211+
"chat": "fireworks::v1::mixtral-8x22b-instruct",
212+
"fastChat": "fireworks::v1::mixtral-8x22b-instruct",
213+
"autocomplete": "fireworks::v1::starcoder-16b"
214+
}
215+
}
216+
```
117217

118-
<Accordion title="Anthropic">
218+
In the configuration above, we:
219+
220+
- Set up a provider override for Fireworks, routing requests for this provider directly to the specified Fireworks endpoint (bypassing Cody Gateway).
221+
- Add two Fireworks models: - `"fireworks::v1::mixtral-8x22b-instruct"` with "edit" and "chat" capabiities - used for "chat"
222+
and "fastChat" - `"fireworks::v1::starcoder-16b"` with "autocomplete" capability - used for "autocomplete".
119223

120224
</Accordion>
225+
226+
<Accordion title="OpenAI"></Accordion>
227+
<Accordion title="Azure OpenAI"></Accordion>
228+
<Accordion title="Generic OpenAI-compatible"></Accordion>
229+
230+
<Accordion title="Google Vertex (Anthropic)"></Accordion>
231+
<Accordion title="Google Vertex (Gemini)"></Accordion>
232+
<Accordion title="Google Vertex (public)"></Accordion>
233+
234+
<Accordion title="AWS Bedrock"></Accordion>
235+
236+
## Self-hosted models
237+
238+
TODO
239+
240+
```
241+
242+
```

0 commit comments

Comments
 (0)