Skip to content
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion samples/managed-llm-provider/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,10 +43,11 @@ For this sample, you will need to provide the following [configuration](https://
> Note that if you are using the 1-click deploy option, you can set these values as secrets in your GitHub repository and the action will automatically deploy them for you.

### `MODEL`
The Model ID of the LLM you are using for your application. For example, `anthropic.claude-3-haiku-20240307-v1:0`.
The Model ID of the LLM you are using for your application. For example, `ai/claude3-haiku`.
```bash
defang config set MODEL
```
For more about model mapping, please see our [Model Mapping documentation](https://docs.defang.io/docs/concepts/managed-llms/openai-access-gateway#model-mapping).

## Deployment

Expand Down
2 changes: 1 addition & 1 deletion samples/managed-llm-provider/compose.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ services:
restart: always
environment:
- ENDPOINT_URL=http://llm/api/v1/chat/completions # endpoint to the Provider Service
- MODEL=anthropic.claude-3-haiku-20240307-v1:0 # LLM model ID used in the Provider Service
- MODEL=ai/claude3-haiku # LLM model ID used in the Provider Service
healthcheck:
test: ["CMD", "python3", "-c", "import sys, urllib.request; urllib.request.urlopen(sys.argv[1]).read()", "http://localhost:8000/"]
interval: 30s
Expand Down
3 changes: 2 additions & 1 deletion samples/managed-llm/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,10 +46,11 @@ For this sample, you will need to provide the following [configuration](https://
> Note that if you are using the 1-click deploy option, you can set these values as secrets in your GitHub repository and the action will automatically deploy them for you.

### `MODEL`
The Model ID of the LLM you are using for your application. For example, `anthropic.claude-3-haiku-20240307-v1:0`.
The Model ID of the LLM you are using for your application. For example, `ai/claude3-haiku`.
```bash
defang config set MODEL
```
For more about model mapping, please see our [Model Mapping documentation](https://docs.defang.io/docs/concepts/managed-llms/openai-access-gateway#model-mapping).

## Deployment

Expand Down
2 changes: 1 addition & 1 deletion samples/managed-llm/compose.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ services:
restart: always
environment:
- ENDPOINT_URL=http://llm/api/v1/chat/completions # endpoint to the gateway service
- MODEL=anthropic.claude-3-haiku-20240307-v1:0 # LLM model ID used for the gateway
- MODEL=ai/claude3-haiku # LLM model ID used for the gateway
- OPENAI_API_KEY=FAKE_TOKEN # the actual value will be ignored when using the gateway, but it should match the one in the llm service
healthcheck:
test: ["CMD", "python3", "-c", "import sys, urllib.request; urllib.request.urlopen(sys.argv[1]).read()", "http://localhost:8000/"]
Expand Down
Loading