Skip to content

Commit c0fac2e

Browse files
committed
revert back to full name for model
1 parent eef075e commit c0fac2e

File tree

4 files changed

+6
-30
lines changed

4 files changed

+6
-30
lines changed

samples/managed-llm-provider/README.md

Lines changed: 2 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,8 @@ You can configure the `MODEL` and `ENDPOINT_URL` for the LLM separately for loca
1414

1515
Ensure you have enabled model access for the model you intend to use. To do this, you can check your [AWS Bedrock model access](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access-modify.html) or [GCP Vertex AI model access](https://cloud.google.com/vertex-ai/generative-ai/docs/control-model-access).
1616

17+
To learn about available LLM models in Defang, please see our [Model Mapping documentation](https://docs.defang.io/docs/concepts/managed-llms/openai-access-gateway#model-mapping).
18+
1719
For more about Managed LLMs in Defang, please see our [Managed LLMs documentation](https://docs.defang.io/docs/concepts/managed-llms/managed-language-models).
1820

1921
### Docker Model Provider
@@ -36,19 +38,6 @@ To run the application locally, you can use the following command:
3638
docker compose -f compose.dev.yaml up --build
3739
```
3840

39-
## Configuration
40-
41-
For this sample, you will need to provide the following [configuration](https://docs.defang.io/docs/concepts/configuration):
42-
43-
> Note that if you are using the 1-click deploy option, you can set these values as secrets in your GitHub repository and the action will automatically deploy them for you.
44-
45-
### `MODEL`
46-
The Model ID of the LLM you are using for your application. For example, `ai/nova-micro`.
47-
```bash
48-
defang config set MODEL
49-
```
50-
For more about model mapping, please see our [Model Mapping documentation](https://docs.defang.io/docs/concepts/managed-llms/openai-access-gateway#model-mapping).
51-
5241
## Deployment
5342

5443
> [!NOTE]

samples/managed-llm-provider/compose.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ services:
88
restart: always
99
environment:
1010
- ENDPOINT_URL=http://llm/api/v1/chat/completions # endpoint to the Provider Service
11-
- MODEL=ai/nova-micro # LLM model ID used in the Provider Service
11+
- MODEL=us.amazon.nova-micro-v1:0 # LLM model ID used in the Provider Service
1212
healthcheck:
1313
test: ["CMD", "python3", "-c", "import sys, urllib.request; urllib.request.urlopen(sys.argv[1]).read()", "http://localhost:8000/"]
1414
interval: 30s

samples/managed-llm/README.md

Lines changed: 2 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,8 @@ You can configure the `MODEL` and `ENDPOINT_URL` for the LLM separately for loca
1717

1818
Ensure you have enabled model access for the model you intend to use. To do this, you can check your [AWS Bedrock model access](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access-modify.html) or [GCP Vertex AI model access](https://cloud.google.com/vertex-ai/generative-ai/docs/control-model-access).
1919

20+
To learn about available LLM models in Defang, please see our [Model Mapping documentation](https://docs.defang.io/docs/concepts/managed-llms/openai-access-gateway#model-mapping).
21+
2022
For more about Managed LLMs in Defang, please see our [Managed LLMs documentation](https://docs.defang.io/docs/concepts/managed-llms/managed-language-models).
2123

2224
### Defang OpenAI Access Gateway
@@ -39,19 +41,6 @@ To run the application locally, you can use the following command:
3941
docker compose -f compose.dev.yaml up --build
4042
```
4143

42-
## Configuration
43-
44-
For this sample, you will need to provide the following [configuration](https://docs.defang.io/docs/concepts/configuration):
45-
46-
> Note that if you are using the 1-click deploy option, you can set these values as secrets in your GitHub repository and the action will automatically deploy them for you.
47-
48-
### `MODEL`
49-
The Model ID of the LLM you are using for your application. For example, `ai/nova-micro`.
50-
```bash
51-
defang config set MODEL
52-
```
53-
For more about model mapping, please see our [Model Mapping documentation](https://docs.defang.io/docs/concepts/managed-llms/openai-access-gateway#model-mapping).
54-
5544
## Deployment
5645

5746
> [!NOTE]

samples/managed-llm/compose.yaml

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ services:
88
restart: always
99
environment:
1010
- ENDPOINT_URL=http://llm/api/v1/chat/completions # endpoint to the gateway service
11-
- MODEL=ai/nova-micro # LLM model ID used for the gateway
11+
- MODEL=us.amazon.nova-micro-v1:0 # LLM model ID used for the gateway
1212
- OPENAI_API_KEY=FAKE_TOKEN # the actual value will be ignored when using the gateway, but it should match the one in the llm service
1313
healthcheck:
1414
test: ["CMD", "python3", "-c", "import sys, urllib.request; urllib.request.urlopen(sys.argv[1]).read()", "http://localhost:8000/"]
@@ -29,8 +29,6 @@ services:
2929
mode: host
3030
environment:
3131
- OPENAI_API_KEY=FAKE_TOKEN # this value must match the one in the app service
32-
- USE_MODEL_MAPPING=false
33-
- DEBUG=true
3432
# if using GCP for BYOC deployment, add these environment variables:
3533
# - GCP_PROJECT_ID=${GCP_PROJECT_ID}
3634
# - GCP_REGION=${GCP_REGION}

0 commit comments

Comments
 (0)