Skip to content

Commit a1ff99f

Browse files
committed
update docs to include model mapping
1 parent 8158798 commit a1ff99f

File tree

2 files changed

+14
-4
lines changed

2 files changed

+14
-4
lines changed

docs/concepts/managed-llms/openai-access-gateway.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,10 @@ services:
3434
Under the hood, when you use the `model` provider, Defang will deploy the **OpenAI Access Gateway** in a private network. This allows you to use the same code for both local development and cloud deployment.
3535
The `x-defang-llm` extension is used to configure the appropriate roles and permissions for your service. See the [Managed Language Models](/docs/concepts/managed-llms/managed-language-models/) page for more details.
3636

37+
## Model Mapping
38+
39+
Defang supports model mapping through the openai-access-gateway on AWS and GCP. This takes a model with a Docker naming convention (e.g. ai/lama3.3) and maps it to the closest matching model name on the target platform. If no such match can be found it can fallback onto a known existing model (e.g. ai/mistral). These environment variables are USE_MODEL_MAPPING (default to true) and FALLBACK_MODEL (no default), respectively.
40+
3741
## Current Support
3842

3943
| Provider | Managed Language Models |

docs/tutorials/deploying-openai-apps-aws-bedrock-gcp-vertex.mdx

Lines changed: 10 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -43,15 +43,16 @@ Add **Defang's [openai-access-gateway](https://github.com/DefangLabs/openai-acce
4343
+ environment:
4444
+ - OPENAI_API_KEY
4545
+ - GCP_PROJECT_ID # if using GCP Vertex AI
46-
+ - GCP_REGION # if using GCP Vertex AI, AWS_REGION not necessary for Bedrock
46+
+ - REGION
4747
```
4848

4949
### Notes:
5050

5151
- The container image is based on [aws-samples/bedrock-access-gateway](https://github.com/aws-samples/bedrock-access-gateway), with enhancements.
5252
- `x-defang-llm: true` signals to **Defang** that this service should be configured to use target platform AI services.
5353
- New environment variables:
54-
- `GCP_PROJECT_ID` and `GCP_REGION` are needed if using **Vertex AI**. (e.g.` GCP_PROJECT_ID` = my-project-456789 and `GCP_REGION` = us-central1)
54+
- `REGION` is the zone where the services runs (for AWS this is equvilent of AWS_REGION)
55+
- `GCP_PROJECT_ID` is needed if using **Vertex AI**. (e.g.` GCP_PROJECT_ID` = my-project-456789 and `REGION` = us-central1)
5556

5657
:::tip
5758
**OpenAI Key**
@@ -106,6 +107,10 @@ You should configure your application to specify the model you want to use.
106107

107108
Choose the correct `MODEL` depending on which cloud provider you are using.
108109

110+
Alternatively, Defang supports model mapping through the openai-access-gateway. This takes a model with a Docker naming convention (e.g. ai/lama3.3) and maps it to
111+
the closest matching one on the target platform. If no such match can be found it can fallback onto a known existing model (e.g. ai/mistral). These environment
112+
variables are USE_MODEL_MAPPING (default to true) and FALLBACK_MODEL (no default), respectively.
113+
109114
:::info
110115
**Choosing the Right Model**
111116

@@ -138,7 +143,7 @@ services:
138143
environment:
139144
- OPENAI_API_KEY
140145
- GCP_PROJECT_ID # required if using Vertex AI
141-
- GCP_REGION # required if using Vertex AI
146+
- REGION
142147
```
143148
144149
---
@@ -148,7 +153,7 @@ services:
148153
| Variable | AWS Bedrock | GCP Vertex AI |
149154
|--------------------|-------------|---------------|
150155
| `GCP_PROJECT_ID` | _(not used)_| Required |
151-
| `GCP_REGION` | _(not used)_| Required |
156+
| `REGION` | Required| Required |
152157
| `MODEL` | Bedrock model ID | Vertex model path |
153158

154159
---
@@ -158,4 +163,5 @@ You now have a single app that can:
158163
- Talk to **AWS Bedrock** or **GCP Vertex AI**
159164
- Use the same OpenAI-compatible client code
160165
- Easily switch cloud providers by changing a few environment variables
166+
:::
161167

0 commit comments

Comments
 (0)