Merge pull request #230 from DefangLabs/linda-fix-llm-docs

Improvements to Managed LLM docs

Author: commit111

blog/2025-04-11-mar-product-updates.md

@@ -25,7 +25,7 @@ Wow - another month has gone by, time flies when you're having fun!
 
 Let us share some important updates regarding what we achieved at Defang in March:
 
-**Managed LLMs:** One of the coolest features we have released in a bit is [support for Managed LLMs (such as AWS Bedrock) through the `x-defang-llm` compose service extension](https://docs.defang.io/docs/concepts/managed-llms/managed-language-models). When coupled with the `defang/openai-access-gateway` service image, Defang offers the easiest way to [migrate your OpenAI-compatible application to cloud-native managed LLMs](https://docs.defang.io/docs/tutorials/deploying-openai-apps-aws-bedrock) without making any changes to your code. Support for GCP and DigitalOcean coming soon.
+**Managed LLMs:** One of the coolest features we have released in a bit is [support for Managed LLMs (such as AWS Bedrock) through the `x-defang-llm` compose service extension](https://docs.defang.io/docs/concepts/managed-llms/managed-language-models). When coupled with the `defang/openai-access-gateway` service image, Defang offers the easiest way to [migrate your OpenAI-compatible application to cloud-native managed LLMs](https://docs.defang.io/docs/tutorials/deploy-openai-apps) without making any changes to your code. Support for GCP and DigitalOcean coming soon.
 
 **Defang Pulumi Provider:** Last month, we announced a preview of the [Defang Pulumi Provider](https://github.com/DefangLabs/pulumi-defang), and this month we are excited to announce that V1 is now available in the [Pulumi Registry](https://www.pulumi.com/registry/packages/defang/). As much as we love Docker, we realize there are many real-world apps that have components that (currently) cannot be described completely in a Compose file. With the Defang Pulumi Provider, you can now leverage [the declarative simplicity of Defang with the imperative power of Pulumi](https://docs.defang.io/docs/concepts/pulumi#when-to-use-the-defang-pulumi-provider).
 

docs/concepts/managed-llms/managed-language-models.md

@@ -8,12 +8,27 @@ sidebar_position: 3000
 
 Each cloud provider offers their own managed Large Language Model services. AWS offers Bedrock, GCP offers Vertex AI, and Digital Ocean offers their GenAI platform. Defang makes it easy to leverage these services in your projects.
 
+## Current Support
+
+| Provider | Managed Language Models |
+| --- | --- |
+| [Playground](/docs/providers/playground#managed-large-language-models) | ✅ |
+| [AWS Bedrock](/docs/providers/aws#managed-large-language-models) | ✅ |
+| [DigitalOcean GenAI](/docs/providers/digitalocean#future-improvements) | ❌ |
+| [GCP Vertex AI](/docs/providers/gcp#managed-large-language-models) | ✅ |
+
 ## Usage
 
 In order to leverage cloud-native managed language models from your Defang services, all you need to do is add the `x-defang-llm` extension to the service config and Defang will configure the approprate roles and permissions for you.
 
 ## Example
 
+:::info
+Ensure you have enabled model access for the model you intend to use:
+* [Configure AWS Bedrock model access](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access-modify.html)
+* [Configure GCP Vertex AI model access](https://cloud.google.com/vertex-ai/generative-ai/docs/control-model-access)
+:::
+
 Assume you have a web service like the following, which uses the cloud native SDK, for example:
 
 ```diff
@@ -26,13 +41,8 @@ Assume you have a web service like the following, which uses the cloud native SD
 
 ## Deploying OpenAI-compatible apps
 
-If you already have an OpenAI-compatible application, Defang makes it easy to deploy on your favourite cloud's managed LLM service. See our [OpenAI Access Gateway](/docs/concepts/managed-llms/openai-access-gateway)
-
-## Current Support
+If you already have an OpenAI-compatible application, Defang makes it easy to deploy on your favourite cloud's managed LLM service. See our [OpenAI Access Gateway](/docs/concepts/managed-llms/openai-access-gateway).
 
-| Provider | Managed Language Models |
-| --- | --- |
-| [Playground](/docs/providers/playground#managed-large-language-models) | ✅ |
-| [AWS Bedrock](/docs/providers/aws#managed-large-language-models) | ✅ |
-| [DigitalOcean GenAI](/docs/providers/digitalocean#future-improvements) | ❌ |
-| [GCP Vertex AI](/docs/providers/gcp#managed-large-language-models) | ✅ |
+:::tip
+Defang has a [*Managed LLM sample*](https://github.com/DefangLabs/samples/tree/main/samples/managed-llm) that uses the OpenAI Access Gateway, and a [*Managed LLM with Docker Model Provider sample*](https://github.com/DefangLabs/samples/tree/main/samples/managed-llm-provider) that uses a Docker Model Provider.
+:::

docs/concepts/managed-llms/openai-access-gateway.md

@@ -9,9 +9,9 @@ sidebar_position: 3000
 Defang makes it easy to deploy on your favourite cloud's managed LLM service with our [OpenAI Access Gateway](https://github.com/DefangLabs/openai-access-gateway). This service sits between your application and the cloud service and acts as a compatibility layer.
 It handles incoming OpenAI requests, translates those requests to the appropriate cloud-native API, handles the native response, and re-constructs an OpenAI-compatible response.
 
-See [our tutorial](/docs/tutorials/deploying-openai-apps-aws-bedrock-gcp-vertex/) which describes how to configure the OpenAI Access Gateway for your application
+See [our tutorial](/docs/tutorials/deploy-openai-apps) which describes how to configure the OpenAI Access Gateway for your application.
 
-## Docker Provider Services
+## Docker Model Provider Services
 
 As of Docker Compose v2.35 and Docker Desktop v4.41, Compose introduces a new service type called `provider` that allows you to declare platform capabilities required by your application.
 For AI models, you use the `model` type to declare model dependencies. This will expose an OpenAI compatible API for your service. Check the [Docker Model Runner documentation](https://docs.docker.com/compose/how-tos/model-runner/) for more details.
@@ -32,8 +32,17 @@ services:
 ```
 
 Under the hood, when you use the `model` provider, Defang will deploy the **OpenAI Access Gateway** in a private network. This allows you to use the same code for both local development and cloud deployment.
+
 The `x-defang-llm` extension is used to configure the appropriate roles and permissions for your service. See the [Managed Language Models](/docs/concepts/managed-llms/managed-language-models/) page for more details.
 
+## Model Mapping
+
+Defang supports model mapping through the [openai-access-gateway](https://github.com/DefangLabs/openai-access-gateway) on AWS and GCP. This takes a model with a Docker naming convention (e.g. `ai/llama3.3`) and maps it to the closest matching model name on the target platform. If no such match can be found, it can fallback onto a known existing model (e.g. `ai/mistral`).
+
+This can be configured through the following environment variables:
+* `USE_MODEL_MAPPING` (default to true) - configures whether or not model mapping should be enabled.
+* `FALLBACK_MODEL` (no default) - configure a model which will be used if model mapping fails to find a target model.
+
 ## Current Support
 
 | Provider | Managed Language Models |

docs/tutorials/deploy-openai-apps/_category_.json

@@ -0,0 +1,5 @@
+{
+  "label": "Deploy OpenAI Apps on Managed LLMs",
+  "position": 45,
+  "collapsible": true
+}

docs/tutorials/deploying-openai-apps-aws-bedrock-gcp-vertex.mdx renamed to docs/tutorials/deploy-openai-apps/aws-bedrock.mdx

@@ -1,15 +1,21 @@
 ---
-title: Deploying your OpenAI Application to AWS Bedrock or GCP Vertex AI
+title: AWS Bedrock
 sidebar_position: 50
+description: Deploy OpenAI Apps to AWS Bedrock using Defang.
 ---
+import React from 'react';
+import {useColorMode} from '@docusaurus/theme-common';
 
-# Deploying your OpenAI Application to AWS Bedrock or GCP Vertex AI
+# Deploy OpenAI Apps to AWS Bedrock
 
-Let's assume you have an app that uses an OpenAI client library and you want to deploy it to the cloud, either on **AWS Bedrock** or **GCP Vertex AI**.  
+Let's assume you have an app that uses an OpenAI client library and you want to deploy it to the cloud on **AWS Bedrock**.  
 
 This tutorial shows you how **Defang** makes it easy.
+:::info
+You must [configure AWS Bedrock model access](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access-modify.html) for each model you intend to use in your AWS account.
+:::
 
-Suppose you start with a compose file like this:
+Suppose you start with a `compose.yaml` file with one `app` service, like this:
 
 ```yaml
 services:
@@ -28,9 +34,7 @@ services:
 
 ## Add an LLM Service to Your Compose File
 
-You need to add a new service that acts as a proxy between your app and the backend LLM provider (Bedrock or Vertex).
-
-Add **Defang's [openai-access-gateway](https://github.com/DefangLabs/openai-access-gateway)** service:
+You can use AWS Bedrock without changing your `app` code by introducing a new [`defangio/openai-access-gateway`](https://github.com/DefangLabs/openai-access-gateway) service. We'll call the new service `llm`. This new service will act as a proxy between your application and AWS Bedrock, and will transparently handle converting your OpenAI requests into AWS Bedrock requests and Bedrock responses into OpenAI responses. This allows you to use AWS Bedrock with your existing OpenAI client SDK.
 
 ```diff
 +  llm:
@@ -42,16 +46,15 @@ Add **Defang's [openai-access-gateway](https://github.com/DefangLabs/openai-acce
 +        mode: host
 +    environment:
 +      - OPENAI_API_KEY
-+      - GCP_PROJECT_ID # if using GCP Vertex AI
-+      - GCP_REGION # if using GCP Vertex AI, AWS_REGION not necessary for Bedrock
++      - REGION
 ```
 
 ### Notes:
 
 - The container image is based on [aws-samples/bedrock-access-gateway](https://github.com/aws-samples/bedrock-access-gateway), with enhancements.
 - `x-defang-llm: true` signals to **Defang** that this service should be configured to use target platform AI services.
 - New environment variables:
-  - `GCP_PROJECT_ID` and `GCP_REGION` are needed if using **Vertex AI**. (e.g.` GCP_PROJECT_ID` = my-project-456789 and `GCP_REGION` = us-central1)
+  - `REGION` is the zone where the services runs (for AWS, this is the equivalent of AWS_REGION)
 
 :::tip
 **OpenAI Key**
@@ -82,7 +85,7 @@ Modify your `app` service to send API calls to the `openai-access-gateway`:
        test: ["CMD", "curl", "-f", "http://localhost:3000/"]
 ```
 
-Now, all OpenAI traffic will be routed through your gateway service and onto AWS Bedrock or GCP Vertex.
+Now, all OpenAI traffic will be routed through your gateway service and onto AWS Bedrock.
 
 ---
 
@@ -98,8 +101,7 @@ You should configure your application to specify the model you want to use.
      environment:
        OPENAI_API_KEY:
        OPENAI_BASE_URL: "http://llm/api/v1"
-+      MODEL: "anthropic.claude-3-sonnet-20240229-v1:0" # for Bedrock
-+      # MODEL: "google/gemini-2.5-pro-preview-03-25" # for Vertex AI
++      MODEL: "anthropic.claude-3-sonnet-20240229-v1:0"
      healthcheck:
        test: ["CMD", "curl", "-f", "http://localhost:3000/"]
 ```
@@ -109,10 +111,15 @@ Choose the correct `MODEL` depending on which cloud provider you are using.
 :::info
 **Choosing the Right Model**
 
-- For **AWS Bedrock**, use a Bedrock model ID (e.g., `anthropic.claude-3-sonnet-20240229-v1:0`) [See available Bedrock models](https://docs.aws.amazon.com/bedrock/latest/userguide/models-supported.html).
-- For **GCP Vertex AI**, use a full model path (e.g., `google/gemini-2.5-pro-preview-03-25`) [See available Vertex models](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/call-vertex-using-openai-library#client-setup)
+- For **AWS Bedrock**, use a Bedrock model ID (e.g., `anthropic.claude-3-sonnet-20240229-v1:0`). [See available Bedrock models](https://docs.aws.amazon.com/bedrock/latest/userguide/models-supported.html).
+:::
 
-# Complete Example Compose File
+Alternatively, Defang supports [model mapping](/docs/concepts/managed-llms/openai-access-gateway/#model-mapping) through the [openai-access-gateway](https://github.com/DefangLabs/openai-access-gateway). This takes a model with a Docker naming convention (e.g. `ai/llama3.3`) and maps it to
+the closest equivalent on the target platform. If no such match can be found, a fallback can be defined to use a known existing model (e.g. `ai/mistral`). These environment
+variables are `USE_MODEL_MAPPING` (default to true) and `FALLBACK_MODEL` (no default), respectively.
+
+
+## Complete Example Compose File
 
 ```yaml
 services:
@@ -124,7 +131,7 @@ services:
     environment:
       OPENAI_API_KEY:
       OPENAI_BASE_URL: "http://llm/api/v1"
-      MODEL: "anthropic.claude-3-sonnet-20240229-v1:0" # or your Vertex AI model path
+      MODEL: "anthropic.claude-3-sonnet-20240229-v1:0"
     healthcheck:
       test: ["CMD", "curl", "-f", "http://localhost:3000/"]
 
@@ -137,25 +144,22 @@ services:
         mode: host
     environment:
       - OPENAI_API_KEY
-      - GCP_PROJECT_ID     # required if using Vertex AI
-      - GCP_REGION         # required if using Vertex AI
+      - REGION
 ```
 
 ---
 
-# Environment Variable Matrix
+## Environment Variable Matrix
 
-| Variable           | AWS Bedrock | GCP Vertex AI |
-|--------------------|-------------|---------------|
-| `GCP_PROJECT_ID`    | _(not used)_| Required      |
-| `GCP_REGION`        | _(not used)_| Required      |
-| `MODEL`             | Bedrock model ID | Vertex model path |
+| Variable           | AWS Bedrock |
+|--------------------|-------------|
+| `REGION`            | Required|
+| `MODEL`             | Bedrock model ID or Docker model name, for example `meta.llama3-3-70b-instruct-v1:0` or `ai/llama3.3` |
 
 ---
 
 You now have a single app that can:
 
-- Talk to **AWS Bedrock** or **GCP Vertex AI**
+- Talk to **AWS Bedrock**
 - Use the same OpenAI-compatible client code
-- Easily switch cloud providers by changing a few environment variables
-
+- Easily switch between models or cloud providers by changing a few environment variables

docs/tutorials/deploy-openai-apps/deploy-openai-apps.mdx

@@ -0,0 +1,12 @@
+---
+title: Deploy OpenAI Apps on Managed LLMs
+sidebar_position: 45
+description: Deploy OpenAI Apps on Managed LLMs
+---
+
+# Deploy OpenAI Apps on Managed LLMs
+
+Defang currently supports using Managed LLMs on AWS Bedrock and GCP Vertex AI. Follow the link below for your specific platform.
+
+- [AWS Bedrock](/docs/tutorials/deploy-openai-apps/aws-bedrock/)
+- [GCP Vertex AI](/docs/tutorials/deploy-openai-apps/gcp-vertex/)