|
| 1 | +--- |
| 2 | +title: Deploying your OpenAI Application to AWS Bedrock or GCP Vertex AI |
| 3 | +sidebar_position: 50 |
| 4 | +--- |
| 5 | + |
| 6 | +# Deploying your OpenAI Application to AWS Bedrock or GCP Vertex AI |
| 7 | + |
| 8 | +Let's assume you have an app that uses an OpenAI client library and you want to deploy it to the cloud, either on **AWS Bedrock** or **GCP Vertex AI**. |
| 9 | + |
| 10 | +This tutorial shows you how **Defang** makes it easy. |
| 11 | + |
| 12 | +Suppose you start with a compose file like this: |
| 13 | + |
| 14 | +```yaml |
| 15 | +services: |
| 16 | + app: |
| 17 | + build: |
| 18 | + context: . |
| 19 | + ports: |
| 20 | + - 3000:3000 |
| 21 | + environment: |
| 22 | + OPENAI_API_KEY: |
| 23 | + healthcheck: |
| 24 | + test: ["CMD", "curl", "-f", "http://localhost:3000/"] |
| 25 | +``` |
| 26 | +
|
| 27 | +--- |
| 28 | +
|
| 29 | +## Add an LLM Service to Your Compose File |
| 30 | +
|
| 31 | +You need to add a new service that acts as a proxy between your app and the backend LLM provider (Bedrock or Vertex). |
| 32 | +
|
| 33 | +Add **Defang's [openai-access-gateway](https://github.com/DefangLabs/openai-access-gateway)** service: |
| 34 | +
|
| 35 | +```diff |
| 36 | ++ llm: |
| 37 | ++ image: defangio/openai-access-gateway |
| 38 | ++ x-defang-llm: true |
| 39 | ++ ports: |
| 40 | ++ - target: 80 |
| 41 | ++ published: 80 |
| 42 | ++ mode: host |
| 43 | ++ environment: |
| 44 | ++ - OPENAI_API_KEY |
| 45 | ++ - GCP_PROJECT_ID # if using GCP Vertex AI |
| 46 | ++ - GCP_REGION # if using GCP Vertex AI, AWS_REGION not necessary for Bedrock |
| 47 | +``` |
| 48 | + |
| 49 | +### Notes: |
| 50 | + |
| 51 | +- The container image is based on [aws-samples/bedrock-access-gateway](https://github.com/aws-samples/bedrock-access-gateway), with enhancements. |
| 52 | +- `x-defang-llm: true` signals to **Defang** that this service should be configured to use target platform AI services. |
| 53 | +- New environment variables: |
| 54 | + - `GCP_PROJECT_ID` and `GCP_REGION` are needed if using **Vertex AI**. (e.g.` GCP_PROJECT_ID` = my-project-456789 and `GCP_REGION` = us-central1) |
| 55 | + |
| 56 | +:::tip |
| 57 | +**OpenAI Key** |
| 58 | + |
| 59 | +You no longer need your original OpenAI API Key. |
| 60 | +We recommend generating a random secret for authentication with the gateway: |
| 61 | + |
| 62 | +```bash |
| 63 | +defang config set OPENAI_API_KEY --random |
| 64 | +``` |
| 65 | +::: |
| 66 | + |
| 67 | +--- |
| 68 | + |
| 69 | +## Redirect Application Traffic |
| 70 | + |
| 71 | +Modify your `app` service to send API calls to the `openai-access-gateway`: |
| 72 | + |
| 73 | +```diff |
| 74 | + services: |
| 75 | + app: |
| 76 | + ports: |
| 77 | + - 3000:3000 |
| 78 | + environment: |
| 79 | + OPENAI_API_KEY: |
| 80 | ++ OPENAI_BASE_URL: "http://llm/api/v1" |
| 81 | + healthcheck: |
| 82 | + test: ["CMD", "curl", "-f", "http://localhost:3000/"] |
| 83 | +``` |
| 84 | + |
| 85 | +Now, all OpenAI traffic will be routed through your gateway service and onto AWS Bedrock or GCP Vertex. |
| 86 | + |
| 87 | +--- |
| 88 | + |
| 89 | +## Selecting a Model |
| 90 | + |
| 91 | +You should configure your application to specify the model you want to use. |
| 92 | + |
| 93 | +```diff |
| 94 | + services: |
| 95 | + app: |
| 96 | + ports: |
| 97 | + - 3000:3000 |
| 98 | + environment: |
| 99 | + OPENAI_API_KEY: |
| 100 | + OPENAI_BASE_URL: "http://llm/api/v1" |
| 101 | ++ MODEL: "anthropic.claude-3-sonnet-20240229-v1:0" # for Bedrock |
| 102 | ++ # MODEL: "google/gemini-2.5-pro-preview-03-25" # for Vertex AI |
| 103 | + healthcheck: |
| 104 | + test: ["CMD", "curl", "-f", "http://localhost:3000/"] |
| 105 | +``` |
| 106 | + |
| 107 | +Choose the correct `MODEL` depending on which cloud provider you are using. |
| 108 | + |
| 109 | +:::info |
| 110 | +**Choosing the Right Model** |
| 111 | + |
| 112 | +- For **AWS Bedrock**, use a Bedrock model ID (e.g., `anthropic.claude-3-sonnet-20240229-v1:0`) [See available Bedrock models](https://docs.aws.amazon.com/bedrock/latest/userguide/models-supported.html). |
| 113 | +- For **GCP Vertex AI**, use a full model path (e.g., `google/gemini-2.5-pro-preview-03-25`) [See available Vertex models](https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/call-vertex-using-openai-library#client-setup) |
| 114 | + |
| 115 | +# Complete Example Compose File |
| 116 | + |
| 117 | +```yaml |
| 118 | +services: |
| 119 | + app: |
| 120 | + build: |
| 121 | + context: . |
| 122 | + ports: |
| 123 | + - 3000:3000 |
| 124 | + environment: |
| 125 | + OPENAI_API_KEY: |
| 126 | + OPENAI_BASE_URL: "http://llm/api/v1" |
| 127 | + MODEL: "anthropic.claude-3-sonnet-20240229-v1:0" # or your Vertex AI model path |
| 128 | + healthcheck: |
| 129 | + test: ["CMD", "curl", "-f", "http://localhost:3000/"] |
| 130 | + |
| 131 | + llm: |
| 132 | + image: defangio/openai-access-gateway |
| 133 | + x-defang-llm: true |
| 134 | + ports: |
| 135 | + - target: 80 |
| 136 | + published: 80 |
| 137 | + mode: host |
| 138 | + environment: |
| 139 | + - OPENAI_API_KEY |
| 140 | + - GCP_PROJECT_ID # required if using Vertex AI |
| 141 | + - GCP_REGION # required if using Vertex AI |
| 142 | +``` |
| 143 | +
|
| 144 | +--- |
| 145 | +
|
| 146 | +# Environment Variable Matrix |
| 147 | +
|
| 148 | +| Variable | AWS Bedrock | GCP Vertex AI | |
| 149 | +|--------------------|-------------|---------------| |
| 150 | +| `GCP_PROJECT_ID` | _(not used)_| Required | |
| 151 | +| `GCP_REGION` | _(not used)_| Required | |
| 152 | +| `MODEL` | Bedrock model ID | Vertex model path | |
| 153 | + |
| 154 | +--- |
| 155 | + |
| 156 | +You now have a single app that can: |
| 157 | + |
| 158 | +- Talk to **AWS Bedrock** or **GCP Vertex AI** |
| 159 | +- Use the same OpenAI-compatible client code |
| 160 | +- Easily switch cloud providers by changing a few environment variables |
| 161 | + |
0 commit comments