rough draft documenting x-defang-llm

Author: jordanstephens

docs/tutorials/deploying-openai-apps-aws-bedrock.mdx

@@ -0,0 +1,95 @@
+---
+title: Deploying your OpenAI application to AWS and using Bedrock
+sidebar_position: 50
+---
+
+# Deploying your OpenAI application to AWS and using Bedrock
+
+Let's assume you have an app which is using one of the OpenAI client libraries and you want to deploy your app to AWS so you can leverage Bedrock. This tutorial will show you how Defang makes it easy.
+
+Assume you have a compose file like this:
+
+```yaml
+services:
+  app:
+    build:
+        context: .
+    ports:
+      - target: 3000
+        published: 3000
+        protocol: tcp
+        mode: ingress
+    environment:
+      OPENAI_API_KEY:
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://localhost:3000/"]
+```
+
+## Add an llm service to your compose file
+
+The first step is to add a new service to your compose file. The `defangio/openai-access-gateway`. This service provides an OpenAI compatible interface to AWS Bedrock. It's easy to configure, first you need to add it to your compose file:
+
+```diff
++  llm:
++    image: defangio/openai-access-gateway
++    x-defang-llm: true
++    ports:
++      - target: 80
++        published: 80
++        protocol: tcp
++        mode: host
++    environment:
++      - OPENAI_API_KEY
++    healthcheck:
++      test: ["CMD", "curl", "-f", "http://localhost/health"]
+```
+
+A few things to note here. First the image is a fork of [aws-samples/bedrock-access-gateway](https://github.com/aws-samples/bedrock-access-gateway), which a few modifications to make it easier to use. The source code is available [here](https://github.com/DefangLabs/openai-access-gateway). Second: the `x-defang-llm` property. Defang uses extensions like this to signal special handling of certain kinds of services. In this case, it signals to Defang that we need to configure the appropriate IAM Roles and Policies to support your application.
+
+## Redirecting application traffic
+
+Then you need to configure your application to redirect traffic to the openai-gateway, like this:
+
+```diff
+ services:
+   app:
+     ports:
+       - target: 3000
+         published: 3000
+         protocol: tcp
+         mode: ingress
+     environment:
+       OPENAI_API_KEY:
++      OPENAI_BASE_URL: "http://llm/api/v1"
++      MODEL: "anthropic.claude-3-sonnet-20240229-v1:0"
+     healthcheck:
+       test: ["CMD", "curl", "-f", "http://localhost:3000/"]
+```
+
+You will also need to configure your application to use one of the bedrock models. We recommend configuring an environment variable called `MODEL` like this:
+
+## Selecting a model
+
+```diff
+ services:
+   app:
+     ports:
+       - target: 3000
+         published: 3000
+         protocol: tcp
+         mode: ingress
+     environment:
+       OPENAI_API_KEY:
+       OPENAI_BASE_URL: "http://llm/api/v1"
++      MODEL: "anthropic.claude-3-sonnet-20240229-v1:0"
+     healthcheck:
+       test: ["CMD", "curl", "-f", "http://localhost:3000/"]
+```
+
+## Enabling bedrock model access
+
+AWS currently requires access to be manually configured on a per-model basis in each account. See this guide for [how to enable model access](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access-modify.html).
+
+## Your OpenAI key
+
+It's worth noting that you no longer need ot use your original OpenAI API key. We do recommend using _something_ in its place, but feel free to generate a new secret and set it with `defang config set OPENAI_API_KEY`.