revisions

Author: jordanstephens

docs/concepts/managed-llms/managed-language-models.md

@@ -26,7 +26,7 @@ Assume you have a web service like the following, which uses the cloud native SD
 
 ## Deploying OpenAI-compatible apps
 
-If you already have an OpenAI-compatible application, Defang makes it easy to deploy on your favourite cloud's managed LLM service. See our [OpenAI Access Gateway](/docs/concepts/openai-access-gateway.md)
+If you already have an OpenAI-compatible application, Defang makes it easy to deploy on your favourite cloud's managed LLM service. See our [OpenAI Access Gateway](/docs/concepts/managed-llms/openai-access-gateway)
 
 ## Current Support
 

docs/concepts/managed-llms/openai-gateway.md renamed to docs/concepts/managed-llms/openai-access-gateway.md

@@ -8,7 +8,7 @@ sidebar_position: 3000
 
 Defang makes it easy to deploy on your favourite cloud's managed LLM service with our [OpenAI Access Gateway](https://github.com/DefangLabs/openai-access-gateway). This service sits between your application and the cloud service and acts as a compatibility layer. It handles incoming OpenAI requests, translates those requests to the appropriate cloud-native API, handles the native response, and re-constructs an OpenAI-compatible response.
 
-See [our tutorial](/docs/tutorials/deploying-openai-apps-aws-bedrock.mdx/) which describes how to configure the OpenAI Access Gateway for your application
+See [our tutorial](/docs/tutorials/deploying-openai-apps-aws-bedrock/) which describes how to configure the OpenAI Access Gateway for your application
 
 ## Current Support
 

docs/providers/aws/aws.md

@@ -72,7 +72,7 @@ When using [Managed Postgres](/docs/concepts/managed-storage/managed-postgres.md
 
 When using [Managed Redis](/docs/concepts/managed-storage/managed-redis.md), the Defang CLI provisions an ElastiCache Redis cluster in your account.
 
-### Managed large language models
+### Managed LLMs
 
 Defang offers integration with managed, cloud-native large language model services with the `x-defang-llm` service extension. Add this extension to any services which use the Bedrock SDKs.
 

docs/providers/gcp.md

@@ -59,7 +59,7 @@ The Provider builds and deploys your services using [Google Cloud Run](https://c
 
 The GCP provider does not currently support storing sensitive config values.
 
-### Managed large language models
+### Managed LLMs
 
 Defang offers integration with managed, cloud-native large language model services with the `x-defang-llm` service extension. Add this extension to any services which use the Bedrock SDKs.
 

docs/providers/playground.md

@@ -20,6 +20,6 @@ Overall, the Defang Playground is very similar to deploying to your own cloud ac
 
 In essence, the Playground does not support any [managed storage](../concepts/managed-storage) services, ie. `x-defang-postgres` and `x-defang-redis` are ignored when deploying to the Playground. You can however run both Postgres and Redis as regular container services for testing purposes.
 
-### Managed large language models
+### Managed LLMs
 
 Defang offers integration with managed, cloud-native large language model services with the `x-defang-llm` service extension when deploying to your own cloud account with BYOC. This extension is not supported in the Defang Playground.

docs/tutorials/deploying-openai-apps-aws-bedrock.mdx

@@ -1,9 +1,9 @@
 ---
-title: Deploying your OpenAI application to AWS and using Bedrock
+title: Deploying your OpenAI application to AWS using Bedrock
 sidebar_position: 50
 ---
 
-# Deploying your OpenAI application to AWS and using Bedrock
+# Deploying your OpenAI application to AWS using Bedrock
 
 Let's assume you have an app which is using one of the OpenAI client libraries and you want to deploy your app to AWS so you can leverage Bedrock. This tutorial will show you how Defang makes it easy.
 
@@ -15,19 +15,16 @@ services:
     build:
         context: .
     ports:
-      - target: 3000
-        published: 3000
-        protocol: tcp
-        mode: ingress
+      - 3000:3000
     environment:
       OPENAI_API_KEY:
     healthcheck:
       test: ["CMD", "curl", "-f", "http://localhost:3000/"]
 ```
 
-## Add an llm service to your compose file
+## Add an LLM service to your compose file
 
-The first step is to add a new service to your compose file. The `defangio/openai-access-gateway`. This service provides an OpenAI compatible interface to AWS Bedrock. It's easy to configure, first you need to add it to your compose file:
+The first step is to add a new service to your compose file: the `defangio/openai-access-gateway`. This service provides an OpenAI compatible interface to AWS Bedrock. It's easy to configure, first you need to add it to your compose file:
 
 ```diff
 +  llm:
@@ -36,15 +33,21 @@ The first step is to add a new service to your compose file. The `defangio/opena
 +    ports:
 +      - target: 80
 +        published: 80
-+        protocol: tcp
 +        mode: host
 +    environment:
 +      - OPENAI_API_KEY
-+    healthcheck:
-+      test: ["CMD", "curl", "-f", "http://localhost/health"]
 ```
 
-A few things to note here. First the image is a fork of [aws-samples/bedrock-access-gateway](https://github.com/aws-samples/bedrock-access-gateway), which a few modifications to make it easier to use. The source code is available [here](https://github.com/DefangLabs/openai-access-gateway). Second: the `x-defang-llm` property. Defang uses extensions like this to signal special handling of certain kinds of services. In this case, it signals to Defang that we need to configure the appropriate IAM Roles and Policies to support your application.
+A few things to note here. First the image is a fork of [aws-samples/bedrock-access-gateway](https://github.com/aws-samples/bedrock-access-gateway), with a few modifications to make it easier to use. The source code is available [here](https://github.com/DefangLabs/openai-access-gateway). Second: the `x-defang-llm` property. Defang uses extensions like this to signal special handling of certain kinds of services. In this case, it signals to Defang that we need to configure the appropriate IAM Roles and Policies to support your application.
+
+:::warning
+**Your OpenAI key**
+
+You no longer need to use your original OpenAI API key. We do recommend using _something_ in its place, but feel free to generate a new secret and set it with `defang config set OPENAI_API_KEY --random`.
+
+This is used to authenticate your application service with the openai-access-gateway.
+:::
+
 
 ## Redirecting application traffic
 
@@ -54,30 +57,23 @@ Then you need to configure your application to redirect traffic to the openai-ac
  services:
    app:
      ports:
-       - target: 3000
-         published: 3000
-         protocol: tcp
-         mode: ingress
+       - 3000:3000
      environment:
        OPENAI_API_KEY:
 +      OPENAI_BASE_URL: "http://llm/api/v1"
-+      MODEL: "anthropic.claude-3-sonnet-20240229-v1:0"
      healthcheck:
        test: ["CMD", "curl", "-f", "http://localhost:3000/"]
 ```
 
-You will also need to configure your application to use one of the bedrock models. We recommend configuring an environment variable called `MODEL` like this:
-
 ## Selecting a model
 
+You will also need to configure your application to use one of the bedrock models. We recommend configuring an environment variable called `MODEL` like this:
+
 ```diff
  services:
    app:
      ports:
-       - target: 3000
-         published: 3000
-         protocol: tcp
-         mode: ingress
+       - 3000:3000
      environment:
        OPENAI_API_KEY:
        OPENAI_BASE_URL: "http://llm/api/v1"
@@ -86,13 +82,11 @@ You will also need to configure your application to use one of the bedrock model
        test: ["CMD", "curl", "-f", "http://localhost:3000/"]
 ```
 
-## Enabling bedrock model access
+:::warning
+**Enabling bedrock model access**
 
 AWS currently requires access to be manually configured on a per-model basis in each account. See this guide for [how to enable model access](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access-modify.html).
-
-## Your OpenAI key
-
-It's worth noting that you no longer need ot use your original OpenAI API key. We do recommend using _something_ in its place, but feel free to generate a new secret and set it with `defang config set OPENAI_API_KEY`.
+:::
 
 ## Complete Example Compose File
 
@@ -102,14 +96,11 @@ services:
     build:
         context: .
     ports:
-      - target: 3000
-        published: 3000
-        protocol: tcp
-        mode: ingress
+      - 3000:3000
     environment:
       OPENAI_API_KEY:
       OPENAI_BASE_URL: "http://llm/api/v1"
-      MODEL: "anthropic.claude-3-sonnet-20240229-v1:0"
+      MODEL: "us:anthropic.claude-3-sonnet-20240229-v1:0"
     healthcheck:
       test: ["CMD", "curl", "-f", "http://localhost:3000/"]
   llm:
@@ -118,11 +109,8 @@ services:
     ports:
       - target: 80
         published: 80
-        protocol: tcp
         mode: host
     environment:
       - OPENAI_API_KEY
-    healthcheck:
-      test: ["CMD", "curl", "-f", "http://localhost/health"]
 
 ```