Skip to content

Commit 8607f87

Browse files
commit111commit111
committed
add managed llm sample
1 parent 4469657 commit 8607f87

File tree

13 files changed

+325
-0
lines changed

13 files changed

+325
-0
lines changed

‎.github/workflows/deploy-changed-samples.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -82,6 +82,7 @@ jobs:
8282
TEST_MB_DB_PASS: ${{ secrets.TEST_MB_DB_PASS }}
8383
TEST_MB_DB_PORT: ${{ secrets.TEST_MB_DB_PORT }}
8484
TEST_MB_DB_USER: ${{ secrets.TEST_MB_DB_USER }}
85+
TEST_MODEL: ${{ secrets.TEST_MODEL }}
8586
TEST_MONGO_INITDB_ROOT_USERNAME: ${{ secrets.TEST_MONGO_INITDB_ROOT_USERNAME }}
8687
TEST_MONGO_INITDB_ROOT_PASSWORD: ${{ secrets.TEST_MONGO_INITDB_ROOT_PASSWORD }}
8788
TEST_NC_DB: ${{ secrets.TEST_NC_DB }}

‎samples/managed-llm/.devcontainer/Dockerfile‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
2+
FROM mcr.microsoft.com/devcontainers/python:alpine3.13

‎samples/managed-llm/.devcontainer/devcontainer.json‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
{
2+
"build": {
3+
"dockerfile": "Dockerfile",
4+
"context": ".."
5+
},
6+
"features": {
7+
"ghcr.io/defanglabs/devcontainer-feature/defang-cli:1.0.4": {},
8+
"ghcr.io/devcontainers/features/docker-in-docker:2": {},
9+
"ghcr.io/devcontainers/features/aws-cli:1": {}
10+
}
11+
}

‎samples/managed-llm/.dockerignore‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
# Default .dockerignore file for Defang
2+
**/__pycache__
3+
**/.direnv
4+
**/.DS_Store
5+
**/.envrc
6+
**/.git
7+
**/.github
8+
**/.idea
9+
**/.next
10+
**/.vscode
11+
**/compose.*.yaml
12+
**/compose.*.yml
13+
**/compose.yaml
14+
**/compose.yml
15+
**/docker-compose.*.yaml
16+
**/docker-compose.*.yml
17+
**/docker-compose.yaml
18+
**/docker-compose.yml
19+
**/node_modules
20+
**/Thumbs.db
21+
Dockerfile
22+
*.Dockerfile
23+
# Ignore our own binary, but only in the root to avoid ignoring subfolders
24+
defang
25+
defang.exe
26+
# Ignore our project-level state
27+
.defang

‎samples/managed-llm/.github/workflows/deploy.yaml‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
name: Deploy
2+
3+
on:
4+
push:
5+
branches:
6+
- main
7+
8+
jobs:
9+
deploy:
10+
environment: playground
11+
runs-on: ubuntu-latest
12+
permissions:
13+
contents: read
14+
id-token: write
15+
16+
steps:
17+
- name: Checkout Repo
18+
uses: actions/checkout@v4
19+
20+
- name: Deploy
21+
uses: DefangLabs/[email protected]
22+
with:
23+
config-env-vars: MODEL
24+
env:
25+
MODEL: ${{ secrets.MODEL }}

‎samples/managed-llm/.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
.env
2+
myenv
3+
__pycache__/

‎samples/managed-llm/README.md‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
# Managed LLM
2+
3+
[![1-click-deploy](https://raw.githubusercontent.com/DefangLabs/defang-assets/main/Logos/Buttons/SVG/deploy-with-defang.svg)](https://portal.defang.dev/redirect?url=https%3A%2F%2Fgithub.com%2Fnew%3Ftemplate_name%3Dsample-managed-llm-template%26template_owner%3DDefangSamples)
4+
5+
This sample application demonstrates the use of Managed LLMs (Large Language Models) with Defang.
6+
7+
The Managed LLM feature, provided by the Defang OpenAI Access Gateway, enables users to switch LLM models in production without altering the app's API interface.
8+
9+
You can configure the `MODEL` and `ENDPOINT_URL` for the LLM separately for local development and production environments. The `MODEL` is the LLM Model ID you are using. The `ENDPOINT_URL` is the bridge that provides authenticated access to the LLM model.
10+
11+
### Defang OpenAI Access Gateway
12+
13+
In the `compose.yaml` file, the `llm-gateway` service is used to route requests to the LLM API model. This is known as the Defang OpenAI Access Gateway.
14+
15+
The `x-defang-llm` property must be set to `true` to enable the feature during deployment with Defang.
16+
17+
## Prerequisites
18+
19+
1. Download [Defang CLI](https://github.com/DefangLabs/defang)
20+
2. (Optional) If you are using [Defang BYOC](https://docs.defang.io/docs/concepts/defang-byoc) authenticate with your cloud provider account
21+
3. (Optional for local development) [Docker CLI](https://docs.docker.com/engine/install/)
22+
23+
## Development
24+
25+
To run the application locally, you can use the following command:
26+
27+
```bash
28+
docker compose -f compose.dev.yaml up --build
29+
```
30+
31+
## Configuration
32+
33+
For this sample, you will need to provide the following [configuration](https://docs.defang.io/docs/concepts/configuration):
34+
35+
> Note that if you are using the 1-click deploy option, you can set these values as secrets in your GitHub repository and the action will automatically deploy them for you.
36+
37+
### `MODEL`
38+
The Model ID of the LLM you are using for your application. For example, `anthropic.claude-3-5-haiku-20241022-v1:0`.
39+
```bash
40+
defang config set MODEL
41+
```
42+
43+
## Deployment
44+
45+
> [!NOTE]
46+
> Download [Defang CLI](https://github.com/DefangLabs/defang)
47+
48+
### Defang Playground
49+
50+
Deploy your application to the Defang Playground by opening up your terminal and typing:
51+
```bash
52+
defang compose up
53+
```
54+
55+
### BYOC
56+
57+
If you want to deploy to your own cloud account, you can [use Defang BYOC](https://docs.defang.io/docs/tutorials/deploy-to-your-cloud).
58+
59+
---
60+
61+
Title: Managed LLM
62+
63+
Short Description: An app using Managed LLMs with Defang's OpenAI Access Gateway.
64+
65+
Tags: LLM, Managed LLM, FastAPI, OpenAI, Python
66+
67+
Languages: Python

‎samples/managed-llm/app/.dockerignore‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
# Default .dockerignore file for Defang
2+
**/__pycache__
3+
**/.direnv
4+
**/.DS_Store
5+
**/.envrc
6+
**/.git
7+
**/.github
8+
**/.idea
9+
**/.next
10+
**/.vscode
11+
**/compose.*.yaml
12+
**/compose.*.yml
13+
**/compose.yaml
14+
**/compose.yml
15+
**/docker-compose.*.yaml
16+
**/docker-compose.*.yml
17+
**/docker-compose.yaml
18+
**/docker-compose.yml
19+
**/node_modules
20+
**/Thumbs.db
21+
Dockerfile
22+
*.Dockerfile
23+
# Ignore our own binary, but only in the root to avoid ignoring subfolders
24+
defang
25+
defang.exe
26+
# Ignore our project-level state
27+
.defang

‎samples/managed-llm/app/Dockerfile‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
FROM python:alpine
2+
3+
# Set working directory
4+
WORKDIR /app
5+
6+
# Copy requirement files first (for better Docker cache)
7+
COPY requirements.txt .
8+
9+
# Install dependencies
10+
RUN pip install --no-cache-dir -r requirements.txt
11+
12+
# Copy the rest of the code
13+
COPY . .
14+
15+
# Expose the port that Uvicorn will run on
16+
EXPOSE 8000
17+
18+
# Run the app with the correct module path using Uvicorn
19+
CMD ["sh", "-c", "uvicorn app:app --host 0.0.0.0 --port 8000"]

‎samples/managed-llm/app/app.py‎

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
import os
2+
import json
3+
import logging
4+
from fastapi import FastAPI, Form, Request
5+
from fastapi.responses import HTMLResponse
6+
import requests
7+
import dotenv
8+
9+
app = FastAPI()
10+
11+
# Load environment variables from .env file
12+
dotenv.load_dotenv()
13+
14+
# Configure basic logging
15+
logging.basicConfig(level=logging.INFO)
16+
17+
# Set the endpoint URL for the chat model
18+
# Here, we use the OpenAI API as an example:
19+
ENDPOINT_URL = os.getenv("ENDPOINT_URL", "https://api.openai.com/v1")
20+
21+
# Get the API key for the LLM
22+
# For development, you can use your local API key. In production, you will need to configure your API key in the LLM gateway service.
23+
def get_api_key():
24+
return os.getenv("OPENAI_API_KEY", "API key not set")
25+
26+
# Home page form
27+
@app.get("/", response_class=HTMLResponse)
28+
async def home():
29+
return """
30+
<html>
31+
<head><title>Ask the Model</title></head>
32+
<body>
33+
<h1>Ask the Magic Backpack 🧙‍♂️🎒</h1>
34+
<form method="post" action="/ask">
35+
<textarea name="prompt" rows="5" cols="60" placeholder="Enter your question here..."></textarea><br><br>
36+
<input type="submit" value="Ask">
37+
</form>
38+
</body>
39+
</html>
40+
"""
41+
42+
# Handle form submission
43+
@app.post("/ask", response_class=HTMLResponse)
44+
async def ask(prompt: str = Form(...)):
45+
headers = {
46+
"Content-Type": "application/json"
47+
}
48+
49+
if not ENDPOINT_URL.startswith("http://localhost"):
50+
API_KEY = get_api_key()
51+
headers["Authorization"] = f"Bearer {API_KEY}"
52+
else:
53+
logging.info("Skipping Authorization header for localhost endpoint.")
54+
55+
payload = {
56+
"model": os.getenv("MODEL", "gpt-4-turbo"),
57+
"messages": [
58+
{"role": "user", "content": prompt}
59+
],
60+
"stream": False
61+
}
62+
63+
# Log request details for debugging
64+
logging.info(f"Sending POST to {ENDPOINT_URL}")
65+
logging.info(f"Request Headers: {headers}")
66+
logging.info(f"Request Payload: {payload}")
67+
68+
response = requests.post(f"{ENDPOINT_URL}/chat/completions", headers=headers, data=json.dumps(payload))
69+
70+
if response.status_code == 200:
71+
data = response.json()
72+
try:
73+
reply = data["choices"][0]["message"]["content"]
74+
except (KeyError, IndexError):
75+
reply = "Model returned an unexpected response."
76+
else:
77+
# Log error details
78+
logging.error(f"Error from server: {response.status_code} - {response.text}")
79+
reply = f"Error: {response.status_code} - {response.text}"
80+
81+
# Return result
82+
return f"""
83+
<html>
84+
<head><title>Ask the Model</title></head>
85+
<body>
86+
<h1>Ask the Magic Backpack 🧙‍♂️🎒</h1>
87+
<form method="post" action="/ask">
88+
<textarea name="prompt" rows="5" cols="60" placeholder="Enter your question here...">{prompt}</textarea><br><br>
89+
<input type="submit" value="Ask">
90+
</form>
91+
<hr>
92+
<h2>Model's Reply:</h2>
93+
<p>{reply}</p>
94+
</body>
95+
</html>
96+
"""
97+
98+
@app.get("/health")
99+
async def health():
100+
return {"status": "ok"}

0 commit comments

Comments
 (0)