working ollama sample

Author: raphaeltm

samples/ollama/.devcontainer/devcontainer.json

@@ -5,6 +5,7 @@
   },
   "features": {
     "ghcr.io/defanglabs/devcontainer-feature/defang-cli:1.0.4": {},
-    "ghcr.io/devcontainers/features/docker-in-docker:2": {}
+    "ghcr.io/devcontainers/features/docker-in-docker:2": {},
+    "ghcr.io/devcontainers/features/node:1": {}
   }
 }

samples/ollama/.github/workflows/deploy.yaml

@@ -17,11 +17,4 @@ jobs:
       uses: actions/checkout@v4
 
     - name: Deploy
-      uses: DefangLabs/[email protected]
-      #REMOVE_ME_AFTER_EDITING - Replace the following line with the list of environment variables you want to pass to the action
-      # or remove the lines if you don't need to pass any environment variables/secrets to the action
-      with:
-        config-env-vars: ENV1 ENV2
-      env:
-        ENV1: ${{ secrets.ENV1 }}
-        ENV2: ${{ secrets.ENV2 }}
+      uses: DefangLabs/[email protected]

samples/ollama/README.md

@@ -1,8 +1,8 @@
-# Starter Sample #REMOVE_ME_AFTER_EDITING
+# Ollama
 
-[1-click deploy](https://portal.defang.dev/redirect?url=https%3A%2F%2Fgithub.com%2Fnew%3Ftemplate_name%3Dsample-<YOUR_SAMPLE_DIR#REMOVE_ME_AFTER_EDITING>-template%26template_owner%3DDefangSamples)
+[1-click deploy](https://portal.defang.dev/redirect?url=https%3A%2F%2Fgithub.com%2Fnew%3Ftemplate_name%3Dsample-ollama-template%26template_owner%3DDefangSamples)
 
-This is a sample that shows the rough structure of an actual Defang sample. This top paragraph should give a bit of context about the project and what it does. The rest of the README should be a guide on how to use the sample. #REMOVE_ME_AFTER_EDITING
+This sample demonstrates how to deploy [Ollama](https://ollama.com/) with Defang, along with a Next.js frontend using the [AI SDK](https://www.npmjs.com/package/ai) for smooth streaming conversations. By default it runs a very small model (`llama3.2:1b`) which can perform well with just a CPU, but we've included lines that you can uncomment in the compose file to enable GPU support and run a larger model like `gemma:7b`. If you want to deploy to a GPU powered instance, you will need to use your own AWS account with [Defang BYOC](https://docs.defang.io/docs/concepts/defang-byoc).
 
 ## Prerequisites
 
@@ -15,18 +15,9 @@ This is a sample that shows the rough structure of an actual Defang sample. This
 To run the application locally, you can use the following command:
 
 ```bash
-# This might be `docker compose -f compose.dev.yaml up` depending on the project. #REMOVE_ME_AFTER_EDITING
-docker compose up
+docker compose -f compose.dev.yaml up
 ```
 
-## Configuration
-#REMOVE_ME_AFTER_EDITING - this section should be removed if there are no configuration values needed. The intro text can probably stay, but the list of configuration values should be updated/removed if there are none.
-
-For this sample, you will need to provide the following [configuration](https://docs.defang.io/docs/concepts/configuration). Note that if you are using the 1-click deploy option, you can set these values as secrets in your GitHub repository and the action will automatically deploy them for you.
-
-### `API_KEY` #REMOVE_ME_AFTER_EDITING
-An explanation of what the env var (`API_KEY`) is, etc.
-
 ## Deployment
 
 > [!NOTE]
@@ -36,6 +27,8 @@ An explanation of what the env var (`API_KEY`) is, etc.
 
 Deploy your application to the defang playground by opening up your terminal and typing `defang up`.
 
+**Keep in mind that the playground does not support GPU instances.**
+
 ### BYOC (AWS)
 
 If you want to deploy to your own cloud account, you can use Defang BYOC:
@@ -45,10 +38,10 @@ If you want to deploy to your own cloud account, you can use Defang BYOC:
 
 ---
 
-Title: Sample Title #REMOVE_ME_AFTER_EDITING
+Title: Ollama
 
-Short Description: A short sentence or two describing the sample. #REMOVE_ME_AFTER_EDITING
+Short Description: Ollama is a tool that lets you easily run large language models.
 
-Tags: Tags, That, Are, Not, Programming, Languages #REMOVE_ME_AFTER_EDITING
+Tags: AI, LLM, ML, Llama, Mistral, Next.js, AI SDK, 
 
-Languages: Programming, Languages, Used #REMOVE_ME_AFTER_EDITING
+Languages: Typescript

samples/ollama/app/.dockerignore

@@ -0,0 +1,15 @@
+services:
+  app:
+    extends:
+      file: compose.yaml
+      service: app
+
+  ui:
+    extends:
+      file: compose.yaml
+      service: ui
+    volumes:
+      - type: bind
+        source: ./ui
+        target: /app
+    command: ["npm", "run", "dev"]

samples/ollama/app/run.sh

@@ -1,20 +1,52 @@
 services:
-  app:
+  ollama:
     # uncomment to add your own domain
     # domainname: example.com
     build:
-      context: ./app
+      context: ./ollama
       dockerfile: Dockerfile
+      shm_size: "16gb"
     ports:
       - target: 8000
-        published: 8000
-        mode: ingress
+        mode: host
     deploy:
       resources:
-        limits:
-          cpus: '2'
-          memory: 4G
+        reservations:
+          cpus: '2.0'
+          memory: 8192M
+          # Uncomment the next two lines to enable GPU support, for example to use gemma:7b
+          # NOTE: this is only supported in [BYOC](https://docs.defang.io/docs/concepts/defang-byoc)
+          # devices:
+          #   - capabilities: ["gpu"]
+    # By Default we load llama3.2:1b because it can run efficiently on a CPU, but you can select
+    # a different model by setting the LOAD_MODEL environment variable. Check the [list](https://ollama.com/library)
+    # for more models. For example, to load gemma:7b, set the LOAD_MODEL environment variable to gemma:7b below.
+    environment:
+      - LOAD_MODEL=llama3.2:1b
+      # - LOAD_MODEL=gemma:7b
     healthcheck:
       # wget or curl required for healthchecks on services with a published port
       # this gets parsed by Defang and provided to the load balancers as well
       test: [ "CMD", "curl", "-s", "http://localhost:8000/" ]
+
+  ui:
+    build:
+      context: ui
+      dockerfile: Dockerfile
+    ports:
+      - mode: ingress
+        target: 3000
+        published: 3000
+    deploy:
+      resources:
+        reservations:
+          memory: 256M
+    healthcheck:
+      # wget or curl required for healthchecks on services with a published port
+      # this gets parsed by Defang and provided to the load balancers as well
+      test: [ "CMD", "curl", "-s", "http://localhost:3000/" ]
+    environment:
+      - OLLAMA_ENDPOINT=http://ollama:8000
+      # Make sure the LOAD_MODEL is the same as the app service
+      - LOAD_MODEL=llama3.2:1b
+      # - LOAD_MODEL=gemma:7b

samples/ollama/compose.dev.yaml

@@ -10,6 +10,7 @@ ARG OLLAMA_MODELS=/usr/share/ollama/.ollama/models
 # Use the build-time arguments to set environment variables
 ENV OLLAMA_HOST=${OLLAMA_HOST}
 ENV OLLAMA_MODELS=${OLLAMA_MODELS}
+ENV LOAD_MODEL=llama3.2:1b
 
 RUN curl -fsSL https://ollama.com/install.sh | sh