Ollama/vLLM

docs/installation/advanced/ollama.md

@@ -0,0 +1,69 @@
+<!-- md:version 6.0 -->
+
+# Ollama
+
+[Ollama](https://docs.ollama.com/) is a framework that allows you to run various LLMs (Large Language Models) on your own hardware.
+
+!!! warning "Ollama vs vLLM"
+
+    This guides showcases a **very basic** Ollama deployment meant for trying out a self-hosted LLM deployed on the same server as SeaTable itself.
+    You should take a look at [vLLM](./vllm.md) in case you plan on deploying a **production-ready** LLM inference/serving engine.
+    Compared to Ollama, vLLM provides much better performance when handling concurrent requests.
+
+## Prerequisites
+
+This guide assumes that you have a fully functioning [SeaTable Server installation](../basic-setup.md) and have successfully installed [SeaTable AI](../components/seatable-ai.md).
+
+### GPU Usage
+
+**Notice:** Please refer to [Ollama's documentation](https://github.com/ollama/ollama/blob/main/docs/docker.md) for instructions regarding GPU usage.
+Depending on your GPU, this will require installing proprietary NVIDIA drivers and the NVIDIA Container Toolkit or adding additional arguments to the `ollama.yml` file shown below.
+
+## Instructions
+
+### Create `ollama.yml`
+
+Create `/opt/seatable-compose/ollama.yml` with the following contents:
+
+```yaml
+services:
+  ollama:
+    image: ollama/ollama:0.11.10
+    restart: unless-stopped
+    container_name: ollama
+    # Comment out the following line if you don't have a GPU
+    gpus: all
+    networks:
+      - backend-seatable-net
+    volumes:
+      - /opt/ollama:/root/.ollama
+```
+
+Afterwards, you should add `ollama.yml` to the `COMPOSE_FILE` variable inside your `.env` file:
+
+```bash
+sed -i "s/COMPOSE_FILE='\(.*\)'/COMPOSE_FILE='\1,ollama.yml'/" /opt/seatable-compose/.env
+```
+
+### Configuration
+
+In order to use Ollama to execute AI-based automation steps inside SeaTable, you must add the following configuration settings to your `.env` file:
+
+```ini
+SEATABLE_AI_LLM_TYPE='ollama_chat'
+SEATABLE_AI_LLM_URL='http://ollama:11434'
+# Choose a model: https://ollama.com/library
+SEATABLE_AI_LLM_MODEL='gemma3:12b'
+```
+
+### Start Ollama
+
+You can now start Ollama by running `docker compose up -d` inside `/opt/seatable-compose`.
+
+In addition, it is necessary to manually download the chosen AI model by executing the following command **once**:
+
+```bash
+docker exec -it ollama ollama pull $MODEL
+```
+
+You are now able to run AI-based automations steps inside SeaTable via your local Ollama deployment!

docs/installation/advanced/vllm.md

@@ -0,0 +1,137 @@
+<!-- md:version 6.0 -->
+
+# vLLM
+
+[vLLM](https://docs.vllm.ai/en/stable/) is an inference and serving engine for LLMs (Large Language Models) that allows you to run AI models on your own hardware.
+Compared to [Ollama](./ollama.md), vLLM provides better throughput under high-concurrency scenarios.
+
+## Prerequisites
+
+This guide assumes that you have a fully functioning [SeaTable Server installation](../basic-setup.md) and have successfully installed [SeaTable AI](../components/seatable-ai.md).
+
+### GPU
+
+!!! note "GPU Usage"
+
+    The exact requirements depend on your specific GPU. Since vLLM does not provide specific instructions, you can refer to [Ollama's documentation](https://github.com/ollama/ollama/blob/main/docs/docker.md) instead.
+    Depending on your GPU, this will require installing proprietary NVIDIA drivers and the NVIDIA Container Toolkit or adding additional arguments to the `vllm.yml` file shown below.
+
+For **Debian**-based systems with **NVIDIA** GPUs, the following steps were carried out to successfully run vLLM:
+
+1. Install the proprietary NVIDIA drivers. The [Debian Wiki](https://wiki.debian.org/NvidiaGraphicsDrivers) contains installation instructions for the latest Debian releases.
+1. Remember to restart your system before proceeding with the following steps.
+2. Install the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html).
+
+### HuggingFace Access Token
+
+By default, vLLM will try to download models from [HuggingFace](https://huggingface.co).
+Some models are only available after accepting license or usage terms.
+This requires you to [generate an access token](https://huggingface.co/docs/hub/security-tokens) that can be used to download models.
+You should store this access token in a safe location since it cannot be displayed again.
+
+## Instructions
+
+### Download `.yml` files
+
+Download the latest `.yml` from GitHub by running the following command:
+
+```bash
+mkdir /opt/seatable-compose && \
+cd /opt/seatable-compose && \
+wget -c https://github.com/seatable/seatable-release/releases/latest/download/seatable-compose.tar.gz -O - | tar -xz -C /opt/seatable-compose && \
+cp -n .env-release .env
+```
+
+### Create `vllm.yml`
+
+Create `/opt/seatable-compose/vllm.yml` with the following contents:
+
+```yaml
+services:
+  vllm:
+    image: vllm/vllm-openai:v0.10.2
+    container_name: vllm
+    restart: unless-stopped
+    runtime: nvidia
+    deploy:
+      resources:
+        reservations:
+          devices:
+            - driver: nvidia
+              count: all
+              capabilities: [gpu]
+    ipc: host
+    command: ["--model", "${VLLM_MODEL:?Variable is not set or empty}", "--enable-log-requests", "--max-model-len", "${VLLM_MAX_MODEL_LEN:-65536}"]
+    environment:
+      - HUGGING_FACE_HUB_TOKEN=${HUGGING_FACE_TOKEN:?Variable is not set or empty}
+      - VLLM_API_KEY=${VLLM_API_KEY:?Variable is not set or empty}
+      - VLLM_DEBUG_LOG_API_SERVER_RESPONSE=${VLLM_DEBUG_LOG_API_SERVER_RESPONSE:-false}
+    networks:
+      - frontend-net
+    volumes:
+      - ~/.cache/huggingface:/root/.cache/huggingface
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
+      interval: 10s
+      start_period: 300s
+    labels:
+      caddy: ${VLLM_HOSTNAME:?Variable is not set or empty}
+      caddy.@denied.not_0: "remote_ip ${VLLM_ALLOWED_IPS:?Variable is not set or empty} private_ranges"
+      caddy.abort: "@denied"
+      caddy.reverse_proxy: "{{upstreams 8000}}"
+```
+
+Afterwards, you should add `vllm.yml` to the `COMPOSE_FILE` variable inside your `.env` file:
+
+```bash
+sed -i "s/COMPOSE_FILE='\(.*\)'/COMPOSE_FILE='\1,vllm.yml'/" /opt/seatable-compose/.env
+```
+
+Make sure that `COMPOSE_FILE` already includes `caddy.yml` since vLLM's port 8000 is not exposed by default.
+
+### vLLM Configuration
+
+Add the following configuration settings to your `.env` file to configure vLLM:
+
+```ini
+VLLM_HOSTNAME='YOUR_VLLM_HOSTNAME'
+
+# Allowed IP addresses (multiple addresses can be separated by spaces)
+VLLM_ALLOWED_IPS='SEATABLE_AI_PUBLIC_IP'
+
+# Set this to a long random string, e.g. generated by "pwgen -s 64 1"
+# This API key must then be used for any API requests
+VLLM_API_KEY=''
+
+HUGGING_FACE_TOKEN='YOUR_HUGGING_FACE_TOKEN'
+
+# Model identifier from HuggingFace (e.g. RedHatAI/gemma-3-12b-it-quantized.w4a16)
+VLLM_MODEL=''
+```
+
+### SeaTable AI Configuration
+
+In order to use vLLM to execute AI-based automation steps inside SeaTable, you must add the following configuration settings to the `.env` file on the host where SeaTable AI is deployed:
+
+```ini
+SEATABLE_AI_LLM_TYPE='hosted_vllm'
+
+SEATABLE_AI_LLM_URL='https://<YOUR_VLLM_HOSTNAME>/v1'
+
+# API key for requests to vLLM (use the same key as above)
+SEATABLE_AI_LLM_KEY=''
+
+# Model identifier from HuggingFace (e.g. RedHatAI/gemma-3-12b-it-quantized.w4a16)
+SEATABLE_AI_LLM_MODEL=''
+```
+
+Remember to restart SeaTable AI after making any changes by running `docker compose up -d` inside `/opt/seatable-compose`.
+
+### Start vLLM
+
+You can now start vLLM by running `docker compose up -d` inside `/opt/seatable-compose`.
+
+Starting vLLM can take several minutes, depending on the model size and your computing resources.
+vLLM will automatically pull down the configured model from [HuggingFace](https://huggingface.co) on first startup.
+
+You are now able to run AI-based automations steps inside SeaTable via your local vLLM deployment!

mkdocs.yml

@@ -182,6 +182,9 @@ nav:
           - SeaTable AI:
               - SeaTable AI (standalone): installation/advanced/seatable-ai-standalone.md
               - AI Token Pricing: installation/advanced/seatable-ai-token-pricing.md
+              - LLM Self-Hosting:
+                  - Ollama: installation/advanced/ollama.md
+                  - vLLM: installation/advanced/vllm.md
           - S3 Object Storage:
               - Configuration: installation/advanced/s3.md
               - Migration: installation/advanced/s3-migration.md