Skip to content

Set up RabbitMQ as Celery broker #1041

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mihow merged 14 commits into from
Nov 14, 2025
Merged

Set up RabbitMQ as Celery broker #1041

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
ac67395
feat: rabbitmq as celery broker, default antenna queue
vanessavmac Nov 8, 2025
c7b0380
Merge branch 'main' into feat/rabbitmq-broker
vanessavmac Nov 8, 2025
9742263
feat: separate CI stack with rabbitmq
vanessavmac Nov 8, 2025
249c058
Merge branch 'feat/rabbitmq-broker' of https://github.com/RolnickLab/…
vanessavmac Nov 8, 2025
1c337cd
fix: broker url trailing slash, database url for CI stack
vanessavmac Nov 8, 2025
53d13eb
Add missing CI changes
vanessavmac Nov 10, 2025
d8aa02d
Merge branch 'main' into feat/rabbitmq-broker
mihow Nov 13, 2025
539fd35
chore: share env vars with django app
mihow Nov 13, 2025
133515d
feat: move celery & debugbpy start command out of compose files
mihow Nov 13, 2025
edbd7d2
feat: update production compose file for rabbitmq & default queue
mihow Nov 13, 2025
efec9f3
chore: explain redundant DATABASE_URL env var
mihow Nov 13, 2025
e8d10ab
feat: use docker override file to activate debuggers
mihow Nov 13, 2025
bfc0c23
Merge branch 'main' into feat/rabbitmq-broker
mihow Nov 14, 2025
82fdfd8
chore: tell agent docs we are using RabbitMQ instead of Redis now
mihow Nov 14, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion .agents/AGENTS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,16 @@ Every call to the AI model API incurs a cost and requires electricity. Be smart
- Always prefer command line tools to avoid expensive API requests (e.g., use git and jq instead of reading whole files)
- Use bulk operations and prefetch patterns to minimize database queries

**Performance Optimization:**
- django-cachalot handles automatic query caching - don't add manual caching layers on top
- Focus on optimizing cold queries first before adding caching
- When ordering by annotated fields, pagination COUNT queries include those annotations - use `.values('pk')` to strip them
- For large tables (>10k rows), consider fuzzy counting using PostgreSQL's pg_class.reltuples

**Git Commit Guidelines:**
- Do NOT include "Generated with Claude Code" in commit messages
- ALWAYS include "Co-Authored-By: Claude <[email protected]>" at the end of commit messages

## Project Overview

Antenna is an Automated Monitoring of Insects ML Platform. It's a collaborative platform for processing and reviewing images from automated insect monitoring stations, maintaining metadata, and orchestrating multiple machine learning pipelines for analysis.
Expand Down Expand Up @@ -273,7 +283,7 @@ Location: `processing_services/` directory contains example implementations

### Celery Task Queue

**Broker & Result Backend:** Redis
**Broker & Result Backend:** RabbitMQ

**Key Tasks:**
- `ami.jobs.tasks.run_job` - Main ML processing workflow
Expand Down
6 changes: 6 additions & 0 deletions .envs/.ci/.django
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,3 +19,9 @@ MINIO_BROWSER_REDIRECT_URL=http://minio:9001

DEFAULT_PROCESSING_SERVICE_NAME=Test Processing Service
DEFAULT_PROCESSING_SERVICE_ENDPOINT=http://ml_backend:2000

# RabbitMQ
# ------------------------------------------------------------------------------
CELERY_BROKER_URL=amqp://rabbituser:rabbitpass@rabbitmq:5672/
RABBITMQ_DEFAULT_USER=rabbituser
RABBITMQ_DEFAULT_PASS=rabbitpass
2 changes: 2 additions & 0 deletions .envs/.ci/.postgres
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,5 @@ POSTGRES_PORT=5432
POSTGRES_DB=ami-ci
POSTGRES_USER=4JXkOnTAeDmDyIapSRrGEE
POSTGRES_PASSWORD=d4xojpnJU3OzPQ0apSCLP1oHR1TYvyMzAlF5KpE9HFL6MPlnbDibwI
# DATABASE_URL is for optional convenience with certain tools (e.g., psql), not used by Django
DATABASE_URL=postgresql://4JXkOnTAeDmDyIapSRrGEE:d4xojpnJU3OzPQ0apSCLP1oHR1TYvyMzAlF5KpE9HFL6MPlnbDibwI@postgres:5432/ami-ci
5 changes: 5 additions & 0 deletions .envs/.local/.django
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,11 @@ REDIS_URL=redis://redis:6379/0
CELERY_FLOWER_USER=QSocnxapfMvzLqJXSsXtnEZqRkBtsmKT
CELERY_FLOWER_PASSWORD=BEQgmCtgyrFieKNoGTsux9YIye0I7P5Q7vEgfJD2C4jxmtHDetFaE2jhS7K7rxaf

# RabbitMQ
CELERY_BROKER_URL=amqp://rabbituser:rabbitpass@rabbitmq:5672/
RABBITMQ_DEFAULT_USER=rabbituser
RABBITMQ_DEFAULT_PASS=rabbitpass

# Attempting to keep Flower from showing workers as offline
# FLOWER_BROKER_API=REDIS_URL
FLOWER_PERSISTENT=True
Expand Down
2 changes: 2 additions & 0 deletions .envs/.local/.postgres
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,5 @@ POSTGRES_PORT=5432
POSTGRES_DB=ami
POSTGRES_USER=xekSryPnqczJXkOnTAeDmDyIapSRrGEE
POSTGRES_PASSWORD=iMRQjJEGflj5xojpnJU3OzPQ0apSCLP1oHR1TYvyMzAlF5KpE9HFL6MPlnbDibwI
# DATABASE_URL is for optional convenience with certain tools (e.g., psql), not used by Django
DATABASE_URL=postgres://xekSryPnqczJXkOnTAeDmDyIapSRrGEE:iMRQjJEGflj5xojpnJU3OzPQ0apSCLP1oHR1TYvyMzAlF5KpE9HFL6MPlnbDibwI@postgres:5432/ami
4 changes: 2 additions & 2 deletions compose/local/django/celery/worker/start
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ set -o nounset
# Note that auto reloading is disabled when debugging, manual restart required for code changes.
if [ "${DEBUGGER:-0}" = "1" ]; then
# exec watchfiles --filter python 'python -m debugpy --listen 0.0.0.0:5679 -m celery -A config.celery_app worker -l INFO'
exec python -Xfrozen_modules=off -m debugpy --listen 0.0.0.0:5679 -m celery -A config.celery_app worker -l INFO
exec python -Xfrozen_modules=off -m debugpy --listen 0.0.0.0:5679 -m celery -A config.celery_app worker --queues=antenna -l INFO
else
exec watchfiles --filter python celery.__main__.main --args '-A config.celery_app worker -l INFO'
exec watchfiles --filter python celery.__main__.main --args '-A config.celery_app worker --queues=antenna -l INFO'
fi
2 changes: 1 addition & 1 deletion compose/production/django/celery/worker/start
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,4 @@ set -o errexit
set -o pipefail
set -o nounset

exec newrelic-admin run-program celery -A config.celery_app worker -l INFO
exec newrelic-admin run-program celery -A config.celery_app worker --queues=antenna -l INFO
3 changes: 3 additions & 0 deletions config/settings/base.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -260,6 +260,7 @@
},
}
}
REDIS_URL = env("REDIS_URL", default=None)

# ADMIN
# ------------------------------------------------------------------------------
Expand Down Expand Up @@ -299,6 +300,8 @@
if USE_TZ:
# https://docs.celeryq.dev/en/stable/userguide/configuration.html#std:setting-timezone
CELERY_TIMEZONE = TIME_ZONE

CELERY_TASK_DEFAULT_QUEUE = "antenna"
# https://docs.celeryq.dev/en/stable/userguide/configuration.html#std:setting-broker_url
CELERY_BROKER_URL = env("CELERY_BROKER_URL")
# https://docs.celeryq.dev/en/stable/userguide/configuration.html#std:setting-result_backend
Expand Down
30 changes: 27 additions & 3 deletions docker-compose.ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,27 @@
# Ensure this launches a separate stack for CI testing to avoid conflicts with local dev setup:
name: antenna-ci

volumes:
ami_ci_postgres_data:
driver: local
minio_ci_data:
driver: local

services:
django: &django
build:
context: .
dockerfile: ./compose/local/django/Dockerfile
image: ami_ci_django
volumes:
- .:/app:z
extra_hosts:
- "host.docker.internal:host-gateway"
depends_on:
- postgres
- redis
- minio-init
- ml_backend
- rabbitmq
env_file:
- ./.envs/.ci/.django
- ./.envs/.ci/.postgres
Expand All @@ -18,6 +31,8 @@ services:
build:
context: .
dockerfile: ./compose/local/postgres/Dockerfile
volumes:
- ami_ci_postgres_data:/var/lib/postgresql/data
env_file:
- ./.envs/.ci/.postgres

Expand All @@ -26,12 +41,20 @@ services:

celeryworker:
<<: *django
image: ami_ci_celeryworker
depends_on:
- rabbitmq
command: /start-celeryworker

rabbitmq:
image: rabbitmq:3-management
env_file:
- ./.envs/.ci/.django

minio:
image: minio/minio:RELEASE.2024-11-07T00-52-20Z
command: minio server --console-address ":9001" /data
volumes:
- "minio_ci_data:/data"
env_file:
- ./.envs/.ci/.django
healthcheck:
Expand All @@ -45,7 +68,8 @@ services:
env_file:
- ./.envs/.ci/.django
depends_on:
- minio
minio:
condition: service_healthy
volumes:
- ./compose/local/minio/init.sh:/etc/minio/init.sh
entrypoint: /etc/minio/init.sh
Expand Down
3 changes: 2 additions & 1 deletion docker-compose.production.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,9 @@ services:
- ./config:/app/config
ports:
- "5001:5000"
extra_hosts:
extra_hosts: # hostname aliases to the IPs of persistant services running outside docker network
- "db:${DATABASE_IP}"
- "rabbitmq:${RABBITMQ_IP}"
- "redis:${REDIS_IP}"
Comment on lines +19 to 22
Copy link
Contributor

@coderabbitai coderabbitai bot Nov 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

Verify RABBITMQ_IP is set in production environment.

The extra_hosts configuration correctly follows the existing pattern for external service resolution. Ensure that the ${RABBITMQ_IP} environment variable is properly defined in your production deployment configuration.

🏁 Script executed:

#!/bin/bash
# Description: Verify RABBITMQ_IP is referenced in production environment files

# Search for RABBITMQ_IP definition in production environment files
rg -n "RABBITMQ_IP" .envs/.production/

Length of output: 0

🏁 Script executed:

#!/bin/bash
# Description: Investigate environment variable configuration structure

# 1. Check if .envs directory exists and its structure
echo "=== Directory structure of .envs/ ==="
fd . .envs/ -type f -name "*" 2>/dev/null | head -20

# 2. Search entire repo for RABBITMQ_IP (not just .envs/.production/)
echo -e "\n=== All occurrences of RABBITMQ_IP in repo ==="
rg -n "RABBITMQ_IP" . 2>/dev/null

# 3. Search for DATABASE_IP and REDIS_IP to understand the pattern
echo -e "\n=== How DATABASE_IP is referenced ==="
rg -n "DATABASE_IP" . 2>/dev/null | head -10

echo -e "\n=== How REDIS_IP is referenced ==="
rg -n "REDIS_IP" . 2>/dev/null | head -10

Length of output: 473

🏁 Script executed:

#!/bin/bash
# Description: Search for environment configuration and documentation

# 1. Look for .env files in repo root
echo "=== .env files in repo root ==="
fd -t f "^\.env" . -max-depth 1 2>/dev/null

# 2. Search for documentation about environment setup
echo -e "\n=== README or documentation files ==="
fd -t f "(README|INSTALL|SETUP|DEPLOY)" . -iname "*" 2>/dev/null | head -10

# 3. Check docker-compose.worker.yml for rabbitmq references
echo -e "\n=== Full extra_hosts section in docker-compose.worker.yml ==="
rg -A 5 "extra_hosts" ./docker-compose.worker.yml 2>/dev/null

# 4. Check for any comments in docker-compose.production.yml about these variables
echo -e "\n=== Comments in docker-compose.production.yml around extra_hosts ==="
rg -B 5 -A 5 "extra_hosts" ./docker-compose.production.yml 2>/dev/null

Length of output: 821

🏁 Script executed:

#!/bin/bash
# Description: Check if Celery worker should connect to RabbitMQ

# 1. Search for celery/rabbitmq configuration in codebase
echo "=== Celery/RabbitMQ references ==="
rg -n -i "celery|rabbitmq|broker" . --type=py --type=yml 2>/dev/null | head -20

# 2. Check docker-compose.worker.yml full configuration
echo -e "\n=== Full docker-compose.worker.yml ==="
cat ./docker-compose.worker.yml 2>/dev/null

Length of output: 887

Add RABBITMQ_IP mapping to docker-compose.worker.yml extra_hosts

The celeryworker service requires RABBITMQ_IP hostname mapping to connect to the external RabbitMQ broker, but this is missing from docker-compose.worker.yml. The celeryworker inherits from the django service which currently only includes DATABASE_IP and REDIS_IP. Add the rabbitmq hostname mapping to lines 16-17 to match the production configuration:

    extra_hosts:
      - "db:${DATABASE_IP}"
      - "rabbitmq:${RABBITMQ_IP}"
      - "redis:${REDIS_IP}"

Without this, the Celery worker will fail to resolve the rabbitmq hostname and cannot connect to the broker.

🤖 Prompt for AI Agents 
In docker-compose.production.yml around lines 19 to 22, the celeryworker service
(via the django service) is missing the rabbitmq hostname mapping in
extra_hosts; add the "rabbitmq:${RABBITMQ_IP}" mapping alongside the existing db
and redis entries so extra_hosts contains db, rabbitmq, and redis mappings to
their respective environment IP variables, ensuring the worker can resolve and
connect to the external RabbitMQ broker.

command: /start
scale: 1 # Can't scale until the load balancer is within the compose config
Expand Down
13 changes: 13 additions & 0 deletions docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ services:
- redis
- minio-init
- ml_backend
- rabbitmq
volumes:
- .:/app:z
env_file:
Expand Down Expand Up @@ -92,6 +93,8 @@ services:
scale: 1
ports: []
command: /start-celeryworker
depends_on:
- rabbitmq

celerybeat:
<<: *django
Expand All @@ -108,6 +111,16 @@ services:
volumes:
- ./data/flower/:/data/

rabbitmq:
image: rabbitmq:3-management
ports:
- "5672:5672"
- "15672:15672"
env_file:
- ./.envs/.local/.django
networks:
- antenna_network

minio:
image: minio/minio:RELEASE.2024-11-07T00-52-20Z
command: minio server --console-address ":9001" /data
Expand Down