Skip to content

[chore] Remove RabbitMQ and Celery, Add TaskIQ, Unify Redis, Add OTLP batching#3019

Merged
junaway merged 63 commits intorelease/v0.69.0from
chore/unify-redis
Dec 11, 2025
Merged

[chore] Remove RabbitMQ and Celery, Add TaskIQ, Unify Redis, Add OTLP batching#3019
junaway merged 63 commits intorelease/v0.69.0from
chore/unify-redis

Conversation

@junaway
Copy link
Copy Markdown
Contributor

@junaway junaway commented Nov 19, 2025

No description provided.

@vercel
Copy link
Copy Markdown

vercel bot commented Nov 19, 2025

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Preview Comments Updated (UTC)
agenta-documentation Ready Ready Preview Comment Dec 11, 2025 6:58pm

@dosubot dosubot bot added the size:XXL This PR changes 1000+ lines, ignoring generated files. label Nov 19, 2025
@dosubot dosubot bot added the Backend label Nov 19, 2025
Copy link
Copy Markdown
Member

@mmabrouk mmabrouk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

might make sense to put as draft

@CLAassistant
Copy link
Copy Markdown

CLAassistant commented Nov 29, 2025

CLA assistant check
All committers have signed the CLA.

jp-agenta and others added 3 commits November 29, 2025 11:53
….specs.md

- Split Redis into two instances:
  - redis-volatile (6379): caches + channels, TTL only, volatile-lru eviction
  - redis-durable (6380): queues + streams, AOF persistence, noeviction policy

- Update environment variables with proper precedence:
  - Per-kind: VALKEY_CACHE_URL, VALKEY_CHANNEL_URL, VALKEY_QUEUE_URL, VALKEY_STREAM_URL
  - Class-level: VALKEY_VOLATILE_URL, VALKEY_DURABLE_URL
  - Global: VALKEY_URL
  - Defaults: redis://redis-volatile:6379 and redis://redis-durable:6380

- Update services:
  - caching.py: Use VALKEY_CACHE_URL (volatile Redis)
  - queues.py (renamed from workers.py): Use VALKEY_QUEUE_URL (durable Redis)
  - streams.py (new): TracingWorker for OTLP ingestion from streams:otlp

- Update docker-compose:
  - Split into redis-volatile and redis-durable services
  - Update queues and streams services to depend on both Redis instances
  - Update volumes for separate data directories

See /sandbox/architecture/redis.split.specs.md for detailed architecture.

🤖 Generated with Claude Code

Co-Authored-By: Claude <noreply@anthropic.com>
@junaway junaway changed the title [chore] Swap rabbitmq and celery for redis streams and taskiq [chore] Remove RabbitMQ and Celery, Add TaskIQ, Unify Redis, Add OTLP batching Dec 11, 2025
@junaway junaway merged commit e05abbd into release/v0.69.0 Dec 11, 2025
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Backend lgtm This PR has been approved by a maintainer size:XXL This PR changes 1000+ lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants