Skip to content

Latest commit

 

History

History
68 lines (43 loc) · 1.78 KB

README.md

File metadata and controls

68 lines (43 loc) · 1.78 KB

Celery Redis Poll Backend

A specialized Redis backend for Celery that disables the default pub/sub mechanism for task result retrieval.

This enables polling based approach by simply calling ready() method on Celery's AsyncResult.

Why Polling Instead of Pub/Sub?

The default Celery Redis backend uses Redis pub/sub for real-time task result notifications. While pub/sub provides immediate updates, it can face challenges in certain scenarios:

  • Deadlocks in highly concurrent/multi-threaded workloads due to single-threaded nature of Redis and Celery clients.
  • Higher overhead with SUBSCRIBE channels.

This backend replaces SUBSCRIBE based backend with simple setter. This means that you need to check ready() == True when getting result.

Features

  • Compatible with Existing Code: Drop-in replacement for the standard Redis backend
  • Resource Efficient: Reduces Redis memory usage by eliminating pub/sub channels

Installation

pip install celery-redis-poll

After installation new backends are automatically registered with Celery:

  • redis+poll
  • rediss+poll
  • redis+cluster_poll
  • rediss+cluster_poll

Usage

Configure your Celery application to use the polling backend:

from celery import Celery

app = Celery('your_app',
             broker='redis://localhost:6379/0',
             backend='redis+poll://localhost:6379/0')

For clustered Redis, use redis+cluster_poll.

Requirements

  • Python >= 3.7
  • Celery >= 5.0.0
  • Redis >= 4.5.0
  • celery-redis-cluster >= 0.1.6

Development

For development, install extra dependencies:

pip install celery-redis-poll[dev]

License

This project is licensed under the MIT License.

Contributing

Contributions are welcome! Please feel free to submit issues and pull requests.