11# Django Lightweight Queue
22
3- DLQ is a lightweight & modular queue and cron system for Django.
3+ DLQ is a lightweight & modular queue and cron system for Django. It powers
4+ millions of production jobs every day at Thread.
5+
6+ ## Installation
7+
8+ ``` shell
9+ pip install django-lightweight-queue[redis]
10+ ```
11+
12+ Currently the only production-ready backends are redis-based, so the ` redis `
13+ extra is essentially required. Additional non-redis backed production-ready
14+ backends are great candidates for community contributions.
415
516## Basic Usage
617
7- ``` python
18+ ``` python
819import time
920from django_lightweight_queue import task
1021
@@ -25,42 +36,50 @@ for more details.
2536All automatically picked up configuration options begin with ` LIGHTWEIGHT_QUEUE_ `
2637and can be found in ` app_settings.py ` . They should be placed in the usual Django
2738settings files, for example:
28- ``` python
39+
40+ ``` python
2941LIGHTWEIGHT_QUEUE_BACKEND = ' django_lightweight_queue.backends.redis.RedisBackend'
3042```
3143
32- If desired, specific configuration overrides can be placed in a standalone python
33- file which passed on the command line.
44+ #### Special Configuration
45+
46+ If desired, specific configuration overrides can be placed in a standalone
47+ python file which passed on the command line. This is useful for applying
48+ customisations for specific servers.
3449
3550For example, given a ` special.py ` containing:
36- ``` python
51+
52+ ``` python
3753LIGHTWEIGHT_QUEUE_REDIS_PORT = 12345
3854```
55+
3956and then running:
57+
4058```
4159$ python manage.py queue_runner --config=special.py
4260```
61+
4362will result in the runner to use the settings from the specified configuration
4463file in preference to settings from the Django environment. Any settings not
4564present in the specified file are inherited from the global configuration.
4665
47- ### Backends
66+ ## Backends
4867
4968There are four built-in backends:
50- - Synchronous (the default): executes the task inline, without any actual queuing
51- - Redis: executes tasks at-most-once using [ Redis] [ redis ] for storage of the
52- enqueued tasks
53- - Reliable Redis: executes tasks at-least-once using [ Redis] [ redis ] for storage
54- of the enqueued tasks
55- - Debug Web: a backend for use in debugging. Instead of running jobs it prints
56- the url to a view that can be used to run a task in a transaction which will
57- be rolled back. This is useful for debugging and optimising tasks.
69+
70+ | Backend | Type | Description |
71+ | -------------- | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
72+ | Synchronous | Development | Executes the task inline, without any actual queuing. |
73+ | Redis | Production | Executes tasks at-most-once using [ Redis] [ redis ] for storage of the enqueued tasks. |
74+ | Reliable Redis | Production | Executes tasks at-least-once using [ Redis] [ redis ] for storage of the enqueued tasks (subject to Redis consistency). Does not guarantee the task _ completes_ . |
75+ | Debug Web | Debugging | Instead of running jobs it prints the url to a view that can be used to run a task in a transaction which will be rolled back. This is useful for debugging and optimising tasks. |
5876
5977[ redis ] : https://redis.io/
6078
6179## Running Workers
6280
6381The queue runner is implemented as a Django management command:
82+
6483```
6584$ python manage.py queue_runner
6685```
@@ -73,37 +92,47 @@ $ python manage.py queue_runner --machine 2 --of 4
7392```
7493
7594Alternatively a runner can be told explicitly which configuration to use:
95+
7696```
7797$ python manage.py queue_runner --exact-configuration --config=special.py
7898```
99+
79100When using ` --exact-configuration ` the number of workers is configured exactly,
80101rather than being treated as the configuration for a pool. Additionally,
81102exactly-configured runners will _ not_ run any cron workers.
82103
83- ### Example
104+ #### Example
84105
85106Given a Django configuration containing:
86- ``` python
107+
108+ ``` python
87109LIGHTWEIGHT_QUEUE_WORKERS = {
88110 ' queue1' 3 ,
89111}
90112```
113+
91114and a ` special.py ` containing:
92- ``` python
115+
116+ ``` python
93117LIGHTWEIGHT_QUEUE_WORKERS = {
94118 ' queue1' 2 ,
95119}
96120```
121+
97122Running any of:
123+
98124```
99125$ python manage.py queue_runner --machine 1 --of 3 # or,
100126$ python manage.py queue_runner --machine 2 --of 3 # or,
101127$ python manage.py queue_runner --machine 3 --of 3
102128```
129+
103130will result in one worker for ` queue1 ` on the current machine, while:
131+
104132```
105133$ python manage.py queue_runner --exact-configuration --config=special.py
106134```
135+
107136will result in two workers on the current machine.
108137
109138## Cron Tasks
@@ -115,7 +144,7 @@ To specify that a management command should be run at a given time, place a
115144` cron.py ` file in the root folder of the Django app which defines the command
116145and which contains a ` CONFIG ` variable:
117146
118- ``` python
147+ ``` python
119148CONFIG = (
120149 {
121150 ' command' ' my_cron_command'
@@ -128,3 +157,9 @@ CONFIG = (
128157 },
129158)
130159```
160+
161+ ## Maintainers
162+
163+ This repository was created by [ Chris Lamb] ( https://github.com/lamby ) at
164+ [ Thread] ( https://www.thread.com/ ) , and continues to be maintained by the [ Thread
165+ engineering team] ( https://github.com/thread ) .
0 commit comments