2
2
3
3
DLQ is a lightweight & modular queue and cron system for Django.
4
4
5
+ ## Installation
6
+
7
+ ``` shell
8
+ pip install django-lightweight-queue[redis]
9
+ ```
10
+
11
+ Currently the only production-ready backends are redis-based, so the ` redis `
12
+ extra is essentially required. Additional non-redis backed production-ready
13
+ backends are great candidates for community contributions.
14
+
5
15
## Basic Usage
6
16
7
- ``` python
17
+ ``` python
8
18
import time
9
19
from django_lightweight_queue import task
10
20
@@ -25,42 +35,50 @@ for more details.
25
35
All automatically picked up configuration options begin with ` LIGHTWEIGHT_QUEUE_ `
26
36
and can be found in ` app_settings.py ` . They should be placed in the usual Django
27
37
settings files, for example:
28
- ``` python
38
+
39
+ ``` python
29
40
LIGHTWEIGHT_QUEUE_BACKEND = ' django_lightweight_queue.backends.redis.RedisBackend'
30
41
```
31
42
32
- If desired, specific configuration overrides can be placed in a standalone python
33
- file which passed on the command line.
43
+ #### Special Configuration
44
+
45
+ If desired, specific configuration overrides can be placed in a standalone
46
+ python file which passed on the command line. This is useful for applying
47
+ customisations for specific servers.
34
48
35
49
For example, given a ` special.py ` containing:
36
- ``` python
50
+
51
+ ``` python
37
52
LIGHTWEIGHT_QUEUE_REDIS_PORT = 12345
38
53
```
54
+
39
55
and then running:
56
+
40
57
```
41
58
$ python manage.py queue_runner --config=special.py
42
59
```
60
+
43
61
will result in the runner to use the settings from the specified configuration
44
62
file in preference to settings from the Django environment. Any settings not
45
63
present in the specified file are inherited from the global configuration.
46
64
47
65
### Backends
48
66
49
67
There are four built-in backends:
50
- - Synchronous (the default): executes the task inline, without any actual queuing
51
- - Redis: executes tasks at-most-once using [ Redis] [ redis ] for storage of the
52
- enqueued tasks
53
- - Reliable Redis: executes tasks at-least-once using [ Redis] [ redis ] for storage
54
- of the enqueued tasks
55
- - Debug Web: a backend for use in debugging. Instead of running jobs it prints
56
- the url to a view that can be used to run a task in a transaction which will
57
- be rolled back. This is useful for debugging and optimising tasks.
68
+
69
+ | Backend | Type | Description |
70
+ | -------------- | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
71
+ | Synchronous | Development | Executes the task inline, without any actual queuing. |
72
+ | Redis | Production | Executes tasks at-most-once using [ Redis] [ redis ] for storage of the enqueued tasks. |
73
+ | Reliable Redis | Production | Executes tasks at-least-once using [ Redis] [ redis ] for storage of the enqueued tasks (subject to Redis consistency). Does not guarantee the task _ completes_ . |
74
+ | Debug Web | Debugging | Instead of running jobs it prints the url to a view that can be used to run a task in a transaction which will be rolled back. This is useful for debugging and optimising tasks. |
58
75
59
76
[ redis ] : https://redis.io/
60
77
61
78
## Running Workers
62
79
63
80
The queue runner is implemented as a Django management command:
81
+
64
82
```
65
83
$ python manage.py queue_runner
66
84
```
@@ -73,37 +91,47 @@ $ python manage.py queue_runner --machine 2 --of 4
73
91
```
74
92
75
93
Alternatively a runner can be told explicitly which configuration to use:
94
+
76
95
```
77
96
$ python manage.py queue_runner --exact-configuration --config=special.py
78
97
```
98
+
79
99
When using ` --exact-configuration ` the number of workers is configured exactly,
80
100
rather than being treated as the configuration for a pool. Additionally,
81
101
exactly-configured runners will _ not_ run any cron workers.
82
102
83
103
### Example
84
104
85
105
Given a Django configuration containing:
86
- ``` python
106
+
107
+ ``` python
87
108
LIGHTWEIGHT_QUEUE_WORKERS = {
88
109
' queue1' : 3 ,
89
110
}
90
111
```
112
+
91
113
and a ` special.py ` containing:
92
- ``` python
114
+
115
+ ``` python
93
116
LIGHTWEIGHT_QUEUE_WORKERS = {
94
117
' queue1' : 2 ,
95
118
}
96
119
```
120
+
97
121
Running any of:
122
+
98
123
```
99
124
$ python manage.py queue_runner --machine 1 --of 3 # or,
100
125
$ python manage.py queue_runner --machine 2 --of 3 # or,
101
126
$ python manage.py queue_runner --machine 3 --of 3
102
127
```
128
+
103
129
will result in one worker for ` queue1 ` on the current machine, while:
130
+
104
131
```
105
132
$ python manage.py queue_runner --exact-configuration --config=special.py
106
133
```
134
+
107
135
will result in two workers on the current machine.
108
136
109
137
## Cron Tasks
@@ -115,7 +143,7 @@ To specify that a management command should be run at a given time, place a
115
143
` cron.py ` file in the root folder of the Django app which defines the command
116
144
and which contains a ` CONFIG ` variable:
117
145
118
- ``` python
146
+ ``` python
119
147
CONFIG = (
120
148
{
121
149
' command' : ' my_cron_command' ,
0 commit comments