You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@@ -41,13 +39,13 @@ In most cases the bottlenecks are look-up experts. In these cases you can easily
41
39
Multithreading
42
40
^^^^^^^^^^^^^^
43
41
44
-
When using the AMQP broker, you can make use of Multi-threading. See the [Uer-Guide, section Multithreading](User-Guide.html#multithreading-beta).
42
+
When using the AMQP broker, you can make use of Multi-threading. See the :ref:`multithreading` section.
45
43
46
44
"Classic" load-balancing (Multiprocessing)
47
45
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
48
46
49
47
Before Multithreading was available in IntelMQ, andin case you use Redis as broker, the only way to do load balancing involves more work.
50
-
Create multiple instances of the same bot and connect them all to the same source and destination bots. Then set the parameter `load_balance` to `true`for the bot which sends the messages to the duplicated bot. Then, the bot sends messages to only one of the destination queues andnot to all of them.
48
+
Create multiple instances of the same bot and connect them all to the same source and destination bots. Then set the parameter ``load_balance`` to ``true`` for the bot which sends the messages to the duplicated bot. Then, the bot sends messages to only one of the destination queues and not to all of them.
51
49
52
50
True Multi*processing* is not available in IntelMQ. See also this :issue:`discussion on a possible enhanced load balancing <186>`.
53
51
@@ -70,36 +68,34 @@ The raw data consumes about 50% - 30% of the messages' size. The size of course
70
68
71
69
You can do this for example by using the *Field Reducer Expert*. The configuration could be:
72
70
73
-
*`type`: `blacklist`
74
-
*`keys`: `raw`
71
+
* ``type``: ``blacklist``
72
+
* ``keys``: ``raw``
75
73
76
74
Other solutions are the *Modify* bot and the *Sieve* bot. The last one is a good choice if you already use it and you only need to add the command:
77
75
78
-
```
79
-
remove raw
80
-
```
76
+
``remove raw``
81
77
82
78
**In the database**
83
79
84
80
In case you store data in the database and you want to keep its size small, you can (periodically) delete the raw data there.
85
81
86
82
To remove the raw data for a events table of a PostgreSQL database, you can use something like:
If the database is big, make sure only update small parts of the database by using an appropriate `WHERE` clause. If you do not see any negative performance impact, you can increase the size of the chunks, otherwise the events in the output bot may queue up. The `id` column can also be used instead of the source's time.
86
+
UPDATE events SET raw = NULL WHERE "time.source" < '2018-07-01';
87
+
88
+
If the database is big, make sure only update small parts of the database by using an appropriate ``WHERE`` clause. If you do not see any negative performance impact, you can increase the size of the chunks, otherwise the events in the output bot may queue up. The ``id`` column can also be used instead of the source's time.
Rather than starting your bot(s) with`intelmqctl start`, try`intelmqctl run [bot]`. This will provide valuable debug output you might not otherwise see, pointing to issues like system configuration errors.
93
+
Rather than starting your bot(s) with ``intelmqctl start``, try ``intelmqctl run [bot]``. This will provide valuable debug output you might not otherwise see, pointing to issues like system configuration errors.
0 commit comments