Merge pull request #2358 from wagner-intevation/bot-library

IEP007 implementation: Library mode

Author: sebix

.github/workflows/unittests.yml

@@ -6,7 +6,6 @@
 name: "Unit tests"
 on:
   push:
-    branches: [develop, maintenance, master]
   pull_request:
     branches: [develop, maintenance]
     paths-ignore:

CHANGELOG.md

@@ -20,6 +20,7 @@ CHANGELOG
 - `intelmq.lib.harmonization`:
   - Changes signature and names of `DateTime` conversion functions for consistency, backwards compatible (PR#2329 by Filip Pokorný).
   - Ensure rejecting URLs with leading whitespaces after changes in CPython (fixes [#2377](https://github.com/certtools/intelmq/issues/2377))
+- `intelmq.lib.bot.Bot`: Allow setting the parameters via parameter on bot initialization.
 
 ### Development
 - CI: pin the Codespell version to omit troubles caused by its new releases (PR #2379).

debian/control

@@ -22,7 +22,8 @@ Build-Depends: debhelper (>= 4.1.16),
                python3-tz,
                quilt,
                rsync,
-               safe-rm
+               safe-rm,
+               python3-pytest-cov
 X-Python3-Version: >= 3.7
 Standards-Version: 3.9.6
 Homepage: https://github.com/certtools/intelmq/

docs/dev/library.rst

@@ -0,0 +1,74 @@
+..
+   SPDX-FileCopyrightText: 2023 Bundesamt für Sicherheit in der Informationstechnik (BSI)
+   SPDX-License-Identifier: AGPL-3.0-or-later
+
+##########################
+Running IntelMQ as Library
+##########################
+
+.. contents::
+
+************
+Introduction
+************
+
+The feature is specified in `IEP007 <https://github.com/certtools/ieps/tree/iep-007/007/>`_.
+
+**********
+Quickstart
+**********
+
+First, import the Python module and a helper. More about the ``BotLibSettings`` later.
+
+.. code-block:: python
+
+   from intelmq.lib.bot import BotLibSettings
+   from intelmq.bots.experts.domain_suffix.expert import DomainSuffixExpertBot
+
+Then we need to initialize the bot's instance.
+We pass two parameters:
+* ``bot_id``: The id of the bot
+* ``settings``: A Python dictionary of runtime configuration parameters, see :ref:`runtime-configuration`.
+  The bot first loads the runtime configuration file if it exists.
+  Then we update them with the ``BotLibSettings`` which are some accumulated settings disabling the logging to files and configure the pipeline so that we can send and receive messages directly to/from the bot.
+  Last by not least, the actual bot parameters, taking the highest priority.
+
+.. code-block:: python
+
+   domain_suffix = DomainSuffixExpertBot('domain-suffix',  # bot id
+                                         settings=BotLibSettings | {
+                                                  'field': 'fqdn',
+                                                  'suffix_file': '/usr/share/publicsuffix/public_suffix_list.dat'}
+
+As the bot is not fully initialized, we can process messages now.
+Inserting a message as dictionary:
+
+.. code-block:: python
+
+   queues = domain_suffix.process_message({'source.fqdn': 'www.example.com'})
+
+The return value is a dictionary of queues, e.g. the output queue and the error queue.
+More details below.
+
+The methods accepts multiple messages as positional argument:
+
+.. code-block:: python
+
+   domain_suffix.process_message({'source.fqdn': 'www.example.com'}, {'source.fqdn': 'www.example.net'})
+   domain_suffix.process_message(*[{'source.fqdn': 'www.example.com'}, {'source.fqdn': 'www.example.net'}])
+
+
+Select the output queue (as defined in `destination_queues`), first message, access the field 'source.domain_suffix':
+
+.. code-block:: python
+
+   >>> output['output'][0]['source.domain_suffix']
+   'com'
+
+*************
+Configuration
+*************
+
+Configuration files are not required to run IntelMQ as library.
+Contrary to IntelMQ normal behavior, if the files ``runtime.yaml`` and ``harmonization.conf`` do not exist, IntelMQ won't raise any errors.
+For the harmonization configuration, internal defaults are loaded.

docs/index.rst

@@ -74,6 +74,7 @@ Getting involved
    :maxdepth: 1
 
    dev/guide
+   dev/library
    dev/data-format
    dev/harmonization-fields
    dev/release-procedure