docs: pipeline configration fixes

fix links
add sections
add missing types and default values
add missing typing hints to Amqp pipeline class

Author: sebix

docs/admin/beta-features.md

@@ -56,78 +56,84 @@ only RabbitMQ as broker has been tested.
 You can change the broker for single bots (set the parameters in the runtime configuration per bot) or for the whole
 botnet (using the global configuration).
 
+### Settings
+
 You need to set the parameter
 `source_pipeline_broker`/`destination_pipeline_broker` to `amqp`. There are more parameters available:
 
-!!! bug
-    This section of the documentation is currently incomplete and will be updated later.
-
 **`destination_pipeline_broker`**
 
 (required, string) `"amqp"`
 
-
 **`destination_pipeline_host`**
 
-()  (default: `'127.0.0.1'`)
+(optional, string)  (default: `'127.0.0.1'`)
 
 **`destination_pipeline_port`**
 
-() (default: 5672)
+(optional, integer) (default: 5672)
 
 **`destination_pipeline_username`**
 
-()
+(optional, string)
 
 **`destination_pipeline_password`**
 
-()
+(optional, string)
+
+**`destination_pipeline_ssl`**
+
+(optional, boolean) (default: false)
 
 **`destination_pipeline_socket_timeout`**
 
-() (default: no timeout)
+(optional, integer) (default: no timeout)
 
 **`destination_pipeline_amqp_exchange`**
 
-() Only change/set this if you know what you do. If set, the destination queues are not declared as queues, but used as routing key. (default: `''`).
+(optional, string) Only change/set this if you know what you do. If set, the destination queues are not declared as queues, but used as routing key. (default: `''`).
 
 **`destination_pipeline_amqp_virtual_host`**
 
-() (default: `'/'`)
+(optional, string) (default: `'/'`)
 
 **`source_pipeline_host`**
 
-() (default: `'127.0.0.1'`)
-
+(optional, string) (default: `'127.0.0.1'`)
 
 **`source_pipeline_port`**
 
-() (default: 5672)
+(optional, port) (default: 5672)
 
 **`source_pipeline_username`**
 
-()
+(optional, string)
 
 **`source_pipeline_password`**
 
-()
+(optional, string)
+
+**`source_pipeline_ssl`**
+
+(optional, boolean) (default: false)
 
 **`source_pipeline_socket_timeout`**
 
-() (default: no timeout)
+(optional, string) (default: no timeout)
 
 **`source_pipeline_amqp_exchange`**
 
-() Only change/set this if you know what you do. If set, the destination queues are not declared as queues, but used as routing key. (default: ['']).
-
+(optional, string) Only change/set this if you know what you do. If set, the destination queues are not declared as queues, but used as routing key. (default: `""`]).
 
 **`source_pipeline_amqp_virtual_host`**
 
-() (default: `'/'`)
+(optional, string) (default: `'/'`)
 
 **`intelmqctl_rabbitmq_monitoring_url`**
 
-() string, see below (default: `"http://{host}:15672"`)
+(optional, string) string, see below (default: `"http://{host}:15672"`)
+
+### Monitoring queues
 
 For getting the queue sizes, `intelmqctl` needs to connect to the monitoring interface of RabbitMQ. If the monitoring
 interface is not available under `http://{host}:15672` you can manually set using the
@@ -137,10 +143,11 @@ user account, make sure to add the tag "monitoring" to it, otherwise IntelMQ can
 
 ![](../static/images/rabbitmq-user-monitoring.png)
 
+### Statistics
+
 Setting the statistics (and cache) parameters is necessary when the local redis is running under a non-default
 host/port. If this is the case, you can set them explicitly:
 
-
 **`statistics_database`**
 
 () `3`

docs/admin/configuration/intelmq.md

@@ -156,30 +156,14 @@ Some information can as well be found in Python's documentation on the used
 
 If the path `_on_error` exists for a bot, the message is also sent to this queue, instead of (only) dumping the file if
 configured to do so.
-
-##### Miscellaneous
-
-**`load_balance`**
-
-(required, boolean) this option allows you to choose the behavior of the queue. Use the following values:
-
-  - **true** - splits the messages into several queues without duplication
-  - **false** - duplicates the messages into each queue - When using AMQP as message broker, take a look at the `multithreading`{.interpreted-text role="ref"} section and the `instances_threads` parameter.
-
-**`rate_limit`**
-
-(required, integer) time interval (in seconds) between messages processing. int value.
-
-**`ssl_ca_certificate`**
-
-(optional, string) trusted CA certificate for IMAP connections (supported by some bots).
+##### Pipeline
 
 **`source_pipeline_broker`**
 
 (optional, string) Allowed values are `redis` and `amqp`. Selects the message broker IntelMQ should use. As this parameter can be overridden by each bot, this allows usage of different broker systems and hosts, as well as switching between them on the same IntelMQ instance. Defaults to `redis`.
 
   - **redis** - Please note that persistence has to be [manually activated](http://redis.io/topics/persistence).
-  - **amqp** - [Using the AMQP broker]() is currently beta but there are no known issues. A popular AMQP broker is [RabbitMQ](https://www.rabbitmq.com/).
+  - **amqp** - [Using the AMQP broker](../beta-features.md#using-amqp-message-broker) is currently beta but there are no known issues. A popular AMQP broker is [RabbitMQ](https://www.rabbitmq.com/).
 
 **`destination_pipeline_broker`**
 
@@ -220,6 +204,22 @@ configured to do so.
 
 (required, integer) broker database that the bot will use to connect and send messages (requirement from
   redis broker).
+##### Miscellaneous
+
+**`load_balance`**
+
+(required, boolean) this option allows you to choose the behavior of the queue. Use the following values:
+
+  - **true** - splits the messages into several queues without duplication
+  - **false** - duplicates the messages into each queue - When using AMQP as message broker, take a look at the `multithreading`{.interpreted-text role="ref"} section and the `instances_threads` parameter.
+
+**`rate_limit`**
+
+(required, integer) time interval (in seconds) between messages processing. int value.
+
+**`ssl_ca_certificate`**
+
+(optional, string) trusted CA certificate for IMAP connections (supported by some bots).
 
 **`http_proxy`**
 

intelmq/lib/pipeline.py

@@ -488,23 +488,23 @@ def receive(self) -> str:
 
 class Amqp(Pipeline):
     queue_args = {'x-queue-mode': 'lazy'}
-    source_pipeline_host = '127.0.0.1'
-    destination_pipeline_host = '127.0.0.1'
-    source_pipeline_db = 2
-    destination_pipeline_db = 2
-    source_pipeline_username = None
-    destination_pipeline_username = None
-    source_pipeline_password = None
-    destination_pipeline_password = None
-    source_pipeline_socket_timeout = None
-    destination_pipeline_socket_timeout = None
-    source_pipeline_amqp_virtual_host = '/'
-    destination_pipeline_amqp_virtual_host = '/'
-    source_pipeline_ssl = False
-    destination_pipeline_ssl = False
-    source_pipeline_amqp_exchange = ""
-    destination_pipeline_amqp_exchange = ""
-    intelmqctl_rabbitmq_monitoring_url = None
+    source_pipeline_host: str = '127.0.0.1'
+    destination_pipeline_host: str = '127.0.0.1'
+    source_pipeline_db: int = 2
+    destination_pipeline_db: int = 2
+    source_pipeline_username: Optional[str] = None
+    destination_pipeline_username: Optional[str] = None
+    source_pipeline_password: Optional[str] = None
+    destination_pipeline_password: Optional[str] = None
+    source_pipeline_socket_timeout: Optional[int] = None
+    destination_pipeline_socket_timeout: Optional[int] = None
+    source_pipeline_amqp_virtual_host: str = '/'
+    destination_pipeline_amqp_virtual_host: str = '/'
+    source_pipeline_ssl: bool = False
+    destination_pipeline_ssl: bool = False
+    source_pipeline_amqp_exchange: str = ""
+    destination_pipeline_amqp_exchange: str = ""
+    intelmqctl_rabbitmq_monitoring_url: Optional[str] = None
 
     def __init__(self, logger, pipeline_args: dict = None, load_balance=False, is_multithreaded=False):
         super().__init__(logger, pipeline_args, load_balance, is_multithreaded)