Sedlak forwarder rework (#60)

* Forwarder: refactor v1

* Forwarder: improve config

* Forwarder: improve main

* Forwarder: removed old files

* Forwarder: updated CMakeLists

* Forwarder: added common

* Forwarder: added forwarder

* Forwarder: added host

* Forwarder: added connection

* Forwarder: added sender

* Forwarder: added message

* Forwarder: improve docs

* Forwarder: resend templates when connection reconnects

* Forwarder: skip hosts with waiting transfers

* Forwarder: changed how reconnections work

* Forwarder: indicate lost messages in SendToAll mode

* Forwarder: fixes in message creation

* Forwarder: use MSG_DONTSIGNAL when sending over socket

* Forwarder: remove debugging printfs

* Forwarder: minor fixes

* Forwarder: atomic sockfd when reconnecting from another thread

* Forwarder: atomic finished flag

* Forwarder: drop unfinished transfers on connection lost for TCP

* Forwarder: fix build on gcc 4.8

* Forwarder: skip data sets with unknown templates

* Forwarder: add non-blocking socket connect

* Forwarder: set export time to current time when forwarding

* Forwarder: minor fixes in Connection

* Forwarder: non-blocking socket connect v2

* Forwarder: fix reconnector

Co-authored-by: Michal Sedlak <[email protected]>

Author: Lukas955

src/plugins/output/forwarder/CMakeLists.txt

@@ -1,19 +1,30 @@
 # Create a linkable module
 add_library(forwarder-output MODULE
     src/main.cpp
-    src/config.h
+    src/Config.h
+    src/Config.cpp
     src/Forwarder.h
-    src/ConnectionManager.h
-    src/ConnectionManager.cpp
-    src/ConnectionParams.h
+    src/Forwarder.cpp
     src/Connection.h
     src/Connection.cpp
-    src/ConnectionBuffer.h
-    src/SyncPipe.h
-    src/IPFIXMessage.h
-    src/MessageBuilder.h
+    src/Host.cpp
+    src/Host.h
+    src/common.h
+    src/common.cpp
+    src/Message.h
+    src/Message.cpp
+    src/Sender.h
+    src/Sender.cpp
+    src/connector/Connector.h
+    src/connector/Connector.cpp
+    src/connector/FutureSocket.h
+    src/connector/FutureSocket.cpp
+    src/connector/Pipe.h
+    src/connector/Pipe.cpp
 )
 
+target_include_directories(forwarder-output PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/src)
+
 install(
     TARGETS forwarder-output
     LIBRARY DESTINATION "${INSTALL_DIR_LIB}/ipfixcol2/"

src/plugins/output/forwarder/README.rst

@@ -1,7 +1,7 @@
 Forwarder (output plugin) [Preview]
 ===================================
 
-This plugin allows forwarding incoming IPFIX messages to other collector in various modes.
+This plugin allows forwarding incoming flow records in IPFIX form to other collector in various modes.
 
 It can be used to broadcast messages to multiple collectors (e.g. a main and a backup collector),
 or to distribute messages across multiple collectors (e.g. for load balancing).
@@ -20,13 +20,18 @@ Example configuration
             <hosts>
                 <host>
                     <name>Subcollector 1</name>
-                    <address>127.0.0.1</address>
-                    <port>4751</port>
+                    <address>192.168.1.2</address>
+                    <port>4739</port>
                 </host>
                 <host>
                     <name>Subcollector 2</name>
+                    <address>192.168.1.3</address>
+                    <port>4739</port>
+                </host>
+                <host>
+                    <name>Subcollector 3</name>
                     <address>localhost</address>
-                    <port>4752</port>
+                    <port>4739</port>
                 </host>
             </hosts>
         </params>
@@ -36,42 +41,48 @@ Parameters
 ----------
 
 :``mode``:
-    The forwarding mode; round robin (messages are sent to one host at time and hosts are cycled through) or all (messages are broadcasted to all hosts)
+    Flow distribution mode. **RoundRobin** (each record will be delivered to one of hosts) or **All** (each record will be delivered to all hosts).
     [values: RoundRobin/All]
 
 :``protocol``:
-    The transport protocol to use
+    The transport protocol to use.
     [values: TCP/UDP]
 
-:``connectionBufferSize``:
-    Size of the buffer of each connection (Warning: number of connections = number of input exporters * number of hosts)
-    [value: number of bytes, default: 4194304]
+:``templatesResendSecs``:
+    Send templates again every N seconds (UDP only).
+    [value: number of seconds, default: 600, 0 = never]
 
-:``templateRefreshIntervalSecs``:
-    Send templates again every N seconds (UDP only)
-    [value: number of seconds, default: 600]
+:``templatesResendPkts``:
+    Send templates again every N packets (UDP only).
+    [value: number of packets, default: 5000, 0 = never]
 
-:``templateRefreshIntervalBytes``:
-    Send templates again every N bytes (UDP only)
-    [value: number of bytes, default: 5000000]
+:``reconnectSecs``:
+    Attempt to reconnect every N seconds in case the connection drops (TCP only).
+    [value: number of seconds, default: 10, 0 = don't wait]
 
-:``reconnectIntervalSecs``:
-    Attempt to reconnect every N seconds in case the connection drops (TCP only)
-    [value: number of seconds, default: 10]
+:``premadeConnections``:
+    Keep N connections open with each host so there is no delay in connecting once a connection is needed.
+    [value: number of connections, default: 5]
 
 :``hosts``:
-    The receiving hosts
+    The receiving hosts.
 
     :``host``:
         :``name``:
-            Optional identification of the host
+            Optional identification of the host.
             [value: string, default: <address>:<port>]
 
         :``address``:
-            The address of the host
+            The address of the host.
             [value: IPv4/IPv6 address or a hostname]
 
         :``port``:
-            The port to connect to
+            The port to connect to.
             [value: port number]
 
+Known limitations
+-----------------
+
+Export time of IPFIX messages is set to the current time when forwarding. This may cause issues with data fields using deltaTime relative to the export time!
+
+