IPFIX output: many code improvements, added documentation

Author: Lukas955

src/plugins/output/ipfix/README.rst

@@ -1,17 +1,32 @@
 IPFIX (output plugin)
 =====================
 
-The plugin outputs incoming IPFIX Data to an IPFIX File or a series of IPFIX Files. 
-
-An IPFIX File is a serialized stream of IPFIX Messages. Simply put, the plugin stores all valid packets received by the collector into one or more IPFIX Files. Although this is not an optimal way to store flows, it can be quite useful for testing purposes - capture flow records into IPFIX File(s) and replay them using ipfixsend2 tool.
-
-After a new file is started, all the (options) templates seen in the (options) template sets of an ODID that are still available are written to the file once the first IPFIX Message corresponding to the ODID arrives with at least one successfully parsed data record. This is necessary so each file can be used individually independent of the (options) template sets from previous files.    
+The plugin writes incoming IPFIX Messages to an IPFIX File or a series of IPFIX
+Files.
 
+An IPFIX File is a serialized stream of IPFIX Messages. Simply put, the plugin
+stores all valid packets received by the collector into one or more IPFIX Files.
+Although this is not an optimal way to store flows, it can be quite useful
+for testing purposes - capture flow records into IPFIX File(s) and replay them
+using ``ipfixsend2`` tool.
 
 Limitations
 -----------
-As there is no session information in raw IPFIX Data, we can't distinguish messages from multiple sessions using the same ODID. For this reason, the same ODID can only be used by one session at a time, and all messages from other sessions using the ODID that's already in use are ignored and a warning message is printed. All ODIDs used by a session are released once the session closes and they can be used by another session. 
 
+In a situation when multiple exporters are using the same ODID at a time, the
+file format can store flow records only from one of the conflicting exporters
+with the given ODID.
+
+This is due to fact that the IPFIX Data Records (i.e. flow records) are
+formatted based on so called (Options) Template definitions (i.e. descriptions
+of flow record structures), which are unique for combination of a Transport
+Session (e.g. connection to a flow exporter) and an ODID. As there is no session
+information in raw IPFIX Messages, we can't distinguish Messages from multiple
+sessions using the same ODID. For this reason, each ODID can be used only by
+one session at a time, and all messages from other sessions using the
+same ODID that's already in use are ignored and a warning message is shown.
+Once a Transport Session is closed, ODIDs used by the session are released
+and can be later reused by another session.
 
 Example configuration
 ---------------------
@@ -22,9 +37,9 @@ Example configuration
         <name>IPFIX output</name>
         <plugin>ipfix</plugin>
         <params>
-            <filename>/tmp/ipfix/%d-%m-%y/%H:%M:%S.ipfix</filename>
+            <filename>/tmp/ipfix/%Y%m%d%H%M%S.ipfix</filename>
             <useLocalTime>false</useLocalTime>
-            <windowSize>60</windowSize>
+            <windowSize>300</windowSize>
             <alignWindows>true</alignWindows>
             <skipUnknownDataSets>true</skipUnknownDataSets>
             <splitOnExportTime>false</splitOnExportTime>
@@ -35,19 +50,42 @@ Parameters
 ----------
 
 :``filename``:
-    The output filename, allows the use of strftime format values to add the export time into the file path.
+    Specifies an absolute path of output files. The path should contain format
+    specifier for day, month, etc. For example "/tmp/ipfix/%Y%m%d%H%M%S.ipfix".
+    See ``strftime``\ (3) for more information.
 
 :``useLocalTime``:
-    Specifies whether the time values in filenames should be in local time instead of UTC. [default: false]
+    Specifies whether the time values in filenames should be in local time
+    instead of UTC. [default: false]
 
 :``windowSize``:
-    The number of seconds before a new file is created. The value of 0 means the file will never split. [default: 0]
+    Specifies the time interval in seconds to rotate files. If the value
+    is "0", all flow will be stored into a single file. [default: 0]
 
 :``alignWindows``:
-    Specifies whether the file should only be split on multiples of windowSize. [default: true]
+    Align file rotation with next N minute interval (true/false). [default: true]
 
 :``skipUnknownDataSets``:
-    Specifies whether data sets with missing template should be left out from the output file. Sequence numbers and message lengths will be adjusted accordingly. [default: false]
+    Specifies whether Data Sets with unknown (Options) Template should be left
+    out from the output file. Sequence numbers and message lengths will be
+    adjusted accordingly. [default: false]
 
 :``splitOnExportTime``:
-    Specifies whether files should be split based on export time instead of system time. [default: false]
+    Specifies whether files should be rotated based on IPFIX Export Time
+    (i.e. timestamp from IPFIX Message header) instead of system time.
+    Warning: If the plugin receives flow records from multiple exporters at time
+    the rotation could be unsteady. [default: false]
+
+Note
+----
+
+After a new IPFIX File is created, all previously seen and still valid (Options)
+Templates of the each ODID are written to the file once the first IPFIX Message
+corresponding to the ODID arrives with at least one successfully parsed data record.
+This is necessary so each file can be used independently of the (Options) Template
+definitions from previous files.
+
+``ipfixsend2`` tool doesn't support sequential reading of multiple IPFIX Files
+right now. However, there is a workaround - you can merge multiple IPFIX Files
+using cat tool e.g. ``cat file1.ipfix file2.ipfix > merge.ipfix`` and then use
+``ipfixsend2 -i merge.ipfix`` to send data.

src/plugins/output/ipfix/src/Config.cpp

@@ -1,7 +1,7 @@
 /**
  * \file src/plugins/output/ipfix/src/Config.cpp
  * \author Michal Sedlak <[email protected]>
- * \brief Config for ipfix output plugin
+ * \brief Config for IPFIX output plugin
  * \date 2019
  */
 
@@ -44,6 +44,7 @@
 #include <stdexcept>
 #include <memory>
 
+/// XML nodes
 enum params_xml_nodes {
     PARAM_FILENAME,
     PARAM_USE_LOCALTIME,
@@ -53,14 +54,15 @@ enum params_xml_nodes {
     PARAM_SPLIT_ON_EXPORT_TIME
 };
 
+/// Description of XML document
 static const struct fds_xml_args args_params[] = {
     FDS_OPTS_ROOT("params"),
-    FDS_OPTS_ELEM(PARAM_FILENAME, "filename", FDS_OPTS_T_STRING, 0),
+    FDS_OPTS_ELEM(PARAM_FILENAME,      "filename",     FDS_OPTS_T_STRING, 0),
     FDS_OPTS_ELEM(PARAM_USE_LOCALTIME, "useLocalTime", FDS_OPTS_T_BOOL, FDS_OPTS_P_OPT),
-    FDS_OPTS_ELEM(PARAM_WINDOW_SIZE, "windowSize", FDS_OPTS_T_UINT, FDS_OPTS_P_OPT),
+    FDS_OPTS_ELEM(PARAM_WINDOW_SIZE,   "windowSize",   FDS_OPTS_T_UINT, FDS_OPTS_P_OPT),
     FDS_OPTS_ELEM(PARAM_ALIGN_WINDOWS, "alignWindows", FDS_OPTS_T_BOOL, FDS_OPTS_P_OPT),
     FDS_OPTS_ELEM(PARAM_SKIP_UNKNOWN_DATASETS, "skipUnknownDataSets", FDS_OPTS_T_BOOL, FDS_OPTS_P_OPT),
-    FDS_OPTS_ELEM(PARAM_SPLIT_ON_EXPORT_TIME, "splitOnExportTime", FDS_OPTS_T_BOOL, FDS_OPTS_P_OPT),
+    FDS_OPTS_ELEM(PARAM_SPLIT_ON_EXPORT_TIME,  "splitOnExportTime",   FDS_OPTS_T_BOOL, FDS_OPTS_P_OPT),
     FDS_OPTS_END
 };
 
@@ -108,8 +110,8 @@ void Config::parse_params(fds_xml_ctx_t *params)
     }
 }
 
-void 
-Config::check_validity() 
+void
+Config::check_validity()
 {
     if (filename.empty()) {
         throw std::invalid_argument("Filename cannot be empty!");

src/plugins/output/ipfix/src/Config.hpp

@@ -47,22 +47,45 @@
 #include <ipfixcol2.h>
 #include <libfds.h>
 
+/// Plugin configuration
 class Config {
-
 private:
+    /**
+     * @brief Set default configuration parameters
+     */
     void set_defaults();
+    /**
+     * @brief Parse XML tree
+     * @param[in] params XML tree
+     * @throws invalid_argument if unexpected node is detected
+     */
     void parse_params(fds_xml_ctx_t *params);
+    /**
+     * @brief Check validity of configuration
+     */
     void check_validity();
 
 public:
+    /// Output file pattern
     std::string filename;
+    /// Use local time instead of UTC time
     bool use_localtime;
+    /// Time interval to rotate files (in seconds)
     uint64_t window_size;
+    /// Rotate files on multiple of time interval
     bool align_windows;
+    /// Skip Data Sets with undefined templates
     bool skip_unknown_datasets;
+    /// Split on IPFIX Export Time instead on system time
     bool split_on_export_time;
 
+    /**
+     * @brief Parse configuration of the IPFIX plugin
+     * @param[in] params Plugin part of the configuration of the collector
+     * @throws runtime_error on error or invalid configuration
+     */
     Config(const char *params);
+    /// Configuration destructor
     ~Config();
 };