Merge pull request #913 from axoflow/version/4.22.0

Version: 4.22.0

Author: alltilla

NEWS.md

@@ -1,4 +1,4 @@
-4.21.0
+4.22.0
 ======
 
 AxoSyslog is binary-compatible with syslog-ng [1] and serves as a drop-in replacement.
@@ -9,72 +9,61 @@ Packages are available in our [APT](https://github.com/axoflow/axosyslog/#deb-pa
 
 Check out the [AxoSyslog documentation](https://axoflow.com/docs/axosyslog-core/) for all the details.
 
+
 ## Features
 
-  * `format_syslog_5424()`: Added new FilterX function for syslog formatting
+  * FilterX `in` operator: Added support for `dict` keys for membership check.
 
-    Usage:
     ```
-    format_syslog_5424(
-      message,
-      add_octet_count=false,
-      pri=expr,
-      timestamp=expr,
-      host=expr,
-      program=expr,
-      pid=expr,
-      msgid=expr
-    )
+    my_dict = {"foo": "foovalue", "bar": "barvalue"};
+    my_needle = "foobar";
+
+    if (my_needle in my_dict) {
+      $MSG = "Found: " + my_dict[my_needle];
+    } else {
+      $MSG = "Not Found";
+    };
     ```
+    ([#888](https://github.com/axoflow/axosyslog/pull/888))
 
-    `message` is the only mandatory argument.
-
-    Fallback values will be used instead of the named arguments
-    if they are not set, or their evaluation fails.
-    ([#875](https://github.com/axoflow/axosyslog/pull/875))
 
-  * `http()` and other threaded destinations: add `worker-partition-autoscaling(yes)`
+## Bugfixes
 
-    When `worker-partition-key()` is used to categorize messages into different batches,
-    the messages are - by default - hashed into workers, which prevents them from being distributed across workers
-    efficiently, based on load.
+  * `parallelize()`: Fixed occasional crashes on high load.
+    ([#904](https://github.com/axoflow/axosyslog/pull/904))
 
-    The new `worker-partition-autoscaling(yes)` option uses a 1-minute statistic to help distribute
-    high-traffic partitions among multiple workers, allowing each worker to maximize its batch size.
+  * `parallelize()`: Fixed unoptimized parallelization with more `workers()` than CPU cores.
+    ([#906](https://github.com/axoflow/axosyslog/pull/906))
 
-    When using this autoscaling option, it is recommended to oversize the number of workers: set it higher than the
-    expected number of partitions.
-    ([#855](https://github.com/axoflow/axosyslog/pull/855))
+  * FilterX `regexp_subst()` function: Fixed capture group references in the replacement argument.
 
-  * `network()`: add `transport(nul-terminated)` to support NULL characters to separate log records instead of the
-    more traditional newline separated format
-    ([#867](https://github.com/axoflow/axosyslog/pull/867))
+    In the case of `global=true`, the value of capture group references were always used from
+    that of the first match, e.g. the 2nd and subseqent matches used an incorrect value.
+    ([#895](https://github.com/axoflow/axosyslog/pull/895))
 
-  * New metrics: `syslogng_output_workers` and `syslogng_output_active_worker_partitions`
+  * `disk-buffer()`: Fixed various bugs.
 
-    Using the new `worker-partition-autoscaling(yes)` option allows producing partition metrics, which can be used
-    for alerting: if the number of active partitions remains higher than the configured number of workers,
-    it may indicate that events are not being batched properly, which can lead to performance degradation.
-    ([#866](https://github.com/axoflow/axosyslog/pull/866))
+    * Fixed potential writes beyond the configured front-cache limit.
+    * Fixed a possible issue where memory usage metrics could reset on reload.
 
-## Bugfixes
+    ([#901](https://github.com/axoflow/axosyslog/pull/901))
 
-  * FilterX `parse_csv()`: fix crash
-    ([#879](https://github.com/axoflow/axosyslog/pull/879))
 
-  * FilterX `metrics_labels()`: fix `+=` operator
-    ([#878](https://github.com/axoflow/axosyslog/pull/878))
+## Other changes
 
-  * `disk-buffer()`: fix memory leaks
-    ([#872](https://github.com/axoflow/axosyslog/pull/872))
+  * `disk-buffer()`: Various smaller improvements and QoL features.
 
-## Other changes
+    * Added detailed debug logging for diskbuffer load and save operations,
+      exposing the internal state of the non-reliable diskbuffer.
+    * Improved front-cache balancing by enforcing the front-cache limit when
+      distributing items between front-cache and front-cache-output.
+    * Improved calculation of abandoned diskbuffer metrics by using existing
+      file header data instead of loading the entire buffer file,
+      significantly reducing reload time impact.
 
-  * `disk-buffer()`: significant performance improvements for the non-reliable disk buffer
-    ([#857](https://github.com/axoflow/axosyslog/pull/857))
+    ([#901](https://github.com/axoflow/axosyslog/pull/901))
+    ([#902](https://github.com/axoflow/axosyslog/pull/902))
 
-  * Performance improvements for memory queues
-    ([#881](https://github.com/axoflow/axosyslog/pull/881))
 
 [1] syslog-ng is a trademark of One Identity.
 
@@ -95,5 +84,5 @@ of AxoSyslog, contribute.
 
 We would like to thank the following people for their contribution:
 
-Andras Mitzki, Attila Szakacs, Balazs Scheidler, László Várady, Szilard Parrag,
-Tamás Kosztyu, shifter
+Andras Mitzki, Attila Szakacs, Balazs Scheidler,
+BenBryzak-brisbaneqldgovau, László Várady, Szilard Parrag, shifter

README.md

@@ -45,7 +45,7 @@ A simple example is to ingest syslog traffic on tcp/514 and write it to a
 file:
 
 ```
-@version: 4.21
+@version: 4.22
 @include "scl.conf"
 
 log {
@@ -85,7 +85,7 @@ docker pull ghcr.io/axoflow/axosyslog:nightly
 > Note: These named packages are automatically updated when a new AxoSyslog package is released. To install a specific version, run `docker pull ghcr.io/axoflow/axosyslog:<version-number>`, for example:
 >
 > ```shell
-> docker pull ghcr.io/axoflow/axosyslog:4.21.0
+> docker pull ghcr.io/axoflow/axosyslog:4.22.0
 > ```
 
 The container images contain a default configuration file which you probably

VERSION.txt

@@ -1 +1 @@
-4.21.0
+4.22.0

contrib/openbsd-packaging/syslog-ng.conf

@@ -2,7 +2,7 @@
 # This should provide behavior similar to OpenBSD's syslog.conf(5).
 # 2021-05-24 millert@openbsd.org
 
-@version: 4.21
+@version: 4.22
 @requires openbsd
 
 options { 

doc/man/dqtool.1.xml

@@ -30,7 +30,7 @@
     <refmeta>
       <refentrytitle>dqtool</refentrytitle>
       <manvolnum>1</manvolnum>
-      <refmiscinfo class="version">4.21</refmiscinfo>
+      <refmiscinfo class="version">4.22</refmiscinfo>
       <refmiscinfo class="source"/>
     </refmeta>
     <refnamediv>
@@ -117,7 +117,7 @@ Mar  3 10:52:05 tristram localprg[1234]: seq: 0000011631, runid: 1267609923, sta
         <link linkend="syslog-ng.8"><command>syslog-ng</command>(8)</link>
       </para>
       <note version="5.0">
-        <para>For the detailed documentation of  see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.37/administration-guide"><command>The  4.21 Administrator Guide</command></link></para>
+        <para>For the detailed documentation of  see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.37/administration-guide"><command>The  4.22 Administrator Guide</command></link></para>
         <para>If you experience any problems or need help with syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"><command>syslog-ng mailing list</command></link>.</para>
         <para>For news and notifications about of syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://syslog-ng.org/blogs/"><command>syslog-ng blogs</command></link>.</para>
       </note>

doc/man/loggen.1.xml

@@ -30,7 +30,7 @@
     <refmeta>
       <refentrytitle>loggen</refentrytitle>
       <manvolnum>1</manvolnum>
-      <refmiscinfo class="version">4.21</refmiscinfo>
+      <refmiscinfo class="version">4.22</refmiscinfo>
       <refmiscinfo class="source"/>
     </refmeta>
     <refnamediv>
@@ -280,7 +280,7 @@
         <link linkend="syslog-ng.conf.5"><command>syslog-ng.conf</command>(5)</link>
       </para>
       <note version="5.0">
-        <para>For the detailed documentation of  see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.37/administration-guide"><command>The  4.21 Administrator Guide</command></link></para>
+        <para>For the detailed documentation of  see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.37/administration-guide"><command>The  4.22 Administrator Guide</command></link></para>
         <para>If you experience any problems or need help with syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"><command>syslog-ng mailing list</command></link>.</para>
         <para>For news and notifications about of syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://syslog-ng.org/blogs/"><command>syslog-ng blogs</command></link>.</para>
       </note>

doc/man/pdbtool.1.xml

@@ -30,7 +30,7 @@
     <refmeta>
       <refentrytitle>pdbtool</refentrytitle>
       <manvolnum>1</manvolnum>
-      <refmiscinfo class="version">4.21</refmiscinfo>
+      <refmiscinfo class="version">4.22</refmiscinfo>
       <refmiscinfo class="source"/>
     </refmeta>
     <refnamediv>
@@ -377,7 +377,7 @@
                     </term>
           <listitem>
             <para>A pattern is added to the output pattern database if at least the specified percentage of log messages from the input logfile match the pattern. For example, if the input logfile contains 1000 log messages and the <parameter>--support=3.0</parameter> option is used, a pattern is created only if the pattern matches at least 3 percent of the log messages (that is, 30 log messages). If patternize does not create enough patterns, try to decrease the support value.</para>
-            <para>Default value: <parameter>4.21</parameter></para>
+            <para>Default value: <parameter>4.22</parameter></para>
           </listitem>
         </varlistentry>
         <varlistentry version="5.0">
@@ -467,7 +467,7 @@
         <link linkend="syslog-ng.8"><command>syslog-ng</command>(8)</link>
       </para>
       <note version="5.0">
-        <para>For the detailed documentation of  see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/index.html"><command>The  4.21 Administrator Guide</command></link></para>
+        <para>For the detailed documentation of  see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/index.html"><command>The  4.22 Administrator Guide</command></link></para>
         <para>If you experience any problems or need help with syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"><command>syslog-ng mailing list</command></link>.</para>
         <para>For news and notifications about of syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://syslog-ng.org/blogs/"><command>syslog-ng blogs</command></link>.</para>
       </note>

doc/man/persist-tool.1.xml

@@ -30,7 +30,7 @@
         <refmeta>
             <refentrytitle>persist-tool</refentrytitle>
             <manvolnum>1</manvolnum>
-            <refmiscinfo class="version">4.21</refmiscinfo>
+            <refmiscinfo class="version">4.22</refmiscinfo>
             <refmiscinfo class="source"/>
         </refmeta>
         <refnamediv>
@@ -130,7 +130,7 @@ affile_sd_curpos(/var/aaa.txt)        OK</synopsis>
             <para><link linkend="syslog-ng.conf.5"> <command>syslog-ng.conf</command>(5)</link></para>
             <para><link linkend="syslog-ng.8"> <command>syslog-ng</command>(8)</link></para>
             <note version="5.0">
-                <para>For the detailed documentation of  see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/index.html"><command>The  4.21 Administrator Guide</command></link></para>
+                <para>For the detailed documentation of  see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/index.html"><command>The  4.22 Administrator Guide</command></link></para>
                 <para>If you experience any problems or need help with syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"><command>syslog-ng mailing list</command></link>.</para>
                 <para>For news and notifications about of syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://syslog-ng.org/blogs/"><command>syslog-ng blogs</command></link>.</para>
             </note>

doc/man/secure-logging.7.xml

@@ -30,7 +30,7 @@
     <refmeta>
       <refentrytitle>secure-logging</refentrytitle>
       <manvolnum>7</manvolnum>
-      <refmiscinfo class="version">4.21</refmiscinfo>
+      <refmiscinfo class="version">4.22</refmiscinfo>
       <refmiscinfo class="source"/>
     </refmeta>
     <refnamediv>
@@ -190,7 +190,7 @@ AgAAAAAAAAA=:5UVybnKL1EAbgC4CLfd8HpgurjREf4LEN61/yWHSD2hbXjRD4QmQdtbwguT1chzdItK
 # entries will be logged to a single file called /var/log/messages.slog
 #
 
-@version: 4.21
+@version: 4.22
 @include "scl.conf"
 
 source s_local {

doc/man/slogencrypt.1.xml

@@ -30,7 +30,7 @@
     <refmeta>
       <refentrytitle>slogencrypt</refentrytitle>
       <manvolnum>1</manvolnum>
-      <refmiscinfo class="version">4.21</refmiscinfo>
+      <refmiscinfo class="version">4.22</refmiscinfo>
       <refmiscinfo class="source"/>
     </refmeta>
     <refnamediv>