Merge pull request #885 from axoflow/version/4.21.0

Author: MrAnno

NEWS.md

@@ -1,8 +1,6 @@
-4.20.1
+4.21.0
 ======
 
-_This is a hotfix release that fixes a rare metric race condition causing a reload time crash._
-
 AxoSyslog is binary-compatible with syslog-ng [1] and serves as a drop-in replacement.
 
 We provide [cloud-ready container images](https://github.com/axoflow/axosyslog/#container-images) and Helm charts.
@@ -11,147 +9,72 @@ Packages are available in our [APT](https://github.com/axoflow/axosyslog/#deb-pa
 
 Check out the [AxoSyslog documentation](https://axoflow.com/docs/axosyslog-core/) for all the details.
 
-
 ## Features
 
-  * Added new metrics
-
-    * `syslogng_window_capacity` and `syslogng_window_available` on `stats(level(3))`.
-      * Shows the `log-iw-size()` value and the current state of the source window, respectively.
-    * `syslogng_window_full_total`
-      * Tracks how many times the window was completely full. This counter will
-        increase any time the destination causes the source to be throttled.
-    * `syslogng_memory_queue_processed_events_total` and `syslogng_disk_queue_processed_events_total`
-      * Counts the number of events processed since startup by each queue.
-    * `syslogng_output_batch_size_...`, `syslogng_output_event_size_...`, `syslogng_output_request_latency_...`
-      * histogram style metrics for `http()`, `otel()` and other threaded destinations
-
-    ([#823](https://github.com/axoflow/axosyslog/pull/823))
-    ([#824](https://github.com/axoflow/axosyslog/pull/824))
-    ([#845](https://github.com/axoflow/axosyslog/pull/845))
-
-  * `clickhouse()` destination: Added JSONCompactEachRow format and new `format` directive
-
-    This update enhances the ClickHouse destination by adding support for the `JSONCompactEachRow`
-    format and introducing a new `format` directive for explicitly selecting the data format.
-
-    **Background**
-    Previously, the destination supported:
-    - `Protobuf` (default when using `proto-var`)
-    - `JSONEachRow` (default when using `json-var`)
-
-    These defaults remain unchanged.
-
-    **What’s new**
-    - Added support for `JSONCompactEachRow` — a more compact, array-based JSON representation (used with `json-var`).
-    - Introduced the `format` directive, allowing manual selection of the desired format:
-      - `JSONEachRow`
-      - `JSONCompactEachRow`
-      - `Protobuf`
-
-    **Example**
-    ```hcl
-    destination {
-      clickhouse (
-        ...
-        json-var(json("$my_filterx_json_var"))
-        format("JSONCompactEachRow")
-        ...
-      );
-    };
-    ```
-
-    **JSONEachRow (each JSON object per line, more readable):**
-
-    ```
-    {"id":1,"name":"foo","value":42}
-    {"id":2,"name":"bar","value":17}
-    ```
-
-    **JSONCompactEachRow (compact array-based row representation):**
-
-    ```
-    [1,"foo",42]
-    [2,"bar",17]
-    ```
-
-    **Validation and error handling**
+  * `format_syslog_5424()`: Added new FilterX function for syslog formatting
 
-    **Invalid format values now produce:**
+    Usage:
     ```
-    Error parsing within destination: invalid format value 'invalid format', possible values:[JSONEachRow, JSONCompactEachRow, Protobuf]
+    format_syslog_5424(
+      message,
+      add_octet_count=false,
+      pri=expr,
+      timestamp=expr,
+      host=expr,
+      program=expr,
+      pid=expr,
+      msgid=expr
+    )
     ```
 
-    **If the data’s actual format doesn’t match the selected format, ClickHouse returns:**
-    ```
-    CANNOT_PARSE_INPUT_ASSERTION_FAILED
-    ```
-    ([#828](https://github.com/axoflow/axosyslog/pull/828))
-
-  * `opentelemetry()` source: Added `keep-alive()` option
+    `message` is the only mandatory argument.
 
-    With this new option, client connections can be kept alive during reload,
-    avoiding unnecessary retry backoffs and other error messages on the client
-    side.
+    Fallback values will be used instead of the named arguments
+    if they are not set, or their evaluation fails.
+    ([#875](https://github.com/axoflow/axosyslog/pull/875))
 
-    The default is `yes`.
+  * `http()` and other threaded destinations: add `worker-partition-autoscaling(yes)`
 
-    ([#832](https://github.com/axoflow/axosyslog/pull/832))
+    When `worker-partition-key()` is used to categorize messages into different batches,
+    the messages are - by default - hashed into workers, which prevents them from being distributed across workers
+    efficiently, based on load.
 
-  * `s3()` destination: Added new `object_key_suffix()` option
+    The new `worker-partition-autoscaling(yes)` option uses a 1-minute statistic to help distribute
+    high-traffic partitions among multiple workers, allowing each worker to maximize its batch size.
 
-    The default suffix is an empty string, to ensure backward compatibility.
-    ([#797](https://github.com/axoflow/axosyslog/pull/797))
+    When using this autoscaling option, it is recommended to oversize the number of workers: set it higher than the
+    expected number of partitions.
+    ([#855](https://github.com/axoflow/axosyslog/pull/855))
 
-  * `http()` and other threaded destinations: add `worker-partition-buckets()` option
+  * `network()`: add `transport(nul-terminated)` to support NULL characters to separate log records instead of the
+    more traditional newline separated format
+    ([#867](https://github.com/axoflow/axosyslog/pull/867))
 
-    This allows the same `worker-partition-key()` to use multiple worker threads.
-    ([#852](https://github.com/axoflow/axosyslog/pull/852))
+  * New metrics: `syslogng_output_workers` and `syslogng_output_active_worker_partitions`
 
+    Using the new `worker-partition-autoscaling(yes)` option allows producing partition metrics, which can be used
+    for alerting: if the number of active partitions remains higher than the configured number of workers,
+    it may indicate that events are not being batched properly, which can lead to performance degradation.
+    ([#866](https://github.com/axoflow/axosyslog/pull/866))
 
 ## Bugfixes
 
-  * `metrics`: Fixed a rare race condition in dynamic metrics.
-    ([#858](https://github.com/axoflow/axosyslog/pull/858))
-
-  * `filterx`: Fixed various memory leaks
-    ([#829](https://github.com/axoflow/axosyslog/pull/829))
-    ([#836](https://github.com/axoflow/axosyslog/pull/836))
-    ([#842](https://github.com/axoflow/axosyslog/pull/842))
-
-  * `filterx`: Fixed a variable synchronization bug.
-    ([#849](https://github.com/axoflow/axosyslog/pull/849))
-
-  * `filterx` `otel_logrecord()`: Fixed not clearing `body` before setting dict and array values.
-    ([#835](https://github.com/axoflow/axosyslog/pull/835))
-
-  * Fixed `keep-alive()` during config reload revert
-    ([#831](https://github.com/axoflow/axosyslog/pull/831))
-
-  * `http()` destination: Fixed batch partitioning in case of templated `body-prefix()`
-    ([#843](https://github.com/axoflow/axosyslog/pull/843))
-
-  * `s3`: Bugfixes and general stability improvements for the `s3` destination driver
-
-    * Fixed a major bug causing data loss if multithreaded upload was enabled via the `upload-threads` option.
-    * Fixed a bug where in certain conditions finished object buffers would fail to upload.
-    * Fixed a bug, where empty chunks were being uploaded, causing errors.
-
-    ([#797](https://github.com/axoflow/axosyslog/pull/797))
-    ([#846](https://github.com/axoflow/axosyslog/pull/846))
-
-  * `filterx`: fix potential use-after-free crashes
-    ([#854](https://github.com/axoflow/axosyslog/pull/854))
+  * FilterX `parse_csv()`: fix crash
+    ([#879](https://github.com/axoflow/axosyslog/pull/879))
 
-  * `disk-buffer()`: fix memory leak when `worker-partition-key()` and `disk-buffer()` are used together
-    ([#853](https://github.com/axoflow/axosyslog/pull/853))
+  * FilterX `metrics_labels()`: fix `+=` operator
+    ([#878](https://github.com/axoflow/axosyslog/pull/878))
 
+  * `disk-buffer()`: fix memory leaks
+    ([#872](https://github.com/axoflow/axosyslog/pull/872))
 
 ## Other changes
 
-  * Improved reload time for large configurations.
-    ([#844](https://github.com/axoflow/axosyslog/pull/844))
+  * `disk-buffer()`: significant performance improvements for the non-reliable disk buffer
+    ([#857](https://github.com/axoflow/axosyslog/pull/857))
 
+  * Performance improvements for memory queues
+    ([#881](https://github.com/axoflow/axosyslog/pull/881))
 
 [1] syslog-ng is a trademark of One Identity.
 
@@ -172,5 +95,5 @@ of AxoSyslog, contribute.
 
 We would like to thank the following people for their contribution:
 
-Andras Mitzki, Attila Szakacs, Balazs Scheidler, Bálint Horváth,
-László Várady, Szilard Parrag, Tamás Kosztyu, shifter
+Andras Mitzki, Attila Szakacs, Balazs Scheidler, László Várady, Szilard Parrag,
+Tamás Kosztyu, shifter

README.md

@@ -45,7 +45,7 @@ A simple example is to ingest syslog traffic on tcp/514 and write it to a
 file:
 
 ```
-@version: 4.20
+@version: 4.21
 @include "scl.conf"
 
 log {
@@ -85,7 +85,7 @@ docker pull ghcr.io/axoflow/axosyslog:nightly
 > Note: These named packages are automatically updated when a new AxoSyslog package is released. To install a specific version, run `docker pull ghcr.io/axoflow/axosyslog:<version-number>`, for example:
 >
 > ```shell
-> docker pull ghcr.io/axoflow/axosyslog:4.20.1
+> docker pull ghcr.io/axoflow/axosyslog:4.21.0
 > ```
 
 The container images contain a default configuration file which you probably

VERSION.txt

@@ -1 +1 @@
-4.20.1
+4.21.0

contrib/openbsd-packaging/syslog-ng.conf

@@ -2,7 +2,7 @@
 # This should provide behavior similar to OpenBSD's syslog.conf(5).
 # 2021-05-24 millert@openbsd.org
 
-@version: 4.20
+@version: 4.21
 @requires openbsd
 
 options { 

doc/man/dqtool.1.xml

@@ -30,7 +30,7 @@
     <refmeta>
       <refentrytitle>dqtool</refentrytitle>
       <manvolnum>1</manvolnum>
-      <refmiscinfo class="version">4.20</refmiscinfo>
+      <refmiscinfo class="version">4.21</refmiscinfo>
       <refmiscinfo class="source"/>
     </refmeta>
     <refnamediv>
@@ -117,7 +117,7 @@ Mar  3 10:52:05 tristram localprg[1234]: seq: 0000011631, runid: 1267609923, sta
         <link linkend="syslog-ng.8"><command>syslog-ng</command>(8)</link>
       </para>
       <note version="5.0">
-        <para>For the detailed documentation of  see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.37/administration-guide"><command>The  4.20 Administrator Guide</command></link></para>
+        <para>For the detailed documentation of  see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.37/administration-guide"><command>The  4.21 Administrator Guide</command></link></para>
         <para>If you experience any problems or need help with syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"><command>syslog-ng mailing list</command></link>.</para>
         <para>For news and notifications about of syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://syslog-ng.org/blogs/"><command>syslog-ng blogs</command></link>.</para>
       </note>

doc/man/loggen.1.xml

@@ -30,7 +30,7 @@
     <refmeta>
       <refentrytitle>loggen</refentrytitle>
       <manvolnum>1</manvolnum>
-      <refmiscinfo class="version">4.20</refmiscinfo>
+      <refmiscinfo class="version">4.21</refmiscinfo>
       <refmiscinfo class="source"/>
     </refmeta>
     <refnamediv>
@@ -280,7 +280,7 @@
         <link linkend="syslog-ng.conf.5"><command>syslog-ng.conf</command>(5)</link>
       </para>
       <note version="5.0">
-        <para>For the detailed documentation of  see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.37/administration-guide"><command>The  4.20 Administrator Guide</command></link></para>
+        <para>For the detailed documentation of  see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.37/administration-guide"><command>The  4.21 Administrator Guide</command></link></para>
         <para>If you experience any problems or need help with syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"><command>syslog-ng mailing list</command></link>.</para>
         <para>For news and notifications about of syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://syslog-ng.org/blogs/"><command>syslog-ng blogs</command></link>.</para>
       </note>

doc/man/pdbtool.1.xml

@@ -30,7 +30,7 @@
     <refmeta>
       <refentrytitle>pdbtool</refentrytitle>
       <manvolnum>1</manvolnum>
-      <refmiscinfo class="version">4.20</refmiscinfo>
+      <refmiscinfo class="version">4.21</refmiscinfo>
       <refmiscinfo class="source"/>
     </refmeta>
     <refnamediv>
@@ -377,7 +377,7 @@
                     </term>
           <listitem>
             <para>A pattern is added to the output pattern database if at least the specified percentage of log messages from the input logfile match the pattern. For example, if the input logfile contains 1000 log messages and the <parameter>--support=3.0</parameter> option is used, a pattern is created only if the pattern matches at least 3 percent of the log messages (that is, 30 log messages). If patternize does not create enough patterns, try to decrease the support value.</para>
-            <para>Default value: <parameter>4.20</parameter></para>
+            <para>Default value: <parameter>4.21</parameter></para>
           </listitem>
         </varlistentry>
         <varlistentry version="5.0">
@@ -467,7 +467,7 @@
         <link linkend="syslog-ng.8"><command>syslog-ng</command>(8)</link>
       </para>
       <note version="5.0">
-        <para>For the detailed documentation of  see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/index.html"><command>The  4.20 Administrator Guide</command></link></para>
+        <para>For the detailed documentation of  see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/index.html"><command>The  4.21 Administrator Guide</command></link></para>
         <para>If you experience any problems or need help with syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"><command>syslog-ng mailing list</command></link>.</para>
         <para>For news and notifications about of syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://syslog-ng.org/blogs/"><command>syslog-ng blogs</command></link>.</para>
       </note>

doc/man/persist-tool.1.xml

@@ -30,7 +30,7 @@
         <refmeta>
             <refentrytitle>persist-tool</refentrytitle>
             <manvolnum>1</manvolnum>
-            <refmiscinfo class="version">4.20</refmiscinfo>
+            <refmiscinfo class="version">4.21</refmiscinfo>
             <refmiscinfo class="source"/>
         </refmeta>
         <refnamediv>
@@ -130,7 +130,7 @@ affile_sd_curpos(/var/aaa.txt)        OK</synopsis>
             <para><link linkend="syslog-ng.conf.5"> <command>syslog-ng.conf</command>(5)</link></para>
             <para><link linkend="syslog-ng.8"> <command>syslog-ng</command>(8)</link></para>
             <note version="5.0">
-                <para>For the detailed documentation of  see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/index.html"><command>The  4.20 Administrator Guide</command></link></para>
+                <para>For the detailed documentation of  see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/index.html"><command>The  4.21 Administrator Guide</command></link></para>
                 <para>If you experience any problems or need help with syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"><command>syslog-ng mailing list</command></link>.</para>
                 <para>For news and notifications about of syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://syslog-ng.org/blogs/"><command>syslog-ng blogs</command></link>.</para>
             </note>

doc/man/secure-logging.7.xml

@@ -30,7 +30,7 @@
     <refmeta>
       <refentrytitle>secure-logging</refentrytitle>
       <manvolnum>7</manvolnum>
-      <refmiscinfo class="version">4.20</refmiscinfo>
+      <refmiscinfo class="version">4.21</refmiscinfo>
       <refmiscinfo class="source"/>
     </refmeta>
     <refnamediv>
@@ -190,7 +190,7 @@ AgAAAAAAAAA=:5UVybnKL1EAbgC4CLfd8HpgurjREf4LEN61/yWHSD2hbXjRD4QmQdtbwguT1chzdItK
 # entries will be logged to a single file called /var/log/messages.slog
 #
 
-@version: 4.20
+@version: 4.21
 @include "scl.conf"
 
 source s_local {

doc/man/slogencrypt.1.xml

@@ -30,7 +30,7 @@
     <refmeta>
       <refentrytitle>slogencrypt</refentrytitle>
       <manvolnum>1</manvolnum>
-      <refmiscinfo class="version">4.20</refmiscinfo>
+      <refmiscinfo class="version">4.21</refmiscinfo>
       <refmiscinfo class="source"/>
     </refmeta>
     <refnamediv>