Merge pull request #847 from axoflow/version/4.20.0

Version: 4.20.0

Author: MrAnno

NEWS.md

@@ -1,8 +1,6 @@
-4.19.1
+4.20.0
 ======
 
-_This is a hotfix release that fixes a config reload time memory leak_
-
 AxoSyslog is binary-compatible with syslog-ng [1] and serves as a drop-in replacement.
 
 We provide [cloud-ready container images](https://github.com/axoflow/axosyslog/#container-images) and Helm charts.
@@ -12,11 +10,142 @@ Packages are available in our [APT](https://github.com/axoflow/axosyslog/#deb-pa
 Check out the [AxoSyslog documentation](https://axoflow.com/docs/axosyslog-core/) for all the details.
 
 
+## Features
+
+  * Added new metrics
+
+    * `syslogng_window_capacity` and `syslogng_window_available` on `stats(level(3))`.
+      * Shows the `log-iw-size()` value and the current state of the source window, respectively.
+    * `syslogng_window_full_total`
+      * Tracks how many times the window was completely full. This counter will
+        increase any time the destination causes the source to be throttled.
+    * `syslogng_memory_queue_processed_events_total` and `syslogng_disk_queue_processed_events_total`
+      * Counts the number of events processed since startup by each queue.
+    * `syslogng_output_batch_size_...`, `syslogng_output_event_size_...`, `syslogng_output_request_latency_...`
+      * histogram style metrics for `http()`, `otel()` and other threaded destinations
+
+    ([#823](https://github.com/axoflow/axosyslog/pull/823))
+    ([#824](https://github.com/axoflow/axosyslog/pull/824))
+    ([#845](https://github.com/axoflow/axosyslog/pull/845))
+
+  * `clickhouse()` destination: Added JSONCompactEachRow format and new `format` directive
+
+    This update enhances the ClickHouse destination by adding support for the `JSONCompactEachRow`
+    format and introducing a new `format` directive for explicitly selecting the data format.
+
+    **Background**
+    Previously, the destination supported:
+    - `Protobuf` (default when using `proto-var`)
+    - `JSONEachRow` (default when using `json-var`)
+
+    These defaults remain unchanged.
+
+    **What’s new**
+    - Added support for `JSONCompactEachRow` — a more compact, array-based JSON representation (used with `json-var`).
+    - Introduced the `format` directive, allowing manual selection of the desired format:
+      - `JSONEachRow`
+      - `JSONCompactEachRow`
+      - `Protobuf`
+
+    **Example**
+    ```hcl
+    destination {
+      clickhouse (
+        ...
+        json-var(json("$my_filterx_json_var"))
+        format("JSONCompactEachRow")
+        ...
+      );
+    };
+    ```
+
+    **JSONEachRow (each JSON object per line, more readable):**
+
+    ```
+    {"id":1,"name":"foo","value":42}
+    {"id":2,"name":"bar","value":17}
+    ```
+
+    **JSONCompactEachRow (compact array-based row representation):**
+
+    ```
+    [1,"foo",42]
+    [2,"bar",17]
+    ```
+
+    **Validation and error handling**
+
+    **Invalid format values now produce:**
+    ```
+    Error parsing within destination: invalid format value 'invalid format', possible values:[JSONEachRow, JSONCompactEachRow, Protobuf]
+    ```
+
+    **If the data’s actual format doesn’t match the selected format, ClickHouse returns:**
+    ```
+    CANNOT_PARSE_INPUT_ASSERTION_FAILED
+    ```
+    ([#828](https://github.com/axoflow/axosyslog/pull/828))
+
+  * `opentelemetry()` source: Added `keep-alive()` option
+
+    With this new option, client connections can be kept alive during reload,
+    avoiding unnecessary retry backoffs and other error messages on the client
+    side.
+
+    The default is `yes`.
+
+    ([#832](https://github.com/axoflow/axosyslog/pull/832))
+
+  * `s3()` destination: Added new `object_key_suffix()` option
+
+    The default suffix is an empty string, to ensure backward compatibility.
+    ([#797](https://github.com/axoflow/axosyslog/pull/797))
+
+  * `http()` and other threaded destinations: add `worker-partition-buckets()` option
+
+    This allows the same `worker-partition-key()` to use multiple worker threads.
+    ([#852](https://github.com/axoflow/axosyslog/pull/852))
+
+
 ## Bugfixes
 
-  * `filterx`: Fixed a memory leak that occurs during reloads.
-    ([#827](https://github.com/axoflow/axosyslog/pull/827))
+  * `filterx`: Fixed various memory leaks
+    ([#829](https://github.com/axoflow/axosyslog/pull/829))
+    ([#836](https://github.com/axoflow/axosyslog/pull/836))
+    ([#842](https://github.com/axoflow/axosyslog/pull/842))
+
+  * `filterx`: Fixed a variable synchronization bug.
+    ([#849](https://github.com/axoflow/axosyslog/pull/849))
+
+  * `filterx` `otel_logrecord()`: Fixed not clearing `body` before setting dict and array values.
+    ([#835](https://github.com/axoflow/axosyslog/pull/835))
+
+  * Fixed `keep-alive()` during config reload revert
+    ([#831](https://github.com/axoflow/axosyslog/pull/831))
+
+  * `http()` destination: Fixed batch partitioning in case of templated `body-prefix()`
+    ([#843](https://github.com/axoflow/axosyslog/pull/843))
+
+  * `s3`: Bugfixes and general stability improvements for the `s3` destination driver
+
+    * Fixed a major bug causing data loss if multithreaded upload was enabled via the `upload-threads` option.
+    * Fixed a bug where in certain conditions finished object buffers would fail to upload.
+    * Fixed a bug, where empty chunks were being uploaded, causing errors.
+
+    ([#797](https://github.com/axoflow/axosyslog/pull/797))
+    ([#846](https://github.com/axoflow/axosyslog/pull/846))
+
+  * `filterx`: fix potential use-after-free crashes
+    ([#854](https://github.com/axoflow/axosyslog/pull/854))
+
+  * `disk-buffer()`: fix memory leak when `worker-partition-key()` and `disk-buffer()` are used together
+    ([#853](https://github.com/axoflow/axosyslog/pull/853))
+
+
+## Other changes
 
+  * Improved reload time for large configurations.
+    ([#844](https://github.com/axoflow/axosyslog/pull/844))
 
 
 [1] syslog-ng is a trademark of One Identity.
@@ -38,5 +167,5 @@ of AxoSyslog, contribute.
 
 We would like to thank the following people for their contribution:
 
-Andras Mitzki, Attila Szakacs, Balazs Scheidler, László Várady,
-Szilard Parrag, Tamás Kosztyu, shifter
+Andras Mitzki, Attila Szakacs, Balazs Scheidler, Bálint Horváth,
+László Várady, Szilard Parrag, Tamás Kosztyu, shifter

README.md

@@ -45,7 +45,7 @@ A simple example is to ingest syslog traffic on tcp/514 and write it to a
 file:
 
 ```
-@version: 4.19
+@version: 4.20
 @include "scl.conf"
 
 log {
@@ -85,7 +85,7 @@ docker pull ghcr.io/axoflow/axosyslog:nightly
 > Note: These named packages are automatically updated when a new AxoSyslog package is released. To install a specific version, run `docker pull ghcr.io/axoflow/axosyslog:<version-number>`, for example:
 >
 > ```shell
-> docker pull ghcr.io/axoflow/axosyslog:4.19.1
+> docker pull ghcr.io/axoflow/axosyslog:4.20.0
 > ```
 
 The container images contain a default configuration file which you probably

VERSION.txt

@@ -1 +1 @@
-4.19.1
+4.20.0

contrib/openbsd-packaging/syslog-ng.conf

@@ -2,7 +2,7 @@
 # This should provide behavior similar to OpenBSD's syslog.conf(5).
 # 2021-05-24 millert@openbsd.org
 
-@version: 4.19
+@version: 4.20
 @requires openbsd
 
 options { 

doc/man/dqtool.1.xml

@@ -30,7 +30,7 @@
     <refmeta>
       <refentrytitle>dqtool</refentrytitle>
       <manvolnum>1</manvolnum>
-      <refmiscinfo class="version">4.19</refmiscinfo>
+      <refmiscinfo class="version">4.20</refmiscinfo>
       <refmiscinfo class="source"/>
     </refmeta>
     <refnamediv>
@@ -117,7 +117,7 @@ Mar  3 10:52:05 tristram localprg[1234]: seq: 0000011631, runid: 1267609923, sta
         <link linkend="syslog-ng.8"><command>syslog-ng</command>(8)</link>
       </para>
       <note version="5.0">
-        <para>For the detailed documentation of  see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.37/administration-guide"><command>The  4.19 Administrator Guide</command></link></para>
+        <para>For the detailed documentation of  see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.37/administration-guide"><command>The  4.20 Administrator Guide</command></link></para>
         <para>If you experience any problems or need help with syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"><command>syslog-ng mailing list</command></link>.</para>
         <para>For news and notifications about of syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://syslog-ng.org/blogs/"><command>syslog-ng blogs</command></link>.</para>
       </note>

doc/man/loggen.1.xml

@@ -30,7 +30,7 @@
     <refmeta>
       <refentrytitle>loggen</refentrytitle>
       <manvolnum>1</manvolnum>
-      <refmiscinfo class="version">4.19</refmiscinfo>
+      <refmiscinfo class="version">4.20</refmiscinfo>
       <refmiscinfo class="source"/>
     </refmeta>
     <refnamediv>
@@ -280,7 +280,7 @@
         <link linkend="syslog-ng.conf.5"><command>syslog-ng.conf</command>(5)</link>
       </para>
       <note version="5.0">
-        <para>For the detailed documentation of  see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.37/administration-guide"><command>The  4.19 Administrator Guide</command></link></para>
+        <para>For the detailed documentation of  see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.37/administration-guide"><command>The  4.20 Administrator Guide</command></link></para>
         <para>If you experience any problems or need help with syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"><command>syslog-ng mailing list</command></link>.</para>
         <para>For news and notifications about of syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://syslog-ng.org/blogs/"><command>syslog-ng blogs</command></link>.</para>
       </note>

doc/man/pdbtool.1.xml

@@ -30,7 +30,7 @@
     <refmeta>
       <refentrytitle>pdbtool</refentrytitle>
       <manvolnum>1</manvolnum>
-      <refmiscinfo class="version">4.19</refmiscinfo>
+      <refmiscinfo class="version">4.20</refmiscinfo>
       <refmiscinfo class="source"/>
     </refmeta>
     <refnamediv>
@@ -377,7 +377,7 @@
                     </term>
           <listitem>
             <para>A pattern is added to the output pattern database if at least the specified percentage of log messages from the input logfile match the pattern. For example, if the input logfile contains 1000 log messages and the <parameter>--support=3.0</parameter> option is used, a pattern is created only if the pattern matches at least 3 percent of the log messages (that is, 30 log messages). If patternize does not create enough patterns, try to decrease the support value.</para>
-            <para>Default value: <parameter>4.19</parameter></para>
+            <para>Default value: <parameter>4.20</parameter></para>
           </listitem>
         </varlistentry>
         <varlistentry version="5.0">
@@ -467,7 +467,7 @@
         <link linkend="syslog-ng.8"><command>syslog-ng</command>(8)</link>
       </para>
       <note version="5.0">
-        <para>For the detailed documentation of  see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/index.html"><command>The  4.19 Administrator Guide</command></link></para>
+        <para>For the detailed documentation of  see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/index.html"><command>The  4.20 Administrator Guide</command></link></para>
         <para>If you experience any problems or need help with syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"><command>syslog-ng mailing list</command></link>.</para>
         <para>For news and notifications about of syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://syslog-ng.org/blogs/"><command>syslog-ng blogs</command></link>.</para>
       </note>

doc/man/persist-tool.1.xml

@@ -30,7 +30,7 @@
         <refmeta>
             <refentrytitle>persist-tool</refentrytitle>
             <manvolnum>1</manvolnum>
-            <refmiscinfo class="version">4.19</refmiscinfo>
+            <refmiscinfo class="version">4.20</refmiscinfo>
             <refmiscinfo class="source"/>
         </refmeta>
         <refnamediv>
@@ -130,7 +130,7 @@ affile_sd_curpos(/var/aaa.txt)        OK</synopsis>
             <para><link linkend="syslog-ng.conf.5"> <command>syslog-ng.conf</command>(5)</link></para>
             <para><link linkend="syslog-ng.8"> <command>syslog-ng</command>(8)</link></para>
             <note version="5.0">
-                <para>For the detailed documentation of  see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/index.html"><command>The  4.19 Administrator Guide</command></link></para>
+                <para>For the detailed documentation of  see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/index.html"><command>The  4.20 Administrator Guide</command></link></para>
                 <para>If you experience any problems or need help with syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"><command>syslog-ng mailing list</command></link>.</para>
                 <para>For news and notifications about of syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://syslog-ng.org/blogs/"><command>syslog-ng blogs</command></link>.</para>
             </note>

doc/man/secure-logging.7.xml

@@ -30,7 +30,7 @@
     <refmeta>
       <refentrytitle>secure-logging</refentrytitle>
       <manvolnum>7</manvolnum>
-      <refmiscinfo class="version">4.19</refmiscinfo>
+      <refmiscinfo class="version">4.20</refmiscinfo>
       <refmiscinfo class="source"/>
     </refmeta>
     <refnamediv>
@@ -190,7 +190,7 @@ AgAAAAAAAAA=:5UVybnKL1EAbgC4CLfd8HpgurjREf4LEN61/yWHSD2hbXjRD4QmQdtbwguT1chzdItK
 # entries will be logged to a single file called /var/log/messages.slog
 #
 
-@version: 4.19
+@version: 4.20
 @include "scl.conf"
 
 source s_local {

doc/man/slogencrypt.1.xml

@@ -30,7 +30,7 @@
     <refmeta>
       <refentrytitle>slogencrypt</refentrytitle>
       <manvolnum>1</manvolnum>
-      <refmiscinfo class="version">4.19</refmiscinfo>
+      <refmiscinfo class="version">4.20</refmiscinfo>
       <refmiscinfo class="source"/>
     </refmeta>
     <refnamediv>