NEWS: v4.21.0

Signed-off-by: László Várady <laszlo.varady@axoflow.com>

Author: github-actions[bot]

NEWS.md

@@ -1,8 +1,6 @@
-4.20.1
+4.21.0
 ======
 
-_This is a hotfix release that fixes a rare metric race condition causing a reload time crash._
-
 AxoSyslog is binary-compatible with syslog-ng [1] and serves as a drop-in replacement.
 
 We provide [cloud-ready container images](https://github.com/axoflow/axosyslog/#container-images) and Helm charts.
@@ -11,147 +9,72 @@ Packages are available in our [APT](https://github.com/axoflow/axosyslog/#deb-pa
 
 Check out the [AxoSyslog documentation](https://axoflow.com/docs/axosyslog-core/) for all the details.
 
-
 ## Features
 
-  * Added new metrics
-
-    * `syslogng_window_capacity` and `syslogng_window_available` on `stats(level(3))`.
-      * Shows the `log-iw-size()` value and the current state of the source window, respectively.
-    * `syslogng_window_full_total`
-      * Tracks how many times the window was completely full. This counter will
-        increase any time the destination causes the source to be throttled.
-    * `syslogng_memory_queue_processed_events_total` and `syslogng_disk_queue_processed_events_total`
-      * Counts the number of events processed since startup by each queue.
-    * `syslogng_output_batch_size_...`, `syslogng_output_event_size_...`, `syslogng_output_request_latency_...`
-      * histogram style metrics for `http()`, `otel()` and other threaded destinations
-
-    ([#823](https://github.com/axoflow/axosyslog/pull/823))
-    ([#824](https://github.com/axoflow/axosyslog/pull/824))
-    ([#845](https://github.com/axoflow/axosyslog/pull/845))
-
-  * `clickhouse()` destination: Added JSONCompactEachRow format and new `format` directive
-
-    This update enhances the ClickHouse destination by adding support for the `JSONCompactEachRow`
-    format and introducing a new `format` directive for explicitly selecting the data format.
-
-    **Background**
-    Previously, the destination supported:
-    - `Protobuf` (default when using `proto-var`)
-    - `JSONEachRow` (default when using `json-var`)
-
-    These defaults remain unchanged.
-
-    **What’s new**
-    - Added support for `JSONCompactEachRow` — a more compact, array-based JSON representation (used with `json-var`).
-    - Introduced the `format` directive, allowing manual selection of the desired format:
-      - `JSONEachRow`
-      - `JSONCompactEachRow`
-      - `Protobuf`
-
-    **Example**
-    ```hcl
-    destination {
-      clickhouse (
-        ...
-        json-var(json("$my_filterx_json_var"))
-        format("JSONCompactEachRow")
-        ...
-      );
-    };
-    ```
-
-    **JSONEachRow (each JSON object per line, more readable):**
-
-    ```
-    {"id":1,"name":"foo","value":42}
-    {"id":2,"name":"bar","value":17}
-    ```
-
-    **JSONCompactEachRow (compact array-based row representation):**
-
-    ```
-    [1,"foo",42]
-    [2,"bar",17]
-    ```
-
-    **Validation and error handling**
+  * `format_syslog_5424()`: Added new FilterX function for syslog formatting
 
-    **Invalid format values now produce:**
+    Usage:
     ```
-    Error parsing within destination: invalid format value 'invalid format', possible values:[JSONEachRow, JSONCompactEachRow, Protobuf]
+    format_syslog_5424(
+      message,
+      add_octet_count=false,
+      pri=expr,
+      timestamp=expr,
+      host=expr,
+      program=expr,
+      pid=expr,
+      msgid=expr
+    )
     ```
 
-    **If the data’s actual format doesn’t match the selected format, ClickHouse returns:**
-    ```
-    CANNOT_PARSE_INPUT_ASSERTION_FAILED
-    ```
-    ([#828](https://github.com/axoflow/axosyslog/pull/828))
-
-  * `opentelemetry()` source: Added `keep-alive()` option
+    `message` is the only mandatory argument.
 
-    With this new option, client connections can be kept alive during reload,
-    avoiding unnecessary retry backoffs and other error messages on the client
-    side.
+    Fallback values will be used instead of the named arguments
+    if they are not set, or their evaluation fails.
+    ([#875](https://github.com/axoflow/axosyslog/pull/875))
 
-    The default is `yes`.
+  * `http()` and other threaded destinations: add `worker-partition-autoscaling(yes)`
 
-    ([#832](https://github.com/axoflow/axosyslog/pull/832))
+    When `worker-partition-key()` is used to categorize messages into different batches,
+    the messages are - by default - hashed into workers, which prevents them from being distributed across workers
+    efficiently, based on load.
 
-  * `s3()` destination: Added new `object_key_suffix()` option
+    The new `worker-partition-autoscaling(yes)` option uses a 1-minute statistic to help distribute
+    high-traffic partitions among multiple workers, allowing each worker to maximize its batch size.
 
-    The default suffix is an empty string, to ensure backward compatibility.
-    ([#797](https://github.com/axoflow/axosyslog/pull/797))
+    When using this autoscaling option, it is recommended to oversize the number of workers: set it higher than the
+    expected number of partitions.
+    ([#855](https://github.com/axoflow/axosyslog/pull/855))
 
-  * `http()` and other threaded destinations: add `worker-partition-buckets()` option
+  * `network()`: add `transport(nul-terminated)` to support NULL characters to separate log records instead of the
+    more traditional newline separated format
+    ([#867](https://github.com/axoflow/axosyslog/pull/867))
 
-    This allows the same `worker-partition-key()` to use multiple worker threads.
-    ([#852](https://github.com/axoflow/axosyslog/pull/852))
+  * New metrics: `syslogng_output_workers` and `syslogng_output_active_worker_partitions`
 
+    Using the new `worker-partition-autoscaling(yes)` option allows producing partition metrics, which can be used
+    for alerting: if the number of active partitions remains higher than the configured number of workers,
+    it may indicate that events are not being batched properly, which can lead to performance degradation.
+    ([#866](https://github.com/axoflow/axosyslog/pull/866))
 
 ## Bugfixes
 
-  * `metrics`: Fixed a rare race condition in dynamic metrics.
-    ([#858](https://github.com/axoflow/axosyslog/pull/858))
-
-  * `filterx`: Fixed various memory leaks
-    ([#829](https://github.com/axoflow/axosyslog/pull/829))
-    ([#836](https://github.com/axoflow/axosyslog/pull/836))
-    ([#842](https://github.com/axoflow/axosyslog/pull/842))
-
-  * `filterx`: Fixed a variable synchronization bug.
-    ([#849](https://github.com/axoflow/axosyslog/pull/849))
-
-  * `filterx` `otel_logrecord()`: Fixed not clearing `body` before setting dict and array values.
-    ([#835](https://github.com/axoflow/axosyslog/pull/835))
-
-  * Fixed `keep-alive()` during config reload revert
-    ([#831](https://github.com/axoflow/axosyslog/pull/831))
-
-  * `http()` destination: Fixed batch partitioning in case of templated `body-prefix()`
-    ([#843](https://github.com/axoflow/axosyslog/pull/843))
-
-  * `s3`: Bugfixes and general stability improvements for the `s3` destination driver
-
-    * Fixed a major bug causing data loss if multithreaded upload was enabled via the `upload-threads` option.
-    * Fixed a bug where in certain conditions finished object buffers would fail to upload.
-    * Fixed a bug, where empty chunks were being uploaded, causing errors.
-
-    ([#797](https://github.com/axoflow/axosyslog/pull/797))
-    ([#846](https://github.com/axoflow/axosyslog/pull/846))
-
-  * `filterx`: fix potential use-after-free crashes
-    ([#854](https://github.com/axoflow/axosyslog/pull/854))
+  * FilterX `parse_csv()`: fix crash
+    ([#879](https://github.com/axoflow/axosyslog/pull/879))
 
-  * `disk-buffer()`: fix memory leak when `worker-partition-key()` and `disk-buffer()` are used together
-    ([#853](https://github.com/axoflow/axosyslog/pull/853))
+  * FilterX `metrics_labels()`: fix `+=` operator
+    ([#878](https://github.com/axoflow/axosyslog/pull/878))
 
+  * `disk-buffer()`: fix memory leaks
+    ([#872](https://github.com/axoflow/axosyslog/pull/872))
 
 ## Other changes
 
-  * Improved reload time for large configurations.
-    ([#844](https://github.com/axoflow/axosyslog/pull/844))
+  * `disk-buffer()`: significant performance improvements for the non-reliable disk buffer
+    ([#857](https://github.com/axoflow/axosyslog/pull/857))
 
+  * Performance improvements for memory queues
+    ([#881](https://github.com/axoflow/axosyslog/pull/881))
 
 [1] syslog-ng is a trademark of One Identity.
 
@@ -172,5 +95,5 @@ of AxoSyslog, contribute.
 
 We would like to thank the following people for their contribution:
 
-Andras Mitzki, Attila Szakacs, Balazs Scheidler, Bálint Horváth,
-László Várady, Szilard Parrag, Tamás Kosztyu, shifter
+Andras Mitzki, Attila Szakacs, Balazs Scheidler, László Várady, Szilard Parrag,
+Tamás Kosztyu, shifter