Merge pull request #782 from axoflow/version/4.17.0

Version: 4.17.0

Author: alltilla

.mailmap

@@ -14,8 +14,6 @@ Attila Szakacs <attila.szakacs@balabit.com> <attila.szakacs@quest.com>
 Attila Szakacs <attila.szakacs@balabit.com> <attila.szakacs@balabit.com>
 Attila Szakacs <attila.szakacs@balabit.com> <attila.szakacs@oneidentity.com>
 Attila Szakacs <attila.szakacs@balabit.com> <45236571+alltilla@users.noreply.github.com>
-Attila Szakacs <attila.szakacs@balabit.com> <szakacs.attila96@gmail.com>
-Attila Szakacs <attila.szakacs@balabit.com> <attila.szakacs@axoflow.com>
 Balint Kovacs <balint.kovacs@balasys.hu> <blint@balabit.hu>
 Balint Kovacs <balint.kovacs@balasys.hu> <blint@blint.hu>
 Fabien Wernli <wernli@in2p3.fr> <wernli@in2p3.fr>
@@ -39,7 +37,6 @@ Lorand Muzamel <lorand.muzamel@balabit.com> <lorand.muzamel@balabit.com>
 Lorand Muzamel <lorand.muzamel@balabit.com> <lorand.muzamel@oneidentity.com>
 László Várady <laszlo.varady@balabit.com> <3130044+MrAnno@users.noreply.github.com>
 László Várady <laszlo.varady@balabit.com> <MrAnno@users.noreply.github.com>
-László Várady <laszlo.varady@balabit.com> <laszlo.varady93@gmail.com>
 László Várady <laszlo.varady@balabit.com> <laszlo.varady@balabit.com>
 Marton ILLES <marton.illes@balabit.com> <marton.illes@balabit.com>
 Marton ILLES <marton.illes@balabit.com> <marci@balabit.com>

NEWS.md

@@ -1,4 +1,4 @@
-4.16.0
+4.17.0
 ======
 
 AxoSyslog is binary-compatible with syslog-ng [1] and serves as a drop-in replacement.
@@ -11,77 +11,93 @@ Check out the [AxoSyslog documentation](https://axoflow.com/docs/axosyslog-core/
 
 ## Features
 
-  * New `$PROTO_NAME` macro: add a $PROTO_NAME macro that expands to "tcp" or
-    "udp" depending on the transport used by syslog-ng.
-    ([#724](https://github.com/axoflow/axosyslog/pull/724))
+  * FilterX `dpath()`: new function to set a potentially non-existing path within a dict
 
+    For example,
+    ```
+    dpath(dict.path.to["complex.n-v"].create) = {...};
+    ```
+    ([#746](https://github.com/axoflow/axosyslog/pull/746))
 
-## Bugfixes
+  * FilterX string slices: support negative indexing
 
-  * Fix memory leaks during configuration parsing
-    ([#755](https://github.com/axoflow/axosyslog/pull/755))
+    For example,
+    ```
+    filterx {
+      str = "example";
+      str[..-2] == "examp";
+      str[-3..] == "ple";
+    };
+    ```
+    ([#780](https://github.com/axoflow/axosyslog/pull/780))
 
-  * `grpc` based destinations: Fixed a race condition around `syslogng_output_grpc_requests_total` metrics
-    ([#754](https://github.com/axoflow/axosyslog/pull/754))
+  * `parallelize()`: add `batch-size()` option and other perf improvements
 
-  * `$PEER_PORT`: fix the value for the PEER_PORT macro, as it was incorrectly
-    reversing the digits in the port value, port 514 would become port 415.
-    ([#724](https://github.com/axoflow/axosyslog/pull/724))
+    The `batch-size()` option specifies, for each input thread, how many consecutive
+    messages should be processed by a single `parallelize()` worker.
 
-  * `google-pubsub()`, `logscale()`, `openobserver()`, `splunk()`, and other batching destinations: fix slowdown
+    This ensures that this many messages preserve their order on the output side
+    and also improves `parallelize()` performance. A value around 100 is recommended.
+    ([#757](https://github.com/axoflow/axosyslog/pull/757))
 
-    The default value of `batch-timeout()` is now 0.
-    This prevents artificial slowdowns in destinations when flow control is enabled
-    and the `log-iw-size()` option of sources is set to a value lower than `batch-lines()`.
+  * `clickhouse-destination()`: new `json-var()` directive
 
-    If you enable `batch-timeout()`, you can further improve batching performance,
-    but you must also adjust the `log-iw-size()` option of your sources accordingly:
+    The new `json-var()` option accepts either a JSON template or a variable containing a JSON string, and sends it to the ClickHouse server in Protobuf/JSON mixed mode (JSONEachRow format). In this mode, type validation is performed by the ClickHouse server itself, so no Protobuf schema is required for communication.
 
-    `log-iw-size / max-connections >= batch-lines * workers`
-    ([#753](https://github.com/axoflow/axosyslog/pull/753))
+    This option is mutually exclusive with `proto-var()`, `server-side-schema()`, `schema()`, and `protobuf-schema()` directives.
 
+    example:
+    ```
+       destination {
+          clickhouse (
+          ...
+          json-var("$json_data");
+    or
+          json-var(json("{\"ingest_time\":1755248921000000000, \"body\": \"test template\"}"))
+          ...
+          };
+       };
+    ```
+    ([#761](https://github.com/axoflow/axosyslog/pull/761))
 
-## FilterX features
+  * FilterX `parse_kv()`: add `stray_words_append_to_value` flag
 
-  * `str_strip()`, `str_lstrip()`, `str_rstrip()`: new string transformation functions to remove leading/trailing whitespaces from a string
-    ([#745](https://github.com/axoflow/axosyslog/pull/745))
+    For example,
+    ```
+    filterx {
+      $MSG = parse_kv($MSG, value_separator="=", pair_separator=" ", stray_words_append_to_value=true);
+    };
 
+    input: a=b b=c d e f=g
+    output: {"a":"b","b":"c d e","f":"g"}
+    ```
+    ([#770](https://github.com/axoflow/axosyslog/pull/770))
 
-## FilterX bugfixes
 
-  * `in` operator: fix crash, that happened, when the list was declared as an operand
+## Bugfixes
 
-    Example:
-    ```
-    if ("test" in ["foo", "bar", "test"]) {
-      $MSG = "YES";
-    };
-    ```
-    ([#759](https://github.com/axoflow/axosyslog/pull/759))
+  * filterx: fix `startswith()`/`endswith()`/`includes()` functions early free
+    ([#772](https://github.com/axoflow/axosyslog/pull/772))
 
-  * `parse_cef()`, `parse_leef()`: Renamed some parsed header fields.
+  * `/string/`: fix escaping
 
-    **These are breaking changes**
+    Strings between / characters are now treated literally. The only exception is `\/`, which can be used to represent a `/` character.
 
-    The motivation behind the renaming is that these names were too
-    generic, and there is a chance to find them in the extensions.
+    This syntax allows you to construct regular expression patterns without worrying about double escaping.
 
-    `parse_cef()`:
-      * `version` -> `cef_version`
-      * `name` -> `event_name`
+    Note: Because of the simplified escaping rules, you cannot represent strings that end with a single `\` character,
+    but such strings would not be valid regular expression patterns anyway.
+    ([#776](https://github.com/axoflow/axosyslog/pull/776))
 
-    `parse_leef()`:
-      * `version` -> `leef_version`
-      * `vendor` -> `vendor_name`
-      * `delimiter` -> `leef_delimiter`
-    ([#748](https://github.com/axoflow/axosyslog/pull/748))
+  * `program()` destination: Fix invalid access of freed log-writer cfg.
+    ([#779](https://github.com/axoflow/axosyslog/pull/779))
 
+  * `ebpf()`: acquire CAP_BPF before loading eBPF programs
 
-## Other changes
+    Previously, when AxoSyslog was compiled with Linux capabilities enabled,
+    the `ebpf()` module was unable to load programs.
+    ([#768](https://github.com/axoflow/axosyslog/pull/768))
 
-  * The base of the AxoSyslog container image is updated to Alpine 3.22
-  ([#758](https://github.com/axoflow/axosyslog/pull/758))
-  * `mongodb()`: support libmongoc v2
 
 
 [1] syslog-ng is a trademark of One Identity.
@@ -103,5 +119,5 @@ of AxoSyslog, contribute.
 
 We would like to thank the following people for their contribution:
 
-Andras Mitzki, Attila Szakacs, Balazs Scheidler, László Várady,
-Tamás Kosztyu
+Andras Mitzki, Attila Szakacs, Balazs Scheidler, Ben Ireland,
+László Várady, Szilard Parrag, Tamás Kosztyu, shifter

README.md

@@ -45,7 +45,7 @@ A simple example is to ingest syslog traffic on tcp/514 and write it to a
 file:
 
 ```
-@version: 4.16
+@version: 4.17
 @include "scl.conf"
 
 log {
@@ -85,7 +85,7 @@ docker pull ghcr.io/axoflow/axosyslog:nightly
 > Note: These named packages are automatically updated when a new AxoSyslog package is released. To install a specific version, run `docker pull ghcr.io/axoflow/axosyslog:<version-number>`, for example:
 >
 > ```shell
-> docker pull ghcr.io/axoflow/axosyslog:4.16.0
+> docker pull ghcr.io/axoflow/axosyslog:4.17.0
 > ```
 
 The container images contain a default configuration file which you probably

VERSION.txt

@@ -1 +1 @@
-4.16.0
+4.17.0

contrib/openbsd-packaging/syslog-ng.conf

@@ -2,7 +2,7 @@
 # This should provide behavior similar to OpenBSD's syslog.conf(5).
 # 2021-05-24 millert@openbsd.org
 
-@version: 4.16
+@version: 4.17
 @requires openbsd
 
 options { 

doc/man/dqtool.1.xml

@@ -30,7 +30,7 @@
     <refmeta>
       <refentrytitle>dqtool</refentrytitle>
       <manvolnum>1</manvolnum>
-      <refmiscinfo class="version">4.16</refmiscinfo>
+      <refmiscinfo class="version">4.17</refmiscinfo>
       <refmiscinfo class="source"/>
     </refmeta>
     <refnamediv>
@@ -117,7 +117,7 @@ Mar  3 10:52:05 tristram localprg[1234]: seq: 0000011631, runid: 1267609923, sta
         <link linkend="syslog-ng.8"><command>syslog-ng</command>(8)</link>
       </para>
       <note version="5.0">
-        <para>For the detailed documentation of  see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.37/administration-guide"><command>The  4.16 Administrator Guide</command></link></para>
+        <para>For the detailed documentation of  see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.37/administration-guide"><command>The  4.17 Administrator Guide</command></link></para>
         <para>If you experience any problems or need help with syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"><command>syslog-ng mailing list</command></link>.</para>
         <para>For news and notifications about of syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://syslog-ng.org/blogs/"><command>syslog-ng blogs</command></link>.</para>
       </note>

doc/man/loggen.1.xml

@@ -30,7 +30,7 @@
     <refmeta>
       <refentrytitle>loggen</refentrytitle>
       <manvolnum>1</manvolnum>
-      <refmiscinfo class="version">4.16</refmiscinfo>
+      <refmiscinfo class="version">4.17</refmiscinfo>
       <refmiscinfo class="source"/>
     </refmeta>
     <refnamediv>
@@ -280,7 +280,7 @@
         <link linkend="syslog-ng.conf.5"><command>syslog-ng.conf</command>(5)</link>
       </para>
       <note version="5.0">
-        <para>For the detailed documentation of  see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.37/administration-guide"><command>The  4.16 Administrator Guide</command></link></para>
+        <para>For the detailed documentation of  see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.37/administration-guide"><command>The  4.17 Administrator Guide</command></link></para>
         <para>If you experience any problems or need help with syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"><command>syslog-ng mailing list</command></link>.</para>
         <para>For news and notifications about of syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://syslog-ng.org/blogs/"><command>syslog-ng blogs</command></link>.</para>
       </note>

doc/man/pdbtool.1.xml

@@ -30,7 +30,7 @@
     <refmeta>
       <refentrytitle>pdbtool</refentrytitle>
       <manvolnum>1</manvolnum>
-      <refmiscinfo class="version">4.16</refmiscinfo>
+      <refmiscinfo class="version">4.17</refmiscinfo>
       <refmiscinfo class="source"/>
     </refmeta>
     <refnamediv>
@@ -377,7 +377,7 @@
                     </term>
           <listitem>
             <para>A pattern is added to the output pattern database if at least the specified percentage of log messages from the input logfile match the pattern. For example, if the input logfile contains 1000 log messages and the <parameter>--support=3.0</parameter> option is used, a pattern is created only if the pattern matches at least 3 percent of the log messages (that is, 30 log messages). If patternize does not create enough patterns, try to decrease the support value.</para>
-            <para>Default value: <parameter>4.16</parameter></para>
+            <para>Default value: <parameter>4.17</parameter></para>
           </listitem>
         </varlistentry>
         <varlistentry version="5.0">
@@ -467,7 +467,7 @@
         <link linkend="syslog-ng.8"><command>syslog-ng</command>(8)</link>
       </para>
       <note version="5.0">
-        <para>For the detailed documentation of  see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/index.html"><command>The  4.16 Administrator Guide</command></link></para>
+        <para>For the detailed documentation of  see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/index.html"><command>The  4.17 Administrator Guide</command></link></para>
         <para>If you experience any problems or need help with syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"><command>syslog-ng mailing list</command></link>.</para>
         <para>For news and notifications about of syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://syslog-ng.org/blogs/"><command>syslog-ng blogs</command></link>.</para>
       </note>

doc/man/persist-tool.1.xml

@@ -30,7 +30,7 @@
         <refmeta>
             <refentrytitle>persist-tool</refentrytitle>
             <manvolnum>1</manvolnum>
-            <refmiscinfo class="version">4.16</refmiscinfo>
+            <refmiscinfo class="version">4.17</refmiscinfo>
             <refmiscinfo class="source"/>
         </refmeta>
         <refnamediv>
@@ -130,7 +130,7 @@ affile_sd_curpos(/var/aaa.txt)        OK</synopsis>
             <para><link linkend="syslog-ng.conf.5"> <command>syslog-ng.conf</command>(5)</link></para>
             <para><link linkend="syslog-ng.8"> <command>syslog-ng</command>(8)</link></para>
             <note version="5.0">
-                <para>For the detailed documentation of  see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/index.html"><command>The  4.16 Administrator Guide</command></link></para>
+                <para>For the detailed documentation of  see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/index.html"><command>The  4.17 Administrator Guide</command></link></para>
                 <para>If you experience any problems or need help with syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"><command>syslog-ng mailing list</command></link>.</para>
                 <para>For news and notifications about of syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://syslog-ng.org/blogs/"><command>syslog-ng blogs</command></link>.</para>
             </note>

doc/man/secure-logging.7.xml

@@ -30,7 +30,7 @@
     <refmeta>
       <refentrytitle>secure-logging</refentrytitle>
       <manvolnum>7</manvolnum>
-      <refmiscinfo class="version">4.16</refmiscinfo>
+      <refmiscinfo class="version">4.17</refmiscinfo>
       <refmiscinfo class="source"/>
     </refmeta>
     <refnamediv>
@@ -190,7 +190,7 @@ AgAAAAAAAAA=:5UVybnKL1EAbgC4CLfd8HpgurjREf4LEN61/yWHSD2hbXjRD4QmQdtbwguT1chzdItK
 # entries will be logged to a single file called /var/log/messages.slog
 #
 
-@version: 4.16
+@version: 4.17
 @include "scl.conf"
 
 source s_local {