Merge pull request #806 from axoflow/version/4.18.0

Version: 4.18.0

Author: MrAnno

.github/workflows/devshell.yml

@@ -115,13 +115,13 @@ jobs:
         if: always() && steps.make_check.outcome == 'failure'
         with:
           name: test-suite-${{ matrix.build-tool }}-${{ matrix.cc }}
-          path: ${{ github.workspace }}/build/test-suite.log
+          path: $GITHUB_WORKSPACE/build/test-suite.log
 
       - name: "Prepare artifact: light-reports"
         id: prepare-light-reports
         if: always() && steps.light.outcome == 'failure'
         run: |
-          REPORTS_DIR=${{ github.workspace }}/tests/light/reports
+          REPORTS_DIR=$GITHUB_WORKSPACE/tests/light/reports
           cp -r ${REPORTS_DIR} /tmp/light-reports
           find /tmp/light-reports -type p,s -print0 | xargs -0 rm -f
           tar -cz -f /tmp/light-reports.tar.gz /tmp/light-reports

NEWS.md

@@ -1,4 +1,4 @@
-4.17.0
+4.18.0
 ======
 
 AxoSyslog is binary-compatible with syslog-ng [1] and serves as a drop-in replacement.
@@ -11,92 +11,58 @@ Check out the [AxoSyslog documentation](https://axoflow.com/docs/axosyslog-core/
 
 ## Features
 
-  * FilterX `dpath()`: new function to set a potentially non-existing path within a dict
+  * `http()`: Added support for templated `headers()`
 
-    For example,
-    ```
-    dpath(dict.path.to["complex.n-v"].create) = {...};
-    ```
-    ([#746](https://github.com/axoflow/axosyslog/pull/746))
-
-  * FilterX string slices: support negative indexing
-
-    For example,
-    ```
-    filterx {
-      str = "example";
-      str[..-2] == "examp";
-      str[-3..] == "ple";
-    };
-    ```
-    ([#780](https://github.com/axoflow/axosyslog/pull/780))
+    In case of batching the templates in `headers()` will be calculated
+    from the first message. Make sure to use `worker-partition-key()` to
+    group similar messages together.
 
-  * `parallelize()`: add `batch-size()` option and other perf improvements
+    Literal dollar signs (`$`) used in `headers()` must be escaped like `$$`.
+    ([#794](https://github.com/axoflow/axosyslog/pull/794))
 
-    The `batch-size()` option specifies, for each input thread, how many consecutive
-    messages should be processed by a single `parallelize()` worker.
+  * FilterX: unary `+` and `-` operators
 
-    This ensures that this many messages preserve their order on the output side
-    and also improves `parallelize()` performance. A value around 100 is recommended.
-    ([#757](https://github.com/axoflow/axosyslog/pull/757))
-
-  * `clickhouse-destination()`: new `json-var()` directive
-
-    The new `json-var()` option accepts either a JSON template or a variable containing a JSON string, and sends it to the ClickHouse server in Protobuf/JSON mixed mode (JSONEachRow format). In this mode, type validation is performed by the ClickHouse server itself, so no Protobuf schema is required for communication.
-
-    This option is mutually exclusive with `proto-var()`, `server-side-schema()`, `schema()`, and `protobuf-schema()` directives.
-
-    example:
+    Useful for dynamic string slicing, for example:
     ```
-       destination {
-          clickhouse (
-          ...
-          json-var("$json_data");
-    or
-          json-var(json("{\"ingest_time\":1755248921000000000, \"body\": \"test template\"}"))
-          ...
-          };
-       };
+    str[..-tempvar]
     ```
-    ([#761](https://github.com/axoflow/axosyslog/pull/761))
+    ([#788](https://github.com/axoflow/axosyslog/pull/788))
 
-  * FilterX `parse_kv()`: add `stray_words_append_to_value` flag
+  * FilterX `parse_csv()`: add `quote_pairs` parameter
+
+    For example:
 
-    For example,
     ```
     filterx {
-      $MSG = parse_kv($MSG, value_separator="=", pair_separator=" ", stray_words_append_to_value=true);
+      str = "sarga,[bogre],'gorbe'";
+      $MSG = parse_csv(str, quote_pairs=["[]", "'"]);
     };
-
-    input: a=b b=c d e f=g
-    output: {"a":"b","b":"c d e","f":"g"}
     ```
-    ([#770](https://github.com/axoflow/axosyslog/pull/770))
+    ([#804](https://github.com/axoflow/axosyslog/pull/804))
 
 
 ## Bugfixes
 
-  * filterx: fix `startswith()`/`endswith()`/`includes()` functions early free
-    ([#772](https://github.com/axoflow/axosyslog/pull/772))
-
-  * `/string/`: fix escaping
+  * FilterX `in` operator: fix crash when left- or right-hand side operand evaluation fails
+    ([#798](https://github.com/axoflow/axosyslog/pull/798))
 
-    Strings between / characters are now treated literally. The only exception is `\/`, which can be used to represent a `/` character.
+  * `in` FilterX operator: Fixed finding `message_value`s in arrays.
+    ([#791](https://github.com/axoflow/axosyslog/pull/791))
 
-    This syntax allows you to construct regular expression patterns without worrying about double escaping.
+  * `python()`: `LogTemplate::format()` now returns a `bytes` object
 
-    Note: Because of the simplified escaping rules, you cannot represent strings that end with a single `\` character,
-    but such strings would not be valid regular expression patterns anyway.
-    ([#776](https://github.com/axoflow/axosyslog/pull/776))
+    In the Python bindings, `LogMessage` is not UTF-8 or Unicode–safe by default.
+    This means developers must explicitly call `decode()` on message fields and handle any decoding errors themselves.
 
-  * `program()` destination: Fix invalid access of freed log-writer cfg.
-    ([#779](https://github.com/axoflow/axosyslog/pull/779))
+    Since `LogTemplate` operates on `LogMessage` fields, this behavior also applies to it.
 
-  * `ebpf()`: acquire CAP_BPF before loading eBPF programs
+    *Breaking change*:
+    When using templates, you now need to decode the result manually. For example:
+    `format().decode("utf-8", errors="backslashreplace")`
+    ([#799](https://github.com/axoflow/axosyslog/pull/799))
 
-    Previously, when AxoSyslog was compiled with Linux capabilities enabled,
-    the `ebpf()` module was unable to load programs.
-    ([#768](https://github.com/axoflow/axosyslog/pull/768))
+  * `in` FilterX operator: Fixed possible memory corruption regarding unreferencing an operand.
+    ([#792](https://github.com/axoflow/axosyslog/pull/792))
 
 
 
@@ -119,5 +85,5 @@ of AxoSyslog, contribute.
 
 We would like to thank the following people for their contribution:
 
-Andras Mitzki, Attila Szakacs, Balazs Scheidler, Ben Ireland,
-László Várady, Szilard Parrag, Tamás Kosztyu, shifter
+Andras Mitzki, Attila Szakacs, Balazs Scheidler, Hofi, László Várady,
+Szilard Parrag, Tamás Kosztyu, shifter

README.md

@@ -45,7 +45,7 @@ A simple example is to ingest syslog traffic on tcp/514 and write it to a
 file:
 
 ```
-@version: 4.17
+@version: 4.18
 @include "scl.conf"
 
 log {
@@ -85,7 +85,7 @@ docker pull ghcr.io/axoflow/axosyslog:nightly
 > Note: These named packages are automatically updated when a new AxoSyslog package is released. To install a specific version, run `docker pull ghcr.io/axoflow/axosyslog:<version-number>`, for example:
 >
 > ```shell
-> docker pull ghcr.io/axoflow/axosyslog:4.17.0
+> docker pull ghcr.io/axoflow/axosyslog:4.18.0
 > ```
 
 The container images contain a default configuration file which you probably

VERSION.txt

@@ -1 +1 @@
-4.17.0
+4.18.0

contrib/openbsd-packaging/syslog-ng.conf

@@ -2,7 +2,7 @@
 # This should provide behavior similar to OpenBSD's syslog.conf(5).
 # 2021-05-24 millert@openbsd.org
 
-@version: 4.17
+@version: 4.18
 @requires openbsd
 
 options { 

doc/man/dqtool.1.xml

@@ -30,7 +30,7 @@
     <refmeta>
       <refentrytitle>dqtool</refentrytitle>
       <manvolnum>1</manvolnum>
-      <refmiscinfo class="version">4.17</refmiscinfo>
+      <refmiscinfo class="version">4.18</refmiscinfo>
       <refmiscinfo class="source"/>
     </refmeta>
     <refnamediv>
@@ -117,7 +117,7 @@ Mar  3 10:52:05 tristram localprg[1234]: seq: 0000011631, runid: 1267609923, sta
         <link linkend="syslog-ng.8"><command>syslog-ng</command>(8)</link>
       </para>
       <note version="5.0">
-        <para>For the detailed documentation of  see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.37/administration-guide"><command>The  4.17 Administrator Guide</command></link></para>
+        <para>For the detailed documentation of  see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.37/administration-guide"><command>The  4.18 Administrator Guide</command></link></para>
         <para>If you experience any problems or need help with syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"><command>syslog-ng mailing list</command></link>.</para>
         <para>For news and notifications about of syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://syslog-ng.org/blogs/"><command>syslog-ng blogs</command></link>.</para>
       </note>

doc/man/loggen.1.xml

@@ -30,7 +30,7 @@
     <refmeta>
       <refentrytitle>loggen</refentrytitle>
       <manvolnum>1</manvolnum>
-      <refmiscinfo class="version">4.17</refmiscinfo>
+      <refmiscinfo class="version">4.18</refmiscinfo>
       <refmiscinfo class="source"/>
     </refmeta>
     <refnamediv>
@@ -280,7 +280,7 @@
         <link linkend="syslog-ng.conf.5"><command>syslog-ng.conf</command>(5)</link>
       </para>
       <note version="5.0">
-        <para>For the detailed documentation of  see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.37/administration-guide"><command>The  4.17 Administrator Guide</command></link></para>
+        <para>For the detailed documentation of  see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.37/administration-guide"><command>The  4.18 Administrator Guide</command></link></para>
         <para>If you experience any problems or need help with syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"><command>syslog-ng mailing list</command></link>.</para>
         <para>For news and notifications about of syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://syslog-ng.org/blogs/"><command>syslog-ng blogs</command></link>.</para>
       </note>

doc/man/pdbtool.1.xml

@@ -30,7 +30,7 @@
     <refmeta>
       <refentrytitle>pdbtool</refentrytitle>
       <manvolnum>1</manvolnum>
-      <refmiscinfo class="version">4.17</refmiscinfo>
+      <refmiscinfo class="version">4.18</refmiscinfo>
       <refmiscinfo class="source"/>
     </refmeta>
     <refnamediv>
@@ -377,7 +377,7 @@
                     </term>
           <listitem>
             <para>A pattern is added to the output pattern database if at least the specified percentage of log messages from the input logfile match the pattern. For example, if the input logfile contains 1000 log messages and the <parameter>--support=3.0</parameter> option is used, a pattern is created only if the pattern matches at least 3 percent of the log messages (that is, 30 log messages). If patternize does not create enough patterns, try to decrease the support value.</para>
-            <para>Default value: <parameter>4.17</parameter></para>
+            <para>Default value: <parameter>4.18</parameter></para>
           </listitem>
         </varlistentry>
         <varlistentry version="5.0">
@@ -467,7 +467,7 @@
         <link linkend="syslog-ng.8"><command>syslog-ng</command>(8)</link>
       </para>
       <note version="5.0">
-        <para>For the detailed documentation of  see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/index.html"><command>The  4.17 Administrator Guide</command></link></para>
+        <para>For the detailed documentation of  see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/index.html"><command>The  4.18 Administrator Guide</command></link></para>
         <para>If you experience any problems or need help with syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"><command>syslog-ng mailing list</command></link>.</para>
         <para>For news and notifications about of syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://syslog-ng.org/blogs/"><command>syslog-ng blogs</command></link>.</para>
       </note>

doc/man/persist-tool.1.xml

@@ -30,7 +30,7 @@
         <refmeta>
             <refentrytitle>persist-tool</refentrytitle>
             <manvolnum>1</manvolnum>
-            <refmiscinfo class="version">4.17</refmiscinfo>
+            <refmiscinfo class="version">4.18</refmiscinfo>
             <refmiscinfo class="source"/>
         </refmeta>
         <refnamediv>
@@ -130,7 +130,7 @@ affile_sd_curpos(/var/aaa.txt)        OK</synopsis>
             <para><link linkend="syslog-ng.conf.5"> <command>syslog-ng.conf</command>(5)</link></para>
             <para><link linkend="syslog-ng.8"> <command>syslog-ng</command>(8)</link></para>
             <note version="5.0">
-                <para>For the detailed documentation of  see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/index.html"><command>The  4.17 Administrator Guide</command></link></para>
+                <para>For the detailed documentation of  see <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/index.html"><command>The  4.18 Administrator Guide</command></link></para>
                 <para>If you experience any problems or need help with syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"><command>syslog-ng mailing list</command></link>.</para>
                 <para>For news and notifications about of syslog-ng, visit the <link xmlns:ns1="http://www.w3.org/1999/xlink" ns1:href="https://syslog-ng.org/blogs/"><command>syslog-ng blogs</command></link>.</para>
             </note>

doc/man/secure-logging.7.xml

@@ -30,7 +30,7 @@
     <refmeta>
       <refentrytitle>secure-logging</refentrytitle>
       <manvolnum>7</manvolnum>
-      <refmiscinfo class="version">4.17</refmiscinfo>
+      <refmiscinfo class="version">4.18</refmiscinfo>
       <refmiscinfo class="source"/>
     </refmeta>
     <refnamediv>
@@ -190,7 +190,7 @@ AgAAAAAAAAA=:5UVybnKL1EAbgC4CLfd8HpgurjREf4LEN61/yWHSD2hbXjRD4QmQdtbwguT1chzdItK
 # entries will be logged to a single file called /var/log/messages.slog
 #
 
-@version: 4.17
+@version: 4.18
 @include "scl.conf"
 
 source s_local {