error[E101]: invalid regular expression

[adedokunk]

I was try to exclude logs but getting the error. i will appreciate any help on this

datadog_exclude_log:
      type: filter
      condition: |
          structured, err = parse_regex(.message, r'^(-|(?P<remote>\S+)) - (-|(?P<user_agent>\S+)) \[(?P<timestamp>.+)\] "(?P<path>(?P<method>\w+) (?P<path>\S+) (?P<protocol>\S+))" (?P<status>\d+) (?P<size>\d+) "(-|(?P<referer>.+))" "(-|(?P<agent>.+))" "(-|(?P<http_x_forwarded_for>.+))" "(-|(?P<host>.+))" "(-|(?P<session_cookie>.+))" "(-|(?P<endpoint.request>\d+\.\d+))" "user\[(-|(?P<http_method>.+))\]"$')
            if err != null {
              log("Unable to parse_regex RAW:" + string!(.message), level: "error")
            }
            else {
              .level == "info" && .logger_name == "INBOUND_REQUEST"
              .status == "DEBUG" || .status == "INFO"
              .request_time = to_float!(.request_time)
              .timestamp = parse_timestamp!(.timestamp, "%d/%b/%Y:%H:%M:%S %z")
            }

Edited: Please use surround blocks with ```

[bfung]

Check the regex syntax please, you can use an online tool like https://regex101.com/ . Make sure to use the Rust flavor.

[adedokunk]

Thank you @bfung i made the correction.

structured, err = parse_regex(.message, r'^(-|(?P<remote>\S+)) - (-|(?P<user_agent>\S+)) \[(?P<timestamp>.+)\] '(?P<RequestPath>(?P<method>\w+)  (?P<status>\d+) ' '(-|(?P<host>.+))' '(-|(?P<session_cookie>.+))' '(-|(?:P<endpoint.request>\d+\.\d+))' 'user\[(-|(?P<http_method>.+))\]'$'))

but still get : Your regular expression does not match the subject string.

[adedokunk]

I'm still getting errors i don't know the reason for that. i need help please:
2025-01-30 04:15:35.137Z 2025-01-30T04:15:35.137808Z ERROR vector::topology::builder: Configuration error. error=Transform "trn_exclude_log": 2025-01-30 04:15:35.138Z error[E203]: syntax error 2025-01-30 04:15:35.138Z ┌─ :1:123 2025-01-30 04:15:35.138Z │ 2025-01-30 04:15:35.138Z 1 │ structured, err = parse_regex(.message, r'^(-|(?P<remote>\S+)) - (-|(?P<user_agent>\S+)) \[(?P<timestamp>.+)\] '(??P<path>\S+)(?P<method>\w+) (?P<status>\d+) ' '(-|(?P<host>.+))' '(-|(?P<session_cookie>.+))' '(-|(?:P<endpoint.request>\d+\.\d+))' 'user\[(-|(?P<http_method>.+))\]'$')) 2025-01-30 04:15:35.138Z │ ^ 2025-01-30 04:15:35.138Z │ │ 2025-01-30 04:15:35.138Z │ unexpected syntax token: "Escape" 2025-01-30 04:15:35.138Z │ expected one of: "\n", "!=", "&&", ")", "*", "+", ",", "-", "/", "<", "<=", "==", ">", ">=", "??", "|", "|", "||" 2025-01-30 04:15:35.138Z │ 2025-01-30 04:15:35.138Z = see language documentation at https://vrl.dev 2025-01-30 04:15:35.138Z = try your code in the VRL REPL, learn more at https://vrl.dev/examples 2025-01-30 04:15:35.138Z Stream closed: EOF

[pront]

This seems like a different kind of error. It has to do with how you plug your VRL source code into your config (maybe YAML?). Please see a multi-line example:

transforms:
  remap0:
    type: remap
    inputs: ["demo_logs_1"]
    source: |
      a = .timestamp
      del(.)
      .log = a

[adedokunk]

Thank you @pront i have a multiple inputs from other transform that am passing into it

[adedokunk]

here is the new error.
`2025-01-30 06:03:37.449Z ┌─ :1:1
2025-01-30 06:03:37.449Z │
2025-01-30 06:03:37.449Z 1 │ ╭ structured, err = parse_regex(.message, /^(-|(?P\S+)) - (-|(?P<user_agent>\S+)) [(?P.+)] "(?P\w+) (?P\S+) (?P.+)" (?P\d+) (-|(?P.+)) (-|(?P<session_cookie>.+)) (-|(?P<endpoint_request>\d+.\d+)) user[(-|(?P<http_method>.+))]/)
2025-01-30 06:03:37.449Z 2 │ │ if err != null {
2025-01-30 06:03:37.449Z 3 │ │ return false
2025-01-30 06:03:37.449Z 4 │ │ }
2025-01-30 06:03:37.449Z · │
2025-01-30 06:03:37.449Z 11 │ │ )
2025-01-30 06:03:37.449Z 12 │ │
2025-01-30 06:03:37.449Z │ ╰^ unexpected error: invalid escape character: \w
2025-01-30 06:03:37.449Z │
2025-01-30 06:03:37.449Z = see language documentation at https://vrl.dev
2025-01-30 06:03:37.449Z = try your code in the VRL REPL, learn more at https://vrl.dev/examples
2025-01-30 06:03:37.449Z
Stream closed: EOF

`

[adedokunk]

When i try to refactor the regex with this code.
datadog_exclude_log: type: filter condition: | structured, err = parse_regex(.message, /^(-|(?P<remote>\S+)) - (-|(?P<user_agent>\S+)) \[(?P<timestamp>.+)\] "(?P<method>\w+) (?P<path>\S+) (?P<protocol>.+)" (?P<status>\d+) (-|(?P<host>.+)) (-|(?P<session_cookie>.+)) (-|(?P<endpoint_request>\d+\.\d+)) user\[(-|(?P<http_method>.+))\]/) if err != null { return false } return ( .level == "info" && .logger_name == "INBOUND_REQUEST" && (.status == "DEBUG" || .status == "INFO") && is_float(try(to_float(.request_time))) && is_timestamp(parse_timestamp(.timestamp, "%d/%b/%Y:%H:%M:%S %z")) ) inputs: - datadog_logs_hosts - datadog_metrics_hosts - datadog_is_log

[adedokunk]

Here is my code block
datadog_exclude_log: type: filter condition: | structured, err = parse_regex(.message, r'^(-|(?P<remote>\S+)) - (-|(?P<user_agent>\S+)) \[(?P<timestamp>.+)\] '(??P<path>\S+)(?P<method>\w+) (?P<status>\d+) ' '(-|(?P<host>.+))' '(-|(?P<session_cookie>.+))' '(-|(?:P<endpoint.request>\d+\.\d+))' 'user\[(-|(?P<http_method>.+))\]'$')) if err != null { return ( .level == "info" && .logger_name == "INBOUND_REQUEST" .status == "DEBUG" || .status == "INFO" .request_time == to_float!(.request_time) .timestamp == parse_timestamp!(.timestamp, "%d/%b/%Y:%H:%M:%S %z") ) } inputs:

[bfung]

Can you post a larger part of the configuration, it looks like there's multiple errors happening...

1. A regex error in datadog_exclude_log configuration
   • Can you post some examples of log lines that should get excluded, and some log lines that shouldn't get excluded?
2. Looks like also a configuration error somewhere in trn_exclude_log?
   • The error message suggests something like unexpected syntax token: "Escape", so maybe an escape character or some quoting issue.

That will help debugging quite a bit.

[adedokunk]

Thank you