Vector - Syslog Logs

[ryansliceup]

Hello guys.
I am trying to capture "severity" from a Cisco device. Is this possible to do? Is there a way to capture the header or can I use regex to parse "Severity notice (5)".

Thanks,
Ryan

13:43:26.102526 IP (tos 0x0, ttl 254, id 1342, offset 0, flags [none], proto UDP (17), length 141)
10.100.1.1.62095 > 10.12.2.241.syslog: [udp sum ok] SYSLOG, length: 113
Facility local7 (23), Severity notice (5)
Msg: 1384: *Mar 4 00:49:56.339: %SYS-5-PRIV_AUTH_PASS: Privilege level set to 15 by csroot on vty0 (10.12.2.233)
0x0000: 3c31 3839 3e31 3338 343a 202a 4d61 7220
0x0010: 2034 2030 303a 3439 3a35 362e 3333 393a
0x0020: 2025 5359 532d 352d 5052 4956 5f41 5554
0x0030: 485f 5041 5353 3a20 5072 6976 696c 6567
0x0040: 6520 6c65 7665 6c20 7365 7420 746f 2031
0x0050: 3520 6279 2063 7372 6f6f 7420 6f6e 2076
0x0060: 7479 3020 2831 302e 3132 2e32 2e32 3333
0x0070: 29

{"host":"10.100.1.1","message":"<189>1384: *Mar 4 00:49:56.339: %SYS-5-PRIV_AUTH_PASS: Privilege level set to 15 by csroot on vty0 (10.12.2.233)","source_ip":"10.100.1.1","source_type":"syslog","timestamp":"2022-01-10T21:43:26.102749856Z"}

[spencergilbert]

I don't believe we can capture the header today. Where is the event you've added at the bottom coming from, and has it been transformed at all?

[ryansliceup]

Spencer, Okay if you cannot capture the header, i can get the severity based on another field in the message. I am thinking grok for this, but am open to suggestions. I am surprised no one has parsed cisco logs using vector before! Thanks, Ryan

…

On Wed, Jan 12, 2022 at 12:59 PM Spencer Gilbert ***@***.***> wrote: I don't believe we can capture the header today. Where is the event you've added at the bottom coming from, and has it been transformed at all? — Reply to this email directly, view it on GitHub <#10780 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AWXADXFA4D2FWPODZ52NJ6DUVW6O3ANCNFSM5LUXBACA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[spencergilbert]

I'm not sure if it was cisco but there have been some users in our discord that used grok/regex to parse "syslog-ish" messages coming from networking devices. If they have a defined format we'd be open to adding a parsing function for the message though.

[ryansliceup]

I do believe they do, let me do some inquiries and I will get back to you. Thanks, Ryan

…

On Wed, Jan 12, 2022 at 2:40 PM Spencer Gilbert ***@***.***> wrote: I'm not sure if it was cisco but there have been some users in our discord that used grok/regex to parse "syslog-ish" messages coming from networking devices. If they have a defined format we'd be open to adding a parsing function for the message though. — Reply to this email directly, view it on GitHub <#10780 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AWXADXG3SG5SZ7QA2C2SK6TUVXKKLANCNFSM5LUXBACA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[ryansliceup]

When I try to use the grok filter this is what I get. Would you recommend I go on discord and ask?

[sources.in_udp]
type = "syslog"
address = "0.0.0.0:514"
mode = "udp"
host_key = "host"

[transforms.cisco_parser]
inputs = ["in_udp"]
type = "grok_parser"
pattern = "<%{INT:priority}>%{NUMBER:seq#}: *%{CISCOTIMESTAMP:log_date}: %{WORD:category}-%{INT:severity}-%{WORD:mnemonic}: %{GREEDYDATA:message}"
#1393: *Mar 5 16:00:36.461: %SYS-5-PRIV_AUTH_PASS: Privilege level set to 15 by csroot on vty0 (10.12.2.233)

[sinks.kafka]
# General
type = "kafka"
inputs = ["in", "in_udp"]
bootstrap_servers = "10.12.2.243:9094"
topic = "vector-lines-0"
compression = "none"
healthcheck = true
encoding = "json"

[sinks.out]
inputs = ["in_udp"]
type = "console"
encoding = "text"

2022-01-13T12:26:06.251687Z WARN transform{component_kind="transform" component_id=cisco_parser component_type=grok_parser component_name=cisco_parser}: vector::internal_events::grok_parser: Grok pattern failed to match. field="<189>1397: *Mar 6 15:32:28.004: %SYS-5-PRIV_AUTH_PASS:[...]" internal_log_rate_secs=30
<189>1397: *Mar 6 15:32:28.004: %SYS-5-PRIV_AUTH_PASS: Privilege level set to 15 by csroot on vty0 (10.12.2.233)

[spencergilbert]

I assume <189>1393: *Mar 5 16:00:36.461: %SYS-5-PRIV_AUTH_PASS: Privilege level set to 15 by csroot on vty0 (10.12.2.233) is the entire .message you're trying to parse? I can spend a few minutes trying to craft it correctly.

I'd also suggest replacing the grok_parser transform (which is deprecated) with a remap transform using the parse_grok() function from VRL

[spencergilbert]

@ryansliceup it looks like this works, but my grok-fu isn't great:

$ parse_grok!(.message, "^<%{POSINT:syslog_pri}>%{NUMBER:seq}: [*]%{CISCOTIMESTAMP:timestamp}: [%]%{WORD:category}-%{INT:severity}-%{WORD:mnemonic}: %{GREEDYDATA:message}")
{ "category": "SYS", "message": "Privilege level set to 15 by csroot on vty0 (10.12.2.233)", "mnemonic": "PRIV_AUTH_PASS", "seq": "1393", "severity": "5", "syslog_pri": "189", "timestamp": "Mar 5 16:00:36.461" }

[ryansliceup]

Hello,
Thanks so much for the information, I appreciate it. As I am new to vector, is this right as it doesn't parse the severity, but it doesn't fail either. I have tried changing the inputs on the sink, and it did not help. What am I doing wrong, I am sure it is something silly.

Thanks,
Ryan

[sources.in_udp]
type = "syslog"
address = "0.0.0.0:514"
mode = "udp"
host_key = "host"

[transforms.cisco_parser]
inputs = ["in_udp"]
type = "remap"
source = '''
parse_grok!(.message, "^<%{POSINT:syslog_pri}>%{NUMBER:seq}: [*]%{CISCOTIMESTAMP:timestamp}: [%]%{WORD:category}-%{INT:severity}-%{WORD:mnemonic}: %{GREEDYDATA:message}")

'''
#1393: *Mar  5 16:00:36.461: %SYS-5-PRIV_AUTH_PASS: Privilege level set to 15 by csroot on vty0 (10.12.2.233)

[sinks.kafka]
# General
type = "kafka"
#inputs = ["in_udp"]
inputs = ["cisco_parser"]
bootstrap_servers = "10.12.2.243:9094"
topic = "vector-lines-0"
compression = "none"
healthcheck = true
encoding = "json"

[sinks.out]
inputs = ["cisco_parser"]
type = "console"
encoding = "json"

[spencergilbert]

Can you share an example event from sources.in_udp (send it directly to sinks.out without parsing) as well as an example event that's currently getting printed by sinks.out (from the current cisco_parser input)?

[ryansliceup]

This is from the following configuration on the console. Is this what you asked? Thanks again for the information, and help!

Output:

<189>1405: *Mar  7 19:52:59.273: %SYS-5-PRIV_AUTH_PASS: Privilege level set to 15 by csroot on vty0 (10.12.2.233)

Configuration:

[sources.in_udp]
    type = "syslog"
    address = "0.0.0.0:514"
    mode = "udp"
    host_key = "host"

[transforms.cisco_parser]
    inputs       = ["in_udp"]
    type         = "remap"
    source = '''
       parse_grok!(.message, "^<%{POSINT:syslog_pri}>%{NUMBER:seq}: [*]%{CISCOTIMESTAMP:timestamp}: [%]%{WORD:category}-%{INT:severity}-%{WORD:mnemonic}: %{GREEDYDATA:message}")

    '''
    #1393: *Mar  5 16:00:36.461: %SYS-5-PRIV_AUTH_PASS: Privilege level set to 15 by csroot on vty0 (10.12.2.233)


[sinks.kafka]
    # General
    type = "kafka"
    #inputs = ["in_udp"]
    inputs = ["cisco_parser"]
    bootstrap_servers = "10.12.2.243:9094"
    topic = "vector-lines-0"
    compression = "none"
    healthcheck = true
    encoding = "json"

[sinks.out]
    inputs = ["in_udp"]
    type = "console"
    encoding = "text"

Then I have modified the configuration to change the inputs from the sink:
Configuration:

[sinks.kafka]
    # General
    type = "kafka"
    #inputs = ["in_udp"]
    inputs = ["cisco_parser"]
    bootstrap_servers = "10.12.2.243:9094"
    topic = "vector-lines-0"
    compression = "none"
    healthcheck = true
    encoding = "json"

[sinks.out]
    inputs = ["cisco_parser"]
    type = "console"
    encoding = "text"

Output:

<189>1406: *Mar  7 19:54:56.165: %SYS-5-PRIV_AUTH_PASS: Privilege level set to 15 by csroot on vty0 (10.12.2.233)

[jszwedko]

Hi @ryansliceup !

You'll want to assign the result of the parse_grok call in your remap transform, otherwise VRL will just drop it. Try:

[transforms.cisco_parser]
    inputs       = ["in_udp"]
    type         = "remap"
    source = '''
       . = parse_grok!(.message, "^<%{POSINT:syslog_pri}>%{NUMBER:seq}: [*]%{CISCOTIMESTAMP:timestamp}: [%]%{WORD:category}-%{INT:severity}-%{WORD:mnemonic}: %{GREEDYDATA:message}")
    '''

You can also preserve the existing fields in . with something like:

[transforms.cisco_parser]
    inputs       = ["in_udp"]
    type         = "remap"
    source = '''
       .| = parse_grok!(.message, "^<%{POSINT:syslog_pri}>%{NUMBER:seq}: [*]%{CISCOTIMESTAMP:timestamp}: [%]%{WORD:category}-%{INT:severity}-%{WORD:mnemonic}: %{GREEDYDATA:message}")
    '''

I'd recommend trying both so you can see the differences.

[StefanSa]

Hi there,
I have once found this with me (grok examples).

# IOS
"message", "%{SYSLOG5424PRI}(%{NUMBER:log_sequence#})?:( %{NUMBER}:)? %{CISCOTIMESTAMPTZ:log_date}: %%{CISCO_REASON:facility}-%{INT:severity_level}-%{CISCO_REASON:facility_mnemonic}: %{GREEDYDATA:message}",
"message", "%{SYSLOG5424PRI}(%{NUMBER:log_sequence#})?:( %{NUMBER}:)? %{CISCOTIMESTAMPTZ:log_date}: %%{CISCO_REASON:facility}-%{CISCO_REASON:facility_sub}-%{INT:severity_level}-%{CISCO_REASON:facility_mnemonic}: %{GREEDYDATA:message}",

# Nexus
"message", "%{SYSLOG5424PRI}(%{NUMBER:log_sequence#})?: %{NEXUSTIMESTAMP:log_date}: %%{CISCO_REASON:facility}-%{INT:severity_level}-%{CISCO_REASON:facility_mnemonic}: %{GREEDYDATA:message}",
"message", "%{SYSLOG5424PRI}(%{NUMBER:log_sequence#})?: %{NEXUSTIMESTAMP:log_date}: %%{CISCO_REASON:facility}-%{CISCO_REASON:facility_sub}-%{INT:severity_level}-%{CISCO_REASON:facility_mnemonic}: %{GREEDYDATA:message}"

[ryansliceup]

Thanks guys. You guys are awesome. Let me check this out!

…

On Sat, Jan 15, 2022, 5:44 AM StefanSa ***@***.***> wrote: Hi there, I have once found this with me (grok examples). # IOS "message", "%{SYSLOG5424PRI}(%{NUMBER:log_sequence#})?:( %{NUMBER}:)? %{CISCOTIMESTAMPTZ:log_date}: %%{CISCO_REASON:facility}-%{INT:severity_level}-%{CISCO_REASON:facility_mnemonic}: %{GREEDYDATA:message}", "message", "%{SYSLOG5424PRI}(%{NUMBER:log_sequence#})?:( %{NUMBER}:)? %{CISCOTIMESTAMPTZ:log_date}: %%{CISCO_REASON:facility}-%{CISCO_REASON:facility_sub}-%{INT:severity_level}-%{CISCO_REASON:facility_mnemonic}: %{GREEDYDATA:message}", # Nexus "message", "%{SYSLOG5424PRI}(%{NUMBER:log_sequence#})?: %{NEXUSTIMESTAMP:log_date}: %%{CISCO_REASON:facility}-%{INT:severity_level}-%{CISCO_REASON:facility_mnemonic}: %{GREEDYDATA:message}", "message", "%{SYSLOG5424PRI}(%{NUMBER:log_sequence#})?: %{NEXUSTIMESTAMP:log_date}: %%{CISCO_REASON:facility}-%{CISCO_REASON:facility_sub}-%{INT:severity_level}-%{CISCO_REASON:facility_mnemonic}: %{GREEDYDATA:message}" — Reply to this email directly, view it on GitHub <#10780 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AWXADXFFYGWXF3JBWYGJF53UWFFXJANCNFSM5LUXBACA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[ryansliceup]

@StefanSa Is this in a different format? I am assuming i have to tweak this to plop it in the config?

[StefanSa]

@ryansliceup
Yes you need to adjust that a bit, because that is an example of logstash, where grok is the same.

.| = parse_grok!(.message, "%{SYSLOG5424PRI}(%{NUMBER:log_sequence#})?:( %{NUMBER}:)? [*]%{CISCOTIMESTAMP:timestamp}: %%{CISCO_REASON:facility}-%{INT:severity_level}-%{CISCO_REASON:facility_mnemonic}: %{GREEDYDATA:message}")

A good tool to debug grok is this one:
https://grokdebug.herokuapp.com/

[jrehm-mmm]

@ryansliceup - did you end up implimenting this in your production environment? I'm considering switching our syslog handling over to vector and am quite surprised as well that there doesn't seem to be too much demand for help with parsing Cisco's plethora of syslog formatting!

[ryansliceup]

I ended up putting rsyslog in front of it for Cisco. It parses better and I was able to aggregate all my servers easier. It worked for my use case at least.

…

On Fri, Mar 17, 2023, 6:11 PM jrehm-mmm ***@***.***> wrote: @ryansliceup <https://github.com/ryansliceup> - did you end up implimenting this in your production environment? I'm considering switching our syslog handling over to vector and am quite surprised as well that there doesn't seem to be too much demand for help with parsing Cisco's plethora of syslog formatting! — Reply to this email directly, view it on GitHub <#10780 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AWXADXFDL3RUHHQFBECR46DW4TOPRANCNFSM5LUXBACA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Disclaimer: This email and its attachments are intended only for the use of the individual or entity to which it is addressed and may contain information which is privileged, confidential, proprietary, or exempt from disclosure under applicable law. If you are not the intended recipient or the person responsible for delivering the message to the intended recipient, you are strictly prohibited from disclosing, distributing, copying, or in any way using this message. If you have received this communication in error, please notify the sender and destroy and delete any copies you may have received. Internet email is not a 100% secure transmission medium. Please understand and observe this lack of security when emailing us. Although we have taken steps to ensure that this email and its attachments are free from any virus, it is the recipient's sole responsibility, in keeping with good computing practice, to ensure that they are actually virus free.