Errors in elastalert logs related to built-in detections

[TheRealPancakes]

Version

2.4.100

Installation Method

Security Onion ISO image

Description

configuration

Installation Type

Standalone

Location

on-prem with Internet access

Hardware Specs

Exceeds minimum requirements

CPU

48

RAM

512G

Storage for /

445.07 GiB

Storage for /nsm

11.64 TiB

Network Traffic Collection

tap

Network Traffic Speeds

Less than 1Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

I'm seeing a lot of ERROR-level logs in /opt/so/log/elastalert/elastalert.log

These errors are related to default (built-in, not custom) detections in /opt/so/rules/elastalert/rules/

I'm not using elastic agents nor any other methods of importing data/logs from Windows or any other OS. I'm wondering if these errors are ignorable, concern-worthy, and/or able to be squelched?

$ grep ERROR /opt/so/log/elastalert/elastalert.log | cut -d ' ' -f6- | sort | uniq -c | sort -rn
   1206 ERROR           elastalert Error running query: RequestError(400, 'verification_exception', 'Found 3 problems\nline 1:12: Unknown column [winlog.channel]\nline 1:80: Unknown column [winlog.provider_name]\nline 1:166: Unknown column [winlog.event_data.threat_name]')
    402 ERROR           elastalert Error running query: RequestError(400, 'verification_exception', 'Found 7 problems\nline 1:11: Unknown column [a3]\nline 1:35: Unknown column [a4]\nline 1:59: Unknown column [a5]\nline 1:83: Unknown column [a6]\nline 1:107: Unknown column [a7]\nline 1:131: Unknown column [a1]\nline 1:155: Unknown column [a2]')
    402 ERROR           elastalert Error running query: RequestError(400, 'verification_exception', 'Found 3 problems\nline 1:11: Unknown column [winlog.channel]\nline 1:95: Unknown column [winlog.event_data.LogonType]\nline 1:129: Unknown column [winlog.event_data.TargetOutboundUserName]')
    402 ERROR           elastalert Error running query: RequestError(400, 'verification_exception', 'Found 3 problems\nline 1:11: Unknown column [winlog.channel]\nline 1:92: Unknown column [winlog.provider_name]\nline 1:146: Unknown column [winlog.event_data.ImagePath]')
    402 ERROR           elastalert Error running query: RequestError(400, 'verification_exception', 'Found 3 problems\nline 1:11: Unknown column [winlog.channel]\nline 1:91: Unknown column [winlog.provider_name]\nline 1:142: Unknown column [winlog.event_data.ServiceName]')
    402 ERROR           elastalert Error running query: RequestError(400, 'verification_exception', 'Found 3 problems\nline 1:11: Unknown column [winlog.channel]\nline 1:74: Unknown column [winlog.event_data.SamAccountName]\nline 1:170: Unknown column [winlog.event_data.TargetUserName]')
    402 ERROR           elastalert Error running query: RequestError(400, 'verification_exception', 'Found 3 problems\nline 1:11: Unknown column [winlog.channel]\nline 1:74: Unknown column [winlog.event_data.AccessMask]\nline 1:138: Unknown column [winlog.event_data.Properties]')
    402 ERROR           elastalert Error running query: RequestError(400, 'verification_exception', 'Found 2 problems\nline 1:12: Unknown column [winlog.channel]\nline 1:241: Unknown column [powershell.file.script_block_text]')
    402 ERROR           elastalert Error running query: RequestError(400, 'verification_exception', 'Found 2 problems\nline 1:12: Unknown column [winlog.channel]\nline 1:240: Unknown column [winlog.event_data.Payload]')
    402 ERROR           elastalert Error running query: RequestError(400, 'verification_exception', 'Found 2 problems\nline 1:12: Unknown column [winlog.channel]\nline 1:240: Unknown column [powershell.file.script_block_text]')
    402 ERROR           elastalert Error running query: RequestError(400, 'verification_exception', 'Found 2 problems\nline 1:11: Unknown column [winlog.channel]\nline 1:99: Unknown column [winlog.provider_name]')
    402 ERROR           elastalert Error running query: RequestError(400, 'verification_exception', 'Found 2 problems\nline 1:11: Unknown column [winlog.channel]\nline 1:129: Unknown column [winlog.event_data.ObjectName]')
    402 ERROR           elastalert Error running query: RequestError(400, 'verification_exception', 'Found 2 problems\nline 1:11: Unknown column [key]\nline 1:39: Unknown column [syscall]')
    402 ERROR           elastalert Error running query: RequestError(400, 'verification_exception', 'Found 2 problems\nline 1:11: Unknown column [Image]\nline 1:40: Unknown column [DestinationIp]')
    402 ERROR           elastalert Error running query: RequestError(400, 'verification_exception', 'Found 1 problem\nline 1:11: Unknown column [winlog.channel]')
    402 ERROR           elastalert Error running query: RequestError(400, 'verification_exception', 'Found 1 problem\nline 1:11: Unknown column [USER]')
    402 ERROR           elastalert Error running query: RequestError(400, 'parsing_exception', 'line 1:28: token recognition error at: \'"\\\\mojo\\.\'')
    402 ERROR           elastalert Error running query: RequestError(400, 'parsing_exception', 'line 1:20: token recognition error at: \'"*momyshark\\?\'')
    402 ERROR           elastalert Error running query: RequestError(400, 'parsing_exception', 'line 1:112: no viable alternative at input \'(winlog.channel:"System" and ((event.code like~ ("5805", "5723")) and (""kali\'')
    402 ERROR           elastalert Error running query: RequestError(400, 'parsing_exception', 'line 1:103: no viable alternative at input \'(winlog.channel:"MSExchange Management" and (((""New\'')
    402 ERROR           elastalert Error running query: RequestError(400, 'parsing_exception', 'line 1:102: no viable alternative at input \'(winlog.channel:"MSExchange Management" and ((""OabVirtualDirectory\'')
    402 ERROR           elastalert Error running query: RequestError(400, 'parsing_exception', 'line 1:102: no viable alternative at input \'(winlog.channel:"MSExchange Management" and ((""New\'')
    401 ERROR           elastalert Error running query: RequestError(400, 'verification_exception', 'Found 4 problems\nline 1:11: Unknown column [winlog.channel]\nline 1:73: Unknown column [winlog.event_data.AccessMask]\nline 1:138: Unknown column [winlog.event_data.ObjectServer]\nline 1:179: Unknown column [winlog.event_data.ObjectType]')
    401 ERROR           elastalert Error running query: RequestError(400, 'parsing_exception', 'line 1:28: token recognition error at: \'"\\\\Windows\\\\__1\\d\'')

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[TheRealPancakes]

This issue persists in 2.4.110

[defensivedepth]

@TheRealPancakes Some of these are expected. With the move to EQL, Elasticsearch does a pre-check on the EQL query - if the query targets fields that don't have any mappings, it will fail the precheck with an error. That is what the unknown_column errors are. Annoying, but not indicative of a deeper issue.

The others are Sigma field mappings issues - we are actively working on those.