Errors in ds-logs-system.syslog-default

[mkay2999]

I've got the following error repeating on the ingest pipeline:
error.message : failed in Fleet agent final_pipeline: field [created] not present as part of path [event.created]

its on a new distributed install from ISO securityonion-2.4.20-20231012.iso

any guidance would be appreciated.

[cm-ops]

If you run so-checkin, does it complete without errors? Are you seeing logs in SOC?

[mkay2999]

I re installed the environment and the error seems to be gone..

[kietatran00]

Version
2.4.20

Installation Method
Security Onion ISO image

Description
other (please provide detail below)

Installation Type
Distributed

Location
on-prem with Internet access

Hardware Specs
Exceeds minimum requirements

CPU
12

RAM
32

Storage for /
1000

Storage for /nsm
1000

Network Traffic Collection
span port

Network Traffic Speeds
1Gbps

Status
Yes, all services on all nodes are running OK

Salt Status
no there are no salt errors

Logs
yes

Details

I'm also getting the same issue as the above user, however multiple re-installs of the manager, search node, and sensor did not seem to fix the issue.

I ran sudo so-checkin and it completed without any errors as well.

Guidelines
I have read the above statement and can confirm my post is relevant to Security Onion 2.4.

[kietatran00]

hey all,

I was able to find the fix for this, the primary offender for the error log was system.syslog dataset which comes from the system integration. In the ingest pipeline for the system integration, it was converting syslog.timestamp to @timestamp. I set the target field for the "Date" processor to event.created

After this change you'll find that you get another error saying that it couldn't parse the timestamp. The syslog timestamps come out looking like this: yyyy-MM-dd'T'HH:mm:ss.SSS'Z. Add that to the formats in the final ingestion pipeline date processor and voila! you'll no longer see the error

[pims848]

Hi,
I was experiencing exactly the same issue that I fixed thx to @kietatran00 fix but I decided to try reinstalling with latest Version: 2.4.30 ... same issue is still there right out of the box without any specific configuration...

I still get millions of system.syslog events with same error.message : [ "failed in Fleet agent final_pipeline: field [created] not present as part of path [event.created]" ]

Thanks for your help on this.