Skip to content

Commit d6fb9d0

Browse files
kcreddykcreddy
authored
Cisco Secure Email Gateway | Update grok pattern to extract additional fields (elastic#4742)
* fix grok to match additional fields * update changelog entry
1 parent 0659c45 commit d6fb9d0

File tree

8 files changed

+854
-62
lines changed

8 files changed

+854
-62
lines changed

‎packages/cisco_secure_email_gateway/changelog.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,9 @@
11
# newer versions go on top
2+
- version: "1.3.1"
3+
changes:
4+
- description: Fix grok pattern to extract additional fields
5+
type: bugfix
6+
link: https://github.com/elastic/integrations/pull/4742
27
- version: "1.3.0"
38
changes:
49
- description: Add an on_failure processor to the date processor.

‎packages/cisco_secure_email_gateway/data_stream/log/_dev/test/pipeline/test-common-consolidated-event.log‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,9 @@
11
<166>Mar 17 18:24:37 consolidated_event: CEF:0|Cisco|C100V Email Security Virtual Appliance|14.0.0-657|ESA_CONSOLIDATED_LOG_EVENT|Consolidated Log Event|5|deviceExternalId=42127C7DDEE76852677B-F80CE8074CD3 ESAMID=1053 ESAICID=134 ESAAMPVerdict=UNKNOWN ESAASVerdict=NEGATIVE ESAAVVerdict=NEGATIVE ESACFVerdict=MATCH endTime=Thu Mar 18 08:04:46 2021 ESADLPVerdict=NOT_EVALUATED dvc=1.128.3.4 ESAAttachmentDetails={'test.txt': {'AMP': {'Verdict': 'FILE UNKNOWN', 'fileHash': '7f843d263304fb0516d6210e9de4fa7f01f2f623074aab6e3ee7051f7b785cfa'}, 'BodyScanner': {'fsize': 10059}}} ESAFriendlyFrom=example.com ESAGMVerdict=NEGATIVE startTime=Thu Mar 18 08:04:29 2021 deviceInboundInterface=Incomingmail deviceDirection=0 ESAMailFlowPolicy=ACCEPT suser=example.com cs1Label=MailPolicy cs1=DEFAULT ESAMFVerdict=NOT_EVALUATED act=QUARANTINED ESAFinalActionDetails=To POLICY cs4Label=ExternalMsgID cs4='<example.com>' ESAMsgSize=11873 ESAOFVerdict=POSITIVE duser=example.com ESAHeloIP=1.128.3.4 cfp1Label=SBRSScore cfp1=None ESASDRDomainAge=27 years 2 months 15 days cs3Label=SDRThreatCategory cs3=N/A cs6Label=SDRRepScore cs6=Weak ESASPFVerdict={'mailfrom': {'result': 'None', 'sender': 'example.com'}, 'helo': {'result': 'None', 'sender': 'postmaster'}, 'pra': {'result': 'None', 'sender': 'example.com'}} sourceHostName=unknown ESASenderGroup=UNKNOWNLIST sourceAddress=1.128.3.4 msg='Testing'
22
<166>Mar 17 18:24:37 consolidated_event: CEF:0|Cisco|C100V Email Security Virtual Appliance|14.0.0-657|ESA_CONSOLIDATED_LOG_EVENT|Consolidated Log Event|5|deviceExternalId=42127C7DDEE76852677B-F80CE8074CD3 ESAMID=1053 ESAICID=134 ESAAMPVerdict=UNKNOWN ESAASVerdict=NEGATIVE ESAAVVerdict=NEGATIVE ESACFVerdict=MATCH endTime=Thu Mar 18 08:04:46 2021 ESADLPVerdict=NOT_EVALUATED dvc=1.128.3.4 ESAAttachmentDetails={'test.txt': {'AMP': {'Verdict': 'FILE UNKNOWN', 'fileHash': '7f843d263304fb0516d6210e9de4fa7f01f2f623074aab6e3ee7051f7b785cfa'}, 'BodyScanner': {'fsize': 10059}}} ESAFriendlyFrom=example.com ESAGMVerdict=NEGATIVE startTime=Thu Mar 18 08:04:29 2021 deviceInboundInterface=Incomingmail deviceDirection=0 ESAMailFlowPolicy=ACCEPT suser=example.com cs1Label=MailPolicy cs1=DEFAULT ESAMFVerdict=NOT_EVALUATED act=QUARANTINED ESAFinalActionDetails=To POLICY cs4Label=ExternalMsgID cs4='<example.com>' ESAMsgSize=11873 ESAOFVerdict=POSITIVE duser=example.com ESAHeloIP=1.128.3.4 cfp1Label=SBRSScore cfp1=95.2 ESASDRDomainAge=27 years 2 months 15 days cs3Label=SDRThreatCategory cs3=N/A cs6Label=SDRRepScore cs6=Weak ESASPFVerdict={'mailfrom': {'result': 'None', 'sender': 'example.com'}, 'helo': {'result': 'None', 'sender': 'postmaster'}, 'pra': {'result': 'None', 'sender': 'example.com'}} sourceHostName=unknown ESASenderGroup=UNKNOWNLIST sourceAddress=1.128.3.4 msg='Testing'
3+
<166>Mar 17 18:24:37 consolidated_event: CEF:0|Cisco|C100V Email Security Virtual Appliance|14.0.2-020|ESA_CONSOLIDATED_LOG_EVENT|Consolidated Log Event|5|deviceExternalId=422084EE64B1B0454D49-AAFBF6B55869 ESAMID=153634 ESAICID=55854 ESADCID=41512 ESAAMPVerdict=NOT_EVALUATED ESAASVerdict=NOT_EVALUATED ESAAVVerdict=NEGATIVE ESACFVerdict=NO_MATCH endTime=Thu Nov 3 14:05:44 2022 ESADLPVerdict=NO_TRIGGER dvc=1.128.3.4 ESAFriendlyFrom=example.com ESAGMVerdict=NEGATIVE startTime=Thu Nov 3 14:05:43 2022 deviceOutboundInterface=OutList deviceDirection=1 ESAMailFlowPolicy=RELAY suser=example.com cs1Label=MailPolicy cs1=DEFAULT cs2Label=SenderCountry cs2=not enabled ESAMFVerdict=NOT_EVALUATED act=DELIVERED cs4Label=ExternalMsgID cs4='<example.com>' ESAMsgSize=8893 ESAOFVerdict=NEGATIVE duser=example.com ESAHeloDomain=example.cisco.com ESAHeloIP=1.128.3.4 ESAReplyTo=example.com cfp1Label=SBRSScore cfp1=not enabled sourceHostName=unknown ESASenderGroup=RELAYLIST sourceAddress=1.128.3.4 msg='[SUSPICIOUS MESSAGE] Everycloud Mailflow Monitor guid: 12312314123' ESAURLDetails={'https://secure-web.cisco.com/213weqs123eqwdasrwqe12rf3efd-dasarfgsaddasdasfdas13rdsdw1e1w31rswd...': {'ExpandedUrl': 'https://www.example.com/example-monitor'}}
4+
<166>Mar 17 18:24:37 consolidated_event: CEF:0|Cisco|C100V Email Security Virtual Appliance|14.0.2-020|ESA_CONSOLIDATED_LOG_EVENT|Consolidated Log Event|5|deviceExternalId=422084EE64B1B0454D49-AAFBF6B55869 ESAMID=164226 ESAICID=62905 ESAAMPVerdict=NOT_EVALUATED ESAASVerdict=NOT_EVALUATED ESAAVVerdict=NOT_EVALUATED ESACFVerdict=NOT_EVALUATED endTime=Mon Nov 14 15:32:05 2022 ESADLPVerdict=NOT_EVALUATED dvc=1.128.3.4 ESAGMVerdict=NOT_EVALUATED startTime=Mon Nov 14 15:32:05 2022 deviceInboundInterface=IncList ESAMailFlowPolicy=ACCEPT suser=example.com cs2Label=SenderCountry cs2=United States ESAMFVerdict=NOT_EVALUATED act=ABORTED ESAFinalActionDetails=Receiving aborted by sender ESAOFVerdict=NOT_EVALUATED ESAHeloDomain=GHYU-TRY-AIMV ESAHeloIP=1.128.3.4 cfp1Label=SBRSScore cfp1=-2.3 sourceHostName=example.cisco.com ESASenderGroup=SUSPECTLIST sourceAddress=1.128.3.4 msg=''
5+
<166>Mar 17 18:24:37 consolidated_event: CEF:0|Cisco|C100V Email Security Virtual Appliance|14.0.2-020|ESA_CONSOLIDATED_LOG_EVENT|Consolidated Log Event|5|deviceExternalId=422084EE64B1B0454D49-AAFBF6B55869 ESAMID=164230 ESAICID=62909 ESADCID=47846 ESAAMPVerdict=NOT_EVALUATED ESAASVerdict=NOT_EVALUATED ESAAVVerdict=NEGATIVE ESACFVerdict=NO_MATCH endTime=Mon Nov 14 15:40:49 2022 ESADLPVerdict=NO_TRIGGER dvc=1.128.3.4 ESAFriendlyFrom=example.com ESAGMVerdict=NEGATIVE startTime=Mon Nov 14 15:40:49 2022 deviceOutboundInterface=OutList deviceDirection=1 ESAMailFlowPolicy=RELAY suser=example.com cs1Label=MailPolicy cs1=DEFAULT cs2Label=SenderCountry cs2=not enabled ESAMFVerdict=NOT_EVALUATED act=DELIVERED cs4Label=ExternalMsgID cs4='<example.com>' ESAMsgSize=7360 ESAOFVerdict=NEGATIVE duser=example.com ESAHeloDomain=example.cisco.com ESAHeloIP=1.128.3.4 ESAReplyTo=example.com cfp1Label=SBRSScore cfp1=not enabled sourceHostName=unknown ESASenderGroup=RELAYLIST sourceAddress=1.128.3.4 msg='Everycloud Mailflow Monitor guid: 34214234232'
6+
<166>Mar 17 18:24:37 consolidated_event: CEF:0|Cisco|C100V Email Security Virtual Appliance|14.0.2-020|ESA_CONSOLIDATED_LOG_EVENT|Consolidated Log Event|5|deviceExternalId=422084EE64B1B0454D49-AAFBF6B55869 ESAMID=164229 ESAICID=62908 ESADCID=47845 ESAAMPVerdict=SKIPPED ESAASVerdict=NEGATIVE ESAAVVerdict=NEGATIVE ESACFVerdict=NO_MATCH endTime=Mon Nov 14 15:40:48 2022 ESADLPVerdict=NOT_EVALUATED dvc=1.128.3.4 ESAFriendlyFrom=example.com ESAGMVerdict=NEGATIVE startTime=Mon Nov 14 15:40:47 2022 deviceInboundInterface=IncList deviceDirection=0 ESAMailFlowPolicy=ACCEPT suser=example.com cs1Label=MailPolicy cs1=DEFAULT cs2Label=SenderCountry cs2=United States ESAMFVerdict=NOT_EVALUATED act=DELIVERED cs4Label=ExternalMsgID cs4='<example.com>' ESAMsgSize=1411 ESAOFVerdict=NEGATIVE duser=example.com ESAHeloDomain=example.cisco.com ESAHeloIP=1.128.3.4 ESAReplyTo=example.com cfp1Label=SBRSScore cfp1=5.2 ESASDRDomainAge=1 month cs3Label=SDRThreatCategory cs3=N/A cs6Label=SDRRepScore cs6=Neutral sourceHostName=example.cisco.com ESASenderGroup=UNKNOWNLIST sourceAddress=1.128.3.4 msg='Everycloud Mailflow Monitor guid: 321514231213'
7+
<166>Mar 17 18:24:37 consolidated_event: CEF:0|Cisco|C100V Email Security Virtual Appliance|14.0.2-020|ESA_CONSOLIDATED_LOG_EVENT|Consolidated Log Event|5|deviceExternalId=422084EE64B1B0454D49-AAFBF6B55869 ESAMID=164231 ESAICID=62910 ESADCID=47847 ESAAMPVerdict=UNKNOWN ESAASVerdict=NEGATIVE ESAAVVerdict=NEGATIVE ESACFVerdict=NO_MATCH endTime=Mon Nov 14 15:43:36 2022 ESADLPVerdict=NOT_EVALUATED dvc=1.128.3.4 ESAAttachmentDetails={'image003.png': {'AMP': {'Verdict': 'FILE UNKNOWN', 'fileHash': '7de9d8514c142887d11821fd30faddc693d192efdd19dfb6459872a1be63dcfa'}, 'BodyScanner': {}}} ESAFriendlyFrom=example.com ESAGMVerdict=NEGATIVE startTime=Mon Nov 14 15:43:29 2022 deviceInboundInterface=IncList deviceDirection=0 ESAMailFlowPolicy=ACCEPT suser=example.com cs1Label=MailPolicy cs1=DEFAULT cs2Label=SenderCountry cs2=United States ESAMFVerdict=NOT_EVALUATED act=DELIVERED cs4Label=ExternalMsgID cs4='<example.com>' ESAMsgSize=352844 ESAOFVerdict=NEGATIVE duser=example.com ESAHeloDomain=example.cisco.com ESAHeloIP=1.128.3.4 cfp1Label=SBRSScore cfp1=5.1 ESASDRDomainAge=1 month cs3Label=SDRThreatCategory cs3=N/A cs6Label=SDRRepScore cs6=Neutral sourceHostName=example.cisco.com ESASenderGroup=UNKNOWNLIST sourceAddress=1.128.3.4 msg='TEST'
8+
<166>Mar 17 18:24:37 consolidated_event: CEF:0|Cisco|C100V Email Security Virtual Appliance|14.3.0-023|ESA_CONSOLIDATED_LOG_EVENT|Consolidated Log Event|5|deviceExternalId=423A4DF759243122B64F-7941F28E57A4 ESAMID=4086421 ESAICID=13956459 ESADCID=2522340 ESAAMPVerdict=UNKNOWN ESAASVerdict=NEGATIVE ESAAVVerdict=NOT_EVALUATED ESACFVerdict=NO_MATCH endTime=Thu Nov 24 13:39:24 2022 ESADKIMVerdict=pass ESADLPVerdict=NOT_EVALUATED dvc=1.128.3.4 ESAAttachmentDetails={'image002.png': {'AMP': {'Verdict': 'FILE UNKNOWN', 'fileHash': '30bf618599d8784ebcf38769f8b524b40dc20d2ba262a1e4052d24711abcd064'}, 'BodyScanner': {}}, 'image001.png': {'AMP': {'Verdict': 'FILE UNKNOWN', 'fileHash': '7de9d8514c142887d11821fd30faddc693d192efdd19dfb6459872a1be63dcfa'}, 'BodyScanner': {}}} ESAFriendlyFrom=example.com ESAGMVerdict=NEGATIVE startTime=Thu Nov 24 13:39:16 2022 deviceInboundInterface=IncomingMail deviceDirection=0 ESAMailFlowPolicy=ACCEPT suser=example.com cs1Label=MailPolicy cs1=DEFAULT cs2Label=SenderCountry cs2=United States ESAMFVerdict=MATCH act=DELIVERED cs4Label=ExternalMsgID cs4='<example.com>' ESAMsgSize=716707 ESAOFVerdict=NEGATIVE duser=example.com ESAHeloDomain=example.cisco.com ESAHeloIP=1.128.3.4 cfp1Label=SBRSScore cfp1=3.5 ESASDRDomainAge=30 days (or greater) cs3Label=SDRThreatCategory cs3=N/A cs6Label=SDRRepScore cs6=Favorable sourceHostName=example.cisco.com ESASenderGroup=ACCEPTLIST sourceAddress=1.128.3.4 msg='RE: SR 312312 : consolidate event log' ESATLSInCipher=KWLDS-RSA-AES256-GCM-SHA384 ESATLSInConnStatus=Success ESATLSInProtocol=TLSv1.2 ESATLSOutCipher=HDKWA-RSA-AES256-JMB-SHA384 ESATLSOutConnStatus=Success ESATLSOutProtocol=TLSv1.2
9+
<166>Mar 17 18:24:37 consolidated_event: CEF:0|Cisco|C100V Email Security Virtual Appliance|14.3.0-023|ESA_CONSOLIDATED_LOG_EVENT|Consolidated Log Event|5|deviceExternalId=423A4DF759243122B64F-7941F28E57A4 ESAMID=4086421 ESAICID=13956459 ESADCID=2522340 ESAAMPVerdict=UNKNOWN ESAASVerdict=NEGATIVE ESAAVVerdict=NOT_EVALUATED ESACFVerdict=NO_MATCH endTime=Thu Nov 24 13:39:24 2022 ESADKIMVerdict=pass ESADLPVerdict=NOT_EVALUATED dvc=1.128.3.4 ESAAttachmentDetails={'image002.png': {'AMP': {'Verdict': 'FILE UNKNOWN', 'fileHash': '30bf618599d8784ebcf38769f8b524b40dc20d2ba262a1e4052d24711abcd064'}, 'BodyScanner': {}}, 'image001.png': {'AMP': {'Verdict': 'FILE UNKNOWN', 'fileHash': '7de9d8514c142887d11821fd30faddc693d192efdd19dfb6459872a1be63dcfa'}, 'BodyScanner': {}}} ESAFriendlyFrom=example.com ESAGMVerdict=NEGATIVE startTime=Thu Nov 24 13:39:16 2022 deviceInboundInterface=IncomingMail deviceDirection=0 ESAMailFlowPolicy=ACCEPT suser=example.com cs1Label=MailPolicy cs1=DEFAULT cs2Label=SenderCountry cs2=United States ESAMFVerdict=MATCH act=DELIVERED cs4Label=ExternalMsgID cs4='<example.com>' ESAMsgSize=716707 ESAOFVerdict=NEGATIVE duser=example.com ESAHeloDomain=example.cisco.com ESAHeloIP=1.128.3.4 cfp1Label=SBRSScore cfp1=3.5 ESASDRDomainAge=30 days (or greater) cs3Label=SDRThreatCategory cs3=N/A cs6Label=SDRRepScore cs6=Favorable sourceHostName=example.cisco.com ESASenderGroup=ACCEPTLIST sourceAddress=1.128.3.4 msg='RE: SR 312312 : consolidate event log' ESATLSInCipher=KWLDS-RSA-AES256-GCM-SHA384 ESATLSInConnStatus=Success ESATLSInProtocol=TLSv1.2 ESATLSOutCipher=HDKWA-RSA-AES256-JMB-SHA384 ESATLSOutConnStatus=Success ESATLSOutProtocol=TLSv1.2 ESADaneHost=testdomain.com ESADaneStatus=success ESADHASource=1.128.3.4 ESADMARCVerdict=TempFailure cs5Label=ESAMsgLanguage cs5=English ESAMARAction={'action':'<>';'succesful_rcpts'='<>';'failed_recipients'='<>';'filename'='<>'} ESAMsgTooBigFromSender=true ESARateLimitedIP=1.128.3.4

0 commit comments

Comments
 (0)