Logs not being properly parsed when shipped from Panorama to Splunk

[NMCPaul]

Describe the bug

Field extractions are not working properly on logs sent from Palo Alto Panorama v10.2.4-h4 to Splunk v9.0.2 with the Palo Alto app/add-on v8.1.0.

Expected behavior

Field extractions and transforms properly parse the data from the logs.

Current behavior

Field extraction are not working correctly. An example of this would be the word “deny” being assign to the transport field rather than the protocol of tcp, or udp, etc. Looking at the transforms.conf file of the add-on, I see the first field in many of the [extract_*] stanzas are defined as “future_use1” as shown below.

FIELDS="future_use1","receive_time","serial_number","log_type","log_subtype","version","generated_time","src_ip","dest_ip","src_translated_ip","dest_translated_ip","rule","src_user","dest_user","app","vsys","src_zone","dest_zone","src_interface","dest_interface","log_forwarding_profile","future_use3","session_id","repeat_count","src_port","dest_port","src_translated_port","dest_translated_port","session_flags","transport","action","misc","threat","raw_category","severity","direction","sequence_number","action_flags","src_location","dest_location","future_use4","content_type","pcap_id","file_hash","cloud_address","url_index","user_agent","file_type","xff","referrer","sender","subject","recipient","report_id","devicegroup_level1","devicegroup_level2","devicegroup_level3","devicegroup_level4","vsys_name","dvc_name","future_use5","src_vm","dest_vm","http_method","tunnel_id","tunnel_monitor_tag","tunnel_session_id","tunnel_start_time","tunnel_type","threat_category","content_version","future_use6"

Each event in the data I am receiving from my firewall begins with a timestamp which is the second field listed in the extraction. I removed “future_use1” from the globalprotect, threat, traffic, userid, correlation, hipmatch, system, and config extractions, and now they seem to be working better.

In conjunction with the above change, I also had to alter the regexes which set the sourcetype on my events. The default configuration looks like the following:

DEST_KEY = MetaData:Sourcetype
REGEX = ^[^,]+,[^,]+,[^,]+,TRAFFIC,
FORMAT = sourcetype::pan:traffic

Since my data starts with the ”receive_time” field, it is necessary to change the regex to the following for the appropriate sourcetypes to be assigned.

REGEX = ^[^,]+,[^,]+,TRAFFIC,

Possible solution

Modify the regexes as noted above.

Steps to reproduce

1. The following Panorama entries (one of each type), when sent from Panorama to Splunk, do not get processed correctly if run through the un-modified plugins:

13:22:49,000702698924,USERID,login,1,2023/08/28 13:22:49,,207.74.235.202,ad\cook370,User-ID-Agent,0,1,2700,0,0,agent,,7270585355236084506,0x0,0,0,0,0,,Panorama,0,,2023/08/28 13:22:46,1,0x0,ad\cook370,,1969-12-31T19:00:00.000-05:00

13:22:51,016401010483,THREAT,file,2562,2023/08/28 13:22:43,198.108.197.113,192.122.186.130,0.0.0.0,0.0.0.0,Allow-Inside-to-Internet-AppDefault-1,,,netflix-base,vsys1,Inside,Outside,ethernet1/14,ethernet1/13,default,2023/08/28 13:22:43,1778831,1,45279,80,0,0,0x2000,tcp,alert,"AAAABSa4Ov8t2867atBdCPWOsUcZeeBkOFUFjD1jACIJVWYDHsoU85K0IqgHd0A",WebP Image File(52275),streaming-media,low,server-to-client,7270675961878125631,0x8000000000000000,Region-198.108.19x.x,United States,,,0,,,195,,,,,,,,0,94,83,0,0,,External-PA-3260-Primary,occ-0-883-3419.1.nflxso.net/dnm/api/v6/tx1O544a9T7n8Z_G12qaboulQQE/AAAABSa4Ov8t2867atBdCPWOsUcZeeBkOFUFjD1jACIJVWYDHsoU85K0IqgHd0A2r3AR40TG7Nu3iYLRXdropQx1yhuer-0YTdZaDem7tkk5Tg.webp?r=030,,,,0,,0,,N/A,N/A,AppThreat-8748-8241,0x0,0,4294967295,,,01adbd9d-cfa3-4bfc-b4d4-b50dace4a9cf,0,,,,,,,,,,,,,,,,,,,,,,,,,,,,,0,2023-08-28T13:22:43.077-04:00,,,,photo-video,media,browser-based,3,"evasive-behavior,consume-big-bandwidth,has-known-vulnerability,pervasive-use",netflix,netflix-base,no,no,

13:24:12,016401010483,GLOBALPROTECT,0,2562,2023/08/28 13:24:03,vsys1,gateway-tunnel-latency,tunnel,,,claerhc,US,DELL3070,97.91.54.255,0.0.0.0,97.91.54.255,0.0.0.0,412d4483-2d72-4a84-ab9e-c6485e4d8682,7JWRL43,6.0.4,any,,1,,,"Pre-tunnel latency: 32ms, Post-tunnel latency: 31ms",success,,0,,0,GP-GW,7270675961866100325,0x8000000000000000,2023-08-28T13:24:03.947-04:00,,,,,,94,83,0,0,,External-PA-3260-Primary,1
 
13:24:12,016401010448,HIPMATCH,0,2562,2023/08/28 13:24:03,ad\claerhc,vsys1,DELL3070,Windows,172.25.3.61,OS-Check-Windows,1,object,,,7270613427142266946,0x8000000000000000,94,83,0,0,,External-PA-3260-Secondary,1,0.0.0.0,412d4483-2d72-4a84-ab9e-c6485e4d8682,7JWRL43,,2023-08-28T13:24:03.952-04:00

13:24:31,000702698924,SYSTEM,general,0,2023/08/28 13:24:31,,general,,0,0,general,informational,"Connection to Update server: updates.paloaltonetworks.com completed successfully, initiated by 10.10.0.135",7270585106127961531,0x0,0,0,0,0,,Panorama,0,0,2023-08-28T13:24:31.000-04:00

Screenshots

n/a

Context

Trying to use data from our PA firewalls in order to set up reports and dashboards.

Your Environment

• Versions used:
  • Panorama v10.2.4-h4
  • Splunk v9.0.2
  • Palo Alto app/add-on v8.1.0.

[NMCPaul]

Adding on to this, I have the following questions:

• Is there some setting in Panorama which changes the log format being sent to Splunk which will make the manual adjustments I made unnecessary?
• If not, are there plans to adjust the Palo Alto Networks Add-on for Splunk to account for the behavior I am experiencing?
• Is there some other configuration I missed which needs to be implemented to make this all work without customization?

[welcome-to-palo-alto-networks]

🎉 Thanks for opening your first issue here! Welcome to the community!

[jeffwalzer]

Any update on this?

[NMCPaul]

Haven't heard anything from anyone yet. Are you running into the same thing?

[jeffwalzer]

Somewhat similar - having issues with events related to pan:firewall_cloud - but seems like no one from Palo has been responding to much of anything

[NMCPaul]

Just curious - is there a better method to get support or an actual reply for this?

[jeffwalzer]

Not that I know of - very disappointing to not hear anything back at all

[jwiley80]

I second that sentiment Jeremy Wiley 678-852-5789

…

________________________________ From: JeffW ***@***.***> Sent: Wednesday, October 4, 2023 9:01:20 AM To: PaloAltoNetworks/Splunk-Apps ***@***.***> Cc: Subscribed ***@***.***> Subject: Re: [PaloAltoNetworks/Splunk-Apps] Logs not being properly parsed when shipped from Panorama to Splunk (Issue #305) Not that I know of - very disappointing to not hear anything back at all — Reply to this email directly, view it on GitHub [github. com], or unsubscribe [github. com]. You are receiving this because you are subscribed to this thread. Message Not that I know of - very disappointing to not hear anything back at all — Reply to this email directly, view it on GitHub [github.com]<https://urldefense.com/v3/__https://github.com/PaloAltoNetworks/Splunk-Apps/issues/305*issuecomment-1746833000__;Iw!!MLsdJ25-fIk!umoavp_d04bPvLp8N19vumA4ZYsaerUBl8ADgtwPNv0cuVsIqtqFbUepuNf0Gm2FMdn6ol0jLe7JPwV6mgDDPf7bHJY$>, or unsubscribe [github.com]<https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AU44XXPGSJVOAXNNKJK3J5TX5VM2BAVCNFSM6AAAAAA4DIBJU6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONBWHAZTGMBQGA__;!!MLsdJ25-fIk!umoavp_d04bPvLp8N19vumA4ZYsaerUBl8ADgtwPNv0cuVsIqtqFbUepuNf0Gm2FMdn6ol0jLe7JPwV6mgDD2AgQpYI$>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>