Fortigate logs not being processed on 2.4

[ebdavison]

Version

2.4.40

Installation Method

Security Onion ISO image

Description

configuration

Installation Type

Standalone

Location

other (please provide detail below)

Hardware Specs

Exceeds minimum requirements

CPU

6

RAM

16GB

Storage for /

588G

Storage for /nsm

588G ( part of / )

Network Traffic Collection

other (please provide detail below)

Network Traffic Speeds

1Gbps to 10Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

Yes, there are additional clues in /opt/so/log/ (please provide detail below)

Detail

I have a fortigate running 7.x and am sending logs to my SO instance. I am receiving logs as verified with tcpdump but they do not show in ES/Kibana. I have enabled the fortigate integration and am sending logs on UDP 9004 using rfc5424 and per this post have followed the advice: #11995 (comment).

tcpdump:

root@edoras:~# tcpdump -i eth0 'host 73.13.4.219 and port 9004' -n -A -c 5
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
00:13:36.168693 IP 73.13.4.219.18702 > 207.244.239.16.9004: UDP, length 866
E..~....8...I.......I.#,.j..<190>1 2024-02-16T00:13:36Z retire01-pafor380-fw - - - - eventtime=1708042415154446820 tz="-0500" logid="0315012546" type="utm" subtype="webfilter" eventtype="urlfilter" level="information" vd="root" urlfilteridx=1 urlfilterlist="default" policyid=2 poluuid="c9db7ef4-a0fc-51ed-ef3c-29c00473297d" policytype="policy" sessionid=8591480 srcip=10.113.11.104 srcport=54356 srccountry="Reserved" srcintf="Retirement.Net" srcintfrole="lan" srcuuid="71352894-9f4f-51ed-4ca9-8d2c764d3262" dstip=35.224.251.107 dstport=443 dstcountry="United States" dstintf="wan1" dstintfrole="wan" dstuuid="499929d8-9f4b-51ed-8350-b255054d723c" proto=6 service="HTTPS" hostname="cloud.iexapis.com" profile="default" action="passthrough" reqtype="direct" url="https://cloud.iexapis.com/" sentbyte=517 rcvdbyte=0 direction="outgoing" msg="URL was allowed because it is in the URL filter list"
00:13:36.168693 IP 73.13.4.219.18702 > 207.244.239.16.9004: UDP, length 959
E.......8...I.......I.#,....<189>1 2024-02-16T00:13:36Z retire01-pafor380-fw - - - - eventtime=1708042415231355121 tz="-0500" logid="0000000020" type="traffic" subtype="forward" level="notice" vd="root" srcip=10.1.10.6 srcname="Polycom64167f08222c" srcport=5060 srcintf="internal" srcintfrole="lan" dstip=199.87.121.10 dstport=5061 dstintf="wan1" dstintfrole="wan" srccountry="Reserved" dstinetsvc="LogMeIn-Other" dstcountry="United States" dstcity="Chicago" dstreputation=5 sessionid=7676757 proto=17 action="accept" policyid=29 policytype="policy" poluuid="a37ed796-1439-51ee-bd20-5fbda05c0822" service="LogMeIn-Other" trandisp="snat" transip=73.13.4.219 transport=65476 duration=106940 sentbyte=18243070 rcvdbyte=11859606 sentpkt=23382 rcvdpkt=26976 appcat="unscanned" sentdelta=20302 rcvddelta=12062 srchwvendor="Poly" devtype="IP Phone" srcfamily="VVX" osname="EOS" srchwversion="401" srcswversion="6.4.3.5814" mastersrcmac="64:16:7f:08:22:2c" srcmac="64:16:7f:08:22:2c" srcserver=0

I do see this in the logs for logstash but it does not look related:

=========================================================================
 Checking log file /opt/so/log/logstash/logstash.log
=========================================================================
[2024-02-15T17:40:46,851][ERROR][logstash.inputs.redis    ] Unexpected error {:message=>"Socket closed", :exception=>OpenSSL::SSL::SSLError, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/connection/ruby.rb:264:in `connect'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/connection/ruby.rb:306:in `connect'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:385:in `establish_connection'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:115:in `block in connect'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:344:in `with_reconnect'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:114:in `connect'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:409:in `ensure_connected'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:269:in `block in process'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:375:in `logging'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:268:in `process'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:161:in `call'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis.rb:270:in `block in send_command'", "org/jruby/ext/monitor/Monitor.java:82:in `synchronize'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis.rb:269:in `send_command'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/commands/scripting.rb:110:in `_eval'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/commands/scripting.rb:97:in `evalsha'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-redis-3.7.0/lib/logstash/inputs/redis.rb:197:in `list_batch_listener'", "org/jruby/RubyMethod.java:130:in `call'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-redis-3.7.0/lib/logstash/inputs/redis.rb:187:in `list_runner'", "org/jruby/RubyMethod.java:124:in `call'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-redis-3.7.0/lib/logstash/inputs/redis.rb:87:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:414:in `inputworker'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:405:in `block in start_input'"]}
[2024-02-15T17:40:46,852][ERROR][logstash.inputs.redis    ] Unexpected error {:message=>"Socket closed", :exception=>OpenSSL::SSL::SSLError, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/connection/ruby.rb:264:in `connect'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/connection/ruby.rb:306:in `connect'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:385:in `establish_connection'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:115:in `block in connect'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:344:in `with_reconnect'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:114:in `connect'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:409:in `ensure_connected'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:269:in `block in process'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:375:in `logging'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:268:in `process'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/client.rb:161:in `call'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis.rb:270:in `block in send_command'", "org/jruby/ext/monitor/Monitor.java:82:in `synchronize'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis.rb:269:in `send_command'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/commands/scripting.rb:110:in `_eval'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/redis-4.8.1/lib/redis/commands/scripting.rb:97:in `evalsha'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-redis-3.7.0/lib/logstash/inputs/redis.rb:197:in `list_batch_listener'", "org/jruby/RubyMethod.java:130:in `call'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-redis-3.7.0/lib/logstash/inputs/redis.rb:187:in `list_runner'", "org/jruby/RubyMethod.java:124:in `call'", "/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-input-redis-3.7.0/lib/logstash/inputs/redis.rb:87:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:414:in `inputworker'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:405:in `block in start_input'"]}

These logs were working perfectly on SO 2.3.x but had to re-install from scratch to get to 2.4 and since then these Fortigate logs have never worked.

Not sure what to check next but really need to get these logs processed.

Please advise

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[hadi-d99]

check this

#11995

same problem for me and it is resolved

[ebdavison]

tried that first and it does not solve it for me. Current I have that set per #11995 already.

[amorphys]

Thank you ! Works like a charm !

[ebdavison]

Still having problems with this one and have resorted to syslog on port 514 as that way the logs are at least arriving and are searchable.

I have spent way to long on this integration and would be happy to parse out the logs in kv format to make them searchable.

I have tried a custom parser and put it in /opt/so/saltstack/local/salt/elasticsearch/files/ingest and it never started up a listener.

Here is what I have in the file I created:

#FIREWALLS
input {
  udp {
    port => 9004
    type => fortigate
  }
}

filter {
  if [type] == "fortigate" {



        kv {
                source => "message"
                exclude_keys => [ "type", "subtype" ] }
                geoip { source => "dst" }
                geoip { source => "dstip" }
                geoip { source => "src" }
                geoip { source => "srcip" }

                        mutate {

    rename => [ "dst", "dst_ip" ]
    rename => [ "dstip", "dst_ip" ]
        rename => [ "dstport", "dst_port" ]
        rename => [ "devname", "device_id" ]
        rename => [ "status", "action" ]
        rename => [ "src", "src_ip" ]
        rename => [ "srcip", "src_ip" ]
        rename => [ "zone", "src_intf" ]
        rename => [ "srcintf", "src_intf" ]
        rename => [ "srcport", "src_port" ]
        rename => [ "rcvd", "byte_recieved" ]
        rename => [ "rcvdbyte", "bytes_recieved" ]
        rename => [ "sentbyte", "bytes_sent" ]
        rename => [ "sent", "bytes_sent" ]
        convert => ["bytes_recieved", "integer"]
        convert => ["bytes_sent", "integer"]
        remove_field => [ "msg" ]

  }


  }
}

After 2 months I need to get this sorted out so the logs are searchable and not just one big message field.

Can someone help me with this???

[jaypfe]

Have you tried using the deprecated standard Fortinet module? I have had success with that one.

[ebdavison]

not sure what that one is to be honest. if you can point me in the right direct, happy to try it out

[TotieBash]

Can you screen shot the result of the following commands:

1. This will tell us if the Elastic Fleet integration and the Elastic agent is configured properly.
   ss -tulpn | grep 9004

2. This will tell us if the firewall section is configured properly. If I where a betting man I think this is your problem. I know this section is confusing to me moving from version 2.3 to 2.4.
   iptables --list -n | grep 9004

3. Finally, this will tell us if the Fortigate syslogs are reaching SO mgmt interface.
   tcpdump -i eth0 -nnA port 9004

see the following for refernce:
#12833
#12245

[ebdavison]

thanks for the response.

Here is the screenshot. I think it is pretty evident with 9004 not listening that something is amiss

[jaypfe]

What role did you assign the custom port group to that you made?

Manager?

[TotieBash]

Base on that screenshot it looks like the the Elastic Fleet integration is missing or misconfigured. Since your we use both "Standalone" install type you should tweak "so-grid-nodes_general"

#12055

[ebdavison]

Wow! Partial success! Thanks!

Well, it never occurred to me that the default listen on localhost was a problem. Changing it to 0.0.0.0 now at least ss is showing that 9004 is listening by carefully reading and following those instructions.

Now, I get a different error about mappings:

[2024-05-04T21:41:27,408][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["create", {:_id=>nil, :_index=>"logs-fortinet_fortigate.log-default", :routing=>nil}, {"message"=>"<189>1 2024-05-04T21:41:25Z fortiagate-fw - - - - eventtime=1714858884321992900 tz=\"-0400\" logid=\"0000000020\" type=\"traffic\" subtype=\"forward\" level=\"notice\" vd=\"root\" srcip=10.1.10.108 srcname=\"PERSON-PC\" srcport=62303 srcintf=\"internal\" srcintfrole=\"lan\" dstip=142.251.167.188 dstport=443 dstintf=\"wan1\" dstintfrole=\"wan\" srccountry=\"Reserved\" dstinetsvc=\"Google-Web\" dstcountry=\"United States\" dstregion=\"California\" dstcity=\"Mountain View\" dstreputation=5 sessionid=8971409 proto=6 action=\"accept\" policyid=1 policytype=\"policy\" poluuid=\"4f776180-9f4b-51ed-cf12-10d98d890ae3\" service=\"Google-Web\" trandisp=\"snat\" transip=1.2.3.4 transport=62303 duration=2565 sentbyte=4991 rcvdbyte=12075 sentpkt=69 rcvdpkt=70 appcat=\"unscanned\" sentdelta=123 rcvddelta=156 srchwvendor=\"HP\" srcfamily=\"Computer\" osname=\"Windows\" srcswversion=\"10\" mastersrcmac=\"9c:7b:ef:3d:71:3d\" srcmac=\"9c:7b:ef:3d:71:3d\" srcserver=0", "tags"=>["fortinet-fortigate", "fortinet-firewall", "forwarded", "elastic-agent", "input-edoras", "beats_input_codec_plain_applied"], "agent"=>{"type"=>"filebeat", "id"=>"ff292607-eacd-4a52-a487-cc4575ae23d8", "ephemeral_id"=>"219c0fb1-7ce6-4516-b3c1-ab5759d8f73b", "name"=>"edoras", "version"=>"8.10.4"}, "data_stream"=>{"namespace"=>"default", "type"=>"logs", "dataset"=>"fortinet_fortigate.log"}, "input"=>{"type"=>"udp"}, "event"=>{"dataset"=>"fortinet_fortigate.log"}, "type"=>"redis-input", "@timestamp"=>2024-05-04T21:41:25.089Z, "ecs"=>{"version"=>"8.0.0"}, "@version"=>"1", "log"=>{"source"=>{"address"=>"73.13.4.219:16426"}}, "elastic_agent"=>{"id"=>"ff292607-eacd-4a52-a487-cc4575ae23d8", "snapshot"=>false, "version"=>"8.10.4"}, "metadata"=>{"input"=>{"beats"=>{"host"=>{"ip"=>"207.244.239.16"}}}, "type"=>"_doc", "raw_index"=>"logs-fortinet_fortigate.log-default", "stream_id"=>"udp-fortinet_fortigate.log-d0631156-c8a4-416c-a54c-147328cbbbf4", "truncated"=>false, "beat"=>"filebeat", "version"=>"8.10.4", "input_id"=>"udp-fortinet_fortigate-d0631156-c8a4-416c-a54c-147328cbbbf4"}}], :response=>{"create"=>{"_index"=>".ds-logs-fortinet_fortigate.log-default-2024.05.04-000001", "_id"=>"VO-ORY8BeYQqngnhCJHk", "status"=>400, "error"=>{"type"=>"document_parsing_exception", "reason"=>"[1:2911] failed to parse field [event.module] of type [constant_keyword] in document with id 'VO-ORY8BeYQqngnhCJHk'. Preview of field's value: 'fortinet_fortigate'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"[constant_keyword] field [event.module] only accepts values that are equal to the value defined in the mappings [fortinet], but got [fortinet_fortigate]"}}}}}
[2024-05-04T21:41:27,408][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["create", {:_id=>nil, :_index=>"logs-fortinet_fortigate.log-default", :routing=>nil}, {"message"=>"<190>1 2024-05-04T21:41:25Z firewall-fw - - - - eventtime=1714858884720352159 tz=\"-0400\" logid=\"0315012546\" type=\"utm\" subtype=\"webfilter\" eventtype=\"urlfilter\" level=\"information\" vd=\"root\" urlfilteridx=1 urlfilterlist=\"default\" policyid=2 poluuid=\"c9db7ef4-a0fc-51ed-ef3c-29c00473297d\" policytype=\"policy\" sessionid=8984096 srcip=10.113.11.118 srcport=53873 srccountry=\"Reserved\" srcintf=\"Retirement.Net\" srcintfrole=\"lan\" srcuuid=\"71352894-9f4f-51ed-4ca9-8d2c764d3262\" dstip=104.237.141.107 dstport=443 dstcountry=\"United States\" dstintf=\"wan1\" dstintfrole=\"wan\" dstuuid=\"499929d8-9f4b-51ed-8350-b255054d723c\" proto=6 service=\"HTTPS\" hostname=\"api.watchmylog.com\" profile=\"default\" action=\"passthrough\" reqtype=\"direct\" url=\"https://api.watchmylog.com/\" sentbyte=284 rcvdbyte=0 direction=\"outgoing\" msg=\"URL was allowed because it is in the URL filter list\"", "tags"=>["fortinet-fortigate", "fortinet-firewall", "forwarded", "elastic-agent", "input-edoras", "beats_input_codec_plain_applied"], "agent"=>{"type"=>"filebeat", "id"=>"ff292607-eacd-4a52-a487-cc4575ae23d8", "ephemeral_id"=>"219c0fb1-7ce6-4516-b3c1-ab5759d8f73b", "name"=>"edoras", "version"=>"8.10.4"}, "data_stream"=>{"namespace"=>"default", "type"=>"logs", "dataset"=>"fortinet_fortigate.log"}, "event"=>{"dataset"=>"fortinet_fortigate.log"}, "input"=>{"type"=>"udp"}, "type"=>"redis-input", "@timestamp"=>2024-05-04T21:41:25.089Z, "ecs"=>{"version"=>"8.0.0"}, "@version"=>"1", "log"=>{"source"=>{"address"=>"1.2.3.4:16426"}}, "elastic_agent"=>{"snapshot"=>false, "id"=>"ff292607-eacd-4a52-a487-cc4575ae23d8", "version"=>"8.10.4"}, "metadata"=>{"input"=>{"beats"=>{"host"=>{"ip"=>"207.244.239.16"}}}, "type"=>"_doc", "raw_index"=>"logs-fortinet_fortigate.log-default", "stream_id"=>"udp-fortinet_fortigate.log-d0631156-c8a4-416c-a54c-147328cbbbf4", "truncated"=>false, "beat"=>"filebeat", "version"=>"8.10.4", "input_id"=>"udp-fortinet_fortigate-d0631156-c8a4-416c-a54c-147328cbbbf4"}}], :response=>{"create"=>{"_index"=>".ds-logs-fortinet_fortigate.log-default-2024.05.04-000001", "_id"=>"Ve-ORY8BeYQqngnhCJHk", "status"=>400, "error"=>{"type"=>"document_parsing_exception", "reason"=>"[1:2548] failed to parse field [event.module] of type [constant_keyword] in document with id 'Ve-ORY8BeYQqngnhCJHk'. Preview of field's value: 'fortinet_fortigate'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"[constant_keyword] field [event.module] only accepts values that are equal to the value defined in the mappings [fortinet], but got [fortinet_fortigate]"}}}}}

[ebdavison]

I think I found the mapping to fix ... attempting this now

[ebdavison]

success!!!!

Thank you @TotieBash

[TotieBash]

glad it all worked out...