How to correctly use Fortigate elastic agent integration with Security Onion?

[soad20000]

Hello,

I am running a standalone on-prem deployment, version 2.4.30 installed from the ISO image, and I am trying to get syslogs sent from my internal/external fortigate firewalls to appear in Kibana. I have added the fortigate integration to the "so-grid-nodes_general" agent policy, which from my understanding is the policy used by my standalone deployment. I also tried adding it to the "FleetServer_<serverName>" policy. I have created a custom host group and custom port group with my fortigate IPs and the port 9004 respectively, (I've also assigned the custom port group to the custom host group) and I've verified that the rule exists with iptables.

I am able to see the packets flowing in on the correct port through tcpdump, here is some example output from tcpdump, taken from the standalone node.

Here is the firewall rule which is in the "INPUT" chain. As you can see, it's receiving traffic.

I added the same rule to the "DOCKER USER" chain just in case, that one isn't receiving traffic though.

Here is an overview of the elastic agent on my standalone node:

Here is the fortigate integration configuration, I also tried using "localhost" and "<server ip address>" as values for the listen address.

I've also restarted the elastic-agent on the server and it's currently running fine:

I've restarted the entire server, twice, as well.

I suspect that I've added the integration incorrectly (maybe to the wrong policy?) because in Kibana all the forti fields are empty.

Here is the output for sudo salt-call state.highstate and sudo so-status

I've also tried sending the logs via TCP instead of UDP, and adding the corresponding firewall rule for that but no luck. The way I understand it is, the logs are being received by SO on port 9004, SO then needs to forward that traffic to port 5055 which is the port for elastic agent data. Once forwarded then they should appear in Kibana right? Here is the relevant section of the policy that is applying to my standalone node which states where the output should go.

revision: 26
outputs:
  default:
    type: logstash
    hosts:
      - '\<server ip>\:5055'
      - '\<server name\>:5055'
      
      <----------------------SNIP--------------------->
  - id: tcp-fortinet_fortigate-877a5b8d-53ee-4d02-bb95-ae995170a27f
    name: fortinet_fortigate-1
    revision: 6
    type: tcp
    use_output: default
    meta:
      package:
        name: fortinet_fortigate
        version: 1.19.0
    data_stream:
      namespace: default
    package_policy_id: 877a5b8d-53ee-4d02-bb95-ae995170a27f
    streams:
      - id: tcp-fortinet_fortigate.log-877a5b8d-53ee-4d02-bb95-ae995170a27f
        data_stream:
          dataset: fortinet_fortigate.log
          type: logs
        host: 'localhost:9004'
        tags:
          - fortinet-fortigate
          - fortinet-firewall
          - forwarded
        publisher_pipeline.disable_host: true
        ssl: null
        framing: rfc6587
  - id: udp-fortinet_fortigate-877a5b8d-53ee-4d02-bb95-ae995170a27f
    name: fortinet_fortigate-1
    revision: 6
    type: udp
    use_output: default
    meta:
      package:
        name: fortinet_fortigate
        version: 1.19.0
    data_stream:
      namespace: default
    package_policy_id: 877a5b8d-53ee-4d02-bb95-ae995170a27f
    streams:
      - id: udp-fortinet_fortigate.log-877a5b8d-53ee-4d02-bb95-ae995170a27f
        data_stream:
          dataset: fortinet_fortigate.log
          type: logs
        host: 'localhost:9004'
        tags:
          - fortinet-fortigate
          - fortinet-firewall
          - forwarded
        publisher_pipeline.disable_host: true

The policy should be applying correctly:

Am I supposed to create a new agent policy with the fortigate integration, and then follow the on-screen instructions to enroll and install my SO node with fleet? I'm a little worried because I am under the impression that security onion 2.4.30 already has the elastic agent installed and I don't want to install it again, overwrite something important and end up breaking my installation. I thought that each agent could only be enrolled in 1 agent policy at a time.

I've been looking through every setting I can think of and I found this setting in the administration section of SO. I changed it to true, but still not getting any results.

Here is the status for the fortigate index, indicating that there are no documents in it:

EDIT 23/11/2023:

I've found what the error is. Logstash can't index the event to Elasticsearch because it's expecting the value of the "event.module" field to be "fortinet" and it comes in as "fortinet_fortigate".

[WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["create", {:_id=>nil, :_index=>"logs-fortinet_fortigate.log-default", :routing=>nil}, {"@version"=>"1", "ecs"=>{"version"=>"8.0.0"}, "message"=> REDACTED error"=>{"type"=>"document_parsing_exception", "reason"=>"[1:2087] failed to parse field [event.module] of type [constant_keyword] in document with id 'C4Fr_osB_bgS-nfCZW0F'. Preview of field's value: 'fortinet_fortigate'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"[constant_keyword] field [event.module] only accepts values that are equal to the value defined in the mappings [fortinet], but got [fortinet_fortigate]"}}}}}

I've tried creating a new index template (cloned from the original "logs-fortinet_fortigate.log") template and applying the following mapping to it:

 "dynamic_templates": [],
 "properties": {
   "event.module": {
     "type": "constant_keyword",
     "value": "fortinet_fortigate"
   }
 }
}

I've changed the index templates priority to be higher than the default one, and I've checked that it's applying to to the corresponding "logs-fortinet_fortigate.log-*" index.

I noticed that in the actual index, the mapping hasn't changed. Even after restarting the logstash container, so it keeps giving me the same error described above. I'm getting closer but damn, I've been at this for 2 days and my thoughts are starting to get tangled up. I'll update this post once I find the solution.

But I still can't see the logs anywhere in Kibana. The default fortigate dashboard comes up empty and the discover section doesn't contain any logs originating from my fortigate appliances. Am I missing something? Or could anyone point me in the right direction please?

EDIT

Ok so I reinstalled the integration. Manually deleted its indices and index templates, and I can now see documents being created from the data being ingested.

The issue now is that all the forti fields are missing from the dashboard. I tested the ingest pipeline, and it seems to be parsing correctly. I've also tried refreshing the index.

Thanks.

[soad20000]

Finally got it to work. So in summary here's what I did (these are two separate issues though, one was the documents not being generated and the other was the fields not appearing).

To fix the issue with the documents not being generated, I removed the integration and reinstalled it.

To fix the issue with the fields not appearing I copied over the index settings I had from my test environment, which is where the fields are defined. I noticed that the fortigate index in my production setup didn't have the same settings as my test setup and copying them over ended up fixing it.

[rjdub2]

In your post, you mentioned that you had to create a new index template, then reinstall the integration to fix the document generation. Do you think the updated template is necessary? Elastic's documentation said the integration was tested with FortiOS 6.x, but not with 7.x. I'm having the same issue with document generation, but haven't tried modifying the template yet. I am running FortiOS 7.x and am curious if this could be the issue.

[soad20000]

Well I can't say for sure in your case. Updating the index template should only change future indices, the changes you make won't apply to existing indices. You'd have to force a rollover on the fortigate index so that it creates a new index with the updated index template. Have you verified that you are receiving traffic on the correct port/interface that you setup in the integration? If your documents aren't appearing even in the discover section, check what the logs from logstash say. It could be that the logs aren't even making it to Elasticsearch, which was the issue in my case.

For reference, here is what the settings for my Fortigate index looks like.

  "index.blocks.read_only_allow_delete": "false",
  "index.refresh_interval": "1s",
  "index.write.wait_for_active_shards": "1",
  "index.mapping.total_fields.limit": "10000",
  "index.mapping.ignore_malformed": "false",
  "index.hidden": "true",
  "index.final_pipeline": ".fleet_final_pipeline-1",
  "index.query.default_field": [
    "cloud.account.id",
    "cloud.availability_zone",
    "cloud.instance.id",
    "cloud.instance.name",
    "cloud.machine.type",
    "cloud.provider",
    "cloud.region",
    "cloud.project.id",
    "cloud.image.id",
    "container.image.name",
    "container.name",
    "container.id",
    "host.architecture",
    "host.hostname",
    "host.id",
    "host.mac",
    "host.name",
    "host.os.family",
    "host.os.kernel",
    "host.os.name",
    "host.os.platform",
    "host.os.version",
    "host.os.build",
    "host.os.codename",
    "host.type",
    "log.source.address",
    "log.flags",
    "log.file.path",
    "log.level",
    "event.message",
    "event.category",
    "event.code",
    "event.kind",
    "event.outcome",
    "event.reference",
    "event.timezone",
    "event.type",
    "input.type",
    "destination.address",
    "destination.as.organization.name",
    "destination.domain",
    "destination.geo.city_name",
    "destination.geo.continent_name",
    "destination.geo.country_iso_code",
    "destination.geo.country_name",
    "destination.geo.name",
    "destination.geo.region_iso_code",
    "destination.geo.region_name",
    "destination.mac",
    "destination.user.email",
    "destination.user.name",
    "dns.id",
    "dns.question.class",
    "dns.question.name",
    "dns.question.type",
    "dns.question.registered_domain",
    "dns.question.subdomain",
    "dns.question.top_level_domain",
    "ecs.version",
    "email.cc.address",
    "email.from.address",
    "email.sender.address",
    "email.to.address",
    "email.subject",
    "error.code",
    "error.message",
    "file.extension",
    "file.name",
    "http.request.method",
    "http.request.referrer",
    "message",
    "network.application",
    "network.direction",
    "network.iana_number",
    "network.transport",
    "network.protocol",
    "observer.egress.interface.name",
    "observer.ingress.interface.name",
    "observer.name",
    "observer.product",
    "observer.serial_number",
    "observer.type",
    "observer.vendor",
    "related.hash",
    "related.hosts",
    "related.user",
    "rule.category",
    "rule.description",
    "rule.id",
    "rule.name",
    "rule.ruleset",
    "rule.uuid",
    "source.address",
    "source.as.organization.name",
    "source.geo.city_name",
    "source.geo.continent_name",
    "source.geo.country_iso_code",
    "source.geo.country_name",
    "source.geo.name",
    "source.geo.region_iso_code",
    "source.geo.region_name",
    "source.mac",
    "source.user.email",
    "source.user.group.name",
    "source.user.name",
    "tags",
    "threat.feed.name",
    "tls.cipher",
    "tls.client.issuer",
    "tls.client.server_name",
    "tls.client.x509.issuer.common_name",
    "tls.client.x509.public_key_algorithm",
    "tls.curve",
    "tls.server.hash.sha1",
    "tls.server.issuer",
    "tls.server.x509.alternative_names",
    "tls.server.x509.issuer.common_name",
    "tls.server.x509.public_key_algorithm",
    "tls.server.x509.serial_number",
    "tls.server.x509.subject.common_name",
    "tls.version",
    "tls.version_protocol",
    "url.domain",
    "url.path",
    "user_agent.original",
    "vulnerability.category",
    "fortinet.file.hash.crc32",
    "fortinet.firewall.acct_stat",
    "fortinet.firewall.acktime",
    "fortinet.firewall.action",
    "fortinet.firewall.activity",
    "fortinet.firewall.addr_type",
    "fortinet.firewall.addrgrp",
    "fortinet.firewall.adgroup",
    "fortinet.firewall.admin",
    "fortinet.firewall.agent",
    "fortinet.firewall.alert",
    "fortinet.firewall.analyticscksum",
    "fortinet.firewall.analyticssubmit",
    "fortinet.firewall.ap",
    "fortinet.firewall.app-type",
    "fortinet.firewall.appact",
    "fortinet.firewall.applist",
    "fortinet.firewall.apprisk",
    "fortinet.firewall.apscan",
    "fortinet.firewall.apsn",
    "fortinet.firewall.apstatus",
    "fortinet.firewall.aptype",
    "fortinet.firewall.attachment",
    "fortinet.firewall.attack",
    "fortinet.firewall.attackcontext",
    "fortinet.firewall.attackcontextid",
    "fortinet.firewall.auditreporttype",
    "fortinet.firewall.auditscore",
    "fortinet.firewall.authgrp",
    "fortinet.firewall.authid",
    "fortinet.firewall.authproto",
    "fortinet.firewall.authserver",
    "fortinet.firewall.bandwidth",
    "fortinet.firewall.banned_rule",
    "fortinet.firewall.banned_src",
    "fortinet.firewall.banword",
    "fortinet.firewall.bibandwidth",
    "fortinet.firewall.botnetdomain",
    "fortinet.firewall.bssid",
    "fortinet.firewall.call_id",
    "fortinet.firewall.carrier",
    "fortinet.firewall.carrier_ep",
    "fortinet.firewall.category",
    "fortinet.firewall.cc",
    "fortinet.firewall.cdrcontent",
    "fortinet.firewall.cert",
    "fortinet.firewall.cert-type",
    "fortinet.firewall.certhash",
    "fortinet.firewall.cfgattr",
    "fortinet.firewall.cfgobj",
    "fortinet.firewall.cfgpath",
    "fortinet.firewall.cfgtid",
    "fortinet.firewall.channeltype",
    "fortinet.firewall.checksum",
    "fortinet.firewall.chgheaders",
    "fortinet.firewall.cldobjid",
    "fortinet.firewall.client_addr",
    "fortinet.firewall.cloudaction",
    "fortinet.firewall.clouduser",
    "fortinet.firewall.command",
    "fortinet.firewall.community",
    "fortinet.firewall.configcountry",
    "fortinet.firewall.connection_type",
    "fortinet.firewall.conserve",
    "fortinet.firewall.constraint",
    "fortinet.firewall.contentdisarmed",
    "fortinet.firewall.contenttype",
    "fortinet.firewall.cookies",
    "fortinet.firewall.crl",
    "fortinet.firewall.crlevel",
    "fortinet.firewall.cveid",
    "fortinet.firewall.daemon",
    "fortinet.firewall.datarange",
    "fortinet.firewall.date",
    "fortinet.firewall.desc",
    "fortinet.firewall.detectionmethod",
    "fortinet.firewall.devcategory",
    "fortinet.firewall.devintfname",
    "fortinet.firewall.devtype",
    "fortinet.firewall.dhcp_msg",
    "fortinet.firewall.dintf",
    "fortinet.firewall.disk",
    "fortinet.firewall.dlpextra",
    "fortinet.firewall.docsource",
    "fortinet.firewall.domainctrldomain",
    "fortinet.firewall.domainctrlname",
    "fortinet.firewall.domainctrlusername",
    "fortinet.firewall.domainfilterlist",
    "fortinet.firewall.ds",
    "fortinet.firewall.dst_int",
    "fortinet.firewall.dstintfrole",
    "fortinet.firewall.dstcountry",
    "fortinet.firewall.dstdevcategory",
    "fortinet.firewall.dstdevtype",
    "fortinet.firewall.dstfamily",
    "fortinet.firewall.dsthwvendor",
    "fortinet.firewall.dsthwversion",
    "fortinet.firewall.dstinetsvc",
    "fortinet.firewall.dstosname",
    "fortinet.firewall.dstosversion",
    "fortinet.firewall.dstssid",
    "fortinet.firewall.dstswversion",
    "fortinet.firewall.dstunauthusersource",
    "fortinet.firewall.dstuuid",
    "fortinet.firewall.duid",
    "fortinet.firewall.eapoltype",
    "fortinet.firewall.encryption",
    "fortinet.firewall.espauth",
    "fortinet.firewall.esptransform",
    "fortinet.firewall.exch",
    "fortinet.firewall.exchange",
    "fortinet.firewall.expectedsignature",
    "fortinet.firewall.expiry",
    "fortinet.firewall.fctemssn",
    "fortinet.firewall.fctuid",
    "fortinet.firewall.field",
    "fortinet.firewall.filefilter",
    "fortinet.firewall.filehashsrc",
    "fortinet.firewall.filtercat",
    "fortinet.firewall.filtername",
    "fortinet.firewall.filtertype",
    "fortinet.firewall.fortiguardresp",
    "fortinet.firewall.forwardedfor",
    "fortinet.firewall.fqdn",
    "fortinet.firewall.frametype",
    "fortinet.firewall.from",
    "fortinet.firewall.fsaverdict",
    "fortinet.firewall.fwserver_name",
    "fortinet.firewall.green",
    "fortinet.firewall.ha_group",
    "fortinet.firewall.ha_role",
    "fortinet.firewall.handshake",
    "fortinet.firewall.hash",
    "fortinet.firewall.hbdn_reason",
    "fortinet.firewall.healthcheck",
    "fortinet.firewall.host",
    "fortinet.firewall.iaid",
    "fortinet.firewall.iccid",
    "fortinet.firewall.icmpcode",
    "fortinet.firewall.icmpid",
    "fortinet.firewall.icmptype",
    "fortinet.firewall.imei",
    "fortinet.firewall.imsi",
    "fortinet.firewall.in_spi",
    "fortinet.firewall.inbandwidth",
    "fortinet.firewall.informationsource",
    "fortinet.firewall.init",
    "fortinet.firewall.initiator",
    "fortinet.firewall.interface",
    "fortinet.firewall.intf",
    "fortinet.firewall.invalidmac",
    "fortinet.firewall.iptype",
    "fortinet.firewall.keyword",
    "fortinet.firewall.kind",
    "fortinet.firewall.kxproto",
    "fortinet.firewall.license_limit",
    "fortinet.firewall.line",
    "fortinet.firewall.log",
    "fortinet.firewall.login",
    "fortinet.firewall.mac",
    "fortinet.firewall.malform_desc",
    "fortinet.firewall.manuf",
    "fortinet.firewall.masterdstmac",
    "fortinet.firewall.mastersrcmac",
    "fortinet.firewall.meshmode",
    "fortinet.firewall.message_type",
    "fortinet.firewall.method",
    "fortinet.firewall.metric",
    "fortinet.firewall.mitm",
    "fortinet.firewall.mode",
    "fortinet.firewall.module",
    "fortinet.firewall.monitor-name",
    "fortinet.firewall.monitor-type",
    "fortinet.firewall.mpsk",
    "fortinet.firewall.msgproto",
    "fortinet.firewall.name",
    "fortinet.firewall.nat",
    "fortinet.firewall.netid",
    "fortinet.firewall.new_status",
    "fortinet.firewall.new_value",
    "fortinet.firewall.newvalue",
    "fortinet.firewall.nf_type",
    "fortinet.firewall.old_status",
    "fortinet.firewall.old_value",
    "fortinet.firewall.oldsn",
    "fortinet.firewall.oldvalue",
    "fortinet.firewall.oldwprof",
    "fortinet.firewall.onwire",
    "fortinet.firewall.opercountry",
    "fortinet.firewall.osname",
    "fortinet.firewall.osversion",
    "fortinet.firewall.out_spi",
    "fortinet.firewall.outbandwidth",
    "fortinet.firewall.outintf",
    "fortinet.firewall.packetloss",
    "fortinet.firewall.passwd",
    "fortinet.firewall.path",
    "fortinet.firewall.peer",
    "fortinet.firewall.peer_notif",
    "fortinet.firewall.phase2_name",
    "fortinet.firewall.phone",
    "fortinet.firewall.phonenumber",
    "fortinet.firewall.plan",
    "fortinet.firewall.policytype",
    "fortinet.firewall.poluuid",
    "fortinet.firewall.poolname",
    "fortinet.firewall.probeproto",
    "fortinet.firewall.process",
    "fortinet.firewall.profile",
    "fortinet.firewall.profile_vd",
    "fortinet.firewall.profilegroup",
    "fortinet.firewall.profiletype",
    "fortinet.firewall.quarskip",
    "fortinet.firewall.quotaexceeded",
    "fortinet.firewall.quotatype",
    "fortinet.firewall.radioband",
    "fortinet.firewall.rate",
    "fortinet.firewall.rawdata",
    "fortinet.firewall.rawdataid",
    "fortinet.firewall.rcvddelta",
    "fortinet.firewall.reason",
    "fortinet.firewall.receivedsignature",
    "fortinet.firewall.red",
    "fortinet.firewall.referralurl",
    "fortinet.firewall.remotewtptime",
    "fortinet.firewall.reporttype",
    "fortinet.firewall.reqtype",
    "fortinet.firewall.request_name",
    "fortinet.firewall.result",
    "fortinet.firewall.role",
    "fortinet.firewall.rsso_key",
    "fortinet.firewall.ruledata",
    "fortinet.firewall.ruletype",
    "fortinet.firewall.scope",
    "fortinet.firewall.security",
    "fortinet.firewall.sensitivity",
    "fortinet.firewall.sensor",
    "fortinet.firewall.sentdelta",
    "fortinet.firewall.seq",
    "fortinet.firewall.serial",
    "fortinet.firewall.serialno",
    "fortinet.firewall.server",
    "fortinet.firewall.session_id",
    "fortinet.firewall.severity",
    "fortinet.firewall.shaperperipname",
    "fortinet.firewall.shaperrcvdname",
    "fortinet.firewall.shapersentname",
    "fortinet.firewall.ski",
    "fortinet.firewall.slamap",
    "fortinet.firewall.slatargetid",
    "fortinet.firewall.sn",
    "fortinet.firewall.snclosest",
    "fortinet.firewall.sndetected",
    "fortinet.firewall.snmeshparent",
    "fortinet.firewall.spi",
    "fortinet.firewall.src_int",
    "fortinet.firewall.srcintfrole",
    "fortinet.firewall.srccountry",
    "fortinet.firewall.srcfamily",
    "fortinet.firewall.srchwvendor",
    "fortinet.firewall.srchwversion",
    "fortinet.firewall.srcinetsvc",
    "fortinet.firewall.srcname",
    "fortinet.firewall.srcssid",
    "fortinet.firewall.srcswversion",
    "fortinet.firewall.srcuuid",
    "fortinet.firewall.sscname",
    "fortinet.firewall.ssid",
    "fortinet.firewall.sslaction",
    "fortinet.firewall.ssllocal",
    "fortinet.firewall.sslremote",
    "fortinet.firewall.stage",
    "fortinet.firewall.stamac",
    "fortinet.firewall.state",
    "fortinet.firewall.status",
    "fortinet.firewall.stitch",
    "fortinet.firewall.subject",
    "fortinet.firewall.submodule",
    "fortinet.firewall.subservice",
    "fortinet.firewall.subtype",
    "fortinet.firewall.switchproto",
    "fortinet.firewall.sync_status",
    "fortinet.firewall.sync_type",
    "fortinet.firewall.sysuptime",
    "fortinet.firewall.tamac",
    "fortinet.firewall.threattype",
    "fortinet.firewall.time",
    "fortinet.firewall.timestamp",
    "fortinet.firewall.to",
    "fortinet.firewall.trace_id",
    "fortinet.firewall.trandisp",
    "fortinet.firewall.translationid",
    "fortinet.firewall.trigger",
    "fortinet.firewall.tunneltype",
    "fortinet.firewall.type",
    "fortinet.firewall.ui",
    "fortinet.firewall.unauthusersource",
    "fortinet.firewall.urlfilterlist",
    "fortinet.firewall.urlsource",
    "fortinet.firewall.urltype",
    "fortinet.firewall.utmaction",
    "fortinet.firewall.utmref",
    "fortinet.firewall.vap",
    "fortinet.firewall.vapmode",
    "fortinet.firewall.vcluster_state",
    "fortinet.firewall.vd",
    "fortinet.firewall.vdname",
    "fortinet.firewall.vendorurl",
    "fortinet.firewall.version",
    "fortinet.firewall.vip",
    "fortinet.firewall.virus",
    "fortinet.firewall.voip_proto",
    "fortinet.firewall.vpn",
    "fortinet.firewall.vpntunnel",
    "fortinet.firewall.vpntype",
    "fortinet.firewall.vulncat",
    "fortinet.firewall.vulnname",
    "fortinet.firewall.vwlquality",
    "fortinet.firewall.vwlservice",
    "fortinet.firewall.wanoptapptype",
    "fortinet.firewall.weakwepiv",
    "fortinet.firewall.xauthgroup",
    "fortinet.firewall.xauthuser"
  ],
  "index.priority": "1",
  "index.number_of_replicas": "1",
  "index.lifecycle.name": "so-logs-fortinet_fortigate.log-logs",
  "index.lifecycle.rollover_alias": "",
  "index.lifecycle.indexing_complete": "true",
  "index.routing.allocation.include._tier_preference": "data_hot",
  "index.default_pipeline": "logs-fortinet_fortigate.log-1.19.0"
}

[rjdub2]

Thanks for getting back to me. I ended up following the suggestion in #11730. I think this has to do with the current 7.x FortiOS where the Elasticsearch integration has only been tested with 6.x.