Integration outputting unparsed logs

[jaypfe]

DISCLAIMER: if this is not the correct area to ask this as it could be an issue with the integration itself, please let me know.

Okay, I am using the Fortinet Forti manager elastic integration to receive logs from my Forti analyzer. I am sending it syslog and it is outputting.

But all the output looks unparsed. It looks like it's just not calling the ingest pipeline at all. As when I run a test document against the pipeline it parses as expected. Looking into the settings I can see it is configured to use its default pipeline. Possibly it could be working as expected, just not indexing the parsed version? I don't see any Fortinet indices created unless I am missing them.

I wanted to bring it to your guy's attention as I feel it could be affecting these types of integrations as a whole rather than just an isolated incident. But if not, I am happy to get it worked out and possibly post in the show and tell area for others if I am fruitful.

[InfosecGoon]

Could you tell us a little more about the integration configuration? Running "sudo so-elastic-agent-inspect" will print the configuration information to the terminal, if you want to cut and paste the part related to the Fortinet data stream.

[jaypfe]

Absolutely! Please find that below. All that has changed from default is the listen address and port. I am sending syslog over port 8765 through a custom host/port group. I also flipped the "Preserve Original Event" switch just in testing, but it seems to have no effect.

---

id: tcp-fortinet_fortimanager-ad4ab7d7-ae06-49b9-ac2d-3b4d54706904
meta:
package:
name: fortinet_fortimanager
version: 2.1.0
name: fortinet_fortimanager-1
package_policy_id: ad4ab7d7-ae06-49b9-ac2d-3b4d54706904
revision: 11
streams:

• data_stream:
  dataset: fortinet_fortimanager.log
  type: logs
  host: 0.0.0.0:8765
  id: tcp-fortinet_fortimanager.log-ad4ab7d7-ae06-49b9-ac2d-3b4d54706904
  publisher_pipeline.disable_host: true
  ssl: null
  tags:
  • preserve_original_event
  • forwarded
  • fortinet_fortimanager-log
    type: tcp
    use_output: default
• data_stream:
  namespace: default
  id: udp-fortinet_fortimanager-ad4ab7d7-ae06-49b9-ac2d-3b4d54706904
  meta:
  package:
  name: fortinet_fortimanager
  version: 2.1.0
  name: fortinet_fortimanager-1
  package_policy_id: ad4ab7d7-ae06-49b9-ac2d-3b4d54706904
  revision: 11
  streams:
  • data_stream:
    dataset: fortinet_fortimanager.log
    type: logs
    host: 0.0.0.0:8765
    id: udp-fortinet_fortimanager.log-ad4ab7d7-ae06-49b9-ac2d-3b4d54706904
    publisher_pipeline.disable_host: true
    tags:
    • forwarded
    • fortinet_fortimanager-log

[InfosecGoon]

What are the event.module and event.dataset for the Fortinet events in Hunt?

[jaypfe]

They are fortinet_fortimanager.log and fortinet_fortimanager respectively.

I just spun up a new RC2 grid just to remove any variables as well.

[InfosecGoon]

Is RC2 behaving the same way?

[jaypfe]

Yes it is, no change.

[EddieN17]

This isn't the greatest solution, but I had a similar problem and my solution was just going into Kibana -> Management -> Integrations -> Installed Integrations -> {integration name} -> Settings then uninstalling and reinstalling until it worked and got mapped properly.

[jaypfe]

I tried that a few times to no avail. How many times did you have to do that? Did you remove all the integrations each time before uninstalling? Or did you just press the reinstall button until it worked?

[Bal33p]

I am running into the same issue with both Darketrace and SentinelOne integrations
Darktrace and SentinelOne integrations not populating data #10821

[jaypfe]

I have noticed this issue as well with the Alienvault OTX integration. It looks like the logs are not parsed but the pipeline works when ran manually. (This integration has had some shard failure as well but not sure if that's related)

[Bal33p]

Same for Palo Alto Firewall Logs

[EddieN17]

This was my solution that seems to work:
Go into Kibana -> Stack Management -> Index Management -> Data Streams and select the integration that is in question. Check under the "Index template" and see if it's so-logs, if so that's where I think the problem is, as that index template doesn't have the mappings for your integration.

Now we need to create our own index template to map our integrations, so check out /opt/so/saltstack/default/salt/elasticsearch/defaults.yaml for some example templates.

Now in the SOC Administration configuration web console, enable advanced settings (there's a warning there, so understand there is some risk for doing this manual fix around that worked for me), and then go to elasticsearch -> advanced and put your template there. Double check the spelling of the managed installed component template of the integration you installed in Kibana, as you'll need to route this newly created index template to those mappings.
Example from my end, a palo alto next gen firewall integration is:

elasticsearch:
  index_settings:
    so-panw-panos:
      index_sorting: False
      index_template:
        index_patterns:
          - "logs-panw.panos-*"
        template:
          settings:
            index:
              number_of_replicas: 0
        composed_of:
          - "logs-panw.panos@package"
          - "logs-panw.panos@custom"
          - "so-fleet_globals-1"
          - "so-fleet_agent_id_verification-1"
        priority: 501
        data_stream:
          hidden: false
          allow_custom_routing: false

Now apply the changes to elasticsearch (manually call or wait for highstate)

Go back to Kibana and check the data stream for your integration and confirmed the index template has changed to your newly created one. Now you need to create a new shard for the mappings to take effect to newly ingested logs (I wasn't able to find a way to fix the unmapped logs). I did this by reinstalling the integration. #10942 (comment)

Results may vary, but worked for me.

[jaypfe]

Yeah same thing. The stream updates as expected, as soon as I hit that reinstall button it stops writing any output.

The EPS reflects its still receiving, so the integration is holding the port open it looks like.

After reinstalling the integration the data steam shows a new index. The new one never gets any docs written to it.

[jaypfe]

If I stick the Fortinet pipeline onto the syslog pipeline and send it via syslog it works like a charm. I just can't seem to the get the integration to write to anything.

So, as of now I have a functional solution to my firewall problem, as jank as it may be.

[dbo266]

I am having the same issue. Please post if you get any more info on how to solve this. Hopefully Security Onion will get it together soon.

[EddieN17]

Can you show the file with the formatting of /opt/so/saltstack/local/pillar/elasticsearch/adv_elasticsearch.sls when you have created the index template as mentioned here:

#10942 (reply in thread)

[jaypfe]

Every time I try to indent it, it ignores the spacing. Here is a screen cap:

[jaypfe]

I believe I have found another workaround, adding the integrations default pipeline to the index that is created by the so-logs template.

1. start from scratch with your unparsed logs flowing (you may need to uninstall the integration and remove the data-steam then reinstall)

2. Navigate to Kibana > Stack Managment > Index Managment > Datateams and select your integration.

3. Click the number below the indices
   

4. Click on the index

5. Go to "Edit settings" and add a new field at the bottom ""index.default_pipeline": "Your Pipeline", like shown
   

6. Then just save, and it should be calling that pipeline and parsing your logs. (You can play with some other settings in the index as well to possibly trim some other meta fields out).

[weslambert]

Have you installed the latest hotfix? If not, you'll want to make sure you are updating before doing anything else.

https://blog.securityonion.net

As mentioned above, you'll need to add the index template for the integration so that it can use the correct component templates which reference the default pipeline.

[jaypfe]

Yes latest hotfix installed. Adding the template as shown with the component templates referenced by the integration created a new index that looked good, but I could not get anything to write to it.

[weslambert]

What is the output of so-elasticsearch-indices-list?

[weslambert]

Please start a new thread instead of jumping into the middle of an existing one. It looks like you started a similar discussion here: #11141

[jaypfe]

I am gonna drop a screen shot as the formatting is the best looking that way. I have docs being written into the index with that workout in place as posted above(I started using the fortigate integration rather than the manager one). The problem for me was when I used the @Package component template when creating the index template. I think it had some settings where something was not matching because it would not write a single doc into it. When I would not use the @pacakge it would write into it unparsed(obv).

If needed I can re-create my issue. I figure there are logs somewhere that may tell where in the pipe things are breaking but I am not sure where they are.

[weslambert]

What is the output of the following?

sudo so-elasticsearch-query .ds-logs-fortinet_fortigate.log-default-2023.02.22-000002/_settings | jq | grep pipeline

[weslambert]

Without adding the default pipeline manually, what default pipeline is configured for the index, and what index templates are associated to it? It shouldn't be the generic so-logs template, but the specific Fortigate template that is being applied, and it should inherit the default pipeline from the component templates.

[jaypfe]

The default pipeline its grabbing is:
"default_pipeline": "logs-fortinet_fortigate.log-1.14.0"

This is from the index template I created using logs-fortinet_fortigate.log@package and @Custom the ladder of which having no settings currently. I also used the standard "so-fleet_globals-1" and "so-fleet_agent_id_verification-1"

Do you want me to dump the config for the component templates and the index?

[weslambert]

It should work like the following:

• The integration is version n
• The index template that matches the data stream pattern points to the @package and @custom component templates and should be created within /opt/so/conf/elasticsearch/templates/index/. Most of the component templates are not stored on disk
• The @package component template should contain the default pipeline relative to the version of the installed integration
• The pipeline should be loaded inside of Elasticsearch; this can be checked with so-elasticsearch-pipelines-list

[jaypfe]

That all seems to be what is taking place. The default index template does not execute but after building one that does with all the correct component templates as shown to do in this thread, they are all show up as you described they should. I simply see nothing being written into the index. I wonder if the default FortiGate component templates are busted in some way.

The problem is in the indexing is my best guess. As the pipeline functions fine when called from the index created by so-logs. Perhaps the patterning? Mappings?

At this point I think I will spin up a new build once you guys drop your next version and see how it shakes out. Wouldnt be able to give a rough estimate on when that could be would you, just as far as days, weeks, or months?

[NexusChak]

Exactly experiencing this. Any update or solution on this?

[8bitjoe]

Same issue with sonicwall logs, data stream pointing at the so-logs index template, not the template that the integration should've installed, pipeline not being used.

[8bitjoe]

is this being looked at for the next update?

[weslambert]

Out-of-the-box support for additional integrations will be added in the next release. Initially we only supported a handful of the integrations.

https://github.com/Security-Onion-Solutions/securityonion/blob/2.4/main/salt/elasticfleet/defaults.yaml#L28-L44

There are some things that we have to do behind the scenes to manage these that doesn't happen by default with a traditional Elastic installation.

[8bitjoe]

Is there anyway you could shine some light on the process you guys do behind the scenes to manage these integrations? Sort of as a temporary work around while the next update comes.
I've been digging into the configurations and i believe it's something along the lines of adding the integration in the list of packages for elasticfleet, something like this:
elasticfleet:
packages:
- sonicwall
Then adding the component and index templates from the elastic integration to the elasticsearch conf folder and running the script 'so-elasticsearch-templates-load'
Am i on the right track?
Collectively we have identified that the datastream that gets created by the elastic agent doesn't select the right template/pipeline, not sure if i have to take a look at the index template pattern matching and priority as well?

Thank you so much for all the support, Security Onion is such a powerful tool, any help from the team is appreciated, and yes i understand it is an open source project, I'm just trying my luck requesting help. Thank you.