RITA and Security Onion 2.3.180

[ng1991]

Hi,

I have started working with Security Onion and been requested to integrate RITA and although I know it isn't fully supported, I have followed the following two guides:

https://docs.securityonion.net/en/2.3/rita.html?highlight=rita
https://docs.securityonion.net/en/2.3/other-supported-logs.html?highlight=rita

These do not use the weslambert build (https://github.com/weslambert/securityonion-rita) is this the better option?

In terms of the current setup I have installed RITA on our forward nodes and I am using the below to create the CSV files:

rita import --rolling /nsm/zeek/logs/current $HOSTNAME
rita show-beacons $HOSTNAME > /nsm/rita/beacons.csv
rita show-exploded-dns $HOSTNAME > /nsm/rita/exploded-dns.csv
rita show-long-connections $HOSTNAME > /nsm/rita/long-connections.csv

If I look at the csv files created they are just a standard comma separated csv file.

I have also enabled rita on the SO minion.sls and can see the configuration is added to the yml file after i restart the filebeat container:

From the log I am not seeing much bar the below however I saw this in another post and this wasnt the issue and it was actually a shard problem (#8352)

{"log.level":"error","@timestamp":"2022-12-07T21:33:22.372Z","log.logger":"input.filestream","log.origin":{"file.name":"filestream/prospector.go","file.line":294},"message":"Error while stopping harvester group: task failures\n\terror while adding new reader to the bookkeeper harvester is already running for file\n\terror while adding new reader to the bookkeeper harvester is already running for file\n\terror while adding new reader to the bookkeeper harvester is already running for file","service.name":"filebeat","id":"rita-dns","prospector":"file_prospector","ecs.version":"1.6.0"}

I am unsure at this point how to troubleshoot this further so any assistance anyone can provide would be appreciated as in theory I cant really see any obvious problems

[ng1991]

Just to add to the above I have also tried the git repo Wes provides and ran that on the manager, I did get the below error:

However after running "sudo salt-call state.apply rita saltenv=custom queue=True" on one of my forward/sensor nodes and then so-rita-start followed by so-rita-update the files have successfully created in the /nsm/rita directory:

I still however dont see any logs and I have added the rita enable to the minion sls and can see the filebeat.yml has the inputs

Anyone else facing similar or have any suggestions to where I am going wrong?

I know this isnt supported so really just looking for any help

Thanks

[ng1991]

Increased log level for filebeat to debug and can see that the files are getting picked up but the inputs, also increased log level on Elasticsearch and logstash to check on the manager and search nodes but didn't see anything that helped further debug issue

[weslambert]

Does a regular Saltstack highstate complete without issue?

if you check the Elasticsearch indices from the CLI (from a search node, or manager node if running a traditional Elastic cluster), do you see an index for RITA logs?

so-elasticsearch-indices-list | grep rita

Do you see any clues in /var/log/logstash/logstash.log?

[weslambert]

Another option is have a look at the pipeline stats for RITA. Do you see anything that indicates records are being processed when running the following command on the search nodes?

for i in rita.beacon rita.connection rita.dns; do sudo so-elasticsearch-pipeline-stats $i; done

[ng1991]

Thanks for getting back to me again.

Unfortunately that returns a count of 0 on the search nodes.

As a test on the forward node I copied the yml file to the local directory and hardcoded the below in so the only input is Rita and it should just dump to file which is working correctly:

filebeat.inputs:
#------------------------------ Log prospector --------------------------------

• type: filestream
  id: rita-beacon
  paths:

  • /nsm/rita/beacons.csv
    exclude_lines: ['^Score', '^Source', '^Domain', '^No results']
    fields:
    module: rita
    dataset: beacon
    category: network
    processors:
  • drop_fields:
    fields: ["source", "prospector", "input", "offset", "beat"]
    fields_under_root: true
    pipeline: "rita.beacon"
    index: "so-rita"

• type: filestream
  id: rita-connection
  paths:

  • /nsm/rita/long-connections.csv
  • /nsm/rita/open-connections.csv
    exclude_lines: ['^Source', '^No results']
    fields:
    module: rita
    dataset: connection
    category: network
    processors:
  • drop_fields:
    fields: ["source", "prospector", "input", "offset", "beat"]
    fields_under_root: true
    pipeline: "rita.connection"
    index: "so-rita"

• type: filestream
  id: rita-dns
  paths:

  • /nsm/rita/exploded-dns.csv
    exclude_lines: ['^Domain', '^No results']
    fields:
    module: rita
    dataset: dns
    category: network
    processors:
  • drop_fields:
    fields: ["source", "prospector", "input", "offset", "beat"]
    fields_under_root: true
    pipeline: "rita.dns"
    index: "so-rita"

#----------------------------- Elasticsearch/Logstash output ---------------------------------

output.file:
path: "/tmp/filebeat"
filename: filebeat

So I do actually see the file it is generated in the container tmp directory when I update the file with new data so the filebeat is working correctly and outputting the data

[weslambert]

Try the following:

• Add id => rita_output to opt/so/saltstack/default/salt/logstash/pipelines/config/so/9801_output_rita.conf
• Restart Logstash
• Wait until you have confirmed new logs are being picked up by Filebeat
• On a search node, check if records have reached the Logstash RITA output configuration using so-logstash-pipeline-stats search | grep -iC 10 rita

[ng1991]

Hi Wes and again really appreciate the help you are providing.

I think the above may of highlighted the problem from the start, on my search nodes when i look in the /opt/so/conf/logstash/pipelines/search/ directory on my search nodes I cant see the rita output which I presume would explain why it never gets to elastic search. Below is a screenshot of the inputs and outputs I can see on the search nodes:

I can see the rita file you mentioned though on the manager and I added the config you requested

Any idea why the SALT would not be pushing this output?

[ng1991]

Think issue has been identified we have a unique need case and a copy of the search sls was made to add a custom output

I am now adding in the rita output as we still need the custom one, the issue was our side so sorry for the wasted time, do you know of a better way of making our custom output more permanent so if new outputs are added we don't miss them?

[ng1991]

Problem for this was caused by a custom modification made to logstash, this stopped new outputs from being published to the search nodes resulting in rita data not being processed and explaining why I saw no errors in the logs.

As we still needed the bespoke output we added the extra missing outputs into our custom search.sls located:

/opt/so/saltstack/local/pillar/logstash/search.sls

We added in the ones that were missing in our custom config from /opt/so/saltstack/default/pillar/logstash/search.sls

More detail can be seen in the chat.

Thank you @weslambert for you help in finding this issue