Extracted files not showing in ZEEK directory

[masedira]

Version

2.4.110

Installation Method

Security Onion ISO image

Description

other (please provide detail below)

Installation Type

Standalone

Location

on-prem with Internet access

Hardware Specs

Meets minimum requirements

CPU

8

RAM

20

Storage for /

300

Storage for /nsm

300

Network Traffic Collection

span port

Network Traffic Speeds

Less than 1Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

Hi all,
I am new to SO and have a basic background in zeek, so apologies if the question is basic.

I am extracting files from zeek and the zeek files log shows the extraction, however when I check the directory it is always empty.

Where can I find these extracted files or where do they go?

Thanks

files.log:
{"ts":1729172370.488029,"fuid":"Fk4Q9k3T7tAhnsqvGi","uid":"CSrC0131sWJx5bXvW1","id.orig_h":"192.168.200.103","id.orig_p":55476,"id.resp_h":"xx.xx.xx.xx","id.resp_p":443,"source":"HTTP","depth":0,"analyzers":["MD5","EXTRACT","SHA1"],"mime_type":"text/plain","duration":0.0,"local_orig":false,"is_orig":false,"seen_bytes":68,"total_bytes":68,"missing_bytes":0,"overflow_bytes":0,"timedout":false,"md5":"44d88612fea8a8f36de82e1278abb02f","sha1":"3395856ce81f2b7382dee72602f798b642f14140","extracted":"/nsm/zeek/extracted/complete/HTTP-Fk4Q9k3T7tAhnsqvGi-44d88612fea8a8f36de82e1278abb02f.com","extracted_cutoff":false}

directory:
[root@securityonion-1 admin]# ls -al /nsm/zeek/extracted/complete/
total 0
drwxrwx---. 2 zeek socore 6 Oct 17 13:39 .
drwxrwx---. 3 zeek socore 22 Oct 17 13:39 ..

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[dougburks]

When Zeek extracts files, they are then analyzed by Strelka:
https://docs.securityonion.net/en/2.4/strelka.html

[masedira]

Hi @dougburks ,

is there a way to configure SO and keep ALL the extracted files somewhere?

The strelka directory only has a handful of files and not all the ones that show as extracted from zeek files log.

[dougburks]

@masedira From https://docs.securityonion.net/en/2.4/strelka.html:

Security Onion checks file hashes before sending to Strelka to avoid analyzing the same file multiple times in a 48 hour period.

Could this explain the files that you are missing? You can take a look at the filecheck.log like this:

grep "Already exists" /opt/so/log/strelka/filecheck.log

[masedira]

Hi @dougburks
first of all, thanks a lot for your clear precise answers, it helped me a lot!

Yes, you are right, the log file shows that the file already exists.

The thing is I want to do a live demo for a customer showing that the files get extracted and stored.

So I want to show the directory /nsm/strelka/processed being empty, then download a test virus file and show that its available in the directory after extraction.

I attempted to remove the files from the strelka directory, and re-download, but the log still shows that the file already exists.

Is there a quick way to clear the file hashes "cache" so that when I download the same file again after deleting it, SO doesn't think it already exists?

=============
[root@securityonion-1 ~]# date
Tue Oct 22 10:48:01 PM UTC 2024
[root@securityonion-1 ~]#

[root@securityonion-1 ~]# ls -ltrh /nsm/strelka/processed/
total 0

[root@securityonion-1 ~]# grep "Already exists" /opt/so/log/strelka/filecheck.log
22-Oct-24 22:48:20 - /nsm/zeek/extracted/complete/HTTP-FLxSwh1K33Ej0MQ5c3-44d88612fea8a8f36de82e1278abb02f.txt Already exists.. removing
[root@securityonion-1 ~]#

[dougburks]

Have you tried removing the files in /nsm/strelka/history/?

[masedira]

Hi @TOoSmOotH
thanks again! this worked perfectly!!

I deleted all files in /nsm/strelka/history and when I re-downloaded the file it was stored / inspected and generated the alert.

I have a very nice demo now thanks to your help! Greatly appreciated!

[Carlos-mb]

Hi, I have the same problem, I can't find the file in strelka/processed

These are similar entries before and after update from .90 to .110:

The file HTTP-FgTK8j1MzI5ETlT3Bc-9839a35b4f8abbf4d36c82848875f727.exe does exist in strelka/processed but the other doesn't exist

[Carlos-mb]

@masedira, could you check file.bytes.missing?

I have other installation with .110 and much more events. I have seen that the problem existed before update to .110

If there are missing bytes, the file is not extracted.

[Carlos-mb]

In the small installation... in the last 24h there is only one file.zeek register. It was not extracted. I see file.bytes.missing: 757645
In the last 24h influxdb reports 0% packet loss and 0% capture loss. Zooming to the exact moment when the file was received, Influxdb reports 0% in both metrics

In the big installation I have found files with 0 bytes missing and not extracted and other files with bytes loss but influxdb reports 0% loss in all metrics.

I have rebooted (sudo reboot) the small installation and it has extracted the only file it has received until now.

I have no more clues :(

[masedira]

Hi Carlos,

in the logs it shows missing_bytes:0 like in the sample in the first post.

I have a zeek standalone server and it is able to extract the files form the same SPAN port and keeps them in a directory.

I am trying to do the same with SO but failing so far.

[TOoSmOotH]

I just tested it and it works fine for me. You may need to clear the history in /nsm/strelka/history if you have seen this file before. We only process files every 3 days. It is safe to delete all the files in /nsm/trelka/history. We do this because if someone sends out a pdf to 500 of their closest colleagues that the sensor does not process the same pdf 500 times. I would also ensure the mime type of the file is listed in the config in soc. We only extract those mime types.