Importing PCAP does not work as expected

[gustavoberman]

Version

2.4.130

Installation Method

Security Onion ISO image

Description

other (please provide detail below)

Installation Type

Standalone

Location

airgap

Hardware Specs

Exceeds minimum requirements

CPU

16

RAM

28

Storage for /

110GB

Storage for /nsm

150GB

Network Traffic Collection

other (please provide detail below)

Network Traffic Speeds

Less than 1Gbps

Status

Yes, all services on all nodes are running OK

Salt Status

No, there are no failures

Logs

No, there are no additional clues

Detail

Hello there!

I'm testing a fresh installation. It's a .130 VM in Proxmox, standalone, airgap. I'm still not capturing anything from net or agents.

This is similar to #14308 behavior

I wanted to test the installation by following https://blog.securityonion.net/2025/02/quick-malware-analysis-smartapesg.html
I waited for ~30 min for Detections to sync (It had the blue clock) and for all status green

Found the pcap at the correct website, imported into security onion using the web interface. After a few moments I can see the generated Alerts:

And I can drilldown on the DLL download event, but when I try to pivot into PCAP it returns:

Checking in PCAP menu looks like this and after some 25 minutes the status does not change, it stays Incomplete:

Any pivot to PCAP turns out the same error. Any click on PCAP number gives same error.

The import looks correct, except that suripcap is empty (perhaps that's a problem?)

du -h -a /nsm/import/
25M	/nsm/import/592f8d93157ae6d82b2fff5c5318dc23/pcap/data.pcap
25M	/nsm/import/592f8d93157ae6d82b2fff5c5318dc23/pcap
0	/nsm/import/592f8d93157ae6d82b2fff5c5318dc23/suricata/suripcap
396K	/nsm/import/592f8d93157ae6d82b2fff5c5318dc23/suricata/eve-2025-03-14-15:18.json
396K	/nsm/import/592f8d93157ae6d82b2fff5c5318dc23/suricata
4.0K	/nsm/import/592f8d93157ae6d82b2fff5c5318dc23/zeek/logs/console.log
4.0K	/nsm/import/592f8d93157ae6d82b2fff5c5318dc23/zeek/logs/packet_filter.log
48K	/nsm/import/592f8d93157ae6d82b2fff5c5318dc23/zeek/logs/loaded_scripts.log
4.0K	/nsm/import/592f8d93157ae6d82b2fff5c5318dc23/zeek/logs/dns.log
4.0K	/nsm/import/592f8d93157ae6d82b2fff5c5318dc23/zeek/logs/known_hosts.log
4.0K	/nsm/import/592f8d93157ae6d82b2fff5c5318dc23/zeek/logs/x509.log
4.0K	/nsm/import/592f8d93157ae6d82b2fff5c5318dc23/zeek/logs/ssl.log
4.0K	/nsm/import/592f8d93157ae6d82b2fff5c5318dc23/zeek/logs/reporter.log
16K	/nsm/import/592f8d93157ae6d82b2fff5c5318dc23/zeek/logs/conn.log
4.0K	/nsm/import/592f8d93157ae6d82b2fff5c5318dc23/zeek/logs/known_services.log
4.0K	/nsm/import/592f8d93157ae6d82b2fff5c5318dc23/zeek/logs/capture_loss.log
4.0K	/nsm/import/592f8d93157ae6d82b2fff5c5318dc23/zeek/logs/analyzer.log
4.0K	/nsm/import/592f8d93157ae6d82b2fff5c5318dc23/zeek/logs/software.log
48K	/nsm/import/592f8d93157ae6d82b2fff5c5318dc23/zeek/logs/files.log
28K	/nsm/import/592f8d93157ae6d82b2fff5c5318dc23/zeek/logs/http.log
4.0K	/nsm/import/592f8d93157ae6d82b2fff5c5318dc23/zeek/logs/weird.log
4.0K	/nsm/import/592f8d93157ae6d82b2fff5c5318dc23/zeek/logs/dpd.log
4.0K	/nsm/import/592f8d93157ae6d82b2fff5c5318dc23/zeek/logs/pe.log
200K	/nsm/import/592f8d93157ae6d82b2fff5c5318dc23/zeek/logs
1.1M	/nsm/import/592f8d93157ae6d82b2fff5c5318dc23/zeek/extracted/HTTP-FDp60y4fYhWUve09y5.exe
672K	/nsm/import/592f8d93157ae6d82b2fff5c5318dc23/zeek/extracted/HTTP-FSL2Ro2shGQGLN0aj4.exe
596K	/nsm/import/592f8d93157ae6d82b2fff5c5318dc23/zeek/extracted/HTTP-F6erOf3gz44wEpCkXk.exe
440K	/nsm/import/592f8d93157ae6d82b2fff5c5318dc23/zeek/extracted/HTTP-F1HPez2i7CI2cALSO2.exe
2.0M	/nsm/import/592f8d93157ae6d82b2fff5c5318dc23/zeek/extracted/HTTP-FQSpuq4Uw8vKzPmTcj.exe
252K	/nsm/import/592f8d93157ae6d82b2fff5c5318dc23/zeek/extracted/HTTP-FQ0hsLTYISpZr4Bbj.exe
80K	/nsm/import/592f8d93157ae6d82b2fff5c5318dc23/zeek/extracted/HTTP-FX8Ss42es2gHYcEIgl.exe
5.1M	/nsm/import/592f8d93157ae6d82b2fff5c5318dc23/zeek/extracted
0	/nsm/import/592f8d93157ae6d82b2fff5c5318dc23/zeek/spool
5.3M	/nsm/import/592f8d93157ae6d82b2fff5c5318dc23/zeek
30M	/nsm/import/592f8d93157ae6d82b2fff5c5318dc23
30M	/nsm/import/

Free space and memory seems good:

df -h / /nsm
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/system-root  109G   25G   85G  23% /
/dev/mapper/nsm-nsm      150G   16G  135G  11% /nsm


free -h
               total        used        free      shared  buff/cache   available
Mem:            26Gi        21Gi       3.0Gi        18Mi       1.9Gi       4.5Gi
Swap:             0B          0B          0B

timedatectl 
               Local time: Fri 2025-03-14 15:29:05 UTC
           Universal time: Fri 2025-03-14 15:29:05 UTC
                 RTC time: Fri 2025-03-14 15:29:05
                Time zone: Etc/UTC (UTC, +0000)
System clock synchronized: no
              NTP service: active
          RTC in local TZ: no

In logs I found perhaps some errors:

ll /opt/so/log/suricata/import/592f8d93157ae6d82b2fff5c5318dc23/console.log 
-rw-r--r--. 1 root root 485 Mar 14 15:18 /opt/so/log/suricata/import/592f8d93157ae6d82b2fff5c5318dc23/console.log

cat /opt/so/log/suricata/import/592f8d93157ae6d82b2fff5c5318dc23/console.log
i: suricata: This is Suricata version 7.0.8 RELEASE running in USER mode
W: runmodes: output module "stats": setup failed
W: counters: stats are enabled but no loggers are active
E: logopenfile: Error opening file: "./stats.log": Permission denied
W: log-pcap: Failed to open directory /nsm/suripcap/1: Not a directory
i: threads: Threads created -> W: 1 FM: 1 FR: 1   Engine started.
i: suricata: Signal Received.  Stopping engine.
i: pcap: read 1 file, 42224 packets, 24649458 bytes

That /nsm/suripcap/1: Not a directory is strange as I can see this:

ll /nsm/suripcap/*
/nsm/suripcap/1:
total 0

/nsm/suripcap/2:
total 0

/nsm/suripcap/3:
total 8
-rw-r--r--. 1 suricata suricata 110 Mar 14 14:14 so-pcap.1741961438
-rw-r--r--. 1 suricata suricata 110 Mar 14 15:14 so-pcap.1741964553

/nsm/suripcap/4:
total 0

/nsm/suripcap/5:
total 0

/nsm/suripcap/6:
total 0

/nsm/suripcap/7:
total 0

I do not know what's the internal proces when you select to pivot into PCAP to try and follow logs

What I find strange also is that PCAP retention says 0.1 days. Perhaps it's removing it after creating it?

Thank you for any help you can give me

Guidelines

• I have read the discussion guidelines at Read before posting! #1720 and assert that I have followed the guidelines.

[gustavoberman]

After 3 hours nothing changed in the status of the PCAPs, so I rebooted the system and waited some minutes to stabilize and obtaining an all green status

Now the PCAP shows status Completed:

But clicking in it shows me nothing:

So I grabbed the same pcap, changed the date of frames to today (thinking that perhaps it helps) and did the import again using the web interface. Waited some minutes, obtained the alerts, pivot into one of that and obtained a similar result:

But this one says Completed from the begining instead of Incomplete

[dougburks]

When you import a pcap, the pcap file itself is copied to /nsm/import/HASH/pcap/data.pcap and that is what is used when you later request PCAP. Looking at your du output above we see this:

25M /nsm/import/592f8d93157ae6d82b2fff5c5318dc23/pcap/data.pcap

Therefore, we know that there is a PCAP file there to pull from.

Looking at your SOC Grid screenshot, it looks like your memory usage is high and you have no swap space:

In this scenario, it is possible that when you request a PCAP job that it is running out of RAM and getting OOM killed by the kernel. To see if this is the case, you can run dmesg or check kernel log files. You can then try adding swap space and/or increasing RAM.

Another thing you can try is to do a fresh installation and choose IMPORT mode instead of STANDALONE mode as shown here:
https://docs.securityonion.net/en/2.4/first-time-users.html

IMPORT mode has much lower memory requirements and is what was used for the blog post:
https://blog.securityonion.net/2025/02/quick-malware-analysis-smartapesg.html

[dougburks]

OK, the next thing I would try is create a new STANDALONE VM and see if that works correctly. If so, then you can compare it to your broken STANDALONE side-by-side looking for any differences.

[gustavoberman]

OK, the next thing I would try is create a new STANDALONE VM and see if that works correctly. If so, then you can compare it to your broken STANDALONE side-by-side looking for any differences.

Sadly, I have the same result. :(
Created a new VM, standalone, airgap, same resources: 28GB RAM, 110GB /, 150GB /nsm but this time when installation finished I addded the swapfile. System status is OK. Detection blue clock finished.

Imported the remcos-rat pcap. I can see the dashboard data just fine.

Alerts are created, but I cannot pivot from alert or hunt to PCAP:

Even if in here it says completed:

Also I cannot obtain data from a PCAP job, I obtain the same result:

So it will be great to try and debug this process

[dougburks]

We've been able to duplicate this problem and created an issue to fix it:
#14426

[dougburks]

@gustavoberman we've just released Security Onion 2.4.140 and it should resolve this issue. Please update and see if that helps.
https://blog.securityonion.net/2025/03/security-onion-24140-now-available.html

[gustavoberman]

@gustavoberman we've just released Security Onion 2.4.140 and it should resolve this issue. Please update and see if that helps. https://blog.securityonion.net/2025/03/security-onion-24140-now-available.html

Awesome! Now it's working. Thanks!

[dougburks]

@gustavoberman Thanks for confirming!