Read PCAP through stenoread from console

[erik4711]

I want to download a full content PCAP with all SMB traffic between a client and server from SO, but I'm only able to read one TCP session at a time using the "PCAP job" feature from the web GUI. Is there some way to provide a wildcard (like *, any or 0-65535) as port number in the PCAP job util. Or even better, can I access Stenographer's stenoread command directly from the console somehow?
For reference, here's the stenoread command I'd like to run:
stenoread host 192.168.4.15 and host 192.168.4.4 and port 445 after 2020-06-09T00:00:00Z before 2020-06-10T00:00:00Z

I'm running Security Onion 2.3.10 (installed with git clone)
Update: My SO machine is installed as an "IMPORT Standalone" node that I've run so-import-pcap on, so I have full access to the host that has the PCAP files.

/erik

[jertel]

You can add an arbitrary job to the SOC PCAP interface (click the + button). Leave the src or dest port blank if you don't want to limit it to a particular port.

Example:

You can also access large PCAPs from the terminal, but this process has not been documented.

Example:

docker exec -it so-sensoroni scripts/stenoquery.sh "host 192.168.4.15 and host 192.168.4.4 and port 445 and after 2020-06-09T00:00:00Z and before 2020-06-10T00:00:00Z" -w /nsm/pcapout/some.pcap

[erik4711]

I'm getting an error message saying The job was unable to complete and will retry within a few minutes. Details are available below [...] No data available for the requested dates when manually entering jobs using the "Add Job" feature you showed. I also see these error messages in opt/so/log/sensoroni/sensoroni.log :

timestamp=2020-12-08T15:25:41.543989804Z level=info message="Discovered pending job" jobId=1006
timestamp=2020-12-08T15:25:41.544075412Z level=info message="Skipping steno processor due to date range conflict" availableDataBeginDate=2020-12-08T15:24:17.378871088Z availableDataEndDate=2020-12-08T15:23:41.544069908Z jobBeginDate=2020-06-09T00:00:00Z jobEndDate=2020-06-10T00:00:00Z jobId=1006
timestamp=2020-12-08T15:25:41.544276661Z level=error message="Failed to process job; job processing aborted" error="No data available for the requested dates" jobId=1006

But clicking on "Show PCAP for this event" from Alerts or Hunt works fine, even if the alerts are from back in June.

Running the stenoquery.sh sounds interesting! Unfortunately it didn't work for me either. I get an error message saying
curl: (7) Failed to connect to 127.0.0.1 port 1234: Connection refused

Running docker exec -it so-sensoroni netstat -ntl shows that I don't have any steno service listening on TCP 1234. Maybe this is because I'm only importing PCAP files, not doing live capture?

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN      
tcp        0      0 0.0.0.0:4505            0.0.0.0:*               LISTEN      
tcp        0      0 0.0.0.0:4506            0.0.0.0:*               LISTEN      
tcp        0      0 0.0.0.0:445             0.0.0.0:*               LISTEN      
tcp        0      0 0.0.0.0:139             0.0.0.0:*               LISTEN      
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      
tcp        0      0 :::443                  :::*                    LISTEN      
tcp        0      0 :::445                  :::*                    LISTEN      
tcp        0      0 :::9822                 :::*                    LISTEN      
tcp        0      0 :::5601                 :::*                    LISTEN      
tcp        0      0 :::514                  :::*                    LISTEN      
tcp        0      0 :::5000                 :::*                    LISTEN      
tcp        0      0 :::139                  :::*                    LISTEN      
tcp        0      0 :::9200                 :::*                    LISTEN      
tcp        0      0 :::80                   :::*                    LISTEN      
tcp        0      0 :::4433                 :::*                    LISTEN      
tcp        0      0 :::4434                 :::*                    LISTEN      
tcp        0      0 :::9300                 :::*                    LISTEN      
tcp        0      0 :::22                   :::*                    LISTEN     

But since "Show PCAP for this event" works perfectly on my imported data I guess I should be able to achieve the same thing also with a manual filter, right?

[jertel]

An import installation does not have the same capabilities as a normal standalone install. So that explains why the "+ Add Job" feature can't find the PCAP, and why the stenoquery.sh script fails.

Since you're dealing with imported PCAPs you can use tcpdump directly on the import host and use the /nsm/import/xyz/pcap/data.pcap file as the input, where xyz is the import UUID that was generated at the time that so-import-pcap was run.

Example:

[root@standalone ~]# so-import-pcap /nsm/pcapout/some.pcap
Processing Import: /nsm/pcapout/some.pcap

• verifying file
• assigning unique identifier to import: dbecfd9696b92f3bf96b0ccbed0b25b9
• analyzing traffic with Suricata
• analyzing traffic with Zeek
• saving PCAP data spanning dates 2020-12-08 through 2020-12-08

Notice in this example the import ID starts with dbecfd...

[erik4711]

Thanks! I didn't know imports were treated different from captured traffic.

Also good that you pointed out the "import ID"! I've been digging into this problem a bit more and found that the so-soc Docker container contains JSON files for all PCAP jobs in "/opt/sensoroni/jobs/so/". I compared the successful ones (created with "Show PCAP for this event") with the failed ones (created with "Add Job") and it seems like the only difference is that the failed ones don't specify any value for the "importId" field inside the filter section.

As you mention my fallback will be to going back to typing a BPF for tcpdump (after having filtered on time with editcap). However, it would be nice if the "Add Job" feature could figure out the value to use for the "importId" field somehow, so that "Add Job" can be used also for imported packets. Maybe a feature request for @sensoroni?

[jertel]

Yes, it's been on my list to add that import ID to the Add Job form. I'll see if I can get it into the next release.

[erik4711]

Updated SO now and tried adding the import ID. Works like a charm, thanks!
Now I have a new problem, what is the best way to figure out which import ID to use? I have imported 30 files, one for each day. Is there a log somewhere that tells me which PCAP file that gets what import ID?

[jmdale83]

^ I would also like to know if this is possible.