Merge pull request #1379 from stratosphereips/develop

Slips v1.1.8

Author: AlyaGomaa

CHANGELOG.md

@@ -1,3 +1,9 @@
+1.1.8 (Mar 31st, 2025)
+- Fix SQLite database errors.
+- Fix CPU and RAM profilers.
+- Fix issue with AsyncModules not shutting down gracefully.
+
+
 1.1.7 (Feb 28th, 2025)
 - Add global P2P support. Thanks to @d-strat
 - Add new "GRE tunnel scan" detections.

README.md

@@ -23,6 +23,7 @@ Slips v1.1.7
 [![License](https://img.shields.io/badge/Blog-Stratosphere-cyan)](https://www.stratosphereips.org/blog/tag/slips)
 [![Discord](https://img.shields.io/discord/761894295376494603?label=&logo=discord&logoColor=ffffff&color=7389D8&labelColor=6A7EC2)](https://discord.gg/zu5HwMFy5C)
 ![Twitter Follow](https://img.shields.io/twitter/follow/StratosphereIPS?style=social)
+
 <hr>
 
 
@@ -53,6 +54,7 @@ Slips is a powerful endpoint behavioral intrusion prevention and detection syste
 
 <img src="https://raw.githubusercontent.com/stratosphereips/StratosphereLinuxIPS/develop/docs/images/slips.gif" width="850px" title="Slips in action.">
 
+---
 
 
 # Introduction
@@ -64,6 +66,7 @@ Slips is supported on Linux, MacOS, and windows dockers only. The blocking featu
 Slips is Python-based and relies on [Zeek network analysis framework](https://zeek.org/get-zeek/) for capturing live traffic and analyzing PCAPs. and relies on
 Redis >= 7.0.4 for interprocess communication.
 
+---
 
 # Usage
 
@@ -102,6 +105,7 @@ cat output_dir/alerts.log
 
 [For a detailed explanation of Slips parameters](https://stratospherelinuxips.readthedocs.io/en/develop/usage.html#slips-parameters)
 
+---
 
 
 # Graphical User Interface
@@ -136,6 +140,7 @@ For more info about the Kalipso interface, check the docs: https://stratospherel
 
 Slips requires Python 3.10.12 and at least 4 GBs of RAM to run smoothly.
 
+---
 
 # Installation
 
@@ -169,6 +174,8 @@ Slips has a [config/slips.yaml](https://github.com/stratosphereips/StratosphereL
 
 [More details about the config file options here]( https://stratospherelinuxips.readthedocs.io/en/develop/usage.html#modifying-the-configuration-file)
 
+---
+
 # Features
 Slips key features are:
 
@@ -184,6 +191,7 @@ Slips key features are:
 * **Detailed Documentation**: Slips provides detailed documentation guiding users through usage instructions for efficient utilization of its features.
 * **Federated learning** Using the feel_project submodule. for more information [check the docs](https://github.com/stratosphereips/feel_project/blob/main/docs/Federated_Learning.md)
 
+---
 
 # Contributing
 
@@ -203,32 +211,47 @@ Check [Slips in GSoC2023](https://github.com/stratosphereips/Google-Summer-of-Co
 You can [join our conversations in Discord](https://discord.gg/zu5HwMFy5C) for questions and discussions.
 We appreciate your contributions and thank you for helping to improve Slips!
 
+---
 
 # Documentation
 [User documentation](https://stratospherelinuxips.readthedocs.io/en/develop/)
 
 [Code docs](https://stratospherelinuxips.readthedocs.io/en/develop/code_documentation.html )
 
+---
+
 # Troubleshooting
 
-If you can't listen to an interface without sudo,
-you can run the following command to let any user use Zeek to listen to an interface not just root.
+If you can't listen to an interface without sudo, foe example when zeek is throwing the following error:
+```bash
+fatal error: problem with interface wlan0 (pcap_error: socket: Operation not permitted (pcap_activate))
+```
+
+you can adjust zeek capabilities using the following command
 
 ```
 sudo setcap cap_net_raw,cap_net_admin=eip /<path-to-zeek-bin/zeek
 ```
 
+
+---
+
 You can [join our conversations in Discord](https://discord.gg/zu5HwMFy5C) for questions and discussions.
 
 Or email us at
 * [email protected]
 * [email protected],
 * [email protected]
 
+---
+
 # License
 
  [GNU General Public License](https://github.com/stratosphereips/StratosphereLinuxIPS/blob/master/LICENCE)
 
+---
+
+
 # Credits
 
 Founder: [Sebastian Garcia](https://github.com/eldraco), [email protected], [email protected].
@@ -248,11 +271,16 @@ Contributors:
 * [Lukas Forst](https://github.com/LukasForst)
 * [Daniel Yang](https://github.com/danieltherealyang)
 
+---
+
+
 # Changelog
 
 https://github.com/stratosphereips/StratosphereLinuxIPS/blob/develop/CHANGELOG.md
 
 
+---
+
 # Demos
 The following videos contain demos of Slips in action in various events:
 
@@ -265,6 +293,7 @@ The following videos contain demos of Slips in action in various events:
 - 2019 OpenAlt, Fantastic Attacks and How Kalipso can Find Them [[video](https://www.youtube.com/watch?v=p2FL2sECpS0&t=1s)]
 - 2016 Ekoparty, Stratosphere IPS. The free machine learning malware detection [[video](https://www.youtube.com/watch?v=IazEdK8R4YI)]
 
+---
 
 # Funding
 We are grateful for the generous support and funding provided by the following organizations:

VERSION

@@ -1 +1 @@
-1.1.7
+1.1.8

config/slips.yaml

@@ -455,7 +455,8 @@ Docker:
 #############################
 Profiling:
   # CPU profiling
-  # enable cpu profiling [yes,no]
+  # enable cpu profiling [true/false]
+  # NOTE: the cpu profiler uses port 9001 to show the results.
   cpu_profiler_enable: false
 
   # Available options are [dev,live]
@@ -466,25 +467,31 @@ Profiling:
   # time. it is accessible from web interface
   cpu_profiler_mode: dev
 
-  # profile all subprocesses in dev mode [yes,no].
-  cpu_profiler_multiprocess: true
+  # decides whether the profiler tracks all processes or only one.
+  # only used  in dev mode [true,false].
+  cpu_profiler_multiprocess: false
 
   # set number of tracer entries (dev mode only)
-  cpu_profiler_dev_mode_entries: 1000000
+  # VizTracer uses a circular buffer to store the entries.
+  # When there are too many entries, it will only store the latest ones
+  # so you know what happened recently.
+  # the more the entries, the more RAM viztracer is going to use.
+  # https://viztracer.readthedocs.io/en/latest/basic_usage.html#circular-buffer-size
+  cpu_profiler_dev_mode_entries: 500000
 
   # set maximum output lines (live mode only)
   cpu_profiler_output_limit: 20
 
   # set the wait time between sampling sequences in seconds (live mode only)
   cpu_profiler_sampling_interval: 20
 
-  # enable memory profiling [yes,no]
+  # enable memory profiling [true,false]
   memory_profiler_enable: false
 
   # set profiling mode [dev,live]
   memory_profiler_mode: live
 
-  # profile all subprocesses [yes,no]
+  # profile all subprocesses [true,false]
   memory_profiler_multiprocess: true
 
 #############################

docker/Dockerfile

@@ -30,7 +30,6 @@ ENV NVM_DIR=/root/.nvm
 SHELL ["/bin/bash", "-c"]
 
 
-# Install wget and add Zeek and redis repositories to our sources.
 RUN apt update && apt install -y --no-install-recommends \
     wget \
     ca-certificates \

docs/images/slips.gif

@@ -85,7 +85,7 @@ Use WASD to zoom in and move left/right.
 ### CPU Profiler Live Mode
 #### Step 1
 Go to slips.yaml and make sure the settings are set correctly.
-```cpu_profiler_enable = yes```
+```cpu_profiler_enable = True```
 ```cpu_profiler_mode = live```
 
 You can also set maximum output lines (live mode only)  to adjust profiler behavior.
@@ -112,15 +112,15 @@ If we print out the data getting sent to the “cpu_profile” redis channel, it
 
 The memory profiler settings are much simpler.
 in slips.yaml, first, enable memory profiling
-```memory_profiler_enable = yes```
+```memory_profiler_enable = True```
 
 
 and set profiling mode:
 
 ```memory_profiler_mode = dev```
 
 now, profile all subprocesses
-```memory_profiler_multiprocess = yes```
+```memory_profiler_multiprocess = True```
 
 #### Step 2
 
@@ -158,9 +158,9 @@ Under the table directory, the files are much simpler. They just show a table of
 Go to ```slips.yaml``` and use the following settings
 
 ```
-memory_profiler_enable = yes
-memory_profiler_mode = yes
-memory_profiler_multiprocess = yes
+memory_profiler_enable = True
+memory_profiler_mode = True
+memory_profiler_multiprocess = True
 ```
 
 ### Step 2

docs/profiling_slips.md

@@ -4,21 +4,21 @@ watchdog==5.0.0
 redis==5.2.1
 urllib3==2.3.0
 pandas==2.2.3
-tzlocal==5.3
+tzlocal==5.3.1
 cabby==0.1.23
 stix2==3.0.1
 certifi==2025.1.31
 tensorflow==2.16.1
 Keras
 validators==0.34.0
 ipwhois==1.2.0
-matplotlib==3.9.4
+matplotlib==3.10.1
 recommonmark==0.7.1
 scikit_learn
 slackclient==2.9.4
 psutil==7.0.0
 six==1.17.0
-pytest==8.3.2
+pytest==8.3.5
 pytest-mock==3.14.0
 pytest-xdist==3.6.1
 scipy==1.15.1
@@ -35,9 +35,9 @@ yappi==1.6.10
 pytest-sugar==1.0.0
 aid_hash
 black==24.10.0
-ruff==0.9.6
+ruff==0.11.2
 pre-commit==4.0.1
-coverage==7.6.12
+coverage==7.7.1
 pyyaml
 pytest-asyncio
 git+https://github.com/SECEF/python-idmefv2.git

install/requirements.txt

@@ -721,9 +721,6 @@ def shutdown_gracefully(self):
                 format_ = self.main.conf.export_labeled_flows_to().lower()
                 self.main.db.export_labeled_flows(format_)
 
-            self.main.profilers_manager.cpu_profiler_release()
-            self.main.profilers_manager.memory_profiler_release()
-
             # if store_a_copy_of_zeek_files is set to yes in slips.yaml
             # copy the whole zeek_files dir to the output dir
             self.main.store_zeek_dir_copy()
@@ -740,6 +737,9 @@ def shutdown_gracefully(self):
                 f"finished in {analysis_time:.2f} minutes"
             )
 
+            self.main.profilers_manager.cpu_profiler_release()
+            self.main.profilers_manager.memory_profiler_release()
+
             self.main.db.close()
             if graceful_shutdown:
                 print(

managers/process_manager.py

@@ -3,11 +3,13 @@
 import os
 import subprocess
 import sys
+from slips_files.common.style import green
 
 
 class ProfilersManager:
     def __init__(self, main):
         self.main = main
+        self.args = self.main.args
         self.read_configurations()
 
     def read_configurations(self):
@@ -50,28 +52,38 @@ def cpu_profiler_init(self):
                 args = sys.argv
                 if args[-1] != "--no-recurse":
                     tracer_entries = str(self.cpu_profiler_dev_mode_entries)
+                    output_file = str(
+                        os.path.join(
+                            self.args.output,
+                            "cpu_profiling_result.json",
+                        )
+                    )
                     viz_args = [
                         "viztracer",
                         "--tracer_entries",
                         tracer_entries,
                         "--max_stack_depth",
-                        "10",
+                        "5",
                         "-o",
-                        str(
-                            os.path.join(
-                                self.args.output,
-                                "cpu_profiling_result.json",
-                            )
-                        ),
+                        output_file,
+                        # viztracer takes -- as a separator between arguments
+                        # to viztracer and positional arguments to your script.
+                        "--",
                     ]
+                    # add slips args
                     viz_args.extend(args)
+                    # add --no-recurse to avoid infinite recursion
                     viz_args.append("--no-recurse")
                     print(
-                        "Starting multiprocess profiling recursive subprocess"
+                        f"Starting multiprocess profiling recursive "
+                        f"subprocess using command: "
+                        f"{green(' '.join(viz_args))}"
                     )
                     subprocess.run(viz_args)
                     exit(0)
             else:
+                # reaching here means slips is now running using the vistracer
+                # command
                 self.cpu_profiler = CPUProfiler(
                     db=self.main.db,
                     output=self.args.output,