Merge pull request #1073 from d-strat/develop

Fides Module

Author: AlyaGomaa

.github/workflows/integration-tests.yml

@@ -31,6 +31,7 @@ jobs:
           - test_dataset.py
           - test_pcap_dataset.py
           - test_zeek_dataset.py
+          - test_fides.py
 
     steps:
     - uses: actions/checkout@v4

.github/workflows/unit-tests.yml

@@ -83,6 +83,8 @@ jobs:
           - test_host_ip_manager.py
           - test_rnn_cc_detection.py
           - test_idea_format.py
+          - test_fides_sqlite_db.py
+          - test_fides_module.py
 
     steps:
     - uses: actions/checkout@v4

.gitignore

@@ -172,3 +172,9 @@ output/
 config-live-macos-*
 dataset-private/*
 appendonly.aof
+/slipsOut/flows.sqlite
+/slipsOut/metadata/info.txt
+/slipsOut/metadata/slips.yaml
+/slipsOut/metadata/whitelist.conf
+/p2p_db.sqlite
+

config/slips.yaml

@@ -78,13 +78,13 @@ parameters:
   # zeek breaks the connection into smaller connections
   tcp_inactivity_timeout: 60
   # Should we delete the previously stored data in the DB when we start?
-  # By default False. Meaning we don't DELETE the DB by default.
+  # By default false. Meaning we don't DELETE the DB by default.
   deletePrevdb: true
   # You can remember the data in all the previous runs of the DB if
-  # you put False.
+  # you put false.
   # Redis will remember as long as the redis server is not down.
   # The persistence is in memory, not disk.
-  # deletePrevdb : False
+  # deletePrevdb : false
   # Set the label for all the flows that are being read.
   # For now only normal and malware directly. No option for setting labels
   # with a filter
@@ -154,7 +154,7 @@ detection:
   #        May lead to false negatives
   evidence_detection_threshold: 0.25
   # Slips can show a popup/notification with every alert.
-  popup_alerts: False
+  popup_alerts: false
 #############################
 modules:
   # List of modules to ignore. By default we always ignore the template!
@@ -198,7 +198,7 @@ threatintelligence:
   # and all TI files are loaded successfully
   # this is usefull if you want to ensure that slips doesn't miss the
   # detection of any blacklisted IPs
-  wait_for_TI_to_finish: False
+  wait_for_TI_to_finish: false
   # Default Path to the folder with files holding malcious IPs
   # All the files in this folder are read and the IPs are considered malicious
   # The format of the files must be, per line: "Number","IP address","Rating",
@@ -275,7 +275,7 @@ exporting_alerts:
   # if your TAXII server is a remote server,
   # you can set the port to 443 or 80.
   port: 1234
-  use_https: False
+  use_https: false
   discovery_path: /services/discovery-a
   inbox_path: /services/inbox-a
   # Collection on the server you want to push stix data to
@@ -299,8 +299,8 @@ exporting_alerts:
 CESNET:
   # Slips supports exporting and importing evidence in the IDEA format to/from
   # warden servers.
-  send_alerts: False
-  receive_alerts: False
+  send_alerts: false
+  receive_alerts: false
   # warden configuration file. For format instructions check
   # yamllint disable-line rule:line-length
   # https://stratospherelinuxips.readthedocs.io/en/develop/exporting.html?highlight=exporting# cesnet-sharing
@@ -346,7 +346,7 @@ Docker:
 Profiling:
   # [11] CPU profiling
   # enable cpu profiling [yes,no]
-  cpu_profiler_enable: False
+  cpu_profiler_enable: false
   # Available options are [dev,live]
   # dev for deterministic profiling. this will give precise information
   # about the CPU usage
@@ -363,16 +363,23 @@ Profiling:
   # set the wait time between sampling sequences in seconds (live mode only)
   cpu_profiler_sampling_interval: 20
   # enable memory profiling [yes,no]
-  memory_profiler_enable: False
+  memory_profiler_enable: false
   # set profiling mode [dev,live]
   memory_profiler_mode: live
   # profile all subprocesses [yes,no]
   memory_profiler_multiprocess: True
 #############################
 web_interface:
   port: 55000
+
+#############################
+global_p2p:
+  # this is the global p2p's trust model. can only be enabled when
+  # running slips on an interface
+  use_fides: false
+
 #############################
 P2P:
   # create p2p.log with additional info about peer communications?
-  create_p2p_logfile: False
-  use_p2p: False
+  create_p2p_logfile: false
+  use_p2p: false

docs/contributing.md

@@ -171,4 +171,107 @@ Once all modules are done processing, EvidenceHandler is killed by the Process m
 
 Using one of these 3 ways
 
-<img src="https://raw.githubusercontent.com/stratosphereips/StratosphereLinuxIPS/develop/docs/images/gw_info.jpg"
+<img src="https://raw.githubusercontent.com/stratosphereips/StratosphereLinuxIPS/develop/docs/images/gw_info.jpg">
+
+
+## Global P2P - Fides contribution notes
+
+Variables used in the trust evaluation and its accompanied processes, such as database-backup in persistent
+SQLite storage and memory persistent
+Redis database of Slips, are strings, integers and floats grouped into custom dataclasses. Aforementioned data classes can
+be found in modules/fidesModule/model. The reader may find that all of the floating variables are in the interval <-1; 1>
+and some of them are between <0; 1>, please refer to the modules/fidesModule/model directory.
+
+The Fides Module is designed to cooperate with a global-peer-to-peer module. The communication is done using Slips' Redis
+channel, for more information please refer to communication and messages sections above.
+
+An example of a message answering Fides-Module's opinion request follows.
+```
+import redis
+
+# connect to redis database 0
+redis_client = redis.StrictRedis(host='localhost', port=6379, db=0)
+
+message = '''
+{
+    "type": "nl2tl_intelligence_response",
+    "version": 1,
+    "data": [
+        {
+            "sender": {
+                "id": "peer1",
+                "organisations": ["org_123", "org_456"],
+                "ip": "192.168.1.1"
+            },
+            "payload": {
+                "intelligence": {
+                    "target": {"type": "server", "value": "192.168.1.10"},
+                    "confidentiality": {"level": 0.8},
+                    "score": 0.5,
+                    "confidence": 0.95
+                },
+                "target": "stratosphere.org"
+            }
+        },
+        {
+            "sender": {
+                "id": "peer2",
+                "organisations": ["org_789"],
+                "ip": "192.168.1.2"
+            },
+            "payload": {
+                "intelligence": {
+                    "target": {"type": "workstation", "value": "192.168.1.20"},
+                    "confidentiality": {"level": 0.7},
+                    "score": -0.85,
+                    "confidence": 0.92
+                },
+                "target": "stratosphere.org"
+            }
+        }
+    ]
+}
+'''
+
+# publish the message to the "network2fides" channel
+channel = "network2fides"
+redis_client.publish(channel, message)
+
+print(f"Message published to channel '{channel}'.")
+```
+
+For more information about message handling, please also refer to modules/fidesModule/messaging/message_handler.py
+and to modules/fidesModule/messaging/dacite/core.py for message parsing.
+
+
+### **Communication**
+The module uses Slips' Redis to receive and send messages related to trust intelligence,
+evaluation of trust in peers and alert message dispatch.
+
+**Used Channels**
+modules/fidesModule/messaging/message_handler.py
+| **Slips Channel Name** | **Purpose**                                                             |
+|-----------------|-------------------------------------------------------------------------|
+| `slips2fides`   | Provides communication channel from Slips to Fides                      |
+| `fides2slips`   | Enables the Fides Module to answer requests from slips2fides            |
+| `network2fides` | Facilitates communication from network (P2P) module to the Fides Module |
+| `fides2network` | Lets the Fides Module request network opinions form network modules     |
+
+For more details, the code [here](https://github.com/stratosphereips/fides/tree/bfac47728172d3a4bbb27a5bb53ceef424e45e4f/fides/messaging) may be read.
+
+
+### **Messages**
+
+| **Message type (data['type'])** | **Channel**     | **Call/Handle**                                                                                                       | **Description**                                                                                       |
+|:-------------------------------:|-----------------|-----------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------|
+|             `alert`             | `slips2fides`   | FidesModule as self.__alerts.dispatch_alert(target=data['target'], confidence=data['confidence'],score=data['score']) | Triggers sending an alert to the network, about given target, which SLips believes to be compromised. |
+|     `intelligence_request`      | `slips2fides`   | FidesModule as self.__intelligence.request_data(target=data['target'])                                                | Triggers request of trust intelligence on given target.                                               |
+|          `tl2nl_alert`          | `fides2network` | call dispatch_alert() of AlertProtocol class instance                                                                 | Broadcasts alert through the network about the target.                                                |
+|  `tl2nl_intelligence_response`  | `fides2network` | NetworkBridge.send_intelligence_response(...)                                                                         | Shares Intelligence with peer that requested it.                                                      |
+|  `tl2nl_intelligence_request`   | `fides2network` | NetworkBridge.send_intelligence_request(...)                                                                          | Requests network intelligence from the network regarding this target.                                 |
+| `tl2nl_recommendation_response` | `fides2network` | NetworkBridge.send_recommendation_response(...)                                                                       | Responds to given request_id to recipient with recommendation on target.                              |
+| `tl2nl_recommendation_request`  | `fides2network` | NetworkBridge.send_recommendation_request(...)                                                                        | Request recommendation from recipients on given peer.                                                 |
+|    `tl2nl_peers_reliability`    | `fides2network` | NetworkBridge.send_peers_reliability(...)                                                                             | Sends peer reliability, this message is only for network layer and is not dispatched to the network.  |
+
+
+Implementations of Fides_Module-network-communication can be found in ```modules/fidesModule/messaging/network_bridge.py```.

docs/fides_module.md

@@ -0,0 +1,98 @@
+# Fides module
+
+The Fides module is an essential component of the Global P2P system in Slips.
+
+
+Traditional network defense systems depend on centralized threat intelligence, which has limitations like single points of failure, inflexibility, and reliance on trust in centralized authorities. Peer-to-peer networks offer an alternative for sharing threat intelligence but face challenges in verifying the trustworthiness of participants, including potential malicious actors.
+
+The Fides Module, based on [Master Theses](https://github.com/stratosphereips/fides/tree/bfac47728172d3a4bbb27a5bb53ceef424e45e4f) on CTU FEL by Lukáš Forst. The goal of this module is to address the challenge of trustworthiness of peers in peer-to-peer networks by providing several trust evaluation models. It evaluates peer behavior, considers membership in trusted organizations, and assesses incoming threat data to determine reliability. Fides aggregates and weights data to enhance intrusion prevention systems, even in adversarial scenarios. Experiments show that Fides can maintain accurate threat intelligence even when 75% of the network is controlled by malicious actors, assuming the remaining 25% are trusted.
+
+The whole architecture is thoroughly documented in the thesis itself, which can be downloaded from the link above.
+
+## Docker direct use
+You can use Slips with Fides Module by allowing it in the Slips config file or by using the following commands.
+
+```
+docker pull stratosphereips/slips
+docker run -it --rm --net=host --cap-add=NET_ADMIN stratosphereips/slips
+```
+
+To be able to use the fides module you should use ```--cap-add=NET_ADMIN```
+
+## Installation:
+
+```
+docker pull stratosphereips/slips
+docker run -it --rm --net=host --use_fides=True stratosphereips/slips
+```
+
+***NOTE***
+
+If you plan on using the Fides Module, lease be aware that it is used only
+if Slips is running on an interface. The `--use_fides=True` is ignored when Slips is run on a file.
+
+### Configuration
+Evaluation model, evaluation thrash-holds and other configuration is located in fides.conf.yml
+
+**Possible threat intelligence evaluation models**
+
+| **Model Name**         | **Description**                                                  |
+|:-----------------------|--------------------------------------------------------------|
+| `average`              | Average Confidence Trust Intelligence Aggregation            |
+| `weightedAverage`      | Weighted Average Confidence Trust Intelligence Aggregation   |
+| `stdevFromScore`       | Standard Deviation From Score Trust Intelligence Aggregation |
+
+## Usage in Slips
+
+Fides is inactive by default in Slips.
+
+To enable it, change ```use_fides=False``` to ```use_fides=True``` in ```config/slips.yaml```.
+
+And start slips on your interface.
+
+## Project sections
+
+The project is built into Slips as a module and uses Redis for communication. Integration with Slips
+is seamless, and it should be easy to adjust the module for use with other IPSs.
+
+ - Slips, the Intrusion Prevention System
+ - Fides Module the trust evaluation module for global p2p interaction
+
+
+## How it works:
+
+Slips interacts with other slips peers for the following purposes:
+
+### Sharing opinion on peers
+
+If a peers A is asked for its opinion on peer B by peer C, peer A sends the aggregated opinion on peer B to peer C, if there is any.
+
+### Asking for an opinion
+
+Newly connected peer will create a base trust by asking ather peers for opinion.
+
+### Dispatching alerts
+
+If a threat so great it may impact whole network, one or more groups, threat alert is
+dispatched to peers, without regard to trust level accumulated on them.
+
+### Answering and receiving requests form global P2P module.
+
+## Logs
+
+Slips contains a minimal log file for reports received by other peers and peer updates in
+```output``` directory if not manually specified using the appropriate slips parameter upon start.
+Custom logger ```modules/fidesModule/utils/logger.py``` is used by Fide Module for internal logging.  Either Slips' logging is used, or the custom logger is defaulted to logging via Python's printing function.
+
+## Limitations
+
+For now, slips supports the trust intelligence evaluation, global p2p is to be implemented.
+
+## Implementation notes and credit
+The mathematical models for trust evaluation were written by Lukáš Forst as part of his theses and can be accessed [here](https://github.com/LukasForst/fides/commits?author=LukasForst).
+
+
+## TLDR;
+
+Slips (meaning Fides Module here) only shares trust level and confidence (numbers) generated by slips about IPs to the network,
+no private information is shared.

docs/images/gw_info.jpg

@@ -18,8 +18,8 @@ This documentation gives an overview how Slips works, how to use it and how to h
 - **Detection modules**. Explanation of detection modules in Slips, types of input and output. See :doc:`Detection modules <detection_modules>`.
 
 - **Architecture**. Internal architecture of Slips (profiles, timewindows), the use of Zeek and connection to Redis. See :doc:`Architecture <architecture>`.
-  
-- **Training with your own data**. Explanation on how to re-train the machine learning system of Slips with your own traffic (normal or malicious).See :doc:`Training <training>`. 
+
+- **Training with your own data**. Explanation on how to re-train the machine learning system of Slips with your own traffic (normal or malicious).See :doc:`Training <training>`.
 
 - **Detections per Flow**. Explanation on how Slips works to make detections on each flow with different techniques. See :doc:`Flow Alerts <flowalerts>`.
 
@@ -41,9 +41,9 @@ This documentation gives an overview how Slips works, how to use it and how to h
 .. toctree::
    :maxdepth: 2
    :hidden:
-   :caption: Slips 
-   
-   self 
+   :caption: Slips
+
+   self
    installation
    usage
    architecture
@@ -59,6 +59,4 @@ This documentation gives an overview how Slips works, how to use it and how to h
    FAQ
    code_documentation
    datasets
-
-
-
+   fides_module