diff --git a/AIDojoCoordinator/docs/Architecture.md b/AIDojoCoordinator/docs/Architecture.md
deleted file mode 100644
index c7e12382..00000000
--- a/AIDojoCoordinator/docs/Architecture.md
+++ /dev/null
@@ -1,20 +0,0 @@
-# Architecture
-
-The NetSecEnv game is using the client-server architecture. The Server runs the [Coordinator](/docs/Coordinator.md) which manages:
- - creation of the game server and communication with agents.
- - processing agent request (see [Actions](/docs/Components.md)) and responses (see [Observations](/docs/Components.md))
- - communication with the game engine and forwarding messages between the agents and the game engine
-
-
-
- ## Network Security Environment
-The environment internally tracks all objects available in the wolrd and their interactions. Following data structures are used for that purpose:
-- `self._ip_to_hostname` - Mapping of `IP`:`host_name`(str) of all nodes in the environment
-- `self._networks` - A `dict` of the networks present in the environment. Keys: `Network` objects, values `set` of `IP` objects.
-- `self._services` - Dict of all services in the environment. Keys: hostname (`str`), values: `set` of `Service` objetcs.
-- `self._data` - Dict of all services in the environment. Keys: hostname (`str`), values `set` of `Service` objetcs.
-
-
-
- ## Agents
- Agents are separate programs that can interact with the NetSecEnv vie TCP sockets. See the [NetSecGameAgents](/NetSecGameAgents) repository for details on the agents.
diff --git a/AIDojoCoordinator/docs/Components.md b/AIDojoCoordinator/docs/Components.md
index 6cc19182..f1da6d14 100644
--- a/AIDojoCoordinator/docs/Components.md
+++ b/AIDojoCoordinator/docs/Components.md
@@ -1,5 +1,5 @@
# Game Components
-Here, you can see the details of all components of the NetSetEnvironment and their usage. These components are located in [game_components.py](/env/game_components.py).
+Here, you can see the details of all components of the NetSetEnvironment and their usage. These components are located in [game_components.py](game_components.py).
## Building blocks
The following classes are used in the game to hold information about the state of the game. They are used both in the [Actions](#actions) and [GameState](#gamestate).
diff --git a/README.md b/README.md
index c944e5c7..1d6f8024 100755
--- a/README.md
+++ b/README.md
@@ -1,10 +1,12 @@
-# NetSecGame
+# Netwrok Security Game
[](https://github.com/stratosphereips/game-states-maker/actions/workflows/python-checks.yml)
[](https://github.com/stratosphereips/game-states-maker/actions/workflows/autotag.yml)
The NetSecGame (Network Security Game) is a framework for training and evaluation of AI agents in the network security tasks (both offensive and defensive). It builds a simulated local network using the [CYST](https://pypi.org/project/cyst/) network simulator, adds many conditions on the environment and can train reinforcement learning (RL) algorithms on how to better attack and defend the network. Examples of implemented agents can be seen in the submodule [NetSecGameAgents](https://github.com/stratosphereips/NetSecGameAgents/tree/main).
-## Install and Dependencies
+The main part of he NetSecGame is the Game coordinator. It creates the enivronemnt, handles the agents and their interactions and coordinates the game(s).
+
+## Installation and Dependencies
To run this code you need an environment and access to cyst code. However, the venv needs to be created for your own user
- If you don't have your environment
@@ -27,21 +29,61 @@ pip install -e .
- If you use conda use
```bash
-conda create --name aidojo python==3.10
+conda create --name aidojo python==3.12
conda activate aidojo
pip install -e .
```
-## Architecture
-The architecture of the environment can be seen [here](docs/Architecture.md).
-
## Components of the NetSecGame Environment
+The architecture of the environment can be seen [here](docs/Architecture.md).
The NetSecGame environment has several components in the following files:
+```
+├── AIDojoGameCoordinator/
+| ├── game_coordinator.py
+| ├── game_components.py
+| ├── global_defender.py
+| ├── worlds/
+| ├── NSGCoordinator.py
+| ├── NSGRealWorldCoordinator.py
+| ├── CYSTCoordinator.py
+| ├── scenarios/
+| ├── tiny_scenario_configuration.py
+| ├── smaller_scenario_configuration.py
+| ├── scenario_configuration.py
+| ├── three_net_configuration.py
+| ├── utils/
+| ├── utils.py
+| ├── log_parser.py
+| ├── gamaplay_graphs.py
+| ├── actions_parser.py
+```
+
+
+### Directory Details
+- `coordinator.py`: Basic coordinator class. Handles agent communication and coordination. **Does not implement dynamics of the world** and must be extended (see examples in `worlds/`).
+- `game_components.py`: Implements a library with objects used in the environment. See [detailed explanation](docs/Components.md) of the game components.
+- `global_defender.py`: Implements global (omnipresent) defender which can be used to stop agents. Simulation of SIEM.
+
+#### **`worlds/`**
+Modules for different world configurations:
+- `NSGCoordinator.py`: Coordinator for the Network Security Game.
+- `NSGRealWorldCoordinator.py`: Real-world NSG coordinator (actions are executed in the *real network*).
+- `CYSTCoordinator.py`: Coordinator for CYST-based simulations (requires CYST running).
+
+#### **`scenarios/`**
+Predefined scenario configurations:
+- `tiny_scenario_configuration.py`: A minimal example scenario.
+- `smaller_scenario_configuration.py`: A compact scenario configuration used for develompent and rapid testing.
+- `scenario_configuration.py`: The main scenario configuration.
+- `three_net_configuration.py`: Configuration for a three-network scenario. Used for evaluation of the model overfitting.
+Implements the network game's configuration of hosts, data, services, and connections. It is taken from [CYST](https://pypi.org/project/cyst/).
+#### **`utils/`**
+Helper modules:
+- `utils.py`: General-purpose utilities.
+- `log_parser.py`: Tools for parsing game logs.
+- `gamaplay_graphs.py`: Tools for visualizing gameplay data.
+- `actions_parser.py`: Parsing and analyzing game actions.
-- File `env/network_security_game.py` implements the game environment
-- File `env/game_components.py` implements a library with objects used in the environment. See [detailed explanation](docs/Components.md) of the game components.
-- File `utils/utils.py` is a collection of utils functions which the agents can use
-- Files in the `env/scenarios` folder, such as `env/scenarios/scenario_configuration.py`. Implements the network game's configuration of hosts, data, services, and connections. It is taken from CYST.
The [scenarios](#definition-of-the-network-topology) define the **topology** of a network (number of hosts, connections, networks, services, data, users, firewall rules, etc.) while the [task-configuration](#task-configuration) is to be used for definition of the exact task for the agent in one of the scenarios (with fix topology).
- Agents compatible with the NetSecGame are located in a separate repository [NetSecGameAgents](https://github.com/stratosphereips/NetSecGameAgents/tree/main)
@@ -52,7 +94,7 @@ The [scenarios](#definition-of-the-network-topology) define the **topology** of
- The action FindServices finds the new services in a host. If in a subsequent call to FindServices there are less services, they completely replace the list of previous services found. That is, each list of services is the final one, and no memory of previous open services is retained.
-### Assumption and Conditions for Actions
+#### Assumption and Conditions for Actions
1. When playing the `ExploitService` action, it is expected that the agent has discovered this service before (by playing `FindServices` in the `target_host` before this action)
2. The `Find Data` action finds all the available data in the host if successful.
3. The `Find Data` action requires ownership of the target host.
@@ -61,16 +103,6 @@ The [scenarios](#definition-of-the-network-topology) define the **topology** of
6. Parameters of `ScanNetwork` and `FindServices` can be chosen arbitrarily (they don't have to be listed in `known_newtworks`/`known_hosts`)
7. The `BlockIP` action needs its three parameters (Source host, Target host, and Blocked host) to be in the controlled list of the Agent.
-### Actions for the defender
-The defender does have the action to block an IP address in a target host.
-
-
-The actions are:
-- BlockIP(). That takes as parameters:
- - "target_host": IP object where the block will be applied.
- - "source_host": IP object from which this action is executed.
- - "blocked_host": IP object to block in ANY direction as seen in the target_host.
-
> [!NOTE]
> The global defender, available in the previous environment versions, will not be supported in the future. To enable backward compatibility, the global defender functionality can be enabled by adding `use_global_defender: True` to the configuration YAML file in the `env` section. This option is disabled by default.
@@ -148,7 +180,7 @@ The system monitors actions and maintains a history of recent ones within the ti
This approach ensures that only repeated or excessive behavior is flagged, reducing false positives while maintaining a realistic monitoring system.
-### Starting the game
+## Starting the game
The environment should be created before starting the agents. The properties of the game, the task and the topology can be either read from a local file or via REST request to the GameDashboard.
#### To start the game with local configuration file
@@ -168,99 +200,108 @@ When created, the environment:
When the game server is created, [agents](https://github.com/stratosphereips/NetSecGameAgents/tree/main) connect to it and interact with the environment. In every step of the interaction, agents submits an [Action](./docs/Components.md#actions) and receives [Observation](./docs/Components.md#observations) with `next_state`, `reward`, `is_terminal`, `end`, and `info` values. Once the terminal state or timeout is reached, no more interaction is possible until the agent asks for a game reset. Each agent should extend the `BaseAgent` class in [agents](https://github.com/stratosphereips/NetSecGameAgents/tree/main).
-## Configuration
+### Configuration
The NetSecEnv is highly configurable in terms of the properties of the world, tasks, and agent interaction. Modification of the world is done in the YAML configuration file in two main areas:
1. Environment (`env` section) controls the properties of the world (taxonomy of networks, maximum allowed steps per episode, probabilities of action success, etc.)
2. Task configuration defines the agents' properties (starting position, goal)
-### Environment configuration
+#### Environment configuration
The environment part defines the properties of the environment for the task (see the example below). In particular:
- `random_seed` - sets seed for any random processes in the environment
- `scenario` - sets the scenario (network topology) used in the task (currently, `scenario1_tiny`, `scenario1_small`, `scenario1` and `three_nets` are available)
-- `max_steps` - sets the maximum number of steps an agent can make before an episode is terminated
-- `store_replay_buffer` - if `True`, interaction of the agents is serialized and stored in a file
+- `save_tajectories` - if `True`, interaction of the agents is serialized and stored in a file
- `use_dynamic_addresses` - if `True`, the network and IP addresses defined in `scenario` are randomly changed at the beginning of **EVERY** episode (the network topology is kept as defined in the `scenario`. Relations between networks are kept, IPs inside networks are chosen at random based on the network IP and mask)
-- ` use_firewall` - if `True` firewall rules defined in `scenario` are used when executing actions. When `False`, the firewall is ignored, and all connections are allowed (Default)
-- `goal_reward` - sets reward which agent gets when it reaches the goal (default 100)
-- `detection_reward` - sets the reward that which agent gets when it is detected (default -50)
-- `step_reward` - sets reward which agent gets for every step taken (default -1)
+- `use_firewall` - if `True` firewall rules defined in `scenario` are used when executing actions. When `False`, the firewall is ignored, and all connections are allowed (Default)
+- `use_global_defender` - if `True`, enables global defendr which is part of the environment and can stop interaction of any playing agent.
+- `rewards`:
+ - `win` - sets reward which agent gets when it reaches the goal (default 100)
+ - `loss` - sets the reward that which agent does not reach it's objective (default -10)
+ - `step_reward` - sets reward which agent gets for every step taken (default -1)
- `actions` - defines the probability of success for every ActionType
```YAML
env:
- random_seed: 42
- scenario: 'scenario1'
- max_steps: 15
- use_dynamic_addresses: False
- use_firewall: True
- goal_reward: 100
- detection_reward: -5
- step_reward: -1
- actions:
- scan_network:
- prob_success: 0.9
- find_services:
- prob_success: 0.9
- exploit_services:
- prob_success: 0.7
- find_data:
- prob_success: 0.8
- exfiltrate_data:
- prob_success: 0.8
+ random_seed: 'random'
+ scenario: 'scenario1'
+ use_global_defender: False
+ use_dynamic_addresses: False
+ use_firewall: True
+ save_trajectories: False
+ rewards:
+ win: 100
+ step: -1
+ loss: -10
+ actions:
+ scan_network:
+ prob_success: 1.0
+ find_services:
+ prob_success: 1.0
+ exploit_service:
+ prob_success: 1.0
+ find_data:
+ prob_success: 1.0
+ exfiltrate_data:
+ prob_success: 1.0
+ block_ip:
+ prob_success: 1.0
```
-## Task configuration
+#### Task configuration
The task configuration part (section `coordinator[agents]`) defines the starting and goal position of the attacker and the type of defender that is used.
-### Attacker configuration (`Attacker`)
+##### Attacker configuration (`[coordinator][agents][Attacker]`)
Configuration of the attacking agents. Consists of three parts:
-1. Goal definition (`goal`) which describes the `GameState` properties that must be fulfilled to award `goal_reward` to the attacker:
- - `known_networks:`(set)
- - `known_hosts`(set)
- - `controlled_hosts`(set)
+1. Goal definition (`goal`) which describes the `GameState` properties that must be fulfilled to award `win` reward to the attacker:
+ - `known_networks:`(list)
+ - `known_hosts`(list)
+ - `controlled_hosts`(list)
- `known_services`(dict)
- `known_data`(dict)
+ - `known_blocks`(dict)
Each of the parts can be empty (not part of the goal, exactly defined (e.g., `known_networks: [192.168.1.0/24, 192.168.3.0/24]`) or include the keyword `random` (`controlled_hosts: [213.47.23.195, random]`, `known_data: {213.47.23.195: [random]}`.
Additionally, if `random` keyword is used in the goal definition,
`randomize_goal_every_episode`. If set to `True`, each keyword `random` is replaced with a randomly selected, valid option at the beginning of **EVERY** episode. If set to `False`, randomization is performed only **once** when the environment is
2. Definition of starting position (`start_position`), which describes the `GameState` in which the attacker starts. It consists of:
- - `known_networks:`(set)
- - `known_hosts`(set)
- - `controlled_hosts`(set)
+ - `known_networks:`(list)
+ - `known_hosts`(list)
+ - `controlled_hosts`(list)
- `known_services`(dict)
- `known_data`(dict)
+ - `known_blocks`(dict)
The initial network configuration must assign at least **one** controlled host to the attacker in the network. Any item in `controlled_hosts` is copied to `known_hosts`, so there is no need to include these in both sets. `known_networks` is also extended with a set of **all** networks accessible from the `controlled_hosts`
3. Definition of maximum allowed amount of steps:
- - `max_steps:`(int)
+ - `max_steps:`(int): defines the maximum allowed number of steps for attackers in **each** episode.
Example attacker configuration:
```YAML
-agents:
- Attacker:
- max_steps: 100
- goal:
- randomize_goal_every_episode: False
- known_networks: []
- known_hosts: []
- controlled_hosts: []
- known_services: {192.168.1.3: [Local system, lanman server, 10.0.19041, False], 192.168.1.4: [Other system, SMB server, 21.2.39421, False]}
- known_data: {213.47.23.195: ["random"]}
- known_blocks: {'all_routers': 'all_attackers'}
-
- start_position:
- known_networks: []
- known_hosts: []
- # The attacker must always at least control the CC if the goal is to exfiltrate there
- # Example of fixing the starting point of the agent in a local host
- controlled_hosts: [213.47.23.195, random]
- # Services are defined as a target host where the service must be, and then a description in the form 'name, type, version, is_local'
- known_services: {}
- known_data: {}
- known_blocks: {}
+coordinator:
+ agents:
+ Attacker:
+ max_steps: 20
+ goal:
+ randomize_goal_every_episode: False
+ known_networks: []
+ known_hosts: []
+ controlled_hosts: []
+ known_services: {192.168.1.3: [Local system, lanman server, 10.0.19041, False], 192.168.1.4: [Other system, SMB server, 21.2.39421, False]}
+ known_data: {213.47.23.195: ["random"]}
+ known_blocks: {'all_routers': 'all_attackers'}
+
+ start_position:
+ known_networks: []
+ known_hosts: []
+ # The attacker must always at least control the CC if the goal is to exfiltrate there
+ # Example of fixing the starting point of the agent in a local host
+ controlled_hosts: [213.47.23.195, random]
+ # Services are defined as a target host where the service must be, and then a description in the form 'name, type, version, is_local'
+ known_services: {}
+ known_data: {}
+ known_blocks: {}
```
-### Defender configuration (`Defender`)
+
+##### Defender configuration (`[coordinator][agents][Defender]`)
Currently, the defender **is** a separate agent.
If you want a defender in the game, you must connect a defender agent. For playing without a defender, leave the section empty.
@@ -270,28 +311,26 @@ Example of defender configuration:
Defender:
goal:
description: "Block all attackers"
- is_any_part_of_goal_random: False
known_networks: []
known_hosts: []
controlled_hosts: []
known_services: {}
known_data: {}
- known_blocks: {'any_routers': 'all_attackers_controlled_hosts'}
+ known_blocks: {}
start_position:
- known_networks: [all_local]
- known_hosts: [all_local]
+ known_networks: []
+ known_hosts: []
controlled_hosts: [all_local]
- known_services: {all_local}
- known_data: {all_local}
+ known_services: {}
+ known_data: {}
blocked_ips: {}
known_blocks: {}
```
+As in other agents, the description is only a text for the agent, so it can know what is supposed to do to win. In the curent implementation, the *Defender* wins, if **NO ATTACKER** reaches their goal.
-As in other agents, the description is only a text for the agent, so it can know what is supposed to do to win. In this example, the goal of the defender is determined by a state where the known blocks can be applied in any router's firewall and must include all the controlled hosts of all the attackers. These are `magic` words that will push the coordinator to check these positions without revealing them to the defender.
-
-## Definition of the network topology
+### Definition of the network topology
The network topology and rules are defined using a [CYST](https://pypi.org/project/cyst/) simulator configuration. Cyst defines a complex network configuration, and this environment does not use all Cyst features for now. CYST components currently used are:
- Server hosts (are a NodeConf in CYST)
@@ -316,22 +355,22 @@ The network topology and rules are defined using a [CYST](https://pypi.org/proje
- which service is the exploit linked to
### Scenarios
-In the current state, we support a single scenario: Data exfiltration to a remote C&C server.
+In the current state, we support a single scenario: Data exfiltration to a remote C&C server. However, extensions can be made by modification of the task configuration.
#### Data exfiltration to a remote C&C
For the data exfiltration we support 3 variants. The full scenario contains 5 clients (where the attacker can start) and 5 servers where the data that is supposed to be exfiltrated can be located. *scenario1_small* is a variant with a single client (the attacker always starts there) and all 5 servers. *scenario1_tiny* contains only a single server with data. The tiny scenario is trivial and intended only for debugging purposes.
| Scenario 1 | Scenario 1 - small | Scenario 1 -tiny |
|---|
-  |  |  |
+  |  |  |
| 3-nets scenario |
|---|
- 
+ 
|
-## Trajectory storing and analysis
+### Trajectory storing and analysis
The trajectory is a sequence of GameStates, Actions, and rewards in one run of a game. It contains the complete information of the actions played by the agent, the rewards observed and their effect on the state of the environment. Trajectory visualization and analysis tools are described in [Trajectory analysis tools](./docs/Trajectory_analysis.md)
Trajectories performed by the agents can be stored in a file using the following configuration:
@@ -341,6 +380,7 @@ env:
```
> [!CAUTION]
> Trajectory files can grow very fast. It is recommended to use this feature on evaluation/testing runs only. By default, this feature is not enabled.
+
## Testing the environment
It is advised after every change you test if the env is running correctly by doing
@@ -353,25 +393,5 @@ This will load and run the unit tests in the `tests` folder.
## Code adaptation for new configurations
The code can be adapted to new configurations of games and for new agents. See [Agent repository](https://github.com/stratosphereips/NetSecGameAgents/tree/main) for more details.
-## Function calling diagram
-
-```
-_handle_world_responses
- ├── _world_response_queue.get() # Called continuously to get a response from the World Response Queue.
- ├── _process_world_response # Called to process the response from the world.
- │ ├── _process_world_response_created # Called if agent status is JoinRequested. Processes agent creation.
- │ ├── _process_world_response_reset_done # Called if agent status is ResetRequested. Processes agent reset.
- │ ├── _remove_player # Called if agent status is Quitting. Removes the agent from the world.
- │ └── _process_world_response_step # Called if agent status is Ready, Playing, or PlayingActive. Processes a game step.
- ├── _answers_queues[agent_id].put() # Called to place the processed response in the agent's answer queue.
- └── asyncio.sleep() # Called to yield control back to the event loop.
-
-_process_world_response
- ├── _process_world_response_created # Called if agent status is JoinRequested. Processes agent creation.
- ├── _process_world_response_reset_done # Called if agent status is ResetRequested. Processes agent reset.
- ├── _remove_player # Called if agent status is Quitting. Removes the agent from the world.
- └── _process_world_response_step # Called if agent status is Ready, Playing, or PlayingActive. Processes a game step.
- ```
-
## About us
This code was developed at the [Stratosphere Laboratory at the Czech Technical University in Prague](https://www.stratosphereips.org/).