Architecture refactoring, .env configuration file and finalizing user registration and login

Author: codeyourweb

.env

@@ -0,0 +1,20 @@
+SENTINELKIT_FRONTEND_HOSTNAME=sentinel-kit.local
+SENTINELKIT_BACKEND_HOSTNAME=backend.sentinel-kit.local
+SENTINELKIT_PMA_HOSTNAME=phpmyadmin.sentinel-kit.local
+SENTINELKIT_KIBANA_HOSTNAME=kibana.sentinel-kit.local
+SENTINELKIT_GRAFANA_HOSTNAME=grafana.sentinel-kit.local
+SENTINELKIT_DATAMONITOR_SERVER_TOKEN=9561ffd1b6de615286b9e52a9d5bc3226970449700c9461bdbe4225730b47b20
+BACKEND_JWT_PASSPHRASE=f164cfc913d2faf65a1b7bc8ccd4aa8b11b5958bce7c20c8cf159a576f8a75f7
+MYSQL_ROOT_PASSWORD=sentinel-kit_r00tp4ssw0rd
+MYSQL_USER=sentinel-kit_user
+MYSQL_PASSWORD=sentinel-kit_passwd
+MYSQL_DATABASE=sentinel-kit_db
+GF_SECURITY_ADMIN_USER=sentinel-kit_grafana_admin
+GF_SECURITY_ADMIN_PASSWORD=sentinel-kit_grafana_password
+SFTP_USER=sentinel-kit_sftp_user
+SFTP_PASSWORD=sentinel-kit_sftp_passwd
+ELASTICSTACK_VERSION=9.2.0
+ELASTICSEARCH_CLUSTER_NAME=sentinel-kit-elasticsearch-cluster
+ELASTICSEARCH_LICENSE=basic
+ELASTICSEARCH_MEMORY_LIMIT=4294967296
+ELASTICSEARCH_PASSWORD=sentinelkit_elastic_passwd

.gitignore

@@ -1,13 +1,19 @@
 /.env.local
 /.env.local.php
 /.env.*.local
+/config/certificates/caddy_server/*
+/config/certificates/elasticsearch/*
+/config/certificates/jwt/*
 /config/yara_ruleset/*
 /config/sigma_ruleset/*
+/data/caddy_logs/*
 /data/ftp_data/*
 /data/grafana/*
+/data/kibana/*
 /data/log_ingest_data/*
 /data/mysql_data/*
 /data/yara_triage_data/*
+/docs/graphs/*.bkp
 /sentinel-kit_server_frontend/node_modules/*
 /sentinel-kit_server_frontend/package-lock.json
 /sentinel-kit_server_backend/migrations/*

README.md

@@ -1,5 +1,5 @@
-![Sentinel Kit](./.github/img/sentinel-kit_logo.png)
-# 🛡️ Sentinel Kit: The Simplified Platform for Incident Response (DFIR/SOC)
+![Sentinel Kit](docs/img/sentinel-kit_logo.png)
+# 🛡️ Sentinel Kit: The Simplified Platform for Incident Response (SOC & DFIR)
 
 ## WARNING: This project is currently in an early stage of development. Not all components have been ported to this repository, and the features are not yet stable enough for production use. 
 ---
@@ -16,7 +16,7 @@ Sentinel Kit is an all-in-one toolkit that covers the entire security incident l
 
 * **Log Collection & Parsing (SIEM Lite)**: Uses **Fluent Bit** for data ingestion and **Elasticsearch** for storage and indexing.
 * **Advanced Analysis & Triage**: Planned integration of **Sigma** rules for log-based detection and **YARA** for suspicious file triage (via upload mechanisms).
-* **Detection and Response (EDR)**: A dedicated agent (integrating into the ecosystem) is planned to provide real-time detection and response functionalities.
+* **Detection and Response (EDR)**: A dedicated agent (integrating into the ecosystem) provide real-time detection and response functionalities. In addition, this optional agent can act as a collection element to forward logs from your workstations to the sentinel-kit server.
 * **Secure Uploads**: Provides a dedicated **SFTP** server for uploading evidence, logs, or suspicious files.
 * **Comprehensive Visualization**: Monitoring dashboards via **Kibana** and **Grafana/Prometheus**.
 
@@ -40,7 +40,17 @@ This project is designed to be deployed in minutes using Docker Compose.
     cd sentinel-kit
     ```
 
-2.  **Launch the Stack:**
+2. **Set the following DNS entry (in hosts file if you are running it locally):**
+```bash
+# OS host file
+127.0.0.1   sentinel-kit.local
+127.0.0.1   backend.sentinel-kit.local
+127.0.0.1   phpmyadmin.sentinel-kit.local
+127.0.0.1   kibana.sentinel-kit.local
+127.0.0.1   grafana.sentinel-kit.local
+```
+
+3.  **Launch the Stack:**
     ```bash
     docker-compose up -d
     ```
@@ -60,10 +70,11 @@ Once the stack is running, you can access the interfaces via the default ports e
 
 | Service | Role | Default Access |
 | :--- | :--- | :--- |
-| **Web Interface** (Frontend) | Access to the main application | `http://localhost:80` or `https://localhost:443` (via Caddy) |
-| **Kibana** | Exploration and visualization of Elastic logs | `http://localhost:5601` |
-| **Grafana** | Monitoring dashboards | `http://localhost:3000` |
-| **phpMyAdmin** | MySQL database management | `http://localhost:8080` |
+| **Web Interface** (Admin frontend) | Access to the admin application | `https://sentinel-kit.local` |
+| **Web API**  | Used for clients<->server communications and admin actions over the web interface | `https://backend.sentinel-kit.local` |
+| **Kibana** | Exploration and visualization of Elastic logs | `http://kibana.sentinel-kit.local` |
+| **Grafana** | Monitoring dashboards | `http://grafana.sentinel-kit.local` |
+| **phpMyAdmin** | MySQL database management | `http://phpmyadmin.sentinel-kit.local` |
 | **SFTP Server** | Secure file/evidence upload | Port `2222` |
 
 ### Default Credentials (Utilities)
@@ -74,39 +85,23 @@ Once the stack is running, you can access the interfaces via the default ports e
 | **MySQL (DB)** | `sentinel-kit_user` | `sentinel-kit_passwd` |
 | **SFTP** | `sentinel-kit_ftp_user` | `sentinel-kit_ftp_passwd` |
 
+All of this can be edited in `.env` file
+
 ---
 
 ## 🛠️ Technical Architecture (via `docker-compose.yml`)
 
 The architecture is modular and relies on the interconnection of several services via the **sentinel-kit-network** network.
-
-### Application Components
-
-* `sentinel-kit-frontend-app`: User Interface.
-* `sentinel-kit-backend-app`: Business logic, API, and data management (depends on MySQL).
-* `sentinel-kit-ftp-server`: Entry point for manual file collection (evidence, YARA/Sigma files).
-* `sentinel-kit-caddy-server`: Reverse proxy managing HTTP/HTTPS access (ports 80/443) and routing to the frontend and  backend.
-
-### Collection & Storage Components
-
-* `sentinel-kit-fluentbit-server`: Log collector (Ingestion on port `24224`) that sends data to Elasticsearch.
-* `sentinel-kit-elasticsearch-db`: Search and log storage engine.
-* `sentinel-kit-mysql-db`: Relational database (for the backend).
-
-### Utility Components (Monitoring & DB)
-
-* `sentinel-kit-kibana-utils`: Visualization of Elasticsearch data (log analysis).
-* `sentinel-kit-prometheus-utils`: Ingestion , parsing, and forwarding metrics collection.
-* `sentinel-kit-grafana-utils`: Metrics visualization (Prometheus) and potentially other data.
-* `sentinel-kit-phpmyadmin-utils`: Web interface for MySQL management (dev / admin).
+![Sentinel-Kit architecture](docs/img/sentinel-kit_network_flow.png)
 
 ## ⚙️ Configuration
 
 Main configurations are located in the `config/` folder: (edit these elements only if you know what you are doing 😊)
 
-* `config/fluentbit_server`: Fluent Bit configuration files (inputs, filters, outputs to Elasticsearch).
 * `config/caddy_server`: Reverse proxy that serve front and back-end web applications
+* `config/certificates`: Contains TLS certification chains for elasticstack, caddy and backend JWT
 * `config/docker-config`: Server stack configuration (dockerfile, entrypoints...).
+* `config/fluentbit_server`: Fluent Bit configuration files (inputs, filters, outputs to Elasticsearch).
 * `config/grafana`: Grafana initial setup (datasources and dashboards).
 * `config/prometheus/prometheus.yml`: Prometheus monitoring targets configuration.
 * `config/sigma_ruleset`: sigma rules used on elasticsearch ingested logs
@@ -116,8 +111,10 @@ Main configurations are located in the `config/` folder: (edit these elements on
 
 Persistent data are located in the `data/` folder:
 
+* `data/caddy_logs`: Store the caddy server access & error logs
 * `data/ftp_data`: Store file uploaded on the SFTP server
 * `data/grafana`: Contains a persistence of your grafana profile if you want to make your own dashboard and customizations
+* `data/kibana`: Kibana user customizations
 * `data/log_ingest_data`: Is designed to forward logs if you don't want to use fluentbit HTTP forwarder
 * `data/mysql_data`: Constains a persistence of the web backend database
 * `data/yara_triage_data`: is used to automatically scan any file placed in this folder  
@@ -129,4 +126,15 @@ Persistent data are located in the `data/` folder:
 To stop and remove the containers, networks, and volumes created by Docker Compose:
 
 ```bash
-docker-compose down -v
+docker-compose down -v
+```
+
+If you want to erase all user data:
+* remove the __content__ of every folder inside `data/`
+* remove the __content__ of `config/certificates/` in caddy_server, elasticsearch and jwt
+* remove the __content__ of `config/grafana`
+* finally, rebuild the stack with the following command:
+
+```bash
+docker-compose up --build --force-recreate
+```

config/caddy_server/Caddyfile

@@ -1,30 +1,81 @@
-:443, localhost {
-    @not_api not path /api/*
+# TLS reverse proxy configuration
+{$SENTINELKIT_BACKEND_HOSTNAME}:443 {
+    tls internal
 
-    log /api/* {
+    log {
         output file /var/log/caddy/backend.log
+        format json
     }
 
-    log @not_api {
+    reverse_proxy sentinel-kit-app-backend:8000
+    encode zstd gzip
+}
+
+{$SENTINELKIT_FRONTEND_HOSTNAME}:443 {
+    tls internal
+    
+    log {
         output file /var/log/caddy/frontend.log
+        format json
     }
 
-    handle /api/* {
-        reverse_proxy sentinel-kit-backend-app:8000
-    }
+    reverse_proxy sentinel-kit-app-frontend:3000
+    encode zstd gzip
+}
+
+{$SENTINELKIT_PMA_HOSTNAME}:443 {
+    tls internal
 
-    handle /bundles/* {
-        reverse_proxy sentinel-kit-backend-app:8000
+    log {
+        output file /var/log/caddy/phpmyadmin.log
+        format json
     }
 
-    handle /_wdt/* {
-        reverse_proxy sentinel-kit-backend-app:8000
+    reverse_proxy sentinel-kit-utils-phpmyadmin:80
+    encode zstd gzip
+}
+
+{$SENTINELKIT_KIBANA_HOSTNAME}:443 {
+    tls internal
+    
+    log {
+        output file /var/log/caddy/kibana.log
+        format json
     }
 
-    handle /_profiler/* {
-        reverse_proxy sentinel-kit-backend-app:8000
-    }    
-    handle @not_api {
-        reverse_proxy sentinel-kit-frontend-app:3000
+    reverse_proxy sentinel-kit-utils-kibana:5601
+    encode zstd gzip
+}
+
+{$SENTINELKIT_GRAFANA_HOSTNAME}:443 {
+    tls internal
+
+    log {
+        output file /var/log/caddy/grafana.log
+        format json
     }
-}
+
+    reverse_proxy sentinel-kit-utils-grafana:3000
+    encode zstd gzip
+}
+
+# HTTP to HTTPS redirection
+{$SENTINELKIT_BACKEND_HOSTNAME}:80 {
+    redir https://{$SENTINELKIT_BACKEND_HOSTNAME}{uri}
+}
+
+{$SENTINELKIT_FRONTEND_HOSTNAME}:80 {
+    redir https://{$SENTINELKIT_FRONTEND_HOSTNAME}{uri}
+}
+
+{$SENTINELKIT_PMA_HOSTNAME}:80 {
+    redir https://{$SENTINELKIT_PMA_HOSTNAME}{uri}
+}
+
+{$SENTINELKIT_KIBANA_HOSTNAME}:80 {
+    redir https://{$SENTINELKIT_KIBANA_HOSTNAME}{uri}
+}
+
+{$SENTINELKIT_GRAFANA_HOSTNAME}:80 {
+    redir https://{$SENTINELKIT_GRAFANA_HOSTNAME}{uri}
+}

config/fluentbit_server/fluent-bit.conf

@@ -25,7 +25,7 @@
 [OUTPUT]
     Name         es
     Match        http.logs
-    Host         sentinel-kit-elasticsearch-db
+    Host         sentinel-kit-db-elasticsearch
     Port         9200
     Index        essai
     Logstash_Format Off

config/grafana/datasources/elasticsearch.yml

@@ -1,9 +1,22 @@
 apiVersion: 1
 
 datasources:
-  - name: Elasticsearch-logs
+  - name: Elasticsearch-logs-es01
     type: elasticsearch
-    url: http://sentinel-kit-elasticsearch-db:9200
+    url: http://sentinel-kit-db-elasticsearch-es01:9200
+    access: proxy
+    isDefault: false
+    version: 9
+    database: '*'
+    jsonData:
+      esVersion: 90
+      timeField: '@timestamp'
+      maxConcurrentShardRequests: 5
+    readOnly: false
+    orgId: 1
+  - name: Elasticsearch-logs-es02
+    type: elasticsearch
+    url: http://sentinel-kit-db-elasticsearch-es02:9200
     access: proxy
     isDefault: false
     version: 9

config/grafana/datasources/prometheus.yml

@@ -3,7 +3,7 @@ apiVersion: 1
 datasources:
   - name: Prometheus-Fluentbit
     type: prometheus
-    url: http://sentinel-kit-prometheus-utils:9090
+    url: http://sentinel-kit-utils-prometheus:9090
     access: proxy
     isDefault: true
     version: 1