Merge pull request #26 from blacklanternsecurity/hackspace-dev

Hackspace Dev

Author: TheTechromancer

.github/workflows/tests.yml

@@ -11,7 +11,7 @@ concurrency:
 
 jobs:
   test:
-    runs-on: ubuntu-latest
+    runs-on: self-hosted
     strategy:
       fail-fast: false
       matrix:
@@ -20,11 +20,11 @@ jobs:
       mongo:
         image: mongo
         ports:
-          - 27017:27017
+          - 127.0.0.1:27017:27017
       redis:
         image: redis
         ports:
-          - 6379:6379
+          - 127.0.0.1:6379:6379
     steps:
       - uses: actions/checkout@v4
       - name: Set up Python

README.md

@@ -21,6 +21,7 @@ BBOT Server is a database and multiplayer hub for all your [BBOT](https://github
     - [x] REST API
     - [x] Python SDK
     - [x] Export to JSON/CSV
+- [x] [AI interaction via MCP](#MCP)
 
 ## Installation
 
@@ -112,7 +113,7 @@ bbctl scan start "evilcorp_subdomains"
 
 You can monitor the scan's progress in several ways:
 
-**Tail asset activity**:
+### Tail asset activity:
 
 This will output an activity whenever a change is detected to an asset, e.g. a change in DNS, new open port, vulnerability, or technology.
 
@@ -121,7 +122,7 @@ This will output an activity whenever a change is detected to an asset, e.g. a c
 bbctl activity tail
 ```
 
-**Tail raw events**:
+### Tail raw events:
 
 If you'd like, you can also tail the raw events as they stream in from the BBOT scan.
 
@@ -130,7 +131,7 @@ If you'd like, you can also tail the raw events as they stream in from the BBOT
 bbctl event tail
 ```
 
-**Check scan status**:
+### Check scan status:
 
 You can monitor or stop an in-progress scan:
 
@@ -151,23 +152,33 @@ You can list targets like so:
 ```bash
 # List targets
 bbctl target list
-```
 
-You can create a target manually:
+# Create a new target
+bbctl target create --seeds seeds.txt --blacklist blacklist.txt
+```
 
 ## Custom triggers
 
-TODO
+You can kick off a custom command or bash script whenever a certain activity happens, such as when a new technology or open port is discovered.
+
+```bash
+# Trigger a custom command whenever a new open port is discovered
+bbctl activity tail --json | jq -r 'select(.type == "PORT_OPENED") | .netloc' | while read netloc
+do
+  echo "New open port at $netloc"
+  ./custom_script.sh "$netloc"
+done
+```
 
 ## Alerting
 
 TODO
 
-## View/export the data
+## Query and Export Data
 
 You can query and export the data even while a scan is running.
 
-**List assets**:
+### Assets
 
 ```bash
 # List assets
@@ -180,7 +191,7 @@ bbctl asset export --csv > assets.csv
 bbctl asset export --json | jq
 ```
 
-**List events**:
+### Events
 
 ```bash
 # List events
@@ -193,19 +204,78 @@ bbctl event export --csv > events.csv
 bbctl event export --json | jq
 ```
 
+### Technologies
+
+```bash
+# List technologies
+bbctl technology list
+
+# List technologies by specific domain
+bbctl technology list --domain evilcorp.com
+```
+
+### Findings
+
+```bash
+# List findings
+bbctl finding list
+
+# Search findings for a certain string
+bbctl finding list --search "IIS"
+```
+
+### Statistics
+
+Overarching statistics are stored for all assets, and can be queried by target or domain.
+
+```bash
+# List stats for all assets
+bbctl asset stats | jq
+
+# List stats for specific domain
+bbctl asset stats --domain evilcorp.com | jq
+```
+
+### MCP
+
+BBOT Server supports chat-based AI interaction via MCP (Model Context Protocol).
+
+The SSE server listens at `http://localhost:8807/v1/mcp/`
+
+`mcp.json` (cursor / vs code):
+```json
+{
+    "mcpServers": {
+        "bbot": {
+            "url": "http://localhost:8807/v1/mcp/"
+        }
+    }
+}
+```
+
+After connecting your AI client to BBOT Server, you can ask it sensible questions like, "Use MCP to get all the bbot findings", "what are the top open ports?", "what else can you do with BBOT MCP?", etc.
+
 ## Screenshots
 
-*Scan editor (terminal UI)*
+*Tailing activities in real time*
+
+![activity-tail](https://github.com/user-attachments/assets/8188f32c-45bc-4f81-bf98-c59adfbdc5df)
+
+*AI Chat interaction via MCP*
+
+![mcp](https://github.com/user-attachments/assets/3997b534-2ed8-4e04-b8c3-a7b42daf4106)
+
+*Scans*
 
-![scan-editor](https://github.com/user-attachments/assets/9c31d2ef-f4f0-4d65-bd45-263a8d16bd7f)
+![scan-run-list](https://github.com/user-attachments/assets/d6ffb6e5-06d7-4439-936a-3d2b1a6306ee)
 
-*Launch and monitor concurrent scans*
+*Technologies*
 
-![scans](https://github.com/user-attachments/assets/7644809f-e111-49f8-b627-c0c77a65110a)
+![technology-list](https://github.com/user-attachments/assets/7b662858-8c08-4bb9-a520-6381d2964dde)
 
-*Realtime asset monitoring*
+*Findings*
 
-![monitor-assets](https://github.com/user-attachments/assets/ed7ac9f2-34e8-4770-a971-49fdf7f77bea)
+![finding-list](https://github.com/user-attachments/assets/3fcbb977-6d47-4dc1-81b7-a26e8e3bc292)
 
 *REST API*
 

bbot_server/agent/agent.py

@@ -10,6 +10,7 @@
 
 from bbot.scanner import Scanner, Preset
 from bbot.scanner.dispatcher import Dispatcher
+from bbot.constants import get_scan_status_name, SCAN_STATUS_NOT_STARTED
 
 from bbot_server.config import BBOT_SERVER_CONFIG
 from bbot_server.errors import BBOTServerValueError
@@ -20,6 +21,7 @@
 default_server_url = BBOT_SERVER_CONFIG.get("url", "http://localhost:8807/v1/")
 default_bbot_preset = BBOT_SERVER_CONFIG.get("agent", {}).get("base_preset", {})
 
+log = logging.getLogger("bbot_server.agent")
 
 VALID_AGENT_COMMANDS = {}
 
@@ -120,7 +122,12 @@ async def start_scan(self, scan_run: dict[str, Any]):
         scan = Scanner(preset=preset, scan_id=scan_run.id, dispatcher=self.dispatcher)
         self._patch_scan(scan)
         self.scan_task = asyncio.create_task(self._start_scan_task(scan))
-        return {"scan_id": scan.id, "scan_status": scan.status, "status": "success"}
+        return {
+            "scan_id": scan.id,
+            "scan_status": scan.status,
+            "scan_status_code": scan._status_code,
+            "status": "success",
+        }
 
     @command
     async def cancel_scan(self, force: bool = False):
@@ -159,7 +166,13 @@ async def _start_scan_task(self, scan):
 
     @command
     async def get_agent_status(self, detailed: bool = False):
-        ret = {"agent_status": self.status, "scan_status": getattr(self.scan, "status", "NOT_RUNNING")}
+        scan_status_code = getattr(self.scan, "_status_code", SCAN_STATUS_NOT_STARTED)
+        scan_status = get_scan_status_name(scan_status_code)
+        ret = {
+            "agent_status": self.status,
+            "scan_status": scan_status,
+            "scan_status_code": scan_status_code,
+        }
         if detailed and self.scan is not None:
             ret["scan_status_detail"] = self.scan.modules_status(detailed=detailed)
         return ret
@@ -262,14 +275,16 @@ async def loop(self):
     async def gratuitous_status_update(self, scan=None, scan_status=None):
         scan_id = getattr(scan, "id", None)
         scan_name = getattr(scan, "name", None)
-        scan_status = scan_status or getattr(scan, "status", "NOT_RUNNING")
+        scan_status_code = getattr(scan, "_status_code", SCAN_STATUS_NOT_STARTED)
+        scan_status = get_scan_status_name(scan_status_code)
         status = {
             "agent_status": self.status,
             "scan_status": scan_status,
+            "scan_status_code": scan_status_code,
             "scan_id": scan_id,
             "scan_name": scan_name,
         }
-        if scan_id:
+        if scan_id and scan is not None:
             try:
                 status["scan_status_detail"] = scan.modules_status(detailed=True)
             except Exception as e:

bbot_server/api/__init__.py

@@ -0,0 +1,3 @@
+from ._fastapi import make_server_app
+
+__all__ = ["make_server_app"]

bbot_server/applets/_base.py

@@ -235,7 +235,13 @@ async def build_indexes(self, model):
             if "indexed" in field.metadata:
                 index = [(fieldname, ASCENDING)]
                 self.log.debug(f"Creating index: {index}")
-                await self.collection.create_index(index, unique=unique)
+                try:
+                    await self.collection.create_index(index, unique=unique)
+                except OperationFailure as e:
+                    if "existing index has the same name" in str(e):
+                        self.log.debug(f"Index {index} already exists, skipping")
+                    else:
+                        raise
             # text indexes
             if "indexed-text" in field.metadata:
                 index = [(fieldname, "text")]
@@ -281,7 +287,7 @@ async def build_indexes(self, model):
                     fields = [fieldname] + fields
                     index = [(fieldname, ASCENDING) for fieldname in fields]
                     self.log.debug(f"Creating compound index: {index}")
-                    await self.collection.create_index(index, unique=unique)
+                    await self.collection.create_index(index, unique=True)
 
     async def register_watchdog_tasks(self, broker):
         # register watchdog tasks
@@ -339,6 +345,7 @@ async def emit_activity(self, *args, **kwargs):
         await self._emit_activity(activity)
 
     async def _emit_activity(self, activity: Activity):
+        self.log.info(f"Emitting activity: {activity.type} - {activity.description}")
         await self.root.message_queue.publish_asset(activity)
 
     def include_app(self, app_class):

bbot_server/applets/agents/agents.py

@@ -5,6 +5,9 @@
 from fastapi import WebSocket
 from contextlib import suppress
 from datetime import datetime, timezone
+
+from bbot.constants import get_scan_status_code, get_scan_status_name, SCAN_STATUS_QUEUED
+
 from bbot_server.models.agent_models import Agent
 from bbot_server.applets._base import BaseApplet, api_endpoint
 
@@ -138,12 +141,22 @@ async def dock(self, websocket: WebSocket, agent_id: UUID4):
                     self.log.debug(f"Server received gratuitous message from agent {agent.name}: {message}")
                     if "agent_status" in message.response:
                         agent_status = message.response["agent_status"]
-                        scan_status = message.response["scan_status"]
+                        scan_status_code = message.response.get("scan_status_code", SCAN_STATUS_QUEUED)
+                        scan_status_code = get_scan_status_code(scan_status_code)
+                        scan_status = get_scan_status_name(scan_status_code)
                         scan_run_id = message.response.get("scan_id", None)
                         scan_name = message.response.get("scan_name", None)
                         if scan_name and scan_run_id:
-                            existing_scan = await self.root.get_scan_run(scan_run_id=scan_run_id)
-                            if existing_scan and existing_scan.status != scan_status:
+                            try:
+                                existing_scan = await self.root.get_scan_run(scan_run_id=scan_run_id)
+                            except self.BBOTServerNotFoundError as e:
+                                self.log.error(f"Error getting scan run {scan_run_id}: {e}")
+                                existing_scan = None
+                            existing_scan_status_code = getattr(existing_scan, "status_code", SCAN_STATUS_QUEUED)
+                            if scan_status_code > existing_scan_status_code:
+                                await self.root.update_scan_run_status(
+                                    scan_run_id=scan_run_id, status_code=scan_status_code
+                                )
                                 await self.emit_activity(
                                     type="SCAN_STATUS",
                                     detail={
@@ -166,7 +179,7 @@ async def dock(self, websocket: WebSocket, agent_id: UUID4):
                     self.log.error(traceback.format_exc())
 
         except Exception as e:
-            self.log.error(f"Error in server-side websocket loop for agent {agent.id}: {e}")
+            self.log.error(f"Error in server-side websocket for agent {agent.id}: {e}")
             self.log.error(traceback.format_exc())
 
         finally:

bbot_server/applets/assets.py

@@ -1,12 +1,12 @@
 # applets imports
-from bbot_server.applets.risk import Risk
 from bbot_server.applets.emails import EmailsApplet
 from bbot_server.applets.export import ExportApplet
 from bbot_server.applets.findings import FindingsApplet
 from bbot_server.applets.dns_links import DNSLinksApplet
 from bbot_server.applets.open_ports import OpenPortsApplet
 from bbot_server.applets.web_screenshots import WebScreenshotsApplet
 from bbot_server.applets.technologies import TechnologiesApplet
+from bbot_server.applets.cloud import CloudApplet
 
 from bbot_server.assets import Asset
 from bbot_server.utils.misc import utc_now
@@ -17,14 +17,14 @@ class AssetsApplet(BaseApplet):
     name = "Assets"
     description = "hostnames and IP addresses discovered during scans"
     include_apps = [
-        FindingsApplet,
         OpenPortsApplet,
         DNSLinksApplet,
         EmailsApplet,
         WebScreenshotsApplet,
         ExportApplet,
-        Risk,
         TechnologiesApplet,
+        CloudApplet,
+        FindingsApplet,
     ]
 
     model = Asset
@@ -153,7 +153,7 @@ async def _get_asset(
         fields: list[str] = None,
     ):
         query = dict(query or {})
-        if type is not None:
+        if type is not None and "type" not in query:
             query["type"] = type
         if host is not None:
             query["host"] = host
@@ -175,3 +175,6 @@ async def _query_assets(self, query: dict, fields: list[str] = None, sort: list[
             cursor = cursor.sort(sort)
         async for asset in cursor:
             yield asset
+
+    async def _update_asset(self, host: str, update: dict):
+        await self.collection.update_one({"host": host}, {"$set": update})

bbot_server/applets/buckets.py

@@ -0,0 +1,6 @@
+from bbot_server.applets._base import BaseApplet
+
+
+class BucketsApplet(BaseApplet):
+    name = "Buckets"
+    description = "storage buckets discovered during scans"