Skip to content

Commit 09b6fb0

Browse files
nate-smalls-s1nate-smalls-s1
authored
Merge pull request #8 from natesmalley/feature/v1.3.0-metadata-improvements
feat: v1.3.0 - Enhanced metadata and repository improvements
2 parents a7d65a1 + 19a3c1a commit 09b6fb0

File tree

18 files changed

+796
-55
lines changed

18 files changed

+796
-55
lines changed

‎CHANGELOG.md‎

Lines changed: 91 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,92 @@
1-
date change
1+
# Changelog
22

3-
09-12-2025 1447 PST Initial Push
3+
All notable changes to the AI-SIEM repository will be documented in this file.
4+
5+
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
6+
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
7+
8+
## [1.3.0] - 2025-01-28
9+
10+
### Added
11+
- Added metadata.yaml files for all workflow components
12+
- AI SIEM workflows for detection enrichment (by Patryk Kostek)
13+
- Abnormal Security audit log ingestion workflow (by Patryk Kostek)
14+
- Added comprehensive metadata.yaml for monitors directory
15+
- Added `search_type` field to all detection metadata files (powerquery | star_rule | watchlist_alert)
16+
- Added Monitors Installation Guide section to main README
17+
18+
### Changed
19+
- Updated all 8 detection metadata.yaml files with accurate descriptions based on actual detection logic
20+
- AzureAD-Entra: Impossible travel, MFA disabled logins, account changes
21+
- O365: File access rates and excessive logons
22+
- Fortinet FortiGate: Virus detection in firewall logs
23+
- Hello World: HTTP 5xx error rate monitoring
24+
- SQL Security: SQL Server Event ID monitoring
25+
- Volume Alerts: Log volume drop detection
26+
- XSOAR Trigger: Lateral movement detection
27+
- Zscaler: DLP engine threat detection
28+
- Consolidated monitors README content into main README
29+
- Updated repository layout descriptions with accurate component counts
30+
31+
### Removed
32+
- Removed duplicate monitors/README.md file
33+
34+
## [1.2.0] - 2025-10-27
35+
36+
### Added
37+
- Three new AI SIEM workflow components (by Patryk Kostek)
38+
- [AI SIEM] Add Event Data to Detection Note
39+
- [AI SIEM] Get Data Source Resources
40+
- Abnormal Security - Audit Log Ingestor
41+
42+
## [1.1.0] - 2025-10-22
43+
44+
### Changed
45+
- Fixed Fortigate typo in directory names and metadata (was: Fortigagte)
46+
47+
### Added
48+
- Pipelines section to README documentation
49+
50+
## [1.0.2] - 2025-10-08
51+
52+
### Added
53+
- AWS CloudTrail dashboard examples
54+
- aws-cloudTrail-trends.conf
55+
- aws-cloudTrail-services-breakdown.conf
56+
57+
## [1.0.1] - 2025-09-14
58+
59+
### Added
60+
- Pipelines directory structure for Observo Transformations
61+
- Updated main branch as default branch
62+
63+
## [1.0.0] - 2025-09-12
64+
65+
### Added
66+
- SECURITY.md file with security policy and vulnerability reporting guidelines
67+
- CHANGELOG.md initial file
68+
- .gitignore configuration
69+
- Monitor files restored:
70+
- log_gen.py - Log generation for testing
71+
- maxmind.py - GeoIP enrichment
72+
- powerquerymonitor.py - PowerQuery monitoring
73+
74+
### Changed
75+
- Set main branch as default (previously master)
76+
- Updated README with repository structure
77+
78+
### Removed
79+
- Removed updated_parsers.zip from parsers/community
80+
- Removed .DS_Store files
81+
82+
## [0.9.0] - 2025-08-26
83+
84+
### Initial Release
85+
- Initial repository structure with 255+ components
86+
- 79 dashboards with metadata
87+
- 8 detection rules with configurations
88+
- 165 parsers (148 community, 17 SentinelOne official)
89+
- 3 Python monitoring scripts
90+
- Workflow templates for automated responses
91+
- Complete metadata.yaml requirements for all component types
92+
- GNU AGPL-3.0 License

‎README.md‎

Lines changed: 38 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,12 +22,12 @@ ai-siem/ # AI SIEM core structure (255+ components)
2222
│ └── community/ # Community-contributed dashboards
2323
├── detections/ # Detection rules (8 detections with metadata)
2424
│ └── community/ # Community-contributed detection rules
25-
├── monitors/ # Monitoring scripts (3 Python monitors)
25+
├── monitors/ # Python monitoring scripts for Dataset Agent (log_gen, maxmind, powerquery)
2626
├── pipelines/ # Prepared for Observo Transformations
2727
├── parsers/ # Parsing logic and configurations (165 parsers)
2828
│ ├── community/ # 148 community parsers (*.conf + metadata)
2929
│ └── sentinelone/ # 17 official marketplace parsers (*.conf + metadata)
30-
└── workflows/ # Automated playbooks and responses (ready for content)
30+
└── workflows/ # Automated playbooks and responses (3 workflows with metadata)
3131
```
3232

3333
---
@@ -77,6 +77,41 @@ Released under the **GNU Affero General Public License v3.0 (AGPL-3.0)** – ens
7777

7878
---
7979

80+
## Monitors Installation Guide
81+
82+
### Dataset Agent Integration
83+
The monitors directory contains Python scripts for use with the Dataset Agent:
84+
- **log_gen.py** - Generate test logs for various vendor formats (Cisco, Windows DNS)
85+
- **maxmind.py** - MaxMind GeoIP enrichment for IP addresses
86+
- **powerquerymonitor.py** - PowerQuery monitoring capabilities
87+
88+
### Installation Steps
89+
1. Copy monitor files to Dataset Agent directory:
90+
```bash
91+
cp monitors/*.py /usr/share/scalyr-agent-2/py/scalyr_agent/builtin_monitors/
92+
```
93+
94+
2. Configure the agent by editing `/etc/scalyr-agent-2/agent.log`:
95+
```json
96+
monitors: [
97+
{
98+
"module": "scalyr_agent.builtin_monitors.log_gen",
99+
"logs": "/tmp/logs/*",
100+
"type_array": "['cisco', 'windows_dns']",
101+
"parser": "json",
102+
"time_pattern": "(?P<date>(\\d+ \\w+ \\d+|\\d+\\/\\d+\\/\\d+)) (?P<time>(\\d{2}:\\d{2}:\\d{2}\\.\\d{3}|\\d+:\\d+:\\d+ \\w+))",
103+
"sampling_rate": ".2"
104+
}
105+
]
106+
```
107+
108+
3. Start the Dataset Agent:
109+
```bash
110+
scalyr-agent-2 start
111+
```
112+
113+
---
114+
80115
## Getting help
81116
Open an issue. Office hours TBD based on requests.
82117

@@ -113,6 +148,7 @@ metadata_details:
113148
purpose: "Detects a specific action from a SentinelOne component or third-party integration"
114149
mitre_tactic_technique: "Provide the MITRE Tactic and Technique (if known)"
115150
datasource: "Name of the dataSource.name field"
151+
search_type: "powerquery | star_rule | watchlist_alert"
116152
usecase_plus: "Explain how combining this data with others enhances detection"
117153
severity: "Information | Low | Medium | High"
118154
expected_alert_scenario: "What alert behavior should users expect?"

‎detections/community/AzureAD-Entra-alerts-latest/metadata.yaml‎

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,12 @@
11
metadata_details:
2-
purpose: "Detection rule migrated from samples-main: AzureAD-Entra-alerts.conf"
3-
mitre_tactic_technique: "To be determined"
4-
datasource: "To be determined"
5-
usecase_plus: "Detection logic from samples-main"
2+
purpose: "Detect suspicious Azure AD/Entra ID activities including account changes, impossible travel, and authentication anomalies"
3+
mitre_tactic_technique: "T1078 - Valid Accounts, T1098 - Account Manipulation"
4+
datasource: "Azure Event Hub"
5+
search_type: "powerquery"
6+
usecase_plus: "Combines multiple Azure AD security detections including impossible traveler analysis using geolocation"
67
severity: "Medium"
7-
expected_alert_scenario: "Based on configured thresholds"
8+
expected_alert_scenario: "Alerts on account creation/deletion, impossible travel (>500km, >1000kph), MFA disabled logins"
89
performance_impact: "Minimal"
9-
tags: detection, alert, migrated, samples-main
10-
version: latest
11-
author: Joel Mora
10+
tags: ["detection", "alert", "azure-ad", "entra-id", "identity", "powerquery"]
11+
version: "latest"
12+
author: "Joel Mora"

‎detections/community/O365-alerts-latest/metadata.yaml‎

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,12 @@
11
metadata_details:
2-
purpose: "Detection rule migrated from samples-main: O365-alerts.conf"
3-
mitre_tactic_technique: "To be determined"
4-
datasource: "To be determined"
5-
usecase_plus: "Detection logic from samples-main"
2+
purpose: "Detect suspicious Office 365 activities including excessive file access and abnormal login patterns"
3+
mitre_tactic_technique: "T1078 - Valid Accounts, T1083 - File and Directory Discovery"
4+
datasource: "Microsoft O365"
5+
search_type: "powerquery"
6+
usecase_plus: "Monitors for high-volume file access and excessive authentication events that may indicate compromise"
67
severity: "Medium"
7-
expected_alert_scenario: "Based on configured thresholds"
8+
expected_alert_scenario: "Alerts on >300 file accesses per user/day and excessive logon attempts"
89
performance_impact: "Minimal"
9-
tags: detection, alert, migrated, samples-main
10-
version: latest
11-
author: Joel Mora
10+
tags: ["detection", "alert", "o365", "office365", "file-access", "authentication", "powerquery"]
11+
version: "latest"
12+
author: "Joel Mora"

‎detections/community/fortinet_fortigate_firewall-latest/metadata.yaml‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,12 @@ metadata_details:
22
purpose: "Detects false positive & true positive virus attempts in the Fortigate Firewalls"
33
mitre_tactic_technique: "Execution > User Execution > Malicious File"
44
datasource: "marketplace-fortinetfortigate-latest"
5+
search_type: "powerquery"
56
usecase_plus: "unmapped.dstip or unmapped.srcip may align with your endpoint data and can be associated with PowerQueries, the 'union' command, and SentinelOne EDR fields like dst.ip.address and src.ip.address"
67
dependency_summary: "dataSource.vendor = 'FortiGate'"
78
severity: "Medium"
89
expected_alert_scenario: "Alert when a virus is detected in Fortigate Firewall (either allowed or blocked); can be tuned to remove all blocked events using status_detail = 'blocked'"
9-
performance_impact: "Minimal"
10+
performance_impact: "Minimal"
11+
tags: ["detection", "alert", "fortinet", "fortigate", "firewall", "malware", "powerquery"]
12+
version: "latest"
13+
author: "Joel Mora"

‎detections/community/hello-world-elevated-error-rate-latest/metadata.yaml‎

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,12 @@
11
metadata_details:
2-
purpose: "Detection rule migrated from samples-main: hello-world-elevated-error-rate.conf"
3-
mitre_tactic_technique: "To be determined"
4-
datasource: "To be determined"
5-
usecase_plus: "Detection logic from samples-main"
2+
purpose: "Detect elevated HTTP error rates indicating potential application issues or attacks"
3+
mitre_tactic_technique: "T1499 - Endpoint Denial of Service"
4+
datasource: "accesslog"
5+
search_type: "powerquery"
6+
usecase_plus: "Monitors web server error responses (5xx status codes) to identify service degradation"
67
severity: "Medium"
7-
expected_alert_scenario: "Based on configured thresholds"
8+
expected_alert_scenario: "Alerts when error count exceeds 120 errors in 4-minute window per host"
89
performance_impact: "Minimal"
9-
tags: detection, alert, migrated, samples-main
10-
version: latest
11-
author: Joel Mora
10+
tags: ["detection", "alert", "web", "error-rate", "availability", "powerquery"]
11+
version: "latest"
12+
author: "Joel Mora"

‎detections/community/sql-security-latest/metadata.yaml‎

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,12 @@
11
metadata_details:
2-
purpose: "Detection rule migrated from samples-main: sql-security.conf"
3-
mitre_tactic_technique: "To be determined"
4-
datasource: "To be determined"
5-
usecase_plus: "Detection logic from samples-main"
2+
purpose: "Monitor SQL Server security events including login failures, permission denials, and integrity violations"
3+
mitre_tactic_technique: "T1078 - Valid Accounts, T1505 - Server Software Component"
4+
datasource: "SQL Server Event Logs"
5+
search_type: "powerquery"
6+
usecase_plus: "Tracks 40+ SQL Server event IDs covering DBCC operations, login attempts, permission violations, and integrity issues"
67
severity: "Medium"
7-
expected_alert_scenario: "Based on configured thresholds"
8+
expected_alert_scenario: "Alerts on SQL Server security events based on EventID matching (18456 login failures, 229/300 permission denied, etc.)"
89
performance_impact: "Minimal"
9-
tags: detection, alert, migrated, samples-main
10-
version: latest
11-
author: Joel Mora
10+
tags: ["detection", "alert", "sql-server", "database", "security", "event-id", "powerquery"]
11+
version: "latest"
12+
author: "Joel Mora"

‎detections/community/volume_alerts_marketplace-latest/metadata.yaml‎

Lines changed: 10 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,12 @@
11
metadata_details:
2-
purpose: "Detection rule migrated from samples-main: volume_alerts_marketplace.conf"
3-
mitre_tactic_technique: "To be determined"
4-
datasource: "To be determined"
5-
usecase_plus: "Detection logic from samples-main"
6-
severity: "Medium"
7-
expected_alert_scenario: "Based on configured thresholds"
2+
purpose: "Monitor log volume metrics to detect data source outages or significant drops in log ingestion"
3+
mitre_tactic_technique: "T1562.002 - Impair Defenses: Disable Windows Event Logging"
4+
datasource: "Log Volume Metrics (logVolume tag)"
5+
search_type: "powerquery"
6+
usecase_plus: "Tracks log volume from critical sources (Cisco Duo, O365, Palo Alto, Proofpoint, Windows) to ensure continuous visibility"
7+
severity: "High"
8+
expected_alert_scenario: "Alerts when log volume drops to zero for 30 minutes or falls below 25% of previous week's average"
89
performance_impact: "Minimal"
9-
tags: detection, alert, migrated, samples-main
10-
version: latest
11-
author: Joel Mora
10+
tags: ["detection", "alert", "log-volume", "monitoring", "availability", "powerquery"]
11+
version: "latest"
12+
author: "Joel Mora"

‎detections/community/xsoar_trigger-latest/metadata.yaml‎

Lines changed: 10 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,12 @@
11
metadata_details:
2-
purpose: "Detection rule migrated from samples-main: xsoar_trigger.conf"
3-
mitre_tactic_technique: "To be determined"
4-
datasource: "To be determined"
5-
usecase_plus: "Detection logic from samples-main"
6-
severity: "Medium"
7-
expected_alert_scenario: "Based on configured thresholds"
2+
purpose: "Detect lateral movement in Windows environments and trigger XSOAR incident response"
3+
mitre_tactic_technique: "T1021 - Remote Services, T1550 - Use Alternate Authentication Material"
4+
datasource: "Windows Event Logs (EventID 1149)"
5+
search_type: "powerquery"
6+
usecase_plus: "Correlates RDP login events to identify lateral movement patterns and automatically creates XSOAR incidents"
7+
severity: "High"
8+
expected_alert_scenario: "Triggers when source IP from public login matches private IP of subsequent login (lateral movement pattern)"
89
performance_impact: "Minimal"
9-
tags: detection, alert, migrated, samples-main
10-
version: latest
11-
author: Joel Mora
10+
tags: ["detection", "alert", "lateral-movement", "windows", "xsoar", "integration", "powerquery"]
11+
version: "latest"
12+
author: "Joel Mora"

‎detections/community/zscaler_http_access-latest/metadata.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,12 @@ metadata_details:
22
purpose: "Detects false positive & true positive attempts in the browser space protected by Zscaler's Internet Protection"
33
mitre_tactic_technique: "Execution > User Execution > Malicious File"
44
datasource: "marketplace-zscalerinternetaccess-latest"
5+
search_type: "powerquery"
56
usecase_plus: "actor.user.name may align with your endpoint data and can be associated with the use of PowerQueries, the 'union' command, and SentinelOne asset data via the 'user.name' field to further enrich details from your identity provider"
67
dependency_summary: "dataSource.vendor = 'Zscaler Internet Access'"
78
severity: "Medium"
89
expected_alert_scenario: "Alert when a threat is detected in Zscaler Internet Protection (either allowed or blocked); can be tuned to exclude all blocked events using status_detail = 'blocked'"
910
performance_impact: "Minimal"
11+
tags: ["detection", "alert", "zscaler", "web-security", "dlp", "powerquery"]
12+
version: "latest"
13+
author: "Joel Mora"

0 commit comments

Comments
 (0)