Sigma Regression Testing

A detection engineering pipeline for developing, validating, deploying, and regression testing Sigma rules. This project includes a curated collection of detection rules for Windows, Linux, and Microsoft 365 environments, along with automated tooling to convert rules to Splunk format, deploy them via REST API, and validate detections using Atomic Red Team tests.

Getting Started

git clone https://github.com/scythe-io/sigma-regression-testing.git
cd sigma-regression-testing
pip install -r requirements.txt

New here? See OVERVIEW.md for a plain-English explanation of the project, pipeline flowcharts, and testing status.

Overview

Metric	Count
Total Rules	77
Windows Rules	42
Linux Rules	17
M365/Cloud Rules	7

Rule Categories

Category	Description	Count
`proc_creation`	Process creation events	47
`file_event`	File system activity	6
`m365_*`	Microsoft 365 audit logs	5
`registry_set`	Registry modifications	4
`net_connection`	Network connections	3
`azure_network`	Azure network firewall changes	2
`web_sharepoint`	SharePoint web activity	2
`azure_application`	Azure application security changes	1
`azure_firewall`	Azure firewall modifications	1
`bitsadmin_mal`	Malicious BITSAdmin activity	1
`dns_query`	DNS query anomalies	1
`file_creation`	File creation events	1
`reg_set`	Registry modifications	1
`sysmon_lockbitv3`	LockBit 3.0 ransomware detection (Sysmon)	1
`wmi_event`	WMI event subscription monitoring	1

Verification status

Last full run — evidence/20260731T150808Z:


Pass rate	52/54 conclusive (96.3%) — 92.9% counting the 2 that cannot run here
Change vs previous run	0 regressions
Attribution	56 of 56 `window_and_host_scoped`
Window overlaps	0
Infrastructure errors	0
Skipped	2 — Microsoft Word is not installed
Remaining failures	2, both environment: Defender rejects `Add-MpPreference`, and Sysmon here does not log network connections for regsvr32
Precision	every deployed rule measured against a generated benign-admin workload
Harness tests	582

No rule defect and no bad mapping is currently known and unfixed. Both remaining failures are properties of this host.

Every result is bounded to the execution window of the atomic that caused it, on the host that ran it. Nothing is excluded from the denominator.

This project once reported 50/57 (87.7%) — a higher number that meant less. Seven of those passes were false, credited to activity on other hosts or from neighbouring tests. The 84.7% above is measured under constraints the old number was never held to; see OVERVIEW.md.

Rule Design Philosophy

These rules are designed with the following principles:

Low False Positive Rate — measured, not asserted. precision-test.py runs every deployed rule against a generated benign-administrator workload; a rule that fires there is a finding.
Behavioral Focus — detect techniques, not just IOCs (no hardcoded hashes or IPs)
Verified by execution — a rule counts as working only when an Atomic Red Team test ran on a live host and the deployed rule fired inside that test's execution window
MITRE ATT&CK Mapped — tagged with relevant technique IDs, and the coverage map is built from run results rather than from the tags

Quick Start

Install Dependencies

# Install all dependencies
pip install -r requirements.txt

# Or install manually
pip install sigma-cli pysigma pysigma-backend-splunk pysigma-pipeline-windows PyYAML requests pywinrm

# Optional: Enable tab completion for regression-test.py
pip install argcomplete
. ./scripts/Enable-TabCompletion.ps1  # PowerShell

Validate Rules

# sigma-cli is included in requirements.txt

# Validate all rules
sigma check sigma_rules/*.yml

Convert to SIEM Format

# Splunk -> splunk_output/savedsearches.conf
python scripts/convert.py --backend splunk

# Elastic -> elastic_output/detection_rules.ndjson (Kibana detection engine)
python scripts/convert.py --backend elastic

Rules are routed to a processing pipeline by platform. A rule with no faithful pipeline for its platform is reported as unmapped and not converted — running an Azure rule through the Windows ECS pipeline does not fail, it just passes the field names through untouched and produces a query that looks valid and matches nothing. Both backends currently convert the same 43 Windows rules; CI fails if that diverges.

Deploying:

# Splunk
export SPLUNK_PASSWORD='...'
python scripts/deploy-to-splunk.py --splunk-host splunk.company.com --dry-run
python scripts/deploy-to-splunk.py --splunk-host splunk.company.com

# Kibana
export ELASTIC_API_KEY='...'
python scripts/deploy-to-elastic.py --kibana-host kibana.company.com

ATT&CK coverage map:

python scripts/generate-navigator-layer.py

Built from regression results, not tags: metadata. A technique is green only when a detection fired during a correlated execution — a rule claiming attack.t1059.001 that never fires shows red. Unattributable passes are marked inconclusive rather than counted as covered.

Repository Structure

.
├── sigma_rules/              # 77 rules; 49 convert and deploy to Splunk
│   ├── proc_creation_*.yml    # Process creation rules
│   ├── file_event_*.yml       # File event rules
│   ├── reg_set_*.yml          # Registry rules
│   ├── net_connection_*.yml   # Network rules
│   ├── m365_*.yml             # Microsoft 365 rules (no pipeline yet -- unmapped)
│   ├── unmapped_rules/        # 64 rules with no ART mapping yet
│   └── unreviewed/            # 14 recovered from backup/, not QA-validated
├── scripts/
│   ├── convert.py                # Convert rules to Splunk or Elastic
│   ├── convert-to-splunk.py      # DEPRECATED shim -> convert.py --backend splunk
│   ├── deploy-to-splunk.py       # Deploy saved searches to Splunk (used by CI)
│   ├── deploy-to-splunk.ps1      # Same, interactive Windows/PowerShell
│   ├── deploy-to-elastic.py      # Import detection rules into Kibana
│   ├── regression-test.py        # Atomic Red Team regression testing (CLI)
│   ├── regression-test-gui.py    # GUI wrapper for regression-test.py
│   ├── precision-test.py         # False-positive testing against a benign baseline
│   ├── benign-workload.py        # Generate the benign baseline rather than hunt for one
│   ├── audit-fields.py           # Find rules keyed on fields the data never has
│   ├── build-atomic-catalog.py   # Rebuild the ART catalog from the target
│   ├── validate-mappings.py      # Verify ART mappings resolve and are deployed
│   ├── triage-agent.py           # AI triage of regression failures
│   ├── remediate.py              # Verify a proposed rule fix by executing it
│   ├── promote-rules.py          # Propose ART mappings for unmapped rules
│   ├── track-history.py          # Record runs, report regressions over time
│   ├── generate-navigator-layer.py # ATT&CK Navigator layer from real results
│   ├── update-readme-stats.py    # Auto-update README statistics
│   └── sigma_qa/                 # Harness support library (21 modules)
│       ├── savedsearches.py       # Parse savedsearches.conf into runnable SPL
│       ├── correlation.py         # Execution windows, query scoping, attribution
│       ├── ingestion.py           # Per-channel ingestion watermark
│       ├── prereqs.py             # Can this test run on this host at all
│       ├── evidence.py            # Per-run evidence capture
│       ├── baseline.py            # Benign-baseline analysis
│       ├── workload.py            # Benign admin command catalog
│       ├── backends.py            # DetectionBackend interface (Splunk, Elastic)
│       ├── splunk_deploy.py       # Saved-search deployment over REST
│       ├── elastic_client.py      # Elasticsearch + Kibana clients
│       ├── field_overrides.py     # Field renames the stock pipelines miss
│       ├── fieldaudit.py          # Which fields do deployed rules depend on
│       ├── triage.py              # Failure taxonomy and evidence assembly
│       ├── remediation.py         # Proposal gates: valid/converts/deploys/detects/precision
│       ├── matcher.py             # ART mapping proposals from detection literals
│       ├── ruleeval.py            # Offline rule evaluator
│       ├── history.py             # Run storage, trends, flaky-test detection
│       ├── credentials.py         # OS credential store
│       └── navigator.py           # ATT&CK layer generation
├── tests/                     # 582 tests -- run with pytest, not unittest
│   ├── art_mapping.yaml       # Atomic Red Team test to rule mappings (incl. bursts)
│   ├── test_correlation.py    # Correlation and evidence tests
│   ├── test_end_to_end.py     # Full runner tests with fake Splunk/WinRM
│   ├── test_remediation.py    # Auto-remediation gates (mostly refusals)
│   └── test_precision.py      # Precision/baseline tests
├── examples/
│   └── remediation-proposal.json  # Proposal format, for use without an API key
├── evidence/                  # Per-run evidence artifacts (gitignored)
├── history/                   # Run history database (gitignored locally)
├── splunk_output/             # Generated Splunk artifacts (auto-updated)
│   ├── savedsearches.conf     # Splunk saved searches (auto-generated)
│   └── conversion_report.json # Conversion statistics
├── wip/                       # Work in progress (not production ready)
│   ├── aurora/                # Aurora EDR integration (coming soon)
│   └── scythe/                # SCYTHE integration (coming soon)
└── .github/workflows/
    ├── sigma-validate.yml       # Rule validation; gates everything downstream
    ├── splunk-pipeline.yml      # Convert, deploy, regression test
    ├── harness-tests.yml        # Harness test suite + four correctness guards
    ├── nightly-regression.yml   # Scheduled run, records history, reports deltas
    └── deploy-rules.yml         # Manual deployment to Splunk and/or Kibana

CI/CD Pipeline

Five workflows. The first three run themselves; the last two are deliberately manual.

   commit a rule change
          │
          ▼
 ┌──────────────────────┐   fails ──► blocks the merge
 │ sigma-validate.yml   │   • sigma check, all three rule directories
 │ every push and PR    │   • release artifact + README stats on main
 └──────────┬───────────┘
            │ on success (main only)
            ▼
 ┌──────────────────────┐
 │ splunk-pipeline.yml  │   • convert for BOTH backends, fail if they diverge
 │ auto after validate  │   • commit savedsearches.conf back to the repo
 └──────────┬───────────┘   • deploy + regression test are OPT-IN
            │
            │  ┌────────────────────────────────────────────────┐
            └─►│ harness-tests.yml    (any scripts/ or tests/)   │
               │  582 tests + five guards that must never break: │
               │   · detection checks stay time-scoped           │
               │   · only FIELD_MISMATCH may edit a rule         │
               │   · no secret is ever passed in argv            │
               │   · every mapped atomic actually exists         │
               │   · no rule references a renamed field          │
               └────────────────────────────────────────────────┘

 ── scheduled ──────────────────────────────────────────────────────
 ┌──────────────────────┐
 │ nightly-regression   │   convert → deploy → audit fields → run the
 │ 07:00 UTC, lab runner│   suite → record → render the DELTA
 └──────────┬───────────┘
            │  a test that used to pass and now fails turns the job red
            ▼
      "T1055.001 regressed after commit abc123"

 ── manual ─────────────────────────────────────────────────────────
 ┌──────────────────────┐   ┌────────────────────────────────────┐
 │ deploy-rules.yml     │   │ scripts/remediate.py               │
 │ Splunk / Kibana/both │   │ a proposed fix must clear 5 gates  │
 │ dry_run on by default│   │ before a PR is opened, and any     │
 └──────────────────────┘   │ gate it cannot MEASURE counts as   │
                            │ failed                             │
                            └────────────────────────────────────┘

The loop that matters: a rule change is validated, converted, deployed, and then executed against — every night — with the result compared to yesterday's. A fix proposed for a failure is itself executed and measured before anyone is asked to review it.

Validation Workflow (`sigma-validate.yml`)

Automatically validates rules on every code change. When validation passes on main, it automatically triggers the Splunk Detection Pipeline.

Triggers:

Push to main branch (when sigma_rules/** changes)
Pull requests to main branch
Manual trigger

Pipeline:

┌─────────────────────────────────────────────────────────────┐
│  VALIDATE JOB (runs on every PR and push)                   │
├─────────────────────────────────────────────────────────────┤
│  1. Checkout code                                           │
│  2. Install Python 3.11 + sigma-cli                         │
│  3. Run: sigma check sigma_rules/*.yml                      │
│  4. If errors found → FAIL (blocks PR merge)                │
│  5. If clean → PASS and display rule count                  │
└─────────────────────────────────────────────────────────────┘
         │                        │
         ▼ (only on merge)        ▼ (only on merge)
┌──────────────────┐   ┌──────────────────────────────────────┐
│  RELEASE JOB     │   │  UPDATE README JOB                   │
├──────────────────┤   ├──────────────────────────────────────┤
│  Package .yml    │   │  Run update-readme-stats.py          │
│  rules as        │   │  Auto-commit updated rule counts     │
│  artifact        │   │  to README.md                        │
└──────────────────┘   └──────────────────────────────────────┘
         │
         │  on completion with conclusion == 'success'
         ▼
┌─────────────────────────────────────────────────────────────┐
│  SPLUNK DETECTION PIPELINE (auto-triggered)                 │
│  see below ↓                                                │
└─────────────────────────────────────────────────────────────┘

Deployment Workflow (`deploy-rules.yml`)

Manual (workflow_dispatch) deployment of converted detections to Splunk, Kibana, or both.

Input	Default	Effect
`backend`	`splunk`	`splunk`, `elastic`, or `both`
`environment`	`staging`	Selects the GitHub environment, so a required reviewer can gate production
`dry_run`	`true`	Reports what would be deployed without writing anything
`enable_alerts`	`false`	Schedules the deployed searches as alerts (Splunk only)

Jobs: validate (sigma check + ART mapping validation) → build (convert, fail on any conversion error) → deploy (connectivity check, then import).

Rules are converted inside the workflow rather than read from splunk_output/ in the repo, so what gets deployed is built from the rule source at that commit — a stale committed savedsearches.conf would otherwise deploy yesterday's detections under today's SHA.

Required secrets: SPLUNK_HOST, SPLUNK_USER, SPLUNK_PASSWORD (and SPLUNK_PORT, SPLUNK_APP if not 8089/search); KIBANA_HOST and ELASTIC_API_KEY for the Elastic path.

Deploying is not verifying. A successful deployment proves the SIEM accepted the query, not that it detects anything. That requires emulation — the regression-test job in splunk-pipeline.yml.

Splunk Detection Pipeline

A complete detection engineering pipeline for deploying Sigma rules to Splunk and validating them with Atomic Red Team tests.

Trigger: Automatically runs after Sigma Rules Validation succeeds on main. Can also be triggered manually with optional deploy/regression flags.

Pipeline Overview

         ┌──────────────────────────────────┐
         │   Sigma Rules Validation (pass)  │
         └────────────────┬─────────────────┘
                          │ workflow_run trigger
                          ▼
┌─────────────────────────────────────────────────────────────┐
│  CONVERT JOB                                                │
├─────────────────────────────────────────────────────────────┤
│  1. Convert Windows rules to Splunk savedsearches.conf      │
│  2. Generate conversion report                              │
│  3. Commit savedsearches.conf back to repository            │
│  4. Upload artifacts                                        │
└─────────────────────────────────────────────────────────────┘
                          │
                          ▼ (on workflow_run or manual deploy)
┌─────────────────────────────────────────────────────────────┐
│  DEPLOY JOB (requires secrets configured)                   │
├─────────────────────────────────────────────────────────────┤
│  1. Download converted artifacts                            │
│  2. Deploy saved searches to Splunk via REST API            │
│  3. Configure alerts (optional)                             │
└─────────────────────────────────────────────────────────────┘
                          │
                          ▼ (manual trigger only)
┌─────────────────────────────────────────────────────────────┐
│  REGRESSION TEST JOB                                        │
├─────────────────────────────────────────────────────────────┤
│  1. Execute Atomic Red Team tests on target endpoint        │
│  2. Query Splunk for triggered rules                        │
│  3. Report coverage and failures                            │
└─────────────────────────────────────────────────────────────┘

Note: The savedsearches.conf file is automatically regenerated and committed to the repository whenever Sigma rules change and pass validation. You can always find the latest converted rules in splunk_output/savedsearches.conf.

Local Usage

Convert rules to Splunk format:

# Install dependencies
pip install sigma-cli pysigma pysigma-backend-splunk pysigma-pipeline-windows PyYAML

# List compatible rules
python scripts/convert-to-splunk.py --list-compatible

# Convert all rules
python scripts/convert-to-splunk.py -i sigma_rules -o splunk_output

Deploy to Splunk:

# Dry run — lists what would be deployed, calls nothing
python scripts/deploy-to-splunk.py --splunk-host splunk.company.com --dry-run

# Check credentials and app before writing anything
export SPLUNK_PASSWORD='...'
python scripts/deploy-to-splunk.py --splunk-host splunk.company.com --test-connection

# Deploy (creates new searches, updates existing ones)
python scripts/deploy-to-splunk.py --splunk-host splunk.company.com

# Deploy one search
python scripts/deploy-to-splunk.py --splunk-host splunk.company.com --only "Conti-style Discovery Burst"

The password comes from SPLUNK_PASSWORD or the OS credential store, never from a flag — argv is readable by every other process on the host, and this pipeline targets machines running Sysmon, so a --splunk-pass would be indexed by the very Splunk it authenticates to.

Alerts are not scheduled unless you pass --enable-alerts. Turning unproven detections into production alerts should be a deliberate act, not the consequence of a forgotten flag.

scripts/deploy-to-splunk.ps1 still exists for interactive Windows use. The Python version is what CI runs, since the runners are Linux.

Run regression tests with Atomic Red Team:

# Dry run (show test cases without executing)
python scripts/regression-test.py --splunk-host splunk.company.com --dry-run --test-config tests/art_mapping.yaml

# Credentials come from the environment, not the command line, so they stay
# out of shell history and process listings.
export SPLUNK_PASSWORD='...'
export WINRM_PASSWORD='...'

# Sequential mode - RECOMMENDED for results you intend to rely on.
# Each atomic gets its own detection window, so nothing overlaps.
python scripts/regression-test.py \
  --splunk-host splunk.company.com \
  --splunk-user admin \
  --target 192.168.1.100 \
  --winrm-user "DOMAIN\Administrator" \
  --test-config tests/art_mapping.yaml \
  --splunk-index main \
  --wait-time 30 \
  --skip-atomic-check

# Batch mode (faster - runs all atomics first, then checks rules).
# Windows are still per-test, but atomics run back to back, so some may
# overlap; any that do are reported as 'window_overlap' rather than passed.
python scripts/regression-test.py \
  --splunk-host splunk.company.com \
  --splunk-user admin \
  --target 192.168.1.100 \
  --winrm-user "DOMAIN\Administrator" \
  --test-config tests/art_mapping.yaml \
  --splunk-index main \
  --wait-time 120 \
  --skip-atomic-check \
  --batch

# Parallel mode (fastest - 5 concurrent atomics, implies --batch).
# Use for smoke tests, NOT for verification: concurrent atomics produce
# overlapping windows, so detections cannot be attributed to one test.
python scripts/regression-test.py \
  --splunk-host splunk.company.com \
  --splunk-user admin \
  --target 192.168.1.100 \
  --winrm-user "DOMAIN\Administrator" \
  --test-config tests/art_mapping.yaml \
  --splunk-index main \
  --wait-time 120 \
  --skip-atomic-check \
  --parallel

Precision testing (false positives):

Regression testing proves a rule fires when the attack runs. It says nothing about whether the rule stays quiet the rest of the time — a rule matching Image|endswith: \net.exe passes every regression test and is unusable in production.

Precision testing needs a baseline window containing activity a rule could plausibly over-match. An idle lab host does not provide one: 12 hours of it yielded 28 distinct process images, almost all agent noise, and every rule was silent — which proved nothing. Generate the activity instead:

# 1. Run ordinary read-only admin commands on the target and record the window
export WINRM_PASSWORD='...'
python scripts/benign-workload.py \
  --target 192.168.1.100 --winrm-user "DOMAIN\Administrator"

# 2. Evaluate every deployed rule against that window
export SPLUNK_PASSWORD='...'
python scripts/precision-test.py \
  --benign-window benign_workload.json \
  --splunk-host splunk.company.com --splunk-user admin \
  --splunk-index main --fail-on-noisy

The workload runs 34 commands across six categories — discovery, WMIC, registry/service queries, PowerShell, file operations, network. They were chosen to overlap the vocabulary of the rule corpus: net localgroup, wmic qfe get, reg query, schtasks /query, netsh advfirewall show. A workload of dir and echo would run cleanly and tell you nothing. Every command is read-only or confined to a scratch directory, and each carries a documented rationale (--list to review them).

Without a generated window, --window-hours evaluates ambient activity instead. Either way the tool reports how much real activity the window contained and refuses to present a thin baseline as evidence of precision — a silent result on an idle host mostly proves the host was idle.

GUI Mode:

A graphical interface is also available for configuring and running tests without building CLI commands by hand:

python scripts/regression-test-gui.py

The GUI provides four tabs (Splunk, Target, Test Settings, Filters), a live color-coded output panel, a Test Connection button to verify Splunk credentials, and a prompt to open the HTML results when the run completes.

Credentials. Save Settings writes non-secret settings to .gui_config.json and stores passwords in the OS credential store (Credential Manager on Windows, Keychain on macOS, Secret Service on Linux) via keyring. Passwords are never written to the config file, and never passed to regression-test.py on the command line — they travel through the subprocess environment. If no credential store is available, saving fails loudly and points you at SPLUNK_PASSWORD / WINRM_PASSWORD rather than silently falling back to plaintext. Any cleartext passwords left in an older .gui_config.json are migrated into the credential store on first launch and removed from the file.

How Verification Works

A rule is only considered working when an Atomic Red Team test actually executed on a live target, generated real telemetry, and the deployed rule fired on that specific activity. Three constraints make that claim meaningful:

Constraint	What it prevents
Time window — queries are bounded to the epoch range in which the atomic actually ran	A historical occurrence of the behavior satisfying the check
Host filter — queries are scoped to the target's Splunk `host` value	The same behavior on a different machine counting as a pass
Direct SPL execution — the rule's underlying query is run with explicit bounds	Inheriting `dispatch.earliest_time = -30d` from `savedsearches.conf`

Every result carries an attribution label recording how strongly it can be trusted:

Attribution	Meaning
`window_and_host_scoped`	Strongest. Detection is attributable to this atomic on this host.
`window_scoped`	Time-bounded, but the target host could not be resolved.
`window_overlap`	Another test's window overlapped this one — not uniquely attributable.
`window_scoped_clock_skew`	Target clock differs from the harness by more than 30s.
`unscoped`	Legacy mode. Not attributable; do not treat as verification.

The runner resolves the target's hostname and measures its clock offset at startup, since detection windows are compared against timestamps the target stamped on its own events.

Evidence

Every run writes evidence/<run_id>/:

evidence/20260730T003910Z/
├── run.json                        # config, target, clock skew, summary
└── 001_<test-slug>/
    ├── meta.json                   # test definition, window, verdict, bounds
    ├── atomic_output.txt           # raw Invoke-AtomicTest stdout/stderr
    ├── matched_events.json         # the events that satisfied each rule
    └── raw_telemetry.json          # on failure: all host telemetry in the window

raw_telemetry.json is what separates "the rule is wrong" from "the event was never logged" — surfaced in reports as telemetry_seen, and counted separately in the summary as failed_no_telemetry so logging gaps are not misread as detection defects.

A pass with the matching event attached is evidence. A pass with only a rule name is a claim.

Test Output:

The regression test script generates two output files:

test_results.json - Machine-readable results with pass/fail status, queries, and timing
test_results.html - Interactive HTML report with filtering and visual summary

The HTML report includes:

Summary cards showing total tests, passed, failed, pass rate, and untested rules count
Visual progress bar for quick pass/fail overview
Filterable table by status (All/Passed/Failed) and search text
Details for each test including expected rules, triggered rules, and missing rules
Clickable Splunk links for each expected rule that open the saved search directly in Splunk (with last 15 minutes time range)
Untested Rules Section showing which rules were not tested and why:
- No test mapping (rule exists but no Atomic test mapped)
- Non-Windows/skipped (Linux, M365, or other non-Windows rules)
- Conversion failed (rules that failed Splunk conversion)
- Test error (tests that encountered errors during execution)

Auditing for rules that can never fire

sigma check proves a rule is well-formed. Conversion proves it compiles. Deployment proves the SIEM accepted it. None of them prove the query names a field the data actually has — and a detection keyed on a field that is never populated does not fail. It returns nothing, quietly, forever.

export SPLUNK_PASSWORD='...'
python scripts/audit-fields.py --splunk-host splunk.company.com \
    --splunk-index main --host-value WORKSTATION01

This found four inert rules in a corpus that was otherwise green:

Field	Populated	The schema calls it	Rules affected
`Computer`	0 of 68	`ComputerName`	Conti-style Discovery Burst
`ScriptBlockText`	0 of 35,560	`ScriptBlock_Text`	3 PowerShell rules

A related defect, found the same way but worse in kind: 46 of 47 deployed rules had lost their logsource.category at conversion. category: dns_query plus an Image match means "that binary made a DNS query"; without the category it means "that binary ran". "DNS Query Request By Regsvr32.EXE" deployed as bare Image="*\\regsvr32.exe" and passed its regression test by matching a process-creation event, having never seen a DNS query.

The cause was a missing pipeline — pySigma's sysmon maps logsource categories to Sysmon event IDs, and it was not installed. A rule matching the wrong event type is worse than one matching nothing: silence gets investigated, a green test does not.

Both are Sigma's canonical names — Computer is the element name in raw EVTX XML, and EventCode 4104 labels its payload ScriptBlock Text:. The Splunk Add-on for Windows renames them on ingest, and the stock splunk_windows pipeline maps EventID and nothing else.

Computer is the worse of the two: in a correlation rule, stats ... by Computer drops every event whose by-field is null, so the rule cannot fire at all rather than merely matching less.

The fix belongs in the conversion pipeline, not the rule — see scripts/sigma_qa/field_overrides.py. That distinction matters: three failing tests with telemetry visibly present on the host look exactly like three rules with bad detection logic, and "loosen the rule until the test passes" would have quietly degraded three correct detections. CI enforces the offline half of this check on every change.

Verifying a correlation rule

A correlation rule fires on an aggregate — N events inside a window, usually grouped by host or parent process. One atomic produces one event, so an ordinary mapping can never trigger one.

A burst entry runs several atomics inside a single remote shell:

  - name: "Conti-style Discovery Burst (correlation)"
    burst:
      - technique_id: "T1016"
        atomic_test_guid: "dafaf052-5508-402d-bf77-51e0700c02e2"
      - technique_id: "T1082"
        atomic_test_guid: "66703791-c902-4560-8770-42b8a91f7667"
      - technique_id: "T1033"
        atomic_test_guid: "4c4959bf-addf-4b4a-be86-8d09cc1857aa"
    expected_rules:
      - "Conti-style Discovery Burst"

One shell is the entire point. Invoke-AtomicTest spawns each payload from the PowerShell process running it, so issuing every step in one remote call gives the resulting processes the same parent and puts them inside one execution window — the shape a group-by + timespan rule matches on. The same atomics written as three separate mappings would each get their own parent and land minutes apart, which is exactly what the rule is designed not to match.

Notes:

technique_id and atomic_test_guid are derived (T1016,T1082,T1033 and the GUIDs joined with +), so a report still identifies precisely what ran.
Each step's GUID is checked individually for duplicates. A GUID reused between a burst and a standalone entry would leave two tests racing for credit on one detection.
Cleanup unwinds in reverse and is timed separately, so the detection window still excludes it.
A burst of one is rejected — that is an ordinary mapping written the hard way.

Caveat worth knowing. pySigma renders timespan: 10m as | bin _time span=10m, and bins align to wall-clock boundaries rather than sliding. A burst straddling a boundary splits across two bins, and neither may reach the threshold. This is a property of the rule, not the harness: a real attacker's sweep has the same blind spot.

Additional Options:

Option	Default	Description
`--wait-time`	40	Ceiling on the per-test ingestion wait, not a fixed sleep
`--min-wait`	2	Floor on that wait. Auto-raised to `pre_buffer + post_buffer` when correlation is on
`--settle-time`	3	Extra wait after ingestion catches up, covering out-of-order arrival
`--no-adaptive-wait`	false	Always sleep the full `--wait-time`. Reproduces the pre-2026-07-30 behaviour
`--lookback-window`	(none)	Deprecated. Applies one window to every test, which disables per-test attribution. Treated as `--legacy-correlation`
`--batch`	false	Run all atomics first, then check rules (faster)
`--parallel`	false	Run 5 atomic tests concurrently via WinRM (implies --batch). Produces overlapping windows — smoke tests only

On waiting. The harness watches Splunk's ingestion watermark rather than guessing a duration. Once an event whose timestamp is after the execution window has been indexed, the pipeline has demonstrably moved past that window, so waiting longer buys nothing.

This matters because ingestion lag is not one number. Measured over 24h on the lab host:

EventCode	p50	p90	p99	max
1 — process creation	1s	2s	6s	18s
3 — network connection	1s	1s	4s	20s
4103 — PowerShell pipeline	1s	27s	55s	69s
4104 — script block	0s	8s	37s	69s

A fixed sleep has to be sized for the slowest event type a test might need, so it is wrong for everything else. 40s was ~6x too long for process-creation tests and too short for the PowerShell script-block rules.

Typical result: 16s per test instead of 40s, with the ceiling unchanged so a stalled forwarder degrades to exactly the old behaviour.

Correlation options:

Option	Default	Description
`--host-value`	(auto)	Splunk `host` value for the target. Auto-detected via `$env:COMPUTERNAME`
`--host-field`	host	Field holding the originating host
`--splunk-index`	(all)	Restrict detection queries to one index
`--pre-buffer`	5	Seconds before atomic start to include in the detection window
`--post-buffer`	10	Seconds after atomic end to include in the detection window
`--include-cleanup`	false	Count detections fired by cleanup activity. Off by default — cleanup is not the attack
`--legacy-correlation`	false	Disable time/host scoping. Results are not attributable; comparison only

Buffers are sized from evidence, not intuition. Across 98 matched events in a full run, no event landed before its window opened and the latest landed 3.8s after it closed — so 5/10 keeps roughly 2.5x margin on the only side that ever saw use. (They were 60/120 until a run showed 56 of 57 windows overlapping, then 10/20.)

Buffers must stay well under the ingestion wait: consecutive detection windows overlap once the gap between tests is smaller than pre + post, and a detection inside two windows cannot be credited to either. Rather than warn about a bad combination, the harness raises the wait floor to the buffer span, which makes the overlap arithmetically impossible. Bounds apply to event time, so buffers never need to absorb ingestion lag — that is what the wait is for.

Evidence options:

Option	Default	Description
`--evidence-dir`	evidence	Directory for per-run evidence artifacts
`--no-evidence`	false	Skip evidence capture
`--splunk-web-port`	8000	Splunk web UI port (for HTML report links)
`--splunk-app`	search	Splunk app context for saved searches
`--test-id`	(all)	Filter by atomic test GUID (can specify multiple)
`--technique`	(all)	Filter by MITRE ATT&CK technique ID, e.g., T1018 (can specify multiple)
`--expected-rule`	(all)	Filter by expected rule name - partial match (can specify multiple)
`--list`	false	List tests instead of running them (works with filters)
`--fields`	name,technique,guid,rules	Fields to show with --list (can specify multiple)
`--format`	table	Output format for --list: table or csv
`--prompt-inputs`	false	Interactively prompt for input arguments
`--inputs-file`	(none)	Load input arguments from YAML file
`--use-defaults`	false	Ignore custom inputs, use ART default values
`--conversion-report`	splunk_output/conversion_report.json	Path to Sigma conversion report (for untested rules tracking)
`--savedsearches`	splunk_output/savedsearches.conf	Path to Splunk savedsearches.conf (for untested rules tracking)
`--skip-untested-report`	false	Skip generating the untested rules section

List available tests:

# List all tests in the config
python scripts/regression-test.py --list --test-config tests/art_mapping.yaml

# List tests for a specific technique
python scripts/regression-test.py --list --technique T1018 --test-config tests/art_mapping.yaml

# List tests with specific fields
python scripts/regression-test.py --list --fields name --fields technique --fields description --test-config tests/art_mapping.yaml

# Export as CSV
python scripts/regression-test.py --list --format csv --test-config tests/art_mapping.yaml > tests.csv

Available fields: name, technique, guid, rules, description, cleanup, inputs

Run specific tests:

# Run a single test by GUID
python scripts/regression-test.py \
  --splunk-host splunk.company.com \
  --test-config tests/art_mapping.yaml \
  --test-id f1bf6c8f-9016-4edf-aff9-80b65f5d711f \
  --dry-run

# Run tests for a specific rule (partial match)
python scripts/regression-test.py \
  --splunk-host splunk.company.com \
  --test-config tests/art_mapping.yaml \
  --expected-rule "Domain Discovery" \
  --dry-run

# Run tests for a specific MITRE technique
python scripts/regression-test.py \
  --splunk-host splunk.company.com \
  --test-config tests/art_mapping.yaml \
  --technique T1018 \
  --dry-run

# Run multiple specific tests
python scripts/regression-test.py \
  --splunk-host splunk.company.com \
  --test-config tests/art_mapping.yaml \
  --test-id f1bf6c8f-9016-4edf-aff9-80b65f5d711f \
  --test-id 80887bec-5a9b-4efc-a81d-f83eb2eb32ab \
  --skip-atomic-check

Custom input arguments:

# Prompt for inputs interactively
python scripts/regression-test.py \
  --splunk-host splunk.company.com \
  --test-config tests/art_mapping.yaml \
  --test-id bc8be0ac-475c-4fbf-9b1d-9fffd77afbde \
  --prompt-inputs \
  --skip-atomic-check

# Load inputs from file
python scripts/regression-test.py \
  --splunk-host splunk.company.com \
  --test-config tests/art_mapping.yaml \
  --inputs-file tests/inputs.yaml \
  --skip-atomic-check

# Use ART defaults (ignore custom inputs in test config)
python scripts/regression-test.py \
  --splunk-host splunk.company.com \
  --test-config tests/art_mapping.yaml \
  --use-defaults \
  --skip-atomic-check

Inputs file format (tests/inputs.yaml):

# By atomic GUID
bc8be0ac-475c-4fbf-9b1d-9fffd77afbde:
  username: "CustomUser"

# Or by test name
"Create Local User Account (PowerShell)":
  username: "AnotherUser"

Tests that cannot run

Before executing anything, the harness asks Atomic Red Team whether the test can run here. The answer splits three ways, and keeping them apart is what makes the pass rate mean something:

Answer	Treatment
Prerequisites met	Run it
Prerequisites not met	Not a detection result. Left out of the denominator, reported with the actual requirement — "Microsoft Word must be installed", not a bare failure
Atomic not found	A defect in this repository's mappings. Counted as a failure, never excused

The asymmetry is deliberate, and the second case is why this exists. Three tests were mapped to GUIDs absent from the installed ART. Nothing ran — and all three passed, because the commands their rules match run on the host regardless:

hostname.exe   2,828 executions / 24h   (118/hour)
whoami.exe       435 executions / 24h   (431 launched by ART's own logger)

A 20-second detection window has a ~65% chance of containing a hostname by coincidence. Window and host scoping — the mechanism this whole harness rests on — cannot help when the noise is inside the window. A test that executes nothing can only pass by accident, so a mapping that names a nonexistent atomic has to be loud.

The headline pass rate is over conclusive tests: those that actually ran. total_tests, skipped_prereq and pass_rate_all_tests all stay in the report. The rule is that nothing is excluded silently, not that nothing is excluded.

Parsing fails open — unrecognised ART output runs the test anyway. Failing closed would skip the whole suite on a wording change and produce an empty run that looks clean.

Costs one WinRM round trip per test (2–5s). --no-prereq-check opts out.

Tracking regressions over time

A single run tells you today's number. The value is the delta.

python scripts/track-history.py --record test_results.json --fail-on-regression
python scripts/track-history.py --trend
python scripts/track-history.py --flaky

.github/workflows/nightly-regression.yml does this on a schedule and renders the result into the GitHub job summary:

### Regressions (1)
| Test                        | Technique   | Now missing              |
|-----------------------------|-------------|--------------------------|
| Process Injection mavinject | T1055.001   | Process Injection Via ... |

A test that used to pass and now fails turns the job red. A run covering a different set of tests is flagged rather than silently compared — a full run diffed against a filtered one would otherwise report "no changes" while most of the corpus was never looked at.

This needs a self-hosted runner inside the lab. A GitHub-hosted machine is not a host your SIEM collects from, so every test would report NO_TELEMETRY and the trend line would be flat and meaningless.

Verifying a proposed fix before anyone reviews it

When a rule fails, the obvious move is to widen it until the test passes. That always works, and it is strictly worse than the original failure: it converts one visible red test into an invisible flood of false positives.

scripts/remediate.py takes a proposed change and tries hard to reject it:

# from AI triage (needs ANTHROPIC_API_KEY)
python scripts/remediate.py --from-triage triage_report.json --splunk-host ... --target ...

# from a hand-written proposal -- no API key needed
# see examples/remediation-proposal.json for the format
python scripts/remediate.py --proposal proposal.json --dry-run

Gate	Checks
`valid`	the edited rule still parses as Sigma
`converts`	it produces a backend query
`deploys`	the SIEM accepts it, under a staging name
`detects`	the mapped atomic makes it fire — recall
`precision`	it stays quiet on the benign baseline

Every gate must pass, and a gate that cannot be measured counts as failed. Without a benign window the precision gate refuses outright rather than reading silence as clean.

Real output from the lab — a proposal that did fix the failing test:

REJECTED: Network Share Enumeration via Net Commands
  [PASS] valid: edited rule parses as Sigma
  [PASS] converts: produced SPL for STAGING - Network Share Enumeration via Net Commands
  [PASS] deploys: Splunk accepted STAGING - Network Share Enumeration via Net Commands
  [PASS] detects: staged rule fired on 1/1 mapped atomic(s)
  [FAIL] precision: benign hits rose 0 -> 2; the fix trades precision for recall

The original and staged rules are measured in a single precision run, so the two counts describe the same window on the same host. Staged searches are deleted afterwards whether the fix is accepted or not. Nothing is written to your working tree; --open-pr creates a branch and is off by default. Only FIELD_MISMATCH failures are ever acted on — every other triage category means the rule was right and something else was wrong.

Installing Atomic Red Team

On the test endpoint, install Atomic Red Team to a system-wide location (required for WinRM access):

# Install Invoke-AtomicRedTeam module to C:\AtomicRedTeam
Set-ExecutionPolicy Bypass -Scope Process -Force
IEX (IWR 'https://raw.githubusercontent.com/redcanaryco/invoke-atomicredteam/master/install-atomicredteam.ps1' -UseBasicParsing)
Install-AtomicRedTeam -getAtomics -Force -InstallPath "C:\AtomicRedTeam"

# Verify installation
Import-Module "C:\AtomicRedTeam\invoke-atomicredteam\Invoke-AtomicRedTeam.psd1" -Force
Get-Command Invoke-AtomicTest

GitHub Actions Setup

Conversion is fully automatic — push changes to sigma_rules/, validation runs, and if it passes the Splunk pipeline triggers automatically to regenerate and commit savedsearches.conf. No manual steps required.

To also enable automated deployment to a live Splunk instance, configure these secrets in your repository:

Secret	Description
`SPLUNK_HOST`	Splunk server hostname
`SPLUNK_PORT`	Management port (default: 8089)
`SPLUNK_USER`	Splunk admin username
`SPLUNK_PASSWORD`	Splunk admin password
`SPLUNK_APP`	Target app (default: search)

Test Mapping Configuration

Define test cases in tests/art_mapping.yaml to map Atomic Red Team tests to expected Sigma rules:

tests:
  - name: "Clear Security Event Log"
    description: "Validates detection of log clearing via wevtutil"
    technique_id: "T1070.001"
    atomic_test_guid: "e6abb60e-26b8-41da-8aae-0c35174b0967"
    expected_rules:
      - "Windows Event Log Manipulation"  # Must match Splunk saved search name exactly
    cleanup: false

  - name: "Create Local User Account"
    description: "Creates a new local user"
    technique_id: "T1136.001"
    atomic_test_guid: "a524ce99-86de-4f6c-88f5-8c3439e21ed5"
    expected_rules:
      - "Local User Account Creation via New-LocalUser PowerShell Cmdlet"
    input_arguments:
      username: "TestUser"
      password: "P@ssw0rd123!"

Important: The expected_rules values must match the Splunk saved search names exactly. These are the Sigma rule titles from the YAML files.

Find Atomic Test GUIDs at atomicredteam.io or by running:

Import-Module "C:\AtomicRedTeam\invoke-atomicredteam\Invoke-AtomicRedTeam.psd1"
Invoke-AtomicTest T1070.001 -ShowDetailsBrief

Aurora EDR Integration

Status: Work in Progress

Aurora integration is under development. See wip/aurora/ for preliminary scripts.

These rules are compatible with Aurora Agent from Nextron Systems.

Manual Installation

Copy rules to Aurora's custom signatures folder:

Copy-Item .\sigma_rules\proc_creation_win_*.yml "C:\Program Files\Aurora-Agent\custom-signatures\"
Restart-Service "Aurora Agent"

Automated sync and deployment tooling coming soon.

Contributing

Adding New Rules

Create a new .yml file in sigma_rules/
Follow the naming convention: <logsource>_<description>.yml
Ensure the rule passes validation: sigma check your-rule.yml
Submit a pull request

Rule Requirements

Must pass sigma check validation
Must include MITRE ATT&CK technique tags (e.g., attack.t1059.001)
Must include falsepositives section
Must include appropriate level (low/medium/high/critical)
Should include filters to reduce false positives

Naming Convention

<logsource>_<platform>_<description>.yml

Examples:
  proc_creation_win_susp_rundll32.yml
  proc_creation_lnx_docker_priv.yml
  file_event_win_archive_creation.yml
  m365_mailbox_delegation.yml

References

License

See LICENSE for details.

Navigation Menu

Uh oh!

Sigma Regression Testing

Getting Started

Overview

Rule Categories

Verification status

Rule Design Philosophy

Quick Start

Install Dependencies

Validate Rules

Convert to SIEM Format

Repository Structure

CI/CD Pipeline

Validation Workflow (`sigma-validate.yml`)

Deployment Workflow (`deploy-rules.yml`)

Splunk Detection Pipeline

Pipeline Overview

Local Usage

How Verification Works

Evidence

Auditing for rules that can never fire

Verifying a correlation rule

Tests that cannot run

Tracking regressions over time

Verifying a proposed fix before anyone reviews it

Installing Atomic Red Team

GitHub Actions Setup

Test Mapping Configuration

Aurora EDR Integration

Manual Installation

Contributing

Adding New Rules

Rule Requirements

Naming Convention

References

License

Uh oh!

FilesExpand file tree

README.md

Latest commit

History

README.md

File metadata and controls

Sigma Regression Testing

Getting Started

Overview

Rule Categories

Verification status

Rule Design Philosophy

Quick Start

Install Dependencies

Validate Rules

Convert to SIEM Format

Repository Structure

CI/CD Pipeline

Validation Workflow (sigma-validate.yml)

Deployment Workflow (deploy-rules.yml)

Splunk Detection Pipeline

Pipeline Overview

Local Usage

How Verification Works

Evidence

Auditing for rules that can never fire

Verifying a correlation rule

Tests that cannot run

Tracking regressions over time

Verifying a proposed fix before anyone reviews it

Installing Atomic Red Team

GitHub Actions Setup

Test Mapping Configuration

Aurora EDR Integration

Manual Installation

Contributing

Adding New Rules

Rule Requirements

Naming Convention

References

License

Validation Workflow (`sigma-validate.yml`)

Deployment Workflow (`deploy-rules.yml`)