Add ScanMalware analyzer. Closes #3974 - #3975
Conversation
Observable analyzer for domain, url and ip, querying the ScanMalware archive of sandboxed URL scans. The API is anonymous, so no key is required and the analyzer is added to the FREE_TO_USE_ANALYZERS playbook. Read-only by design: it never submits an observable for a new scan, because a submitted URL becomes publicly listed in the archive. The data model maps only the scanner's structured security_verdict. An explicit high or critical risk level sets evaluation to malicious, with reliability taken from the confidence the scanner reported; a low or medium level sets nothing, because an absence of findings is not a statement of trust. max_results bounds the scan list and also the Certificate Transparency domain list for an IP, which the API returns unlimited: 8.8.8.8 alone was a 30 KB report before the cap and is 5.9 KB after, with the real total preserved.
|
Docs PR, covering both |
There was a problem hiding this comment.
Pull request overview
This PR introduces a new ScanMalware observable analyzer that performs read-only enrichment for domain, url, and ip observables via the ScanMalware public API, registers it via Django data migrations, and adds unit tests to validate key behaviors (including report size bounding and data model mapping rules).
Changes:
- Added
ScanMalwareobservable analyzer implementation with optional scan-detail fetching and bounded result sizes. - Registered the analyzer (config + default parameters) and added it to
FREE_TO_USE_ANALYZERSvia migrations. - Added unit tests covering endpoint selection, “404 means no data”, optional detail fetching, CT truncation, and data model mapping behavior.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
api_app/analyzers_manager/observable_analyzers/scanmalware.py |
New observable analyzer implementation (host/IP enrichment, optional latest-scan detail fetch, data model mapping). |
api_app/analyzers_manager/migrations/0197_analyzer_config_scan_malware.py |
Data migration registering the analyzer, parameters, and default values. |
api_app/playbooks_manager/migrations/0069_add_scan_malware_to_free_to_use.py |
Adds ScanMalware to the FREE_TO_USE_ANALYZERS playbook config. |
tests/api_app/analyzers_manager/unit_tests/observable_analyzers/test_scanmalware.py |
New unit tests for routing, bounded report output, error handling, and data model mapping behavior. |
Suppressed comments (1)
tests/api_app/analyzers_manager/unit_tests/observable_analyzers/test_scanmalware.py:166
test_scan_details_are_optionalonly asserts thatscan_idis present; it doesn’t assert that the verdict extracted from/result/{scan_id}is included inlatest_scan_details.security_verdict. Adding an assertion here ensures the most important structured field is actually propagated into the report.
with_details = self._analyzer("example.com", Classification.DOMAIN, fetch_scan_details=True).run()
self.assertEqual(
with_details["latest_scan_details"]["scan_id"],
"10b0b675-a02f-4e63-b4f1-6d0d721f0321",
)
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| AI = { | ||
| "scan_id": "10b0b675-a02f-4e63-b4f1-6d0d721f0321", | ||
| "results": {"classification": "benign"}, | ||
| } | ||
|
|
||
|
|
||
| def _route(url, *args, **kwargs): | ||
| if "/domain/stats/" in url: | ||
| return MockUpResponse(STATS, 200) | ||
| if "/scans" in url: | ||
| return MockUpResponse(SCANS, 200) | ||
| if "/ct/ip/" in url: | ||
| return MockUpResponse(CT_IP, 200) | ||
| if "/search/smql" in url: | ||
| return MockUpResponse(SMQL, 200) | ||
| if "/ioc/" in url: | ||
| return MockUpResponse(IOC, 200) | ||
| if "/ai/" in url: | ||
| return MockUpResponse(AI, 200) | ||
| # 404 means "nothing recorded for this observable", which is an answer. | ||
| return MockUpResponse({}, 404) |
| { | ||
| "python_module": { | ||
| "module": "scanmalware.ScanMalware", | ||
| "base_path": "api_app.analyzers_manager.observable_analyzers", | ||
| }, | ||
| "name": "fetch_scan_details", | ||
| "type": "bool", | ||
| "description": "Also fetch indicators and AI classification for the most recent scan. One extra request per run.", | ||
| "is_secret": False, | ||
| "required": False, | ||
| }, |
Both from review feedback on this PR.
The mocked routing table did not answer /result/{scan_id}, so
latest_scan_details.security_verdict was always None in the route-driven tests
and an API shape change there would not have failed anything. The mock now
serves it and the test asserts the verdict survives the trip.
fetch_scan_details was documented as "one extra request per run" when it makes
three (/result, /ioc, /ai). Operators size rate limits from that number.
| confidence = verdict.get("confidence") | ||
| if isinstance(confidence, (int, float)): | ||
| data_model.reliability = max(0, min(10, round(confidence / 10))) | ||
|
|
round() is banker's rounding, so a confidence of 85 mapped to 8 while 86 mapped to 9. Use floor division, as abuseipdb does with abuseConfidenceScore: reliability N now means at least N0% confident, and a confidence is never rounded up. Reliability is averaged across analyzers, so an analyzer that rounds where its neighbours floor biases the aggregate. The previous test asserted 75 -> 8, one of the values where both rules agree, so it could not have caught this.
|
Fixed in 491467d. The rounding was wrong, though not for the stated reason: I used floor division rather than round-half-up, to match The old test could not have caught this: it asserted |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.
Suppressed comments (1)
Previously missed (1) — in code that hasn't changed since the last review.
api_app/analyzers_manager/observable_analyzers/scanmalware.py:198
__scan_detailsdocstring says a miss is reported as null, but when there are no scans (or a scan row lacksscan_id) the function returns{}. That produces an empty object in the report instead ofnull, and also forceslatest_scan_detailsto exist even when there is nothing to report.
if not scans:
return {}
scan_id = scans[0].get("scan_id")
if not scan_id:
|
please show a screenshot of a working analysis with its JSON results then I guess we are done |
…egories
Two mapping bugs found by running the analyzer against the live archive
rather than against fixtures.
RISK_LEVELS_MEANING_MALICIOUS was ("high", "critical"). Sampling the live
API gives low 74, malicious 14, high 9, medium 2, and no critical at all,
so a page the scanner calls outright Malicious at 95% confidence set no
evaluation, while the value being checked for is one the API has not been
observed to emit. The vocabulary had been taken from the SMQL `verdict`
filter enum (LOW_RISK/MODERATE_RISK/HIGH_RISK), which is a different field
from security_verdict.risk_level.
Tags were derived from risk_factors alone. A live verdict of "High Risk
(Credential Phishing on disposable hosting)" carrying
threat_categories: ["Credential Phishing"] was tagged with nothing, because
its only risk factor named the flagged IPs. threat_categories is the
structured field and is now read first, with the verdict string and the
risk factors kept as prose fallbacks.
Three tests added, each failing against the previous code.
|
Screenshot of a run against the live archive, IntelOwl v6.8.0: 
The observable is The DataModel produced from it: 
Building this found two mapping bugs, fixed in
Both have tests that fail against the previous code. 18 tests pass. |
|
waiting current release status to be solved before merging |
Description
Adds the
ScanMalwareobservable analyzer fordomain,urlandip. The API is anonymous, so nokey is required and the analyzer is added to
FREE_TO_USE_ANALYZERS.Read-only by design: it never submits an observable for a new scan, because a submitted URL becomes
publicly listed in the archive.
Data model. Only the scanner's structured
security_verdictis mapped. An explicitmalicious,criticalorhighrisk level setsevaluation = malicious, withreliabilitytaken from the confidence thescanner reported. A
lowormediumlevel sets nothing, because an absence of findings is not astatement of trust. The AI endpoint returns model prose and is surfaced in the report but never
converted into an evaluation.
Live results:
https://github.io/malicious, reliability 7,['phishing']github.comReport size is bounded.
max_resultscaps the scan list, and also the Certificate Transparencydomain list for an IP, which the endpoint returns unlimited:
8.8.8.8alone was a 30 KB reportbefore the cap and is 5.9 KB after, with
totalpreserved so the cut is visible.Raw JSON of a finished analysis
{ "observable": "github.com", "host": "github.com", "link": "https://scanmalware.com/search?q=domain:github.com", "stats": {"total_scans": 178, "completed_scans": 174, "first_scan": "2025-09-24T17:45:44", "latest_scan": "2026-08-26T09:04:26", "scans_7d": 2}, "scans": [{"scan_id": "10b0b675-…", "url": "https://ctif.hagezi.org", "final_url": "https://github.com/hagezi/dns-servers", "matched_on": ["final_url"], "title": "HaGeZi DNS"}], "latest_scan_details": {"security_verdict": {"verdict": "Low Risk", "risk_level": "low", "confidence": 72, "overall_score": 28}, "iocs": {…}, "ai_classification": {…}} }Checklist
developdumppluginand added as a data migrationFREE_TO_USE_ANALYZERS, since it needs no API keyurlattribute created for Health ChecksDataModelcreatedanalyzers_managersuite passesMigrations
0197_analyzer_config_scan_malwareand0069_add_scan_malware_to_free_to_usewereapplied against an empty database, not only an existing one.
Disclosure: I maintain ScanMalware, so this is a vendor-submitted analyzer.