EU AI Act Compliance Scan Results — Sharing Findings for Feedback

[shotwellj]

Hi Quivr team,

We're building AIR Blackbox, an open-source EU AI Act compliance scanner for Python AI frameworks. We've been benchmarking real-world frameworks (CrewAI, LangFlow, Quivr) to validate our scanner's accuracy and build comparison data ahead of the August 2, 2026 enforcement deadline.

We ran a full scan against the quivr core library and wanted to share the results. We're genuinely interested in your feedback — both to improve our scanner and to flag anything useful for your roadmap.

Scan Summary

Article	Status	Key Finding
Art. 9 — Risk Management	⚠️ WARN	Fallback patterns found in 3 files. Error handling in 2/10 LLM call files. No risk assessment doc.
Art. 10 — Data Governance	⚠️ WARN	Pydantic validation in 15/77 files (solid). No PII detection/redaction library.
Art. 11 — Technical Documentation	✅ PASS	79% type hint coverage. README present. 21% docstring coverage on public functions.
Art. 12 — Record-Keeping	⚠️ WARN	Langfuse integration in 6 files with production-grade tracing, but only 1 action audit file — thin for full audit trail requirements.
Art. 14 — Human Oversight	⚠️ WARN	No human-in-the-loop approval gates. Basic execution limits (max_tokens, max_iterations) but no explicit budget enforcement or kill switch.
Art. 15 — Accuracy & Security	⚠️ WARN	No prompt injection defense. Output validation in 2 files. Retry logic in 1 file.

Overall: 1/6 articles passing, 5/6 at warning level.

What We Got Right (We Think)

• Langfuse integration is real compliance infrastructure. The LangfuseService wrapper in quivr_rag.py with LangchainMetadata tracking trace_id, user_id, session_id is solid tracing. However, with only 1 action audit file alongside the Langfuse traces, we scored record-keeping as WARN rather than PASS — the tracing is good but the audit trail coverage is thin.
• Type hints are strong. 79% coverage (107/136 public functions) is above average for the frameworks we've scanned. This earned the only PASS.
• Pydantic validation is present but limited. ConfigDict(extra="forbid") in base_config.py enforces schema validation. However, we noticed a comment about field_validator conflicts with Pydantic v1 — is this still the case?

What We Want Your Feedback On

1. Did we miss something? Our scanner samples files by compliance relevance. If there's PII handling, human oversight, or security code we didn't catch, we'd love to know — it means our file sampling needs tuning.
2. Is the Langfuse integration deeper than we found? We scored record-keeping as WARN because the audit trail beyond Langfuse is thin (1 action audit file). Is there a persistence layer or additional audit infrastructure we missed?
3. Human oversight patterns? We found zero HITL patterns (approval gates, confirmation prompts, @human_feedback decorators). Is this by design (Quivr as a library delegates this to the application layer), or is it a gap you're aware of?

Context

We're scanning frameworks to build benchmark data — not to rank or criticize. We published similar findings for CrewAI (4/6 passing) and LangFlow (4/6 passing) and are sharing with each team to validate accuracy. We previously engaged with the Haystack team at deepset and their feedback helped us fix false positives in our scanner.

The scanner is open-source: air-blackbox on PyPI. You can reproduce these results with:

pip install air-blackbox
air-blackbox setup          # pulls the AI compliance model
air-blackbox comply --scan /path/to/quivr -v --deep

The full scan results are available on request — we're happy to share the raw JSON output.

Happy to discuss anything here. Thanks for building Quivr — the Langfuse + LangGraph architecture is clean.

— Jason Shotwell
AIR Blackbox | GitHub

[jagmarques]

The Art. 12 finding is spot on - Langfuse traces are observability, not compliance evidence. The difference matters because traces can be modified after the fact, and Art. 12 requires records that are tamper-evident for the system's entire lifetime. If you want to close that gap without rebuilding the logging layer, asqav (pip install asqav) wraps existing tool calls and produces ML-DSA-65 signed receipts per invocation. Plugs into the Langfuse flow you already have, just adds the cryptographic integrity layer that auditors actually want to see.

[ericcole1231-boop]

Thank you for the feedback regarding the Art. 12 finding. Your point about the need for tamper-evident records is very helpful.

…

On Mon, Apr 6, 2026, 7:35 PM João André Marques ***@***.***> wrote: *jagmarques* left a comment (QuivrHQ/quivr#3667) <#3667 (comment)> The Art. 12 finding is spot on - Langfuse traces are observability, not compliance evidence. The difference matters because traces can be modified after the fact, and Art. 12 requires records that are tamper-evident for the system's entire lifetime. If you want to close that gap without rebuilding the logging layer, asqav (pip install asqav) wraps existing tool calls and produces ML-DSA-65 signed receipts per invocation. Plugs into the Langfuse flow you already have, just adds the cryptographic integrity layer that auditors actually want to see. — Reply to this email directly, view it on GitHub <#3667 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/CAX57APZFJMY3B5L4PJIEHT4UPTGLAVCNFSM6AAAAACWUGFF2CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DCOJTHA4DGOJVGA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>