Skip to content

Commit 091c70d

Browse files
Copilotimran-siddique
Copilot
and
imran-siddique
authored
docs: add competitive comparison page (vs NeMo, Guardrails AI, LiteLLM, Portkey)
* Initial plan * docs: add COMPARISON.md competitive comparison page and link from README Co-authored-by: imran-siddique <45405841+imran-siddique@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: imran-siddique <45405841+imran-siddique@users.noreply.github.com>
1 parent b5478bb commit 091c70d

File tree

2 files changed

+173
-0
lines changed

2 files changed

+173
-0
lines changed

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -186,6 +186,7 @@ Full methodology, per-adapter breakdowns, and memory profiling: **[BENCHMARKS.md
186186
- **[Azure Deployment Guides](docs/deployment/README.md)** — AKS, Azure AI Foundry, Container Apps, OpenClaw sidecar
187187
- **[NIST RFI Mapping](docs/nist-rfi-mapping.md)** — Question-by-question mapping to NIST AI Agent Security RFI (2026-00206)
188188
- [OWASP Compliance Mapping](docs/OWASP-COMPLIANCE.md)
189+
- [Competitive Comparison](docs/COMPARISON.md) — vs. NeMo Guardrails, Guardrails AI, LiteLLM, Portkey
189190
- [CSA Agentic Trust Framework Mapping](docs/CSA-ATF-PROPOSAL.md)
190191
- [Performance Benchmarks](BENCHMARKS.md)
191192
- [Changelog](CHANGELOG.md)

‎docs/COMPARISON.md‎

Lines changed: 172 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,172 @@
1+
# Competitive Comparison: Agent Governance Toolkit vs. Alternatives
2+
3+
> **TL;DR:** They guard LLM outputs. We govern agent actions. Complementary, not competing.
4+
5+
---
6+
7+
## Overview
8+
9+
When evaluating agent security tooling, developers often encounter [NeMo Guardrails](https://github.com/NVIDIA/NeMo-Guardrails), [Guardrails AI](https://github.com/guardrails-ai/guardrails), [LiteLLM](https://github.com/BerriAI/litellm), and [Portkey](https://portkey.ai/). These are widely-used, well-regarded tools — but they solve a fundamentally **different problem**.
10+
11+
| Tool | Core Focus | Primary User |
12+
|------|-----------|--------------|
13+
| **Agent Governance Toolkit** | Agent action governance, identity, sandboxing, SRE | Platform / security teams deploying autonomous agents |
14+
| NeMo Guardrails | Conversational rail constraints on LLM responses | Developers building chatbots and dialog systems |
15+
| Guardrails AI | LLM output validation and structured data extraction | Developers needing reliable structured outputs from LLMs |
16+
| LiteLLM | Unified LLM API gateway / proxy | Teams managing multi-provider LLM access |
17+
| Portkey | LLM observability, caching, and routing gateway | Teams optimizing LLM cost, reliability, and visibility |
18+
19+
---
20+
21+
## Feature Comparison
22+
23+
| Feature | Agent Governance Toolkit | NeMo Guardrails | Guardrails AI | LiteLLM | Portkey |
24+
|---------|:----------------------:|:---------------:|:-------------:|:-------:|:-------:|
25+
| **Agent action governance** | ✅ | ❌ | ❌ | ❌ | ❌ |
26+
| **LLM output validation** | ✅ (via [content-policy adapters](../packages/agent-os/)) | ✅ | ✅ | ✅ | ✅ |
27+
| **Agent identity (cryptographic)** | ✅ Ed25519 / SPIFFE | ❌ | ❌ | ❌ | ❌ |
28+
| **Execution sandboxing** | ✅ 4-tier rings | ❌ | ❌ | ❌ | ❌ |
29+
| **SRE (SLOs / error budgets)** | ✅ | ❌ | ❌ | ❌ | ❌ |
30+
| **Inter-agent trust mesh** | ✅ | ❌ | ❌ | ❌ | ❌ |
31+
| **Least-privilege capability model** | ✅ | ❌ | ❌ | ❌ | ❌ |
32+
| **Deterministic pre-execution enforcement** | ✅ < 0.1 ms | ❌ | ❌ | ❌ | ❌ |
33+
| **Chaos / replay testing** | ✅ | ❌ | ❌ | ❌ | ❌ |
34+
| **OWASP Agentic Top 10 coverage** | **10 / 10** | ~2 / 10 ¹ | ~1 / 10 ¹ | ~0 / 10 ¹ | ~1 / 10 ¹ |
35+
| **Framework integrations** | **12+** | 3 (LangChain, NeMo-based, custom) | 2 (LangChain, custom) | N/A (gateway) | N/A (gateway) |
36+
| **LLM provider routing / caching** | ❌ | ❌ | ❌ | ✅ | ✅ |
37+
| **Works alongside existing tools** | ✅ | ✅ | ✅ | ✅ | ✅ |
38+
39+
> ¹ **OWASP scoring methodology:** Each tool was assessed against the ten [OWASP Agentic Top 10 (2026)](https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/) risk categories. A risk is counted as "covered" only when the tool provides a mitigation that addresses the root cause of that risk category (not merely partial or indirect coverage). Scores for NeMo, Guardrails AI, LiteLLM, and Portkey are approximate because none of those tools publish explicit OWASP Agentic Top 10 mappings; they are based on a good-faith review of each tool's documented capabilities as of early 2026.
40+
41+
---
42+
43+
## Detailed Breakdown
44+
45+
### NeMo Guardrails (NVIDIA)
46+
47+
**What it does:** Adds conversational guardrails to LLM-based chatbots — blocking off-topic requests, enforcing dialog flows (Colang), and filtering harmful outputs in real time.
48+
49+
**Where it excels:**
50+
- Chatbot safety and topicality constraints
51+
- Structured dialog flow control (Colang DSL)
52+
- Programmable input/output filters
53+
54+
**What it doesn't cover:**
55+
- Governing *what an agent does* (tool calls, sub-agent spawning, file writes, API invocations)
56+
- Agent identity or authentication between agents
57+
- Runtime privilege rings or sandboxing
58+
- SRE / reliability patterns (SLOs, circuit breakers)
59+
- OWASP Agentic Top 10 risks beyond output filtering (~ASI-05)
60+
61+
**Best used:** Alongside the Agent Governance Toolkit when you want chatbot-level dialog safety **and** full agentic action governance.
62+
63+
---
64+
65+
### Guardrails AI
66+
67+
**What it does:** Validates and coerces LLM outputs into structured formats (JSON schemas, Pydantic models) — ensuring outputs conform to expected shapes and correcting them via re-prompting when they don't.
68+
69+
**Where it excels:**
70+
- Reliable structured data extraction from LLM responses
71+
- Output schema enforcement and type coercion
72+
- Re-prompting pipelines for malformed outputs
73+
74+
**What it doesn't cover:**
75+
- Any form of pre-execution action governance
76+
- Agent identity or trust between agents
77+
- Execution sandboxing or privilege rings
78+
- SRE / error budgets
79+
80+
**Best used:** As a companion for output parsing. The Agent Governance Toolkit handles what an agent *does*; Guardrails AI handles what an LLM *says*.
81+
82+
---
83+
84+
### LiteLLM
85+
86+
**What it does:** Provides a unified API gateway that abstracts over 100+ LLM providers behind a single OpenAI-compatible interface — including routing, load balancing, spend tracking, and basic content moderation hooks.
87+
88+
**Where it excels:**
89+
- Multi-provider LLM management from a single API
90+
- Spend tracking and budget enforcement per model/team
91+
- Basic content policy hooks at the LLM call level
92+
93+
**What it doesn't cover:**
94+
- Agent-level governance (pre-execution policy checks on tool calls, spawns, etc.)
95+
- Agent identity, trust scoring, or zero-trust mesh
96+
- Execution sandboxing
97+
- SRE patterns (SLOs, chaos testing, circuit breakers)
98+
99+
**Best used:** As a transparent LLM proxy in front of any provider while the Agent Governance Toolkit enforces what the calling agent is allowed to do.
100+
101+
---
102+
103+
### Portkey
104+
105+
**What it does:** A production LLM gateway providing observability, semantic caching, routing fallbacks, and prompt management — focused on LLM operational reliability and cost optimization.
106+
107+
**Where it excels:**
108+
- LLM call observability and tracing
109+
- Semantic caching to reduce cost
110+
- Routing fallbacks across providers
111+
- Prompt versioning and A/B testing
112+
113+
**What it doesn't cover:**
114+
- Agent action governance (tool calls are invisible to Portkey)
115+
- Agent identity or cryptographic attestation
116+
- Execution sandboxing or privilege isolation
117+
- SRE / reliability engineering at the *agent* level
118+
119+
**Best used:** As a telemetry and cost-optimization layer for LLM calls while the Agent Governance Toolkit enforces governance on the agent's actions.
120+
121+
---
122+
123+
## The Key Distinction
124+
125+
```
126+
LLM Output Layer (NeMo, Guardrails AI, Portkey, LiteLLM)
127+
└─ "Did the model say something safe / structured / on-topic?"
128+
129+
Agent Action Layer (Agent Governance Toolkit)
130+
└─ "Should this agent be allowed to execute this action right now?"
131+
```
132+
133+
These two layers are **complementary, not competing**. A fully governed agentic system typically needs both:
134+
135+
1. **Agent Governance Toolkit** — enforces *what agents do* before every tool call, spawn, or API invocation, with cryptographic identity, privilege rings, SRE reliability, and full OWASP Agentic Top 10 coverage.
136+
2. **An output validator** (Guardrails AI, NeMo) — ensures the LLM's *words* conform to the format and safety rules you need.
137+
3. **An LLM gateway** (LiteLLM, Portkey) — routes, caches, and observes the underlying model calls.
138+
139+
---
140+
141+
## OWASP Agentic Top 10 Coverage Detail
142+
143+
| Risk | Agent Governance Toolkit | NeMo Guardrails | Guardrails AI | LiteLLM | Portkey |
144+
|------|:------------------------:|:---------------:|:-------------:|:-------:|:-------:|
145+
| ASI-01 Agent Goal Hijacking | ✅ Policy engine blocks unauthorized goal changes | ⚠️ Partial (dialog rails) | ❌ | ❌ | ❌ |
146+
| ASI-02 Excessive Capabilities | ✅ Capability model enforces least-privilege | ❌ | ❌ | ❌ | ❌ |
147+
| ASI-03 Identity & Privilege Abuse | ✅ Ed25519 / SPIFFE zero-trust identity | ❌ | ❌ | ❌ | ❌ |
148+
| ASI-04 Uncontrolled Code Execution | ✅ 4-tier execution rings + sandboxing | ❌ | ❌ | ❌ | ❌ |
149+
| ASI-05 Insecure Output Handling | ✅ Content policies validate all outputs | ✅ Output filters | ✅ Schema validation | ⚠️ Basic hooks | ❌ |
150+
| ASI-06 Memory Poisoning | ✅ Episodic memory with integrity checks | ❌ | ❌ | ❌ | ❌ |
151+
| ASI-07 Unsafe Inter-Agent Communication | ✅ Encrypted channels + trust gates | ❌ | ❌ | ❌ | ❌ |
152+
| ASI-08 Cascading Failures | ✅ Circuit breakers + SLO enforcement | ❌ | ❌ | ⚠️ Retries only | ⚠️ Fallback routing |
153+
| ASI-09 Human-Agent Trust Deficit | ✅ Full audit trails + flight recorder | ❌ | ❌ | ⚠️ Logging | ⚠️ Observability |
154+
| ASI-10 Rogue Agents | ✅ Kill switch + ring isolation + anomaly detection | ❌ | ❌ | ❌ | ❌ |
155+
156+
---
157+
158+
## Summary
159+
160+
If your question is:
161+
162+
- *"How do I stop my agent from calling tools it shouldn't?"* → **Agent Governance Toolkit**
163+
- *"How do I ensure my LLM always returns valid JSON?"* → **Guardrails AI**
164+
- *"How do I add topicality constraints to my chatbot?"* → **NeMo Guardrails**
165+
- *"How do I route across 100+ LLM providers with one API?"* → **LiteLLM**
166+
- *"How do I observe and cache my LLM calls?"* → **Portkey**
167+
168+
For production agentic systems, you likely need the Agent Governance Toolkit **plus** one or more of the above tools working together.
169+
170+
---
171+
172+
*See also: [OWASP Compliance Mapping](OWASP-COMPLIANCE.md) · [Architecture Overview](../README.md#architecture) · [Quick Start](../QUICKSTART.md)*

0 commit comments

Comments
 (0)