EvalOps

All

39 repositories

open-associate-skills
Public
Skills bundle for simulating a top‑decile VC associate: market mapping, sourcing, diligence, memos, and portfolio ops.
skills salesforce diligence sourcing venture-capital vc agent-skills market-mapping investment-memo portfolio-ops
Python
•0•0•0•0•Updated Jan 29, 2026Jan 29, 2026
max
Public
Max - AI Agent Dashboard built with Next.js, TypeScript, and Tailwind CSS
TypeScript
•0•1•0•0•Updated Dec 27, 2025Dec 27, 2025
dspy-micro-agent
Public
Minimal agent runtime built with DSPy modules and a thin Python loop. Includes CLI, FastAPI server, and eval harness with OpenAI/Ollama support.
python agent cli ai openai fastapi dspy llm ollama agent-runtime
Python
•7•68•0•0•Updated Dec 22, 2025Dec 22, 2025
petri-tribunal
Public
Multi-critic alignment auditing: Petri + LLM Tribunal integration for robust AI safety evaluation
Python
•0•1•0•0•Updated Dec 16, 2025Dec 16, 2025
tokio-hotel
Public
Ephemeral, cancellable task rooms with scoped resources for Tokio runtimes
rust async concurrency tracing tokio cancellation resource-management
Rust
•
MIT License
•0•0•0•0•Updated Dec 6, 2025Dec 6, 2025
keep
Public
PoC zero-trust access stack with Google SSO, Envoy, OPA, and device attestation
Go
•
Other
•0•0•0•8•Updated Nov 24, 2025Nov 24, 2025
nimbus
Public
Self-hosted CI infrastructure optimized for AI evaluation workloads. Run evals on bare metal with Firecracker isolation, built-in observability, and zero cloud …
ci smith github-actions firecracker
Python
•0•7•0•1•Updated Nov 10, 2025Nov 10, 2025
cursor-triplane
Public
Cursor-style tri-plane RL stack: Firecracker microVM envs, Ray inference, FP8 MoE trainer
Python
•0•1•0•0•Updated Nov 8, 2025Nov 8, 2025
agent-pm
Public
Agent PM: OpenAI Agents-powered product management orchestrator with automated PRDs, tickets, and comms.
product-management fastapi openai-agents evalops agent-pm
Python
•
MIT License
•2•4•0•0•Updated Oct 21, 2025Oct 21, 2025
provenance
Public
Agent provenance & risk analytics API: FastAPI service with Semgrep detection, Redis, analytics dashboards
redis security analytics fastapi streamlit semgrep
Python
•0•0•0•0•Updated Oct 17, 2025Oct 17, 2025
cognitive-dissonance-dspy
Public
A multi-agent LLM system for detecting and resolving cognitive dissonance.
Python
•22•274•0•1•Updated Oct 14, 2025Oct 14, 2025
meta-secdev-agent
Public
Static analysis meta agent for security and tenant isolation workflows
Python
•0•0•0•0•Updated Oct 13, 2025Oct 13, 2025
github-ona
Public
Organizational Network Analysis tool for GitHub repositories - analyze collaboration patterns through PR and review data
Python
•0•0•0•0•Updated Oct 11, 2025Oct 11, 2025
env-drift-detector
Public
Detect environment variable drift across monorepo services using OpenAI Codex SDK - catches naming inconsistencies, missing vars, and undocumented secrets
devops monorepo environment-variables configuration-management openai developer-tools codex drift-detection
TypeScript
•0•1•0•0•Updated Oct 8, 2025Oct 8, 2025
tenant-isolation-guard
Public
Detect missing tenant isolation in multi-tenant FastAPI apps using Codex SDK - prevents cross-tenant data leaks by catching missing workspace filters
security sqlalchemy multi-tenant typescript saas openai codex fastapi tenant-isolation
TypeScript
•0•0•0•0•Updated Oct 8, 2025Oct 8, 2025
grimoire
Public
The AI coding agent built for the terminal.
TypeScript
•
MIT License
•9.7k•0•0•0•Updated Oct 4, 2025Oct 4, 2025
tinker-eval-loop
Public
Evaluation-driven LoRA fine-tuning loop using Tinker API - automatically improve models through iterative training and evaluation cycles
machine-learning evaluation lora tinker fine-tuning llm evalops inspect-ai
Python
•0•0•0•0•Updated Oct 2, 2025Oct 2, 2025
stack
Public
A cohesive Transformers stack built on PyTorch with uv for reproducible dependency management
Python
•0•0•0•0•Updated Sep 30, 2025Sep 30, 2025
mocktopus
Public
🐙 Multi-armed mocks for LLM apps - Drop-in replacement for OpenAI/Anthropic APIs for deterministic testing
python testing continuous-integration mock-server evaluation pytest openai developer-tools testing-tools deterministic
Python
•
Other
•0•6•0•0•Updated Sep 24, 2025Sep 24, 2025
metaethical_breach_dspy
Public
DSPy implementation for detecting metaethical breaches in AI systems through systematic evaluation of moral reasoning patterns
benchmark machine-learning evaluation ai-safety ethics ai-alignment dspy moral-reasoning
Python
•
MIT License
•0•0•0•0•Updated Sep 23, 2025Sep 23, 2025
dspy-lean-prover-hint-clipping
Public
DSPy + Lean (mock) iterative prover with hint clipping; sweeps on clipping vs KL, noise, sparsity; scalable dataset generator; curated training; frozen tools.
experiment evaluation program-synthesis dataset rl lean clipping variance-reduction ppo tool-use
Python
•0•2•0•0•Updated Sep 19, 2025Sep 19, 2025
EvalATS
Public
Modern, AI-powered Applicant Tracking System with evaluation focus
TypeScript
•2•4•0•0•Updated Sep 18, 2025Sep 18, 2025
override-cascade-dspy
Public
DSPy framework for detecting and preventing safety override cascades in LLM systems. Research-grade implementation for studying when completion urgency override…
python machine-learning ai-safety cognitive-architecture research-framework dspy llm-evaluation llm-safety safety-research override-cascade
Python
•
MIT License
•0•5•0•0•Updated Sep 14, 2025Sep 14, 2025
founder-email-optimizer
Public
DSPy-powered email optimization for startup founders: drop in your 3 best emails, get optimized outreach for new leads
Python
•1•39•0•0•Updated Sep 14, 2025Sep 14, 2025
llmcc
Public
LLM-native compiler toolchain - implementing 'LLM ≈ probabilistic compiler' with real OpenAI integration, structured outputs, and constrained decoding
typescript ai compiler json-schema openai code-generation llm structured-outputs
TypeScript
•
MIT License
•0•1•0•0•Updated Sep 13, 2025Sep 13, 2025
congress-bill-search
Public
High-quality congressional bill search with hybrid BM25+vector similarity using DuckDB, TEI embeddings, and GovInfo API. Local deployment with Docker.
docker search-engine embeddings tei bm25 text-search reranking congressional-bills vector-search duckdb
Python
•0•2•0•0•Updated Sep 11, 2025Sep 11, 2025
orbit-agent
Public
A brutally honest "high‑orbit" startup advisor you can text or run from the CLI. Built with DSPy, it provides opinionated, YC-style advice and financial tools f…
python cli finance ai sms startup entrepreneurship advisor founder y-combinator
Python
•0•20•0•0•Updated Sep 10, 2025Sep 10, 2025
folie-a-deux-dspy
Public
Iterative LLM agreement training framework using DSPy MIPROv2 - exploring consensus formation in AI systems
Python
•
MIT License
•0•5•0•1•Updated Sep 8, 2025Sep 8, 2025
proofcite
Public
Cite or shut up. LLM answers without verifiable sources are liabilities. Extractive RAG with line-anchored citations that fails closed when confidence is low. D…
citations tf-idf compliance deterministic regulatory rag extractive-qa fail-closed line-anchored
Python
•0•0•0•0•Updated Sep 7, 2025Sep 7, 2025
eval2otel
Public
Library to convert AI evaluation results to OpenTelemetry GenAI semantic conventions for observability and monitoring
typescript ai telemetry evaluation observability opentelemetry genai
TypeScript
•
MIT License
•0•2•3•0•Updated Sep 1, 2025Sep 1, 2025