Project Spec: An Agent-Native SDK for Data UIs

An agent-native SDK for building data UIs from typed schemas.

---

1. Philosophy

Core thesis

The labelling primitive that the modern post-training, evals, and agent-development loop needs does not exist as a library. Heavyweight platforms (Argilla, Label Studio) optimize for a discrete-project workflow that no longer matches how teams work. Vibe-coded one-off UIs (the Hamel/Shreya school) prove the demand but fragment the artifact. There is room for a small, opinionated SDK that turns typed schemas into constrained data UIs, composes into agent loops natively, and produces Hub-native artifacts.

What this is

• An SDK you import. Not a hosted product. Not an app. Not a platform.
• Schema-first: typed Python classes drive UI generation.
• Agent-native: the loop primitive (link → handoff → state-diff → continue) is in the core, not bolted on.
• Local-first: defaults to 127.0.0.1, no server-side state required.
• Hub-native: sessions serialize to Hub-compatible artifacts; the Hub is the only system of record.
• Standalone-capable: works perfectly well without an agent in the loop.

What this is not

• Not a platform with workspaces, users, roles, or auth.
• Not a hosted SaaS.
• Not a project-management tool for labelling teams.
• Not an end-to-end pipeline.
• Not a competitor to Gradio (sibling library, shared aesthetics, different loop).
• Not a replacement for Label Studio / CVAT / Argilla for users who actually want those products.

Design principles

• Tools, not platforms. Sessions are ephemeral by default; persistence is opt-in via the Hub.
• Generality at the workflow layer, specificity at the primitive layer. Inverse of Argilla.
• Strong opinions on the schema vocabulary and component set. No opinions on storage, auth, queues, team workflows — those compose externally.
• The Hub is the system of record, always. No parallel workspace concept.
• Small core, curated extensions. Resist feature creep. The component library is the moat because it's small.
• Optimize for casual contribution. A new component should be a single-file PR.
• Lead docs with the standalone case. Agent integration is the second example, not the first.
• Capability-grounded vocabulary, audience-routed framing. Core types describe what the SDK does; docs route by audience.
• Resist the platform pull. Every requested team/queue/auth feature is a composable adjacency, not a core feature.

Positioning tagline

An agent-native SDK for building data UIs from typed schemas.

Eight words. The constant. Use it everywhere.

Audience routing (docs only, not API)

• MLEs training models
• SMEs assessing responses
• MCP/agent developers iterating on servers
• Researchers auditing synthetic data
• Anyone with a CSV who needs to grade things

Each gets a complete, runnable example. None previews the others.

---

2. Functional Scope

2.1 Schema definition

Users (or agents) define a session by declaring a typed schema. The schema is the source of truth for the UI, the state shape, the artifact format, and validation.

• Pydantic models or @dataclass-style classes (decide during exploration)
• typing.Annotated markers parameterize components and add UI metadata
• Schemas compose: a session can include a list of sub-schemas (e.g. a queue of items each with their own form)
• Schemas are serializable (JSON Schema export for agent consumption and cross-language clients)
• Schemas are versioned (artifact records schema version for reproducibility)

2.2 Component library (v1 candidates)

Kept deliberately small. Each is a typed, persistable, validated input primitive.

• Trajectory viewer — agent step list, tool calls, observations; per-step annotation slots
• Rubric grid — N criteria × M items, scores per cell, optional rationale
• Pairwise comparator — A vs B with preference + structured rationale
• Schema prober — render any JSON Schema as a form, capture call+response
• Diff reviewer — side-by-side or unified text/JSON diff with comments
• Annotation queue — paginated list of items, each rendered with a sub-schema

Plus a small set of leaf inputs that compose into the above:

• Text, markdown, code, numeric, boolean, single-select, multi-select, slider, score, free-text rationale, file upload (deferred to v2 if scope-pressed)

2.3 Session lifecycle

• session = LabelingSession(schema=..., data=...) — instantiate
• session.serve() — start a local HTTP server, return URL
• session.read() — return current state (the API the agent's MCP tool calls)
• session.diff(since=...) — return state changes since a marker (for efficient agent polling)
• session.complete() — event/future the agent awaits via elicitation
• session.artifact() — return the serialized artifact (schema + state + provenance)
• session.push_to_hub(repo_id) — publish to the Hub
• LabelingSession.from_hub(repo_id) — load a published session back

2.4 Artifact format

A Hub-resident folder containing:

• schema.json — the typed schema definition
• state.json — the current/final state
• events.jsonl — append-only log of state transitions (provenance)
• agent_trace.jsonl — if agent-driven, the agent's reasoning trace
• manifest.toml — version, timestamps, author, schema hash
• README.md — auto-generated summary, viewable on the Hub

Artifact is:

• Diff-able (events.jsonl makes session-to-session comparison trivial)
• Reproducible (schema + initial data + events = deterministic final state)
• Loadable (one line to rehydrate)
• Shareable (Hub URL)

2.5 Agent loop primitives

The SDK ships the pieces an agent needs to drive a session. Not the agent itself.

• Link emission — session.serve() returns a URL the agent can hand off
• State reading — session.read() / session.diff() for the agent's MCP tool
• Completion signal — session.complete() resolves when the human signals done
• Elicitation glue — helper that wraps the link + completion event in an MCP elicitation request
• Optional MCP server wrapper — a tiny FastMCP server exposing start_session, read_state, complete_session as tools, so any MCP-capable agent can drive sessions without bespoke integration

Agent integration is recipes in docs, not framework lock-in. The SDK works the same whether the caller is Claude Code, Codex, Aider, smolagents, a bare LangGraph node, or a Jupyter cell.

2.6 Runtime

• Default: local HTTP server on 127.0.0.1:<random-port>
• Optional: HF Spaces deployment for shared/persistent sessions
• Optional: Gradio share-link-style tunneling for ad-hoc external review
• No remote-by-default mode. No telemetry by default.

2.7 Standalone usage

Same SDK, no agent:

1. Write a schema in a Python file
2. python my_session.py → opens browser at local URL
3. Label, grade, audit, whatever
4. Ctrl-C → artifact saved locally or pushed to Hub
5. Optional .from_hub() to resume

The agent loop is one valid caller; the human-only loop is another. Both first-class.

---

3. High-Level Architecture

Layered structure

1. Schema layer — typed schemas, Annotated markers, validation, JSON Schema export
2. Component layer — the curated UI primitives, each a self-contained module
3. Session layer — runtime state, event log, completion signaling
4. Server layer — HTTP + WebSocket (likely on top of Gradio's serving infrastructure or FastAPI directly; explore)
5. Artifact layer — serialization, manifest generation, Hub integration
6. Agent layer — elicitation helpers, optional MCP server wrapper, polling/diff utilities

What we build vs. what we sit on

Sit on:

• Gradio's serving stack where possible (FastAPI core, Spaces deployment, share-link tunneling)
• Pydantic for schema validation
• HF Datasets / Hub Python client for artifact storage
• FastMCP for the optional MCP server wrapper

Build:

• The typed-schema-to-UI generation pipeline
• The six component primitives
• The session model (state + events + completion)
• The artifact format and Hub integration
• The agent-loop helpers (elicitation, state-diff)
• Docs, recipes, examples

Open architectural questions (to discover)

• Pydantic models vs. @dataclass vs. our own decorator: which gives the best agent-ergonomics?
• Build on Gradio's runtime or run parallel? Tradeoff between leverage and architectural freedom.
• WebSocket-based state sync vs. REST polling vs. SSE: which simplifies the agent's state-diff tool?
• How much of the UI is React vs. server-rendered HTML? Affects bundle size, extensibility, contribution friction.
• Component extensibility model: how does a third party add a new component without forking?
• Schema versioning and migration: how do we handle artifact load when the schema has evolved?
• Multi-user collaboration on a single session: in scope for v1, deferred, or never?

---

4. Agent-Loop Flow (Canonical)

The reference flow, documented as the default integration pattern:

1. Agent (in any terminal coding tool) determines a human handoff is needed.
2. Agent constructs a typed schema describing what the human should do.
3. Agent calls start_session(schema, data) via the SDK's MCP server (or the Python API directly).
4. SDK returns {url, session_id}.
5. Agent issues an MCP elicitation request: "Please complete the task at {url}. Reply 'done' when finished."
6. Human visits URL, completes the form, browser shows "all done" state.
7. Human replies to elicitation: "done".
8. Agent calls read_state(session_id) and receives the completed artifact.
9. Agent continues its work, optionally publishing the artifact to the Hub.

Works in every terminal agent that supports elicitation (which is the entire MCP-compliant ecosystem). Works in chat-panel hosts too (Claude, ChatGPT) — they get inline link rendering and elicitation flows for free.

MCP Apps as a deployment target

Not the unifying primitive. A chat-panel-hosted client can render the session inline as an MCP App if both the client and SDK support it, but the SDK does not require it and the canonical flow does not depend on it. Treat MCP Apps emission as an enhancement for the chat-panel cohort, not a default.

---

5. Three Canonical Recipes (Docs)

Each recipe is a complete, copy-pasteable example. These ship as the headline demos.

Recipe 1: Synthetic data audit

• Inputs: a Distilabel pipeline's output (HF dataset)
• Schema: side-by-side prompt/response viewer + quality score + failure-mode multi-select + notes
• Flow: agent surfaces suspect items, SME audits, agent retrains judge filter
• Artifact: annotated dataset on the Hub

Recipe 2: Post-training preference grading

• Inputs: a DPO preference dataset where the reward model disagrees with humans
• Schema: pairwise comparator + preference + rubric grid for why
• Flow: MLE grades disagreements, agent updates rubric and reward model
• Artifact: preference dataset + versioned rubric on the Hub

Recipe 3: MCP server iteration

• Inputs: an MCP server under development
• Schema: schema prober per tool + pass/fail/ambiguous + suggested description rewrite
• Flow: agent exercises the server, developer reviews findings, iterates descriptions
• Artifact: reproducible test suite on the Hub

Additional recipes (later): RAG retrieval review, prompt eval grading, trajectory annotation for RL environments, dataset spot-check.

---

6. Out-of-Scope (Explicitly)

To protect the small-core discipline:

• User management, roles, permissions, SSO
• Hosted runtime as a paid service
• Persistent queues, project dashboards, team workflows
• Multi-tenant deployment
• Pixel-level annotation (defer to CVAT via export adapters if needed)
• Video timeline annotation (same)
• Pre-built integrations with every annotation vendor
• An end-to-end training pipeline
• A judge model, an LLM API client, an eval harness
• Agent orchestration logic (the agent is the caller, not the callee)

If a user wants any of the above, they compose it. The SDK stays a library.

---

7. Relationship to Gradio

• Sibling library, not extension. Different loop, different primitive vocabulary.
• Shared aesthetics where it makes sense. Typography, color, baseline component styling.
• Shared serving infrastructure where it makes sense. Explore using Gradio's FastAPI layer + share-link tunneling rather than rebuilding.
• Interoperable in the same Python process. A user mixing demo-shape and labelling-shape code should not feel friction.
• Long-term relationship unresolved. Three plausible futures (stay sibling, get absorbed as gradio.label, philosophical convergence). Hold loosely; let usage inform.

---

8. Relationship to Argilla

• Not a replacement; a different category. Argilla is a deployed app; this is a library you import.
• Complementary positioning. Argilla covers static-schema annotation projects; this covers dynamic, ephemeral, agent-driven sessions.
• Honest about the cautionary tale. Argilla's platform-shape is the failure mode this design avoids. Say it in docs/talks without disrespect.
• No migration path planned. Different enough that direct migration isn't meaningful. Recipes can demonstrate equivalent workflows.

---

9. Unknowns to Discover

These are real questions the implementation phase must answer. None blocks getting started; all need answers before v1.0.

Schema and API

• Pydantic vs. dataclass vs. custom decorator — which is most agent-ergonomic?
• How are nested/composite schemas best expressed? (List of items, each with its own schema.)
• What's the right vocabulary of Annotated markers? Erring small to start.
• How do we handle schema evolution and artifact migration?
• Can the schema be defined in JSON Schema directly (for non-Python callers / agents emitting JSON)?

Runtime

• Build on Gradio's serving stack or run parallel?
• WebSocket / SSE / polling for state sync?
• React frontend vs. server-rendered HTML vs. hybrid (HTMX-style)?
• How small can we get the JS bundle for the local-server case?

Component design

• What does a third-party component PR look like? Single file? Convention over configuration?
• How do components compose? Slots? Children? Explicit composition API?
• Styling: Tailwind-style utility classes, CSS modules, or component-scoped styles?
• Accessibility baseline: WCAG AA from day one, or accept v1 gaps?

Agent integration

• Should the SDK ship its own MCP server or leave it as a recipe?
• How does the agent know what schema to write? Do we ship "schema templates" the agent can pick from?
• What's the right read_state granularity — full state, diff, or query-able?
• Should completion signaling support partial completion / pause-resume?

Hub integration

• What's the canonical Hub repo structure for a session artifact? New repo type or a folder convention?
• Should sessions be loadable as HF Datasets directly, or remain a distinct artifact type?
• How does versioning work on the Hub — branches, tags, commit history?
• Private session artifacts: respect Hub permissions as-is.

Standalone UX

• Can a user write a single-file Python script and python my_label.py Just Works?
• CLI entry point? hf-data new, hf-data run, hf-data publish?
• Notebook integration: how does this feel in a Jupyter cell?

Ecosystem

• Adapters: do we ship import/export for Label Studio JSON, CVAT XML, Hub datasets at v1?
• Distilabel integration: explicit pipeline step, or just a recipe?
• TRL / verifiers integration: same question.
• VS Code / Cursor extensions: out of scope for v1, but worth thinking about discoverability.

Project and contribution

• License: Apache 2.0 (matches Gradio, TRL, Datasets).
• Repo structure: monorepo with core + components, or split?
• CI from day one or after first stable release?
• Docs site: MkDocs (Gradio's choice) or Docusaurus or HF docs builder?
• How do we handle the "audience routing" in docs technically — duplicate examples or canonical examples with audience preambles?

Naming

• Project name: TBD. Working title for now. Should be:
  • Memorable
  • Pronounceable in English and other major languages
  • Available as PyPI package, GitHub org/repo, domain (if applicable)
  • Not directly evocative of "labelling" (we want the breadth) or "agent" (we want the standalone case)
  • Capability-grounded, not audience-grounded

Risks and watch-fors

• Scope creep toward platform-shape. Re-read this spec quarterly; refuse features that drift.
• Gradio team politics around naming, positioning, and sibling-status. Surface early, often.
• MCP ecosystem evolution: if elicitation gets replaced or deprecated, the canonical flow needs an alternative.
• The "vibe coding hangover": ship reliability and security from day one to avoid the trust deficit hitting the project unfairly.
• Argilla comparisons in the wild: have a one-sentence clean answer ready.

---

10. Success Criteria (6-Month Horizon)

Not commitments. Signals that the bet is working.

• 300+ GitHub stars
• 50+ session artifacts published to the Hub by non-team users
• 5+ external case studies (blog posts, conference talks, Twitter threads from users)
• 3+ external contributors with merged component PRs
• At least one integration into a major OSS post-training pipeline (TRL, verifiers, SkyRL, Distilabel)
• Mention in a Hamel/Shreya/Karpathy/Lütke/Anthropic/HF blog or post
• 80%+ of recipe issues resolved with "use this recipe" rather than "we'll add that feature"

If 4+ of these hit by month 6, double down. If <2, re-evaluate scope and positioning.

---

11. What Comes Next

1. Naming. Pick a working name.
2. Schema-API spike. Sketch the typed-schema → form generation in ~200 lines, decide Pydantic vs. dataclass vs. decorator.
3. One component, end-to-end. Build the rubric grid as the canonical first component, including artifact serialization.
4. Local-server prototype. session.serve() + browser flow + session.read() working.
5. Standalone recipe. A real user (someone outside the team) labels something with it.
6. Agent recipe. Same flow, driven from Claude Code via elicitation.
7. Hub artifact roundtrip. Push, view, load back.
8. Second component. Trajectory viewer or pairwise comparator, to validate the extension model.
9. Public preview. Repo, docs site, first recipes published.
10. Iterate based on real usage. Don't add components or features speculatively; let the recipes pull them in.

Each step is a checkpoint where the spec can be revised. The spec is not law; the eight-word tagline is.