feat(py/tools): add conform CLI with multi-runtime support (#4593)

## Summary

Refactors the parallel conformance runner into a dedicated private package (`conform`) with a clean module architecture and `asyncio.Queue` worker pool for parallel execution.

<img width="1364" height="776" alt="Screenshot 2026-02-11 at 2 33 49 PM" src="https://github.com/user-attachments/assets/12beabb3-ad99-4cd0-808b-ff3d167dd548" />

<img width="1081" height="473" alt="Screenshot 2026-02-11 at 2 34 28 PM" src="https://github.com/user-attachments/assets/37b3dd6d-031f-4113-9cfc-70edfcc5ece1" />


### Subcommands

| Command | Purpose |
|---------|---------|
| `conform check-model [PLUGIN...] [--all] [-j N] [-v] [--runtime NAME]` | Run model conformance tests in parallel |
| `conform check-plugin` | Lint-time check that plugins have conformance files |
| `conform list` | Show plugins and env-var readiness |

### Key changes

- **Module architecture** — `types.py` contains shared types (Status, PluginResult, Runtime Protocol) to eliminate circular imports
- **Runtime Protocol** — abstracts runtime differences (Python, JS, Go) behind a Protocol; configurable via `[tool.conform.runtimes.*]` in pyproject.toml
- **asyncio.Queue worker pool** — natural backpressure instead of semaphore; N workers pull from the queue
- **Error log** — Rich Panels showing last 15 lines per failure (full with `-v`)
- **CLI polish** — yellow section headers, cyan args via rich-argparse; bare `conform` shows help + env-var table
- **MkDocs eng docs** — Material theme (blue), covers architecture, usage, config, multi-runtime
- **py/bin/conform** — wrapper script for convenience
- **Blocking I/O audit** — all async paths verified safe
- **Releasekit fixes** — added `conform` to `internal_tools` group (fixes ungrouped_packages and publish_classifier_consistency warnings); added Changelog URL to genkit `pyproject.toml`

### Conformance test results

Running `genkit dev:test-model` directly for `google-genai` with credentials:

| Model | Tests | Result |
|-------|-------|--------|
| `gemini-2.5-flash` | Tool Request, Structured Output, Multiturn, System Role, Image Base64, Image URL, Video YouTube | **7/7 ✅** |
| `gemini-2.5-pro` | All 7 tests | 0/7 ❌ (aborted — race condition) |
| `gemini-3-pro-preview` | All 10 tests | 0/10 ❌ (aborted — race condition) |
| `imagen-4.0-generate-001` | Image Output | 0/1 ❌ (aborted — race condition) |
| `gemini-2.5-flash-preview-tts` | TTS | 0/1 ❌ (aborted — race condition) |

**Summary: 7 passed, 19 failed** — All 7 `gemini-2.5-flash` tests pass. The 19 failures are all "aborted" due to a **genkit CLI race condition**: `genkit dev:test-model` dispatches test requests for models before the Python reflection server has finished registering their actions, causing 404s on `/api/runAction`. The last model in the sequence (`gemini-2.5-flash`) passes because the server is fully ready by then.

### Known upstream issues (not addressed in this PR)

1. **`_base_async.py` graceful shutdown** — The `dev_runner` raises `RuntimeError: User coroutine finished without a result.` when SIGTERM cancels a conformance entry point. This is a core framework bug that needs a separate PR. It doesn't affect test correctness but produces noisy tracebacks on shutdown.

2. **Genkit CLI action discovery race** — `genkit dev:test-model` starts testing models before the Python runtime has registered all actions with the reflection server. Models tested early get 404s and are marked "aborted". This is a timing issue in the Node.js CLI, not the Python SDK.

### Lint status

`bin/lint` passes with 0 errors.

Author: yesudeep

py/GEMINI.md

@@ -34,6 +34,7 @@
   | Broad type ignores | No `# type: ignore` without codes | ✅ Automated |
   | Python classifiers | All packages have 3.10-3.14 classifiers | ✅ Automated |
   | Namespace `__init__.py` | Plugins must not have `__init__.py` in `genkit/` or `genkit/plugins/` | ✅ Automated |
+  | Model conformance specs | Model plugins have `model-conformance.yaml` + `conformance_entry.py` | ✅ Automated |
 
   **Release Checks** (`py/bin/release_check`):
 

py/bin/check_consistency

@@ -38,6 +38,8 @@
 #  17. Python version classifiers (3.10-3.14)
 #  18. Namespace package __init__.py (plugins must not have __init__.py in genkit/ or genkit/plugins/)
 #  19. Test file basename uniqueness (no duplicate basenames across test directories)
+#  20. CHANGELOG.md files exist
+#  21. Model conformance spec coverage (model plugins → conformance spec + entry point)
 
 set -euo pipefail
 
@@ -59,7 +61,7 @@ PY_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)"
 cd "$PY_DIR"
 
 # Total number of checks
-TOTAL_CHECKS=20
+TOTAL_CHECKS=21
 
 echo -e "${BLUE}=== Genkit Python Consistency Checks ===${NC}"
 echo ""
@@ -675,6 +677,18 @@ else
   echo -e "  ${YELLOW}!${NC} Some packages are missing CHANGELOG.md files (warnings only)"
 fi
 echo ""
+
+# -----------------------------------------------------------------------------
+# Check 21: Model Conformance Spec Coverage
+# -----------------------------------------------------------------------------
+echo -e "${BLUE}[21/$TOTAL_CHECKS] Checking model conformance spec coverage...${NC}"
+# Delegated to the Python-based ``conform check-model`` tool which performs
+# the same checks (model_info.py scanning + additional-model-plugins from
+# TOML config) and prints output in the same colored format.
+if ! uv run --active conform check-plugin; then
+  ERRORS=$((ERRORS + 1))
+fi
+echo ""
 # -----------------------------------------------------------------------------
 # Summary
 # -----------------------------------------------------------------------------

py/bin/conform

@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+# Wrapper script for the ``conform`` tool.
+#
+# Usage:
+#   py/bin/conform check-model --all      Run all model conformance tests
+#   py/bin/conform check-plugin           Check plugins have conformance files
+#   py/bin/conform list                   List plugins and env-var readiness
+#   py/bin/conform --help                 Show help
+
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PY_DIR="$(cd "${SCRIPT_DIR}/.." && pwd)"
+
+exec uv run --directory "${PY_DIR}" --active conform "$@"

py/bin/test-model-conformance

@@ -187,6 +187,24 @@ phase are independent and should run in parallel** for fastest completion.
 |------|-------------|------------|---------|--------|
 | `runner-script` | Shell script to orchestrate per-plugin conformance test runs | All Phase 1 tasks | `py/bin/test-model-conformance` | ✅ Done |
 
+### Phase 2.5: Spec Audit + Model Updates ✅ COMPLETE
+
+| Task | Description | File(s) | Status |
+|------|-------------|---------|--------|
+| `audit-specs` | Verified all 11 plugin specs against official provider documentation (Feb 11, 2026). Fixed model names, corrected Supports flags, added missing models. Total: 24 models across 11 plugins. | All `model-conformance.yaml` files | ✅ Done |
+
+**Changes made during audit:**
+
+| Plugin | Before | After | Changes |
+|--------|--------|-------|---------|
+| **anthropic** | 2 models | 4 models | Added claude-sonnet-4-5, claude-opus-4-6 |
+| **deepseek** | 1 model (no structured-output) | 2 models | Added structured-output to chat, added deepseek-reasoner (no tools) |
+| **xai** | 1 model (grok-3, legacy) | 2 models | Replaced grok-3 → grok-4-fast-non-reasoning, added grok-2-vision-1212 |
+| **mistral** | 1 model (no vision) | 2 models | Added vision tests, added mistral-large-latest |
+| **amazon-bedrock** | Missing structured-output | Fixed | Added structured-output, streaming-structured-output |
+| **cloudflare** | Missing tool-request | Fixed | Added tool-request, streaming-multiturn |
+| **ollama** | Missing tool-request, vision | Fixed | Added tool-request, input-image-base64 |
+
 ### Phase 3: Validation ⏳ PENDING
 
 | Task | Description | Depends On | File(s) | Status |

py/engdoc/model-conformance-roadmap.md

@@ -136,6 +136,14 @@ validate-docs:
 
 # --- Model Conformance -------------------------------------------------
 
-# Run model conformance tests.
+# Run model conformance tests in parallel (see `conform check-model --help`).
 test-conformance *ARGS:
-    "{{ py_dir }}/bin/test-model-conformance" {{ ARGS }}
+    uv run --directory "{{ py_dir }}" --active conform check-model {{ ARGS }}
+
+# Check that every model plugin has conformance files.
+check-conformance:
+    uv run --directory "{{ py_dir }}" --active conform check-plugin
+
+# List available conformance plugins and env-var readiness.
+list-conformance:
+    uv run --directory "{{ py_dir }}" --active conform list

py/justfile

@@ -42,6 +42,8 @@ dependencies = [
   "genkit-plugin-mistral",
   "genkit-plugin-huggingface",
   "genkit-plugin-observability",
+  # Internal tools (private, not published)
+  "conform",
   "liccheck>=0.9.2",
   "setuptools>=75.0.0,<82",                   # Required by liccheck (provides pkg_resources, removed in setuptools 82+)
   "mcp>=1.25.0",
@@ -233,10 +235,12 @@ genkit-plugin-observability         = { workspace = true }
 genkit-plugin-ollama                = { workspace = true }
 genkit-plugin-vertex-ai             = { workspace = true }
 genkit-plugin-xai                   = { workspace = true }
+# Internal tools (private, not published)
+conform = { workspace = true }
 
 [tool.uv.workspace]
 exclude = ["*/shared", "samples/sample-test", "testapps/*"]
-members = ["packages/*", "plugins/*", "samples/*"]
+members = ["packages/*", "plugins/*", "samples/*", "tools/conform"]
 
 
 # Ruff checks and formatting.
@@ -453,6 +457,7 @@ root = [
   # Tools
   "tools/releasekit/src", # For releasekit package imports
   "tools/releasekit",     # For test imports (pythonpath = ["."])
+  "tools/conform/src",    # For conform package imports
 ]
 
 # Pyright type checking configuration.
@@ -499,6 +504,7 @@ extraPaths = [
   "plugins/xai/src",
   # Tools
   "tools/releasekit/src",
+  "tools/conform/src",
 ]
 pythonVersion = "3.10"
 reportMissingImports = true
@@ -547,6 +553,7 @@ project_includes = [
   # Tools
   "tools/releasekit/src/**/*.py",
   "tools/releasekit/tests/**/*.py",
+  "tools/conform/src/**/*.py",
 ]
 
 # Search path for first-party code import resolution.
@@ -563,6 +570,7 @@ search-path = [
   # Tools
   "tools/releasekit/src",
   "tools/releasekit",
+  "tools/conform/src",
 ]
 # Ignore missing imports for namespace packages - pyrefly can't resolve PEP 420
 # namespace packages but these imports work at runtime.

py/pyproject.toml

@@ -47,7 +47,7 @@ google_plugins = [
   "genkit-plugin-google-genai",
   "genkit-plugin-vertex-ai",
 ]
-internal_tools = []
+internal_tools = ["conform"]
 samples = [
   "dev-local-vectorstore-hello",
   "framework-context-demo",