feat(cts): add spec-driven behavioural contract test suite

Introduce a second contract test suite for the Lance namespace REST API
that exercises operation *semantics* in-process, complementing — not
replacing — the existing wire-level WireMock suite. The two suites now
have distinct, non-overlapping concerns and a clean directory layout.

What's new
----------
* Authoritative spec under docs/src/cts-contracts/ split by domain:
  main.yaml, namespace.yaml, table.yaml, data.yaml, index.yaml,
  tag.yaml, transaction.yaml. Each case declares pre-conditions
  (`given`), the request (`when`) and the expected outcome (`then`:
  success / error_code / 4xx / 409 …) plus required capabilities.
* JSON Schema (ci/cts/cts-contracts.schema.json) and strict linter
  (ci/cts/lint_contracts.py) enforcing single-file ownership per
  operation, capability validity and shape correctness.
* Capability model: ci/cts/capabilities.py plus the per-impl manifest
  ci/cts/capabilities.directory.txt; cases requiring an unsupported
  capability are skipped at runtime instead of failing.
* ci/cts/contract_loader.py parses & validates contracts;
  ci/cts/gen_contract_tests.py renders one Rust test module per
  operation under rust/lance-namespace-cts/tests/contracts/ (43
  modules), post-processed with `rustfmt --edition 2024` so
  `cargo fmt --check` is a fixed point and `--check` mode catches
  drift in CI.
* New workspace member rust/lance-namespace-cts/ hosting the
  in-process harness — Fixtures, ContractCaller, Capabilities,
  assert_contract_{ok,error} — driving DirectoryNamespace from the
  sibling `lance` repo via a cargo path dependency. No cdylib, no
  network, runs as a normal `cargo test` target.

Disambiguation renames (the name "contract" now refers to the new
behavioural suite; the wire-level suite is consistently "wiremock"):
* ci/cts/gen_client_tests.py            -> gen_wiremock_tests.py
* templates/{rust,python,java_*}_contract.mustache
                                        -> *_wiremock.mustache
* rust   tests/contract.rs              -> tests/wiremock.rs
* python tests/test_contract.py         -> tests/test_wiremock.py
* java   WireMockContractIT.java        -> WireMockIT.java

Build / CI surface
------------------
* Make: new `gen-cts-behavior` and `test-cts-behavior` targets;
  default `test-cts` now runs spec-lint + behavioural; opt-in alias
  `test-cts-wiremock` keeps the historical (multi-language) meaning.
* GitHub Actions: contract-tests.yml runs the behavioural suite on
  every PR; the WireMock job becomes opt-in (push to long-lived
  branches and workflow_dispatch only) to keep PR latency low.
* CONTRIBUTING.md gains a "Contract Tests (CTS)" section documenting
  the two suites, how to author a behavioural case, and the
  lint / codegen / run loop.

Quality gates (all green locally)
---------------------------------
  cargo fmt --check              OK
  cargo clippy -D warnings       OK
  gen_contract_tests --check     OK
  lint_contracts --strict        OK
  make test-cts                  170 passed
  make test-cts-wiremock         49 (rust) + 50 (python) + 49 (java)

Author: XuQianJin-Stars

.github/workflows/contract-tests.yml

@@ -5,6 +5,7 @@ on:
     branches: [main, "feat/**", "feature/**"]
     paths:
       - "docs/src/spec.yaml"
+      - "docs/src/cts-contracts/**"
       - "ci/**"
       - "java/**"
       - "python/**"
@@ -13,11 +14,18 @@ on:
   pull_request:
     paths:
       - "docs/src/spec.yaml"
+      - "docs/src/cts-contracts/**"
       - "ci/**"
       - "java/**"
       - "python/**"
       - "rust/**"
       - ".github/workflows/contract-tests.yml"
+  workflow_dispatch:
+    inputs:
+      run_wiremock:
+        description: "Run the legacy WireMock client-conformance matrix"
+        type: boolean
+        default: false
 
 concurrency:
   group: ${{ github.workflow }}-${{ github.ref }}
@@ -90,12 +98,92 @@ jobs:
   # ─────────────────────────────────────────────────────────────
 
   # ─────────────────────────────────────────────────────────────
-  # Job 3: Client conformance — matrix over java / python / rust.
-  #         Each language generates WireMock stubs from the spec
-  #         and runs its own thin contract runner independently.
+  # Job 3: Behavioural contract conformance — drives the spec-driven
+  #         CTS YAML bundle through the in-process Rust harness in
+  #         `rust/lance-namespace-cts` against `DirectoryNamespace`.
+  #         This is the primary PR-blocking signal; it covers behaviour
+  #         (state changes, error codes, capability gating) — not just
+  #         HTTP wire shape.  Runs on every push and PR.
+  # ─────────────────────────────────────────────────────────────
+  behavior-conformance:
+    name: Behaviour Conformance (Rust in-process)
+    runs-on: ubuntu-latest
+    needs: spec
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Setup Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v3
+
+      - name: Install Python workspace (loader/lint/codegen need pyyaml + jsonschema)
+        run: uv sync --all-packages
+
+      - name: Setup Rust
+        uses: dtolnay/rust-toolchain@stable
+        with:
+          toolchain: stable
+
+      # `lance-namespace-cts` pulls in `lance-namespace-impls` from the sibling
+      # `lance` repository via cargo `path`, which transitively depends on
+      # `lance-encoding`.  `lance-encoding`'s build script invokes `protoc` to
+      # compile its `.proto` files, so the GitHub-hosted runner needs the
+      # protobuf compiler on PATH before we run `cargo test`.  Locally this
+      # works because contributors typically have `protoc` installed; CI does
+      # not, so we install it explicitly here.
+      - name: Install protoc
+        uses: arduino/setup-protoc@v3
+        with:
+          version: "27.x"
+          repo-token: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Cache Rust build
+        uses: Swatinem/rust-cache@v2
+        with:
+          workspaces: "rust -> rust/target"
+
+      - name: Lint contract bundle (strict)
+        run: uv run python ci/cts/lint_contracts.py --strict
+
+      - name: Verify generated tests are up-to-date
+        run: uv run python ci/cts/gen_contract_tests.py --check
+
+      - name: Run behavioural contract suite
+        run: make test-cts-behavior
+
+      - name: Upload behavioural test logs
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: contract-test-behavior
+          path: rust/lance-namespace-cts/target/nextest/**/*.xml
+          if-no-files-found: ignore
+
+  # ─────────────────────────────────────────────────────────────
+  # Job 4: Client conformance (legacy WireMock) — matrix over
+  #         java / python / rust.  Each language generates WireMock
+  #         stubs from the spec and runs its thin contract runner.
+  #
+  #         As of P5 (the CI-finalisation phase of the behavioural CTS rollout)
+  #         this job is **opt-in**: it runs on push to `main` only and
+  #         on manual workflow_dispatch with `run_wiremock=true`. It is
+  #         skipped on pull_request events and on push to feature
+  #         branches, to keep PR/dev-branch feedback focused on the
+  #         (much faster) behavioural job above.
   # ─────────────────────────────────────────────────────────────
   client-conformance:
     name: Client Conformance (${{ matrix.lang }})
+    # Opt-in only: WireMock matrix is expensive (java + python + rust),
+    # so we restrict push triggering to `main` and otherwise require an
+    # explicit workflow_dispatch with run_wiremock=true.  PRs never run
+    # this job; the behavioural suite above is the PR-blocking signal.
+    if: >-
+      (github.event_name == 'push' && github.ref == 'refs/heads/main') ||
+      (github.event_name == 'workflow_dispatch' && inputs.run_wiremock)
     runs-on: ubuntu-latest
     needs: spec
     strategy:
@@ -169,7 +257,7 @@ jobs:
 
       # ── Run contract tests ──────────────────────────────────────
       - name: Run contract tests
-        run: make test-client-${{ matrix.lang }}
+        run: make test-clients-wiremock-${{ matrix.lang }}
 
       # ── Upload reports ──────────────────────────────────────────
       - name: Upload test reports

.github/workflows/spec.yml

@@ -54,10 +54,12 @@ jobs:
           make gen
           # Contract tests (CTS) live alongside generated client code and are
           # wiped by `make clean` (which uses `rm -rf <module>/**`).  Re-run the
-          # CTS test generator so the working tree matches the committed sources.
-          # Use gen-client-tests (not gen-cts) to avoid downloading the WireMock
-          # jar, which is only needed for actually running the contract tests.
-          make gen-client-tests
+          # WireMock-layer test generator so the working tree matches the
+          # committed sources.  We use `gen-wiremock-tests` rather than the
+          # full `gen-cts` target so this job stays light: we only need the
+          # generated source files to diff against — actually *running* the
+          # tests would also require downloading the WireMock standalone jar.
+          make gen-wiremock-tests
       - name: Check no difference in codegen
         run: |
           output=$(git diff)

CONTRIBUTING.md

@@ -144,6 +144,91 @@ You can also run `make <command>-<language>-<module>` inside a language folder t
 - `make gen-rust-reqwest-client`: codegen and lint the Rust reqwest client module
 - `make build-java-springboot-server`: build the Java Spring Boot server module
 
+## Contract Tests (CTS)
+
+This repository ships **two** contract test suites; together they form the
+Compatibility Test Suite (CTS):
+
+| Suite | Source of truth | What it verifies | Driver | Default? |
+|---|---|---|---|---|
+| **Behavioural CTS** | `docs/src/cts-contracts/*.yaml` | Operation semantics: state transitions, error codes, capability gating | In-process Rust harness in `rust/lance-namespace-cts` against `DirectoryNamespace` | ✅ runs on every PR via `make test-cts` |
+| **WireMock CTS** | `docs/src/spec.yaml` | HTTP wire shape: paths, methods, status codes, JSON serde | Per-language thin runner (Java / Python / Rust) against a WireMock standalone jar | ⚠️ opt-in via `make test-cts-wiremock`; CI runs it on push to long-lived branches and on manual dispatch only |
+
+Design background and implementation progress notes for the behavioural
+CTS are maintained outside this repository (in the maintainer's working
+notes); the entry points below describe the **stable** authoring
+contract and should be sufficient to add cases without consulting them.
+
+### Authoring a Behavioural Contract Case
+
+1. **Pick the right domain file** under `docs/src/cts-contracts/`:
+   `namespace.yaml` / `table.yaml` / `data.yaml` / `index.yaml` /
+   `tag.yaml` / `transaction.yaml`. Each operation's contracts must
+   live in exactly one file (lint enforces this).
+2. **Write the case** in YAML using the schema in
+   `ci/cts/cts-contracts.schema.json`. A minimal shape is:
+   ```yaml
+   - id: drop_namespace_nonexistent_must_404
+     description: Dropping a non-existent namespace must surface
+                  NamespaceNotFound (4).
+     given:
+       state: empty_catalog
+     when:
+       request:
+         id: ["{{NsMissing}}"]
+     then:
+       error_code: 4
+   ```
+   Capabilities the case requires (e.g. `supports_table_tags`) go in
+   `requires_capabilities`; the harness skips the case at runtime
+   when an implementation does not advertise them.
+3. **Run lint locally**:
+   ```
+   uv run python ci/cts/lint_contracts.py --strict
+   ```
+   `--strict` checks that every `(operation, error_code)` row in
+   `docs/src/namespace/operations/errors.md` is covered by at least
+   one case. CI runs lint in `--strict` mode.
+4. **Regenerate the Rust test sources**:
+   ```
+       make gen-cts-behavior     # rewrites rust/lance-namespace-cts/tests/contracts/
+   ```
+   The generated files **are** checked in (so a fresh checkout
+   `cargo test`s without manual steps); CI verifies they are not stale
+   via `gen_contract_tests.py --check`.
+5. **Run the suite**:
+   ```
+   make test-cts-behavior    # cargo test -p lance-namespace-cts
+   ```
+   `SKIP: missing capabilities: [...]` lines indicate cases that an
+   implementation advertises it does not support — those are
+   informational, not failures.
+
+### Adding a New Operation Field or Capability
+
+- **New request field referenced by a case?** Extend
+  `_render_request_literal` in `ci/cts/gen_contract_tests.py` with
+  the field's optionality and Rust literal shape; mirror the
+  Mustache template `ci/cts/templates/rust_inproc_contract.mustache`
+  if a new request struct needs importing.
+- **New operation?** Register it in `_SUPPORTED_OPS` (also in
+  `gen_contract_tests.py`), add a forwarder method on
+  `ContractCaller` and `InProcessDirectoryCaller` in
+  `rust/lance-namespace-cts/src/caller.rs`, then write the YAML
+  cases. Lint + `--check` will tell you what is missing.
+- **New capability?** Add the ID to
+  `ci/cts/capabilities.directory.txt` (or whichever target's
+  capability file applies) and reference it from
+  `requires_capabilities` on cases.
+
+### When to Use the Legacy WireMock Suite
+
+Run `make test-cts-wiremock` (or `make test-clients-wiremock-{java,python,rust}`)
+when changing **client serialization** or **HTTP wire shape**. It is
+not a substitute for the behavioural suite for namespace
+implementations; behavioural contracts should always be authored
+under `docs/src/cts-contracts/`.
+
 ## Documentation
 
 ### Setup

Makefile

@@ -120,25 +120,25 @@ $(WIREMOCK_JAR):
 	  https://repo1.maven.org/maven2/org/wiremock/wiremock-standalone/$(WIREMOCK_VER)/wiremock-standalone-$(WIREMOCK_VER).jar \
 	  -o $(WIREMOCK_JAR)
 
-# Generate client contract test files for all 4 clients (Rust, Python, Java Apache, Java Async).
+# Generate WireMock contract test files for all 4 clients (Rust, Python, Java Apache, Java Async).
 # Depends on gen-wiremock so the mappings directory already exists when the script runs.
-gen-client-tests: gen-wiremock
-	uv run python ci/cts/gen_client_tests.py \
+gen-wiremock-tests: gen-wiremock
+	uv run python ci/cts/gen_wiremock_tests.py \
 		--mappings-dir $(WIREMOCK_MAPPINGS) \
-		--out-rust rust/lance-namespace-reqwest-client/tests/contract.rs \
-		--out-python python/lance_namespace_urllib3_client/tests/test_contract.py \
-		--out-java-apache java/lance-namespace-apache-client/src/test/java/org/lance/namespace/client/apache/cts/WireMockContractIT.java \
-		--out-java-async java/lance-namespace-async-client/src/test/java/org/lance/namespace/client/async/cts/WireMockContractIT.java
+		--out-rust rust/lance-namespace-reqwest-client/tests/wiremock.rs \
+		--out-python python/lance_namespace_urllib3_client/tests/test_wiremock.py \
+		--out-java-apache java/lance-namespace-apache-client/src/test/java/org/lance/namespace/client/apache/cts/WireMockIT.java \
+		--out-java-async java/lance-namespace-async-client/src/test/java/org/lance/namespace/client/async/cts/WireMockIT.java
 	# Apply spotless formatting to the generated Java test files so the committed
 	# version satisfies spotless:check on CI (which does not run apply beforehand).
 	cd java && ./mvnw -q spotless:apply \
 		-pl lance-namespace-apache-client,lance-namespace-async-client -am \
 		-DskipTests || true
 
-.PHONY: gen-client-tests
+.PHONY: gen-wiremock-tests
 
 # Run all CTS generation steps
-gen-cts: $(AUTO_OVERLAY) $(SPEC_MERGED) gen-wiremock gen-client-tests $(WIREMOCK_JAR)
+gen-cts: $(AUTO_OVERLAY) $(SPEC_MERGED) gen-wiremock gen-wiremock-tests $(WIREMOCK_JAR)
 	@echo "CTS artifacts generated in $(CTS_OUT)"
 
 .PHONY: gen-cts
@@ -154,7 +154,7 @@ gen-cts: $(AUTO_OVERLAY) $(SPEC_MERGED) gen-wiremock gen-client-tests $(WIREMOCK
 # files into the freshly produced trees.
 #
 # Finally, run `test-compile` on the two Java client modules so the freshly
-# written `WireMockContractIT.java` is actually compiled (the per-module
+	# written `WireMockIT.java` is actually compiled (the per-module
 # `build` targets above only ran `mvn install`, which happens *before* the
 # contract tests are written).
 build-cts: build-rust build-python build-java gen-cts
@@ -172,6 +172,7 @@ test-spec-lint: verify-spec-untouched
 		--fail-severity error \
 		--format junit \
 		--output build/spectral-report.xml
+	uv run python ci/cts/lint_contracts.py
 
 .PHONY: test-spec-lint
 
@@ -185,37 +186,73 @@ test-schemathesis: $(SPEC_MERGED)
 
 .PHONY: test-schemathesis
 
-# Java client contract tests
-test-client-java: gen-wiremock $(WIREMOCK_JAR)
+# Java client WireMock contract tests
+test-clients-wiremock-java: gen-wiremock $(WIREMOCK_JAR)
 	cd java && ./mvnw -pl lance-namespace-apache-client,lance-namespace-async-client \
-		test -Dtest="WireMockContractIT" \
+		test -Dtest="WireMockIT" \
 		-Dsurefire.failIfNoSpecifiedTests=false \
 		--no-transfer-progress
 
-.PHONY: test-client-java
+.PHONY: test-clients-wiremock-java
 
-# Python client contract tests
-test-client-python: gen-wiremock $(WIREMOCK_JAR)
+# Python client WireMock contract tests
+test-clients-wiremock-python: gen-wiremock $(WIREMOCK_JAR)
 	cd python && uv run pytest \
-		lance_namespace_urllib3_client/tests/test_contract.py \
+		lance_namespace_urllib3_client/tests/test_wiremock.py \
 		-v --tb=short
 
-.PHONY: test-client-python
+.PHONY: test-clients-wiremock-python
 
-# Rust client contract tests
-test-client-rust: gen-wiremock $(WIREMOCK_JAR)
+# Rust client WireMock contract tests
+test-clients-wiremock-rust: gen-wiremock $(WIREMOCK_JAR)
 	cd rust/lance-namespace-reqwest-client && \
-		cargo test --test contract -- --nocapture
+		cargo test --test wiremock -- --nocapture
+
+.PHONY: test-clients-wiremock-rust
+
+# All client WireMock tests
+test-clients-wiremock: test-clients-wiremock-java test-clients-wiremock-python test-clients-wiremock-rust
+.PHONY: test-clients-wiremock
+
+# Default `make test-cts` now runs the behavioural-contract suite
+# (in-process Rust harness against `DirectoryNamespace`).  WireMock
+# was historically the only client-conformance signal but covers
+# only HTTP wire shape, not behaviour, so it now ships as an
+# explicitly opt-in alias `make test-cts-wiremock` (== the existing
+# `test-clients-wiremock` target).  CI runs both; PR-blocking is
+# limited to the behavioural job (see .github/workflows/contract-tests.yml).
+test-cts: test-spec-lint test-cts-behavior
+.PHONY: test-cts
+
+# Opt-in WireMock alias \u2014 kept so the historical `make test-cts`
+# meaning is one keystroke away and so existing local muscle memory
+# keeps working.  Equivalent to `make test-clients-wiremock`.
+test-cts-wiremock: test-spec-lint build-cts test-clients-wiremock
+.PHONY: test-cts-wiremock
+
+# ============================================================
+# Behavioural-contract CTS (Phase 2+, see the maintainer's
+# behavioural-contract design notes — kept outside this repo)
+#
+# These targets drive the in-process Rust harness in
+# `rust/lance-namespace-cts`, which depends via cargo path on the
+# sibling `lance` repository's `lance-namespace-impls` crate.  No
+# WireMock, no JVM, no Spring Boot — just `cargo test`.
+# ============================================================
 
-.PHONY: test-client-rust
+# Regenerate the per-operation contract test sources from
+# `docs/src/cts-contracts/*.yaml`.  Idempotent; safe to re-run.
+gen-cts-behavior:
+	uv run python ci/cts/gen_contract_tests.py
 
-# All client tests
-test-clients: test-client-java test-client-python test-client-rust
-.PHONY: test-clients
+.PHONY: gen-cts-behavior
 
-# All CTS tests (client-side; add test-schemathesis when a server is running)
-test-cts: test-spec-lint build-cts test-clients
-.PHONY: test-cts
+# Run the in-process behavioural contract suite.  Depends on
+# gen-cts-behavior so a fresh checkout works without manual steps.
+test-cts-behavior: gen-cts-behavior
+	cd rust && cargo test -p lance-namespace-cts --test cts -- --nocapture
+
+.PHONY: test-cts-behavior
 
 # Verify spec was not modified
 verify-spec-untouched: