socialfoundations
diff --git a/‎.gitignore‎
Lines changed: 7 additions & 1 deletion b/‎.gitignore‎
Lines changed: 7 additions & 1 deletion
diff --git a/‎README.md‎
Lines changed: 104 additions & 48 deletions b/‎README.md‎
Lines changed: 104 additions & 48 deletions
diff --git a/‎docs/index.rst‎
Lines changed: 1 addition & 0 deletions b/‎docs/index.rst‎
Lines changed: 1 addition & 0 deletions
diff --git a/‎docs/updates.md‎
Lines changed: 134 additions & 0 deletions b/‎docs/updates.md‎
Lines changed: 134 additions & 0 deletions
diff --git a/‎folktexts/__init__.py‎
Lines changed: 1 addition & 1 deletion b/‎folktexts/__init__.py‎
Lines changed: 1 addition & 1 deletion
diff --git a/‎folktexts/benchmark.py‎
Lines changed: 48 additions & 4 deletions b/‎folktexts/benchmark.py‎
Lines changed: 48 additions & 4 deletions
diff --git a/‎folktexts/classifier/__init__.py‎
Lines changed: 1 addition & 0 deletions b/‎folktexts/classifier/__init__.py‎
Lines changed: 1 addition & 0 deletions
@@ -180,4 +180,10 @@ docs/_build/
 docs/notebooks/
 
 # Claude Code project context (local, not shared)
-CLAUDE.md
+CLAUDE.md
+
+# Local-only validation / debugging artifacts (kept out of the repo)
+scripts/
+divergences/
+results/VALIDATION_STATUS.md
+results/phase7-edge-cases/
@@ -34,6 +34,7 @@ Check out the following sub-pages:
    :maxdepth: 1
 
    Readme file <readme>
+   Updates <updates>
    API reference <source/modules>
    Example notebooks <notebooks>
 
 
@@ -0,0 +1,134 @@
+# Updates
+
+Release notes summarising user-visible changes between versions. Older changes
+not yet listed here can be reconstructed from the git log.
+
+## v0.4.0 — vLLM backend
+
+`folktexts` v0.4.0 introduces local inference via [vLLM] alongside the existing
+HuggingFace `transformers` backend, typically delivering a 5–30× throughput
+improvement on GPU benchmarks while preserving the full score-extraction
+contract (multiple-choice, direct-numeric, and reasoning prompting).
+
+[vLLM]: https://docs.vllm.ai/
+
+### What's new
+
+- **`VLLMClassifier`**: a new top-K-logprobs classifier in
+  `folktexts.classifier.vllm_classifier`, parallel to
+  `TransformersLLMClassifier`. Both feed the same QA decoders, so result
+  semantics are unchanged.
+- **`load_vllm_model`** in `folktexts.llm_utils`: helper that initialises a
+  vLLM `LLM` engine + tokenizer with sensible defaults for this benchmark
+  (BF16, `gpu_memory_utilization=0.85`, `logprobs_mode="processed_logprobs"`).
+- **CLI flag `--inference-backend {transformers,vllm}`**: selects the local
+  backend. **Default is now `vllm`.** Pass `--inference-backend transformers`
+  to fall back to the previous path; the transformers code is unchanged and
+  remains a fully supported alternative.
+- **vLLM-specific CLI flags**: `--gpu-memory-utilization`, `--max-model-len`,
+  `--vllm-dtype`, `--tensor-parallel-size`. The CLI auto-derives a
+  `max_model_len` from `--context-size + ReasoningQA.max_new_tokens + 256`
+  when the user does not pass `--max-model-len` explicitly.
+- **Optional install group `[vllm]`**: `pip install folktexts[vllm]` pulls in
+  the vLLM wheel. The base install is unchanged for users on the transformers
+  path.
+
+### Architecture
+
+- **Two classifiers, one decoder.** `VLLMClassifier`,
+  `TransformersLLMClassifier`, and `WebAPILLMClassifier` all hand answers to
+  the QA-decoder methods on `MultipleChoiceQA`, `DirectNumericQA`, and
+  `ReasoningQA`. The new helper `decode_topk_logprobs_to_risk_estimate` in
+  `folktexts.llm_utils` factors out the top-K decoding logic shared by vLLM
+  and the WebAPI; the transformers path (which has full-vocab logits)
+  bypasses this helper, as before.
+- **Backend dispatch.** `Benchmark.make_*_benchmark(...)` accepts a
+  `backend=` argument (`"transformers"`, `"vllm"`, `"webapi"`, or
+  `None` for autodetect). When `None`, autodetect uses
+  `str → webapi`, duck-typed `LLM-shaped → vllm`, else `transformers`.
+- **`VLLMClassifier.__hash__`** includes a `"vllm"` tag so cached result
+  paths (`results.bench-{hash}.json`) cannot collide with transformers runs
+  of the same model. Predictions can drift by ~1e-3 across backends due to
+  attention-kernel differences; mixing them in one CSV would be a silent
+  mistake.
+- **Numeric mode** uses vLLM's `allowed_token_ids` to restrict generation to
+  digit tokens (mirroring the transformers `digits_only=True` mask).
+  Multiple-choice mode runs unmasked; the QA decoder's prefix-variant
+  matching handles renormalisation across answer letters.
+
+### Cluster runtime requirements (B200 / Hopper / vllm 0.20.1 wheel)
+
+The vLLM 0.20.1 wheel is built against CUDA 13. On clusters where the
+default toolkit is older, two environment steps are required for any
+vLLM invocation:
+
+```bash
+source /etc/profile.d/modules.sh
+module load cuda/13.2          # provides libcudart.so.13
+export VLLM_USE_DEEP_GEMM=0    # skips an FP8 warmup that needs deep_gemm
+                               # (not on PyPI); harmless on BF16 models
+```
+
+Without these, `import vllm._C` and engine init both crash on Hopper+
+GPUs.
+
+### Validation
+
+The migration was validated across 38 cross-backend cells covering the
+paper's Table 1 (8 models × 2-4 modes), a modern + thinking-model sweep
+(`gemma-3-1b-it`, `Qwen3-1.7B`, `Qwen3-4B`, `Qwen3-4B-Instruct-2507`,
+`Qwen3-4B-Thinking-2507`), and a chat-template extension on
+`Mistral-7B-Instruct-v0.2` and `Yi-34B-Chat`. Multi-seed stability was
+verified across 4 seeds × 2 backends on Llama-3-8B-Instruct and
+Qwen3-Thinking-2507.
+
+**36/38 cells fall within the strict gates** `|ΔAUC| ≤ 0.015` and
+`|ΔECE| ≤ 0.025`. The two remaining outliers are characterised:
+
+- `Llama-3-8B` base × `numeric` (zero-shot): vLLM `+0.017` AUC, `−0.041`
+  ECE — vLLM is slightly *better*. The model is essentially near-random
+  on this prompt (TF AUC 0.559); the delta is within the kernel-noise
+  band of a near-random model.
+- `Qwen3-1.7B` × `chat-MCQ`: vLLM `+0.190` AUC, `+0.265` ECE — vLLM
+  is *much* better. The transformers path collapses to 3 unique scores
+  on this combination; vLLM produces 425 unique scores with broad
+  spread. The bug is on the transformers side and does not reproduce on
+  Qwen3-4B / Qwen3-4B-Instruct / Qwen3-4B-Thinking-2507.
+
+Phase 7 robustness checks (1-row DataFrame, sequential model swap in
+the same Python process, near- and over-cap inputs, tied-logit
+cross-backend agreement, and OOM clean failure) all pass.
+
+### Backwards compatibility
+
+- The CLI accepts the same flags as before plus the new
+  `--inference-backend` / `--gpu-memory-utilization` /
+  `--max-model-len` / `--vllm-dtype` / `--tensor-parallel-size`. All new
+  flags have safe defaults; existing scripts work unchanged on the
+  vLLM backend, or on the transformers backend with
+  `--inference-backend transformers`.
+- `Benchmark.make_*_benchmark(...)` accepts an optional `backend=` kwarg.
+  Existing callers that pass `model=` as a HuggingFace `PreTrainedModel`
+  continue to be routed to `TransformersLLMClassifier`.
+- Result CSVs from previous runs (transformers) are not invalidated; the
+  new vLLM hash tag means vLLM runs save to a fresh path rather than
+  overwriting transformers numbers.
+
+### Migration notes
+
+If you previously installed `folktexts` and want to use the new vLLM
+backend:
+
+```bash
+pip install --upgrade 'folktexts[vllm]'
+# or, from a checkout:
+pip install -e .[vllm]
+```
+
+Then either accept the new default (`vllm`) or stay on transformers
+explicitly:
+
+```bash
+run_acs_benchmark --model <path> --task ACSIncome --data-dir <path> \
+    --inference-backend transformers
+```
@@ -1,6 +1,6 @@
 from ._version import __version__, __version_info__
 from .acs import ACSDataset, ACSTaskMetadata
 from .benchmark import Benchmark, BenchmarkConfig
-from .classifier import LLMClassifier, TransformersLLMClassifier, WebAPILLMClassifier
+from .classifier import LLMClassifier, TransformersLLMClassifier, VLLMClassifier, WebAPILLMClassifier
 from .qa_interface import DirectNumericQA, MultipleChoiceQA, ReasoningQA
 from .task import TaskMetadata
@@ -15,7 +15,7 @@
 from ._utils import hash_dict, is_valid_number, get_current_timestamp
 from .acs.acs_dataset import ACSDataset
 from .acs.acs_tasks import ACSTaskMetadata
-from .classifier import LLMClassifier, TransformersLLMClassifier, WebAPILLMClassifier
+from .classifier import LLMClassifier, TransformersLLMClassifier, VLLMClassifier, WebAPILLMClassifier
 from .dataset import Dataset
 from .evaluation import evaluate_predictions
 from .plotting import render_evaluation_plots, render_fairness_plots
@@ -443,6 +443,8 @@ def make_acs_benchmark(
         data_dir: str | Path = None,
         max_api_rpm: int = None,
         config: BenchmarkConfig = BenchmarkConfig.default_config(),
+        backend: str | None = None,
+        model_name_or_path: str | Path | None = None,
         **kwargs,
     ) -> Benchmark:
         """Create a standardized calibration benchmark on ACS data.
@@ -509,8 +511,36 @@ def make_acs_benchmark(
             tokenizer=tokenizer,
             max_api_rpm=max_api_rpm,
             config=config,
+            backend=backend,
+            model_name_or_path=model_name_or_path,
         )
 
+    @staticmethod
+    def _resolve_backend(*, backend: str | None, model) -> str:
+        """Pick the inference backend for this benchmark run.
+
+        Explicit `backend` overrides autodetection. Autodetection rules:
+        - `model` is a string -> "webapi" (model ID for litellm).
+        - `model` is a `vllm.LLM`-like object (has `.generate` and `.get_tokenizer`) -> "vllm".
+        - Otherwise -> "transformers".
+        """
+        if backend is not None:
+            backend = backend.lower()
+            if backend not in {"transformers", "vllm", "webapi"}:
+                raise ValueError(
+                    f"Unknown inference backend '{backend}'. "
+                    f"Expected one of: 'transformers', 'vllm', 'webapi'."
+                )
+            return backend
+
+        if isinstance(model, str):
+            return "webapi"
+        if hasattr(model, "generate") and hasattr(model, "get_tokenizer"):
+            # Duck-typed vLLM `LLM`. transformers models also expose `.generate`,
+            # but not `.get_tokenizer` — that's the discriminator.
+            return "vllm"
+        return "transformers"
+
     @staticmethod
     def _configure_task_question(task: TaskMetadata, config: BenchmarkConfig) -> None:
         """Pick the Q&A interface (reasoning / numeric / multiple-choice) on `task`."""
@@ -645,6 +675,8 @@ def make_benchmark(
         tokenizer: AutoTokenizer = None,    # WebAPI models have no local tokenizer
         max_api_rpm: int = None,
         config: BenchmarkConfig = BenchmarkConfig.default_config(),
+        backend: str | None = None,
+        model_name_or_path: str | Path | None = None,
         **kwargs,
     ) -> Benchmark:
         """Create a calibration benchmark from a given configuration.
@@ -710,8 +742,9 @@ def make_benchmark(
         if max_api_rpm is not None and isinstance(model, str):
             llm_inference_kwargs["max_api_rpm"] = max_api_rpm
 
-        # Create LLMClassifier object
-        if isinstance(model, str):
+        resolved_backend = cls._resolve_backend(backend=backend, model=model)
+
+        if resolved_backend == "webapi":
             llm_clf = WebAPILLMClassifier(
                 model_name=model,
                 task=task,
@@ -720,7 +753,18 @@ def make_benchmark(
             )
             logging.info(f"Using webAPI model: {model}")
 
-        else:
+        elif resolved_backend == "vllm":
+            llm_clf = VLLMClassifier(
+                llm=model,
+                tokenizer=tokenizer,
+                task=task,
+                model_name_or_path=model_name_or_path,
+                encode_row=encode_row_function,
+                **llm_inference_kwargs,
+            )
+            logging.info(f"Using local vLLM model: {llm_clf.model_name}")
+
+        else:  # transformers
             llm_clf = TransformersLLMClassifier(
                 model=model,
                 tokenizer=tokenizer,
 
@@ -1,3 +1,4 @@
 from .base import LLMClassifier  # noqa: F401
 from .transformers_classifier import TransformersLLMClassifier  # noqa: F401
+from .vllm_classifier import VLLMClassifier  # noqa: F401
 from .web_api_classifier import WebAPILLMClassifier  # noqa: F401