From c15b5a2b35f5e42fac4a7d5642e73b7a6bdfb477 Mon Sep 17 00:00:00 2001 From: jimbojw Date: Thu, 19 Feb 2026 14:49:31 -0500 Subject: [PATCH 1/6] docs: Add AGENTS.md files --- AGENTS.md | 128 ++++++++++++++++++++++++++++++++++++++++++++ frontend/AGENTS.md | 109 +++++++++++++++++++++++++++++++++++++ functions/AGENTS.md | 86 +++++++++++++++++++++++++++++ scripts/AGENTS.md | 62 +++++++++++++++++++++ utils/AGENTS.md | 83 ++++++++++++++++++++++++++++ 5 files changed, 468 insertions(+) create mode 100644 AGENTS.md create mode 100644 frontend/AGENTS.md create mode 100644 functions/AGENTS.md create mode 100644 scripts/AGENTS.md create mode 100644 utils/AGENTS.md diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 000000000..8f9e283d0 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,128 @@ +# AGENTS.md — Deliberate Lab (root) + +Deliberate Lab is a platform for running online research experiments on +human + LLM group dynamics. This file orients AI coding assistants to the +monorepo structure, conventions, and workflows. + +## Keeping AGENTS.md in sync + +These `AGENTS.md` files document conventions that should be followed by +default. If a user request conflicts with the guidance here, **raise the +concern** — ask which takes precedence (their idea or the documented +convention) before proceeding. Whichever direction is chosen, update the +relevant `AGENTS.md` file(s) to match so documentation and code stay in +sync. + +## Architecture + +This is an npm workspaces monorepo with three packages: + +| Workspace | Path | Purpose | +|-----------|------|---------| +| `@deliberation-lab/utils` | `utils/` | Shared TypeScript types, validation, and utilities | +| `functions` | `functions/` | Firebase Cloud Functions (HTTP endpoints + Firestore triggers) | +| `deliberate-lab` | `frontend/` | Lit Element + MobX single-page app (Webpack) | + +Other top-level directories: + +- `firestore/` — Firestore security rules, database rules, and indexes +- `docs/` — Jekyll documentation site (GitHub Pages) +- `scripts/` — pip-installable Python API client (`deliberate_lab`) with auto-generated Pydantic types + a Node.js doctor script (see `scripts/AGENTS.md`) +- `emulator_test_config/` — Static config for Firebase emulator imports + +### Dependency graph + +``` +utils ──► frontend + │ + └────► functions +``` + +`utils` is a shared library consumed by both `frontend` and `functions`. +**Always build `utils` first** — the other two workspaces depend on it. + +## Getting started + +- **Node ≥22** is required (see `.nvmrc`) +- Install all dependencies from the repo root: `npm ci` +- Run everything locally: `./run_locally.sh` + +> [!IMPORTANT] +> Always run npm commands from the **repository root** using the `--workspace` +> (or `-w`) flag. Do **not** `cd` into subdirectories. +> +> ```sh +> npm run build -w utils +> npm test -w functions +> npm run start -w frontend +> ``` +> +> This matches the convention used in `cloudbuild.yaml` and ensures +> consistent dependency resolution via npm workspaces. + +## Linting & formatting + +- **Prettier** formats `.json`, `.ts`, `.html`, `.scss`, and `.css` files +- **ESLint** with `@typescript-eslint`; `@typescript-eslint/no-explicit-any` + is set to `error` — do not use `any` +- **Husky** + **lint-staged** runs Prettier and ESLint on pre-commit +- Frontend files get browser globals; everything else gets Node globals + +## CI + +`cloudbuild.yaml` drives all builds. The `_DEPLOYMENT_TYPE` substitution +variable controls which steps run: + +| Value | What it does | +|-------|-------------| +| `test` | Lint, format check, and unit tests for all workspaces (no deploy) | +| `functions` | Build + deploy Cloud Functions | +| `frontend` | Build + deploy frontend to App Engine | +| `rules` | Deploy Firestore security rules | +| `indexes` | Deploy Firestore indexes | +| `all` | All of the above | + +GitHub Actions (`.github/workflows/ci.yaml`) also runs a **schema sync +check**: if types in `utils` change, `docs/assets/api/schemas.json` and +`scripts/deliberate_lab/types.py` must be regenerated or CI will fail. +Run `npm run update-schemas` from the repo root to fix this. + +## Testing + +Each workspace has its own `npm test`: + +```sh +npm test -w utils # Jest; unit tests colocated with source +npm test -w functions # Jest; requires Java 21 for Firebase emulator +npm test -w frontend # Jest +``` + +Functions tests run against the Firebase emulator using +`firebase-test.json`. The integration test in +`cohort_definitions.integration.test.ts` is large and slow. + +## Firebase config + +| File | Purpose | +|------|---------| +| `firebase.json` | Local dev emulator config | +| `firebase-test.json` | Emulator config for CI / test runs | +| `.firebaserc.example` | Template for project aliases (copy to `.firebaserc`) | +| `firestore/firestore.rules` | Firestore security rules | +| `firestore/database.rules.json` | Realtime Database rules | +| `firestore/storage.rules` | Cloud Storage rules | +| `firestore/indexes.json` | Firestore composite indexes | + +## Stage system + +Experiments are composed of ordered **stages** (chat, survey, chip +negotiation, ranking, etc.). The `StageKind` enum in +`utils/src/stages/stage.ts` lists all stage types. + +Adding a new stage type touches **all three workspaces**: + +1. **`utils/src/stages/`** — types, validation, manager, prompts +2. **`functions/src/stages/`** — backend endpoint + trigger logic +3. **`frontend/src/components/stages/`** — config, preview, and answer UI components + +See each workspace's `AGENTS.md` for detailed guidance. diff --git a/frontend/AGENTS.md b/frontend/AGENTS.md new file mode 100644 index 000000000..3b4c89412 --- /dev/null +++ b/frontend/AGENTS.md @@ -0,0 +1,109 @@ +# AGENTS.md — `frontend` + +Lit Element + MobX single-page application built with Webpack. +See also `frontend/README.md` for additional context on routing, +Firebase setup, and experiment configuration. + +## Build & test + +From the **repository root**: + +```sh +npm run start -w frontend # Build + serve (dev, localhost:4201) +npm run build -w frontend # Dev build only +npm run build:prod -w frontend # Production build +npm test -w frontend # Run Jest tests +``` + +For local dev, copy config files first (or use `run_locally.sh` which +handles this automatically): + +```sh +cp frontend/firebase_config.example.ts frontend/firebase_config.ts +cp frontend/index.example.html frontend/index.html +``` + +## Component architecture + +### `src/components/` — Feature components + +Organized into 17 subdirectories by feature area: + +| Directory | Purpose | +|-----------|---------| +| `stages/` | Stage config, preview, and answer components (~87 files) | +| `experiment_builder/` | Experiment creation/editing UI | +| `experiment_dashboard/` | Experiment monitoring dashboard | +| `experimenter/` | Experimenter-facing views | +| `participant_view/` | Participant-facing experiment UI | +| `chat/` | Chat interface components | +| `header/` | App header (includes routing logic) | +| `sidenav/` | Side navigation (includes routing logic) | +| `login/` | Authentication UI | +| `settings/` | App settings | +| `gallery/` | Experiment gallery/templates | +| `progress/` | Progress indicators | +| `popup/` | Modal/popup components | +| `participant_profile/` | Participant profile display | +| `avatar_picker/` | Avatar selection UI | +| `admin/` | Admin panel | +| `shared/` | Shared component utilities | + +### `src/pair-components/` — Reusable primitives + +Standalone UI primitives: `button`, `icon`, `icon_button`, `textarea`, +`textarea_template`, `tooltip`, `menu`, `info_popup`. These are +general-purpose and not tied to project state. + +`shared.css` in this directory provides base styles used across primitives. + +## Service layer + +MobX-based services in `src/services/`: + +| Service | Role | +|---------|------| +| `firebase.service.ts` | Firebase connection (Firestore, Auth) | +| `auth.service.ts` | Authentication and login state | +| `experiment.manager.ts` | Experiment data management (largest service) | +| `experiment.editor.ts` | Experiment editing state | +| `experiment.service.ts` | Current experiment subscription | +| `participant.service.ts` | Participant state and stage progress | +| `participant.answer.ts` | Participant answer management | +| `cohort.service.ts` | Cohort management | +| `router.service.ts` | App routing and page definitions | +| `home.service.ts` | Home page experiment list | +| `settings.service.ts` | App settings | +| `admin.service.ts` | Admin operations | +| `analytics.service.ts` | Google Analytics | +| `presence.service.ts` | Participant online/offline presence | + +## Stage components + +Stage UI components live in `src/components/stages/`. Each stage type +typically has three components: + +| Pattern | Purpose | +|---------|---------| +| `_config.ts` | Experimenter-facing configuration editor | +| `_preview.ts` | Participant-facing stage view | +| `_answer.ts` | Answer/response display component | + +## Styling + +- **Material 3 Design** via SASS variables in `src/sass/`: + - `_colors.scss` — color palettes and theme tokens + - `_common.scss` — shared mixins and layout utilities + - `_typescale.scss` — typography scale definitions +- `src/pair-components/shared.css` — base styles for primitives +- Use SASS variables and mixins — **do not hardcode** colors, spacing, or + font values + +## Key files + +| File | Role | +|------|------| +| `src/app.ts` | Root app component and page rendering | +| `src/index.ts` | App entry point | +| `src/service_provider.ts` | MobX service dependency injection | +| `src/shared/` | Shared config, constants, and utilities | diff --git a/functions/AGENTS.md b/functions/AGENTS.md new file mode 100644 index 000000000..255465144 --- /dev/null +++ b/functions/AGENTS.md @@ -0,0 +1,86 @@ +# AGENTS.md — `functions` + +Firebase Cloud Functions backend: HTTP callable endpoints and Firestore +document triggers. All functions are registered in `src/index.ts`. + +## Build & test + +From the **repository root**: + +```sh +npm run build -w functions # Build with esbuild +npm test -w functions # Run all tests (unit + emulator) +npm run typecheck -w functions +``` + +Emulator tests require **Java 21** for the Firebase emulator. The test +runner uses `firebase-test.json` (in the repo root) as the emulator config. + +Unit-only tests (no emulator): + +```sh +npx jest --testPathIgnorePatterns= -w functions +``` + +The integration test in `cohort_definitions.integration.test.ts` is large +and slow; it exercises full experiment lifecycle flows. + +## File naming conventions + +| Pattern | Purpose | +|---------|---------| +| `.endpoints.ts` | HTTP callable functions (registered in `src/index.ts`) | +| `.utils.ts` | Business logic and helpers | +| `.utils.test.ts` | Unit tests for business logic | + +### Stage-specific files (`src/stages/`) + +Stage backend logic lives in `src/stages/` and follows these patterns: + +| Pattern | Purpose | +|---------|---------| +| `.endpoints.ts` | Callable endpoints for the stage | +| `.utils.ts` | Stage-specific business logic | +| `.utils.test.ts` | Tests | + +New stage endpoints must be exported from `src/index.ts`. + +## Trigger system + +Firestore document triggers live in `src/triggers/`. See +`src/triggers/README.md` for the full list of triggers and the Firestore +document paths they listen on. + +Key trigger files: + +| File | Listens to | +|------|-----------| +| `participant.triggers.ts` | Participant document create/update | +| `stage.triggers.ts` | Participant stage data and public stage data updates | +| `chat.triggers.ts` | Public and private chat message creation | +| `chip.triggers.ts` | Chip transaction creation | +| `agent_participant.triggers.ts` | Agent participant lifecycle events | +| `presence.triggers.ts` | Realtime Database presence changes | + +## Agent (LLM) participants + +- `agent.utils.ts` — core agent orchestration logic +- `agent_participant.utils.ts` — manages how LLM agents participate in + experiments (stage completion, API calls, answer extraction) +- `structured_prompt.utils.ts` — prompt construction for agent participants + and mediators + +## Data access + +`src/data.ts` is the Firestore data access layer. It provides functions +like `getExperimentDownload` and `getExperimentLogs` using the Firebase +Admin SDK. **Do not write raw Firestore calls in endpoint files** — use or +extend the data access layer instead. + +## Endpoint conventions + +- Endpoints are defined in `*.endpoints.ts` files and exported from + `src/index.ts` +- Each endpoint returns structured responses +- The Express app setup is in `src/app.ts` +- The `src/dl_api/` directory contains the external REST API layer diff --git a/scripts/AGENTS.md b/scripts/AGENTS.md new file mode 100644 index 000000000..1bda89fd5 --- /dev/null +++ b/scripts/AGENTS.md @@ -0,0 +1,62 @@ +# AGENTS.md — `scripts` (Python Client) + +This directory contains the **Deliberate Lab Python client** — a +pip-installable SDK for the REST API — plus a standalone Node.js +diagnostic script. + +## Package structure + +| File | Purpose | +|------|---------| +| `deliberate_lab/client.py` | `Client` class — full REST API client | +| `deliberate_lab/types.py` | ⚠️ **Auto-generated** Pydantic models — do not edit by hand | +| `deliberate_lab/__init__.py` | Public API surface (`Client`, `APIError`, all types) | +| `doctor.js` | Node.js diagnostic script (not part of the Python package) | + +## ⚠️ `types.py` is auto-generated + +`types.py` is generated from JSON schemas produced by the `utils` workspace. +The full pipeline: + +``` +utils/src/export-schemas.ts → docs/assets/api/schemas.json → types.py +``` + +To regenerate, run from the **repository root**: + +```sh +npm run update-schemas +``` + +This builds `utils`, exports JSON schemas, then runs `datamodel-codegen` +to produce Pydantic v2 models. **Never edit `types.py` manually** — your +changes will be overwritten. + +## Development + +Requires **Python 3.12+** and uses **uv** for dependency management: + +```sh +cd scripts +uv sync # Install dependencies +uv run pyright deliberate_lab/ # Type check +``` + +## Installation (as a user) + +```sh +pip install git+https://github.com/PAIR-code/deliberate-lab.git#subdirectory=scripts +``` + +## Usage + +```python +import deliberate_lab as dl + +client = dl.Client() # Uses DL_API_KEY environment variable +experiments = client.list_experiments() +data = client.export_experiment("experiment-id") +``` + +The `Client` class supports `env="prod"` or `env="dev"` (default) to +target production or local emulator endpoints. diff --git a/utils/AGENTS.md b/utils/AGENTS.md new file mode 100644 index 000000000..4df5a7df6 --- /dev/null +++ b/utils/AGENTS.md @@ -0,0 +1,83 @@ +# AGENTS.md — `@deliberation-lab/utils` + +Shared TypeScript types, validation functions, and utilities consumed by +both `frontend` and `functions`. Changes here can cause cascading breakage — +always rebuild and run tests before committing. + +## Build & test + +From the **repository root**: + +```sh +npm run build -w utils # Build with tsup (outputs to dist/) +npm test -w utils # Run Jest tests (colocated with source) +npm run typecheck -w utils +``` + +`utils` must be rebuilt before `frontend` or `functions` pick up changes. +During local dev, `run_locally.sh` starts a watcher automatically. + +## File naming conventions + +### General entities + +| Pattern | Purpose | +|---------|---------| +| `.ts` | Type definitions and interfaces | +| `.validation.ts` | Runtime validation functions | +| `.test.ts` | Tests (colocated with source) | + +### Stage files (`src/stages/`) + +Each stage type follows a consistent naming pattern: + +| Pattern | Purpose | +|---------|---------| +| `_stage.ts` | Stage config types, participant answer types, public data types | +| `_stage.validation.ts` | Validation for stage configs and answers | +| `_stage.manager.ts` | `BaseStageHandler` subclass for agent actions and prompt display | +| `_stage.prompts.ts` | Prompt construction helpers for LLM agents | +| `_stage.utils.ts` | Stage-specific utility functions | +| `_stage.test.ts` | Tests | + +Not every stage type has all of these files — only the ones it needs. + +## How to add a new stage type + +1. **Define the `StageKind`** — add an entry to the `StageKind` enum in + `src/stages/stage.ts` +2. **Create stage files** — add files in `src/stages/` following the naming + pattern above (at minimum `_stage.ts` and + `_stage.validation.ts`) +3. **Add types to the unions** — update the `StageConfig`, + `StageParticipantAnswer`, and `StagePublicData` union types in + `src/stages/stage.ts` +4. **Add transfer migration entry** — update + `STAGE_KIND_REQUIRES_TRANSFER_MIGRATION` in `src/stages/stage.ts` +5. **Register the handler** — add a handler instance in + `src/stages/stage.handler.ts` (or import the default `BaseStageHandler`) +6. **Export JSON schema** — register the new types in + `src/export-schemas.ts` so consumers can validate JSON payloads +7. **Update sibling workspaces** — implement backend logic in + `functions/src/stages/` and UI components in + `frontend/src/components/stages/`; register any new endpoints in + `functions/src/index.ts` + +## Variables system + +`variables.ts`, `variables.utils.ts`, and `variables.template.ts` implement +a template variable system used in prompts and stage descriptions. +`variables.schema.utils.ts` handles schema-level variable processing. +Variables are defined as `VariableDefinition` objects and resolved at +runtime via `resolveTemplateVariables`. + +## Key files + +| File | Role | +|------|------| +| `src/index.ts` | Public API barrel file — all exports | +| `src/stages/stage.ts` | `StageKind` enum, base types, union types | +| `src/stages/stage.handler.ts` | `BaseStageHandler` class for stage actions | +| `src/export-schemas.ts` | JSON schema generation for the docs site | +| `src/structured_prompt.ts` | Structured prompt types (mediator + participant) | +| `src/structured_output.ts` | Structured output parsing | From 01b55310bc3bd69022ad810a306d2e14cccec742 Mon Sep 17 00:00:00 2001 From: jimbojw Date: Thu, 19 Feb 2026 14:57:39 -0500 Subject: [PATCH 2/6] docs: Add backlinks, common pitfalls and other details to AGENTS.md files --- AGENTS.md | 31 ++++++++++++++++++++++++++++--- frontend/AGENTS.md | 7 +++++-- functions/AGENTS.md | 15 ++++++++++++++- scripts/AGENTS.md | 2 ++ utils/AGENTS.md | 8 ++++++++ 5 files changed, 57 insertions(+), 6 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 8f9e283d0..b501c099e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -36,6 +36,8 @@ Other top-level directories: utils ──► frontend │ └────► functions + · + ·····► scripts/types.py (auto-generated via npm run update-schemas) ``` `utils` is a shared library consumed by both `frontend` and `functions`. @@ -48,8 +50,9 @@ utils ──► frontend - Run everything locally: `./run_locally.sh` > [!IMPORTANT] -> Always run npm commands from the **repository root** using the `--workspace` -> (or `-w`) flag. Do **not** `cd` into subdirectories. +> Always run **npm** commands from the **repository root** using the +> `--workspace` (or `-w`) flag. Do **not** `cd` into subdirectories for +> npm operations. > > ```sh > npm run build -w utils @@ -59,6 +62,10 @@ utils ──► frontend > > This matches the convention used in `cloudbuild.yaml` and ensures > consistent dependency resolution via npm workspaces. +> +> Python tooling (`uv`, `pyright`) in `scripts/` is the exception — those +> commands expect to run from the `scripts/` directory where +> `pyproject.toml` lives. ## Linting & formatting @@ -85,7 +92,8 @@ variable controls which steps run: GitHub Actions (`.github/workflows/ci.yaml`) also runs a **schema sync check**: if types in `utils` change, `docs/assets/api/schemas.json` and `scripts/deliberate_lab/types.py` must be regenerated or CI will fail. -Run `npm run update-schemas` from the repo root to fix this. +Run `npm run update-schemas` from the repo root to fix this (requires +Python 3.12+ and `uv` — see `scripts/AGENTS.md` for setup). ## Testing @@ -126,3 +134,20 @@ Adding a new stage type touches **all three workspaces**: 3. **`frontend/src/components/stages/`** — config, preview, and answer UI components See each workspace's `AGENTS.md` for detailed guidance. + +## Common pitfalls + +1. **Forgetting to rebuild `utils`** — `frontend` and `functions` consume + the compiled output in `utils/dist/`. After changing `utils` source, + rebuild it (`npm run build -w utils`) before testing downstream + workspaces, or changes won't be picked up. +2. **Running `npm` from a subdirectory** — always run from the repo root + with `-w ` (see above). +3. **Editing `scripts/deliberate_lab/types.py` by hand** — this file is + auto-generated and will be overwritten by `npm run update-schemas`. +4. **Missing Java 21 for `functions` tests** — the Firebase emulator + requires Java 21. Unit-only tests can be run without it via + `npm run test:unit -w functions`. +5. **Forgetting to run `npm run update-schemas`** — if you change types + in `utils`, CI will fail the schema sync check until schemas are + regenerated. diff --git a/frontend/AGENTS.md b/frontend/AGENTS.md index 3b4c89412..c1ee98ee9 100644 --- a/frontend/AGENTS.md +++ b/frontend/AGENTS.md @@ -4,6 +4,8 @@ Lit Element + MobX single-page application built with Webpack. See also `frontend/README.md` for additional context on routing, Firebase setup, and experiment configuration. +> See also: [root AGENTS.md](../AGENTS.md) for monorepo-wide conventions. + ## Build & test From the **repository root**: @@ -52,8 +54,9 @@ Organized into 17 subdirectories by feature area: ### `src/pair-components/` — Reusable primitives Standalone UI primitives: `button`, `icon`, `icon_button`, `textarea`, -`textarea_template`, `tooltip`, `menu`, `info_popup`. These are -general-purpose and not tied to project state. +`textarea_template`, `tooltip`, `menu`, `info_popup`. Also contains +`types.ts` (shared primitive types) and `utils.ts` (helper functions). +These are general-purpose and not tied to project state. `shared.css` in this directory provides base styles used across primitives. diff --git a/functions/AGENTS.md b/functions/AGENTS.md index 255465144..34f062c4f 100644 --- a/functions/AGENTS.md +++ b/functions/AGENTS.md @@ -3,6 +3,8 @@ Firebase Cloud Functions backend: HTTP callable endpoints and Firestore document triggers. All functions are registered in `src/index.ts`. +> See also: [root AGENTS.md](../AGENTS.md) for monorepo-wide conventions. + ## Build & test From the **repository root**: @@ -19,7 +21,7 @@ runner uses `firebase-test.json` (in the repo root) as the emulator config. Unit-only tests (no emulator): ```sh -npx jest --testPathIgnorePatterns= -w functions +npm run test:unit -w functions ``` The integration test in `cohort_definitions.integration.test.ts` is large @@ -45,6 +47,17 @@ Stage backend logic lives in `src/stages/` and follows these patterns: New stage endpoints must be exported from `src/index.ts`. +## Source directory overview + +| Directory | Purpose | +|-----------|---------| +| `src/stages/` | Stage-specific backend logic | +| `src/triggers/` | Firestore document triggers (see `src/triggers/README.md`) | +| `src/chat/` | Chat-specific utilities | +| `src/dl_api/` | External REST API layer | +| `src/api/` | Internal API utilities | +| `src/utils/` | Shared backend helper functions | + ## Trigger system Firestore document triggers live in `src/triggers/`. See diff --git a/scripts/AGENTS.md b/scripts/AGENTS.md index 1bda89fd5..d53cd9bd9 100644 --- a/scripts/AGENTS.md +++ b/scripts/AGENTS.md @@ -4,6 +4,8 @@ This directory contains the **Deliberate Lab Python client** — a pip-installable SDK for the REST API — plus a standalone Node.js diagnostic script. +> See also: [root AGENTS.md](../AGENTS.md) for monorepo-wide conventions. + ## Package structure | File | Purpose | diff --git a/utils/AGENTS.md b/utils/AGENTS.md index 4df5a7df6..3304c07f0 100644 --- a/utils/AGENTS.md +++ b/utils/AGENTS.md @@ -4,6 +4,8 @@ Shared TypeScript types, validation functions, and utilities consumed by both `frontend` and `functions`. Changes here can cause cascading breakage — always rebuild and run tests before committing. +> See also: [root AGENTS.md](../AGENTS.md) for monorepo-wide conventions. + ## Build & test From the **repository root**: @@ -41,6 +43,12 @@ Each stage type follows a consistent naming pattern: | `_stage.test.ts` | Tests | Not every stage type has all of these files — only the ones it needs. +For example: + +- **Minimal** (`tos_stage`): `.ts`, `.validation.ts`, `.manager.ts`, + `.prompts.ts` — just the basics +- **Full** (`survey_stage`): `.ts`, `.validation.ts`, `.manager.ts`, + `.prompts.ts`, `.prompts.test.ts` — includes prompt tests ## How to add a new stage type From cf2f87f5718eab0da0b6c574e12f34b658c040d3 Mon Sep 17 00:00:00 2001 From: jimbojw Date: Fri, 20 Feb 2026 11:41:47 -0500 Subject: [PATCH 3/6] build: Mark generated types.py file as AUTO-GENERATED to discourage direct editing --- package.json | 2 +- scripts/AGENTS.md | 1 + scripts/deliberate_lab/types.py | 3 ++ scripts/update_schemas.sh | 63 +++++++++++++++++++++++++++++++++ 4 files changed, 68 insertions(+), 1 deletion(-) create mode 100755 scripts/update_schemas.sh diff --git a/package.json b/package.json index c8ef20897..b44375e9d 100644 --- a/package.json +++ b/package.json @@ -30,7 +30,7 @@ "scripts": { "doctor": "node scripts/doctor.js", "prepare": "husky", - "update-schemas": "npm run build --workspace=utils && npx tsx utils/src/export-schemas.ts && npx prettier --write docs/assets/api/schemas.json && cd scripts && uv run datamodel-codegen --input ../docs/assets/api/schemas.json --output deliberate_lab/types.py --input-file-type jsonschema --reuse-model --collapse-root-models --use-union-operator --use-title-as-name --use-one-literal-as-default --use-default --use-annotated --use-standard-collections --target-python-version 3.12 --output-model-type pydantic_v2.BaseModel --allow-population-by-field-name --custom-file-header '# pyright: reportInvalidTypeForm=false\n# pytype: disable=invalid-function-definition\n# pylint: disable=missing-module-docstring,missing-class-docstring,invalid-name,too-few-public-methods' && uv run black deliberate_lab/ && uv run pyright deliberate_lab/" + "update-schemas": "./scripts/update_schemas.sh" }, "workspaces": [ "frontend", diff --git a/scripts/AGENTS.md b/scripts/AGENTS.md index d53cd9bd9..080b47755 100644 --- a/scripts/AGENTS.md +++ b/scripts/AGENTS.md @@ -13,6 +13,7 @@ diagnostic script. | `deliberate_lab/client.py` | `Client` class — full REST API client | | `deliberate_lab/types.py` | ⚠️ **Auto-generated** Pydantic models — do not edit by hand | | `deliberate_lab/__init__.py` | Public API surface (`Client`, `APIError`, all types) | +| `update_schemas.sh` | Schema regeneration pipeline (called by `npm run update-schemas`) | | `doctor.js` | Node.js diagnostic script (not part of the Python package) | ## ⚠️ `types.py` is auto-generated diff --git a/scripts/deliberate_lab/types.py b/scripts/deliberate_lab/types.py index 86d654f70..6cdb4b60c 100644 --- a/scripts/deliberate_lab/types.py +++ b/scripts/deliberate_lab/types.py @@ -1,3 +1,6 @@ +# AUTO-GENERATED FILE — DO NOT EDIT BY HAND. +# Regenerate via: npm run update-schemas (from repo root). +# # pyright: reportInvalidTypeForm=false # pytype: disable=invalid-function-definition # pylint: disable=missing-module-docstring,missing-class-docstring,invalid-name,too-few-public-methods diff --git a/scripts/update_schemas.sh b/scripts/update_schemas.sh new file mode 100755 index 000000000..47f010b6b --- /dev/null +++ b/scripts/update_schemas.sh @@ -0,0 +1,63 @@ +#!/usr/bin/env bash +# update_schemas.sh — Regenerate Python types from TypeScript schemas. +# +# Usage: Run from the repository root: +# npm run update-schemas +# # or directly: +# ./scripts/update_schemas.sh +# +# Prerequisites: +# - Node.js ≥22 +# - Python 3.12+ with uv (https://docs.astral.sh/uv/) + +set -euo pipefail + +REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)" +cd "$REPO_ROOT" + +# Step 1: Build utils so exported types are up to date. +echo "==> Building utils..." +npm run build --workspace=utils + +# Step 2: Export JSON schemas from TypeScript type definitions. +echo "==> Exporting JSON schemas..." +npx tsx utils/src/export-schemas.ts + +# Step 3: Format the exported schema file. +echo "==> Formatting schemas.json..." +npx prettier --write docs/assets/api/schemas.json + +# Step 4: Generate Pydantic v2 models from the JSON schemas. +echo "==> Generating Python types..." +cd scripts +uv run datamodel-codegen \ + --input ../docs/assets/api/schemas.json \ + --output deliberate_lab/types.py \ + --input-file-type jsonschema \ + --reuse-model \ + --collapse-root-models \ + --use-union-operator \ + --use-title-as-name \ + --use-one-literal-as-default \ + --use-default \ + --use-annotated \ + --use-standard-collections \ + --target-python-version 3.12 \ + --output-model-type pydantic_v2.BaseModel \ + --allow-population-by-field-name \ + --custom-file-header '# AUTO-GENERATED FILE — DO NOT EDIT BY HAND. +# Regenerate via: npm run update-schemas (from repo root). +# +# pyright: reportInvalidTypeForm=false +# pytype: disable=invalid-function-definition +# pylint: disable=missing-module-docstring,missing-class-docstring,invalid-name,too-few-public-methods' + +# Step 5: Format the generated Python code. +echo "==> Formatting Python code..." +uv run black deliberate_lab/ + +# Step 6: Typecheck the generated Python code. +echo "==> Typechecking Python code..." +uv run pyright deliberate_lab/ + +echo "==> Done. schemas.json and types.py are up to date." From bc91e821d5eee3f75cbf0c98d82bb0183644cac6 Mon Sep 17 00:00:00 2001 From: jimbojw Date: Thu, 26 Feb 2026 13:45:24 -0500 Subject: [PATCH 4/6] docs: Add missing details in AGENTS.md files --- AGENTS.md | 27 ++++++++++++++++++++++++--- frontend/AGENTS.md | 23 +++++++++++++++++++++++ functions/AGENTS.md | 18 +++++++++++++++--- scripts/AGENTS.md | 14 ++++++++++++-- utils/AGENTS.md | 19 +++++++++++++++++-- 5 files changed, 91 insertions(+), 10 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index b501c099e..521c8770c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -48,6 +48,13 @@ utils ──► frontend - **Node ≥22** is required (see `.nvmrc`) - Install all dependencies from the repo root: `npm ci` - Run everything locally: `./run_locally.sh` +- Diagnose setup problems: `npm run doctor` + +`run_locally.sh` copies required config files (`.firebaserc`, +`firebase_config.ts`, `index.html`) if they are missing, builds `utils` and +`functions`, starts file watchers for both, launches the Firebase emulators +(with seed data from `emulator_test_config/`), and serves the frontend at +`http://localhost:4201`. > [!IMPORTANT] > Always run **npm** commands from the **repository root** using the @@ -70,9 +77,11 @@ utils ──► frontend ## Linting & formatting - **Prettier** formats `.json`, `.ts`, `.html`, `.scss`, and `.css` files -- **ESLint** with `@typescript-eslint`; `@typescript-eslint/no-explicit-any` - is set to `error` — do not use `any` -- **Husky** + **lint-staged** runs Prettier and ESLint on pre-commit +- **ESLint** uses the **flat config** format (`eslint.config.mjs`); the + project does **not** have a `.eslintrc.*` file +- `@typescript-eslint/no-explicit-any` is set to `error` — do not use `any` +- **Husky** + **lint-staged** runs both Prettier and ESLint on pre-commit + for the same file set (`*.{json,ts,html,scss,css}`) - Frontend files get browser globals; everything else gets Node globals ## CI @@ -121,6 +130,18 @@ Functions tests run against the Firebase emulator using | `firestore/storage.rules` | Cloud Storage rules | | `firestore/indexes.json` | Firestore composite indexes | +## Import convention + +Both `frontend` and `functions` import from `utils` using the npm workspace +package name: + +```ts +import {StageKind, ExperimentConfig} from '@deliberation-lab/utils'; +``` + +Do **not** use relative paths to reach into `utils/src/` — always import +from `@deliberation-lab/utils`. + ## Stage system Experiments are composed of ordered **stages** (chat, survey, chip diff --git a/frontend/AGENTS.md b/frontend/AGENTS.md index c1ee98ee9..8253a3942 100644 --- a/frontend/AGENTS.md +++ b/frontend/AGENTS.md @@ -60,12 +60,20 @@ These are general-purpose and not tied to project state. `shared.css` in this directory provides base styles used across primitives. +> [!NOTE] +> Primitives use a **mix** of `.css` and `.scss` for styles. `button`, +> `icon`, `icon_button`, `textarea`, `textarea_template`, and `tooltip` +> use plain `.css`; `menu` and `info_popup` use `.scss`. Follow the +> existing pattern when modifying a primitive. + ## Service layer MobX-based services in `src/services/`: | Service | Role | |---------|------| +| `service.ts` | Abstract `Service` base class (all services extend this) | +| `initialization.service.ts` | Orchestrates app startup (initializes analytics, Firebase, routing) | | `firebase.service.ts` | Firebase connection (Firestore, Auth) | | `auth.service.ts` | Authentication and login state | | `experiment.manager.ts` | Experiment data management (largest service) | @@ -81,6 +89,11 @@ MobX-based services in `src/services/`: | `analytics.service.ts` | Google Analytics | | `presence.service.ts` | Participant online/offline presence | +Services are wired together via `src/service_provider.ts`, which +creates all service instances and injects dependencies. Components access +services through the service provider — see existing components for the +pattern. + ## Stage components Stage UI components live in `src/components/stages/`. Each stage type @@ -110,3 +123,13 @@ typically has three components: | `src/index.ts` | App entry point | | `src/service_provider.ts` | MobX service dependency injection | | `src/shared/` | Shared config, constants, and utilities | + +## Common pitfalls + +1. **Hardcoding colors or spacing** — use SASS variables and mixins from + `src/sass/`. Do not hardcode hex colors, pixel sizes, or font values. +2. **Mixing CSS formats in pair-components** — check the existing style file + format (`.css` vs `.scss`) before editing a primitive's styles. +3. **Forgetting to register a new service** — new services must be + instantiated in `src/service_provider.ts` and extend the `Service` base + class. diff --git a/functions/AGENTS.md b/functions/AGENTS.md index 34f062c4f..18ad23a87 100644 --- a/functions/AGENTS.md +++ b/functions/AGENTS.md @@ -1,7 +1,8 @@ # AGENTS.md — `functions` -Firebase Cloud Functions backend: HTTP callable endpoints and Firestore -document triggers. All functions are registered in `src/index.ts`. +Firebase Cloud Functions backend (built with **esbuild**): HTTP callable +endpoints and Firestore document triggers. All functions are registered in +`src/index.ts`. > See also: [root AGENTS.md](../AGENTS.md) for monorepo-wide conventions. @@ -95,5 +96,16 @@ extend the data access layer instead. - Endpoints are defined in `*.endpoints.ts` files and exported from `src/index.ts` - Each endpoint returns structured responses -- The Express app setup is in `src/app.ts` +- `src/app.ts` initializes the `StageManager` (from `utils`) which maps + stage types to their handler logic - The `src/dl_api/` directory contains the external REST API layer + (key-authenticated HTTP endpoints for programmatic access) + +## Common pitfalls + +1. **Writing raw Firestore calls in endpoint files** — use or extend the + data access layer in `src/data.ts` instead. +2. **Forgetting to export new endpoints from `src/index.ts`** — Cloud + Functions will silently ignore unregistered functions. +3. **Missing Java 21 for emulator tests** — use `npm run test:unit -w + functions` to run unit tests without the emulator. diff --git a/scripts/AGENTS.md b/scripts/AGENTS.md index 080b47755..e5734c0a6 100644 --- a/scripts/AGENTS.md +++ b/scripts/AGENTS.md @@ -14,7 +14,7 @@ diagnostic script. | `deliberate_lab/types.py` | ⚠️ **Auto-generated** Pydantic models — do not edit by hand | | `deliberate_lab/__init__.py` | Public API surface (`Client`, `APIError`, all types) | | `update_schemas.sh` | Schema regeneration pipeline (called by `npm run update-schemas`) | -| `doctor.js` | Node.js diagnostic script (not part of the Python package) | +| `doctor.js` | Node.js health-check script: verifies Node version, config files, and build outputs (run via `npm run doctor` from repo root) | ## ⚠️ `types.py` is auto-generated @@ -35,7 +35,12 @@ This builds `utils`, exports JSON schemas, then runs `datamodel-codegen` to produce Pydantic v2 models. **Never edit `types.py` manually** — your changes will be overwritten. -## Development +> [!NOTE] +> The pipeline requires `utils` to be built first. If you have changed +> `utils` source, `npm run update-schemas` will rebuild it automatically +> via the `update_schemas.sh` script. + +## Build & test Requires **Python 3.12+** and uses **uv** for dependency management: @@ -63,3 +68,8 @@ data = client.export_experiment("experiment-id") The `Client` class supports `env="prod"` or `env="dev"` (default) to target production or local emulator endpoints. + +## Testing + +There are no automated tests for the Python client. The end-to-end schema +sync is validated by the CI schema check (see root `AGENTS.md`). diff --git a/utils/AGENTS.md b/utils/AGENTS.md index 3304c07f0..5b19ac6bf 100644 --- a/utils/AGENTS.md +++ b/utils/AGENTS.md @@ -76,8 +76,11 @@ For example: `variables.ts`, `variables.utils.ts`, and `variables.template.ts` implement a template variable system used in prompts and stage descriptions. `variables.schema.utils.ts` handles schema-level variable processing. -Variables are defined as `VariableDefinition` objects and resolved at -runtime via `resolveTemplateVariables`. + +Variables are defined as `VariableDefinition` objects (name, description, +default value) and referenced in text using double-brace syntax (e.g., +`{{variable_name}}`). At runtime, `resolveTemplateVariables` replaces +placeholders with their resolved values. ## Key files @@ -89,3 +92,15 @@ runtime via `resolveTemplateVariables`. | `src/export-schemas.ts` | JSON schema generation for the docs site | | `src/structured_prompt.ts` | Structured prompt types (mediator + participant) | | `src/structured_output.ts` | Structured output parsing | + +## Common pitfalls + +1. **Forgetting to update union types** — when adding a new stage, you must + add it to the `StageConfig`, `StageParticipantAnswer`, and + `StagePublicData` unions in `src/stages/stage.ts`. +2. **Forgetting to rebuild** — `frontend` and `functions` consume compiled + output from `utils/dist/`. After changing source, rebuild with + `npm run build -w utils` or rely on the watcher started by + `run_locally.sh`. +3. **Editing `scripts/deliberate_lab/types.py` by hand** — this file is + auto-generated. Run `npm run update-schemas` instead. From ae7557d0d5b028f55df55a413e93048a5e327548 Mon Sep 17 00:00:00 2001 From: jimbojw Date: Thu, 26 Feb 2026 13:58:01 -0500 Subject: [PATCH 5/6] docs: Add information revealed by git history analysis as useful for an AI agent working on this project --- frontend/AGENTS.md | 1 + functions/AGENTS.md | 54 +++++++++++++++++++++++++++++++++++++++++++-- scripts/AGENTS.md | 12 ++++++++++ utils/AGENTS.md | 25 +++++++++++++++++++++ 4 files changed, 90 insertions(+), 2 deletions(-) diff --git a/frontend/AGENTS.md b/frontend/AGENTS.md index 8253a3942..cf5896490 100644 --- a/frontend/AGENTS.md +++ b/frontend/AGENTS.md @@ -122,6 +122,7 @@ typically has three components: | `src/app.ts` | Root app component and page rendering | | `src/index.ts` | App entry point | | `src/service_provider.ts` | MobX service dependency injection | +| `src/shared/callables.ts` | Typed wrappers for all Cloud Function calls (must be updated when adding new callable endpoints) | | `src/shared/` | Shared config, constants, and utilities | ## Common pitfalls diff --git a/functions/AGENTS.md b/functions/AGENTS.md index 18ad23a87..ae7bee43a 100644 --- a/functions/AGENTS.md +++ b/functions/AGENTS.md @@ -55,10 +55,19 @@ New stage endpoints must be exported from `src/index.ts`. | `src/stages/` | Stage-specific backend logic | | `src/triggers/` | Firestore document triggers (see `src/triggers/README.md`) | | `src/chat/` | Chat-specific utilities | -| `src/dl_api/` | External REST API layer | -| `src/api/` | Internal API utilities | +| `src/dl_api/` | External REST API layer (see below) | +| `src/api/` | Internal API utilities (LLM provider integrations) | | `src/utils/` | Shared backend helper functions | +### Key files + +| File | Role | +|------|---------| +| `src/index.ts` | Registers all Cloud Functions (callables + triggers) | +| `src/app.ts` | Initializes `StageManager` — maps stage types to handlers | +| `src/data.ts` | Firestore data access layer (reads/exports) | +| `src/participant.utils.ts` | Participant lifecycle — stage progression, transfers, cohort assignment (~1400 lines, the largest and most complex backend file) | + ## Trigger system Firestore document triggers live in `src/triggers/`. See @@ -101,6 +110,43 @@ extend the data access layer instead. - The `src/dl_api/` directory contains the external REST API layer (key-authenticated HTTP endpoints for programmatic access) +### REST API structure (`src/dl_api/`) + +The REST API is an Express app exposed as a single Cloud Function. It +uses API key authentication (not Firebase Auth) for server-to-server +access. + +| File | Purpose | +|------|---------| +| `dl_api.endpoints.ts` | Express app setup, middleware, route registration | +| `experiments.dl_api.ts` | `/v1/experiments/` route handlers | +| `cohorts.dl_api.ts` | `/v1/experiments/:id/cohorts/` route handlers | +| `dl_api.utils.ts` | Auth middleware and request validation | +| `dl_api_key.utils.ts` | API key creation, verification, and revocation | +| `dl_api.test.utils.ts` | Shared test setup utilities | + +## How to add a new REST API endpoint + +Adding a REST API endpoint touches multiple files across workspaces: + +1. **Implement the route handler** — add a function in the appropriate + `*.dl_api.ts` file (or create a new one for a new resource type) +2. **Register the route** — add the Express route in + `dl_api.endpoints.ts` +3. **Update the OpenAPI spec** — add the endpoint to + `docs/assets/api/openapi.yaml` +4. **Add a Python client method** — update + `scripts/deliberate_lab/client.py` so the SDK exposes the new endpoint +5. **Write integration tests** — add tests in + `*.dl_api.integration.test.ts` + +> [!NOTE] +> REST API endpoints (`/v1/...`) use API key auth and are for +> server-to-server use. **Frontend callables** (used by the web app) are +> separate — they are Firebase `onCall` functions registered in +> `src/index.ts` with corresponding wrappers in +> `frontend/src/shared/callables.ts`. + ## Common pitfalls 1. **Writing raw Firestore calls in endpoint files** — use or extend the @@ -109,3 +155,7 @@ extend the data access layer instead. Functions will silently ignore unregistered functions. 3. **Missing Java 21 for emulator tests** — use `npm run test:unit -w functions` to run unit tests without the emulator. +4. **Firestore race conditions** — concurrent writes to the same document + (e.g., multiple participants updating public stage data simultaneously) + are a recurring source of bugs. Use Firestore **transactions** for + read-modify-write operations on shared documents. diff --git a/scripts/AGENTS.md b/scripts/AGENTS.md index e5734c0a6..1296aa47a 100644 --- a/scripts/AGENTS.md +++ b/scripts/AGENTS.md @@ -69,6 +69,18 @@ data = client.export_experiment("experiment-id") The `Client` class supports `env="prod"` or `env="dev"` (default) to target production or local emulator endpoints. +## REST API connection + +The Python client in `client.py` is the SDK for the REST API implemented +in `functions/src/dl_api/`. When a new REST API endpoint is added, the +matching Python method should be added here. The API spec lives in +`docs/assets/api/openapi.yaml`. + +``` +functions/src/dl_api/ → docs/assets/api/openapi.yaml → client.py +(implementation) (API spec) (Python SDK) +``` + ## Testing There are no automated tests for the Python client. The end-to-end schema diff --git a/utils/AGENTS.md b/utils/AGENTS.md index 5b19ac6bf..58a017b31 100644 --- a/utils/AGENTS.md +++ b/utils/AGENTS.md @@ -19,6 +19,12 @@ npm run typecheck -w utils `utils` must be rebuilt before `frontend` or `functions` pick up changes. During local dev, `run_locally.sh` starts a watcher automatically. +> [!TIP] +> If you change **any exported type** in `utils`, you must also run +> `npm run update-schemas` from the repo root to regenerate +> `docs/assets/api/schemas.json` and `scripts/deliberate_lab/types.py`. +> CI will fail the schema sync check if these are out of date. + ## File naming conventions ### General entities @@ -71,6 +77,25 @@ For example: `frontend/src/components/stages/`; register any new endpoints in `functions/src/index.ts` +## How to modify an existing stage + +Modifying an existing stage is the most common type of change. The files +you need to touch depend on what you're changing: + +| What you're changing | Files to update | +|---------------------|-----------------| +| Stage config options | `utils/src/stages/_stage.ts` (types), `_stage.validation.ts` (defaults), and the frontend config editor in `frontend/src/components/stages/_config.ts` | +| Participant answer shape | `utils/src/stages/_stage.ts` (answer type + union in `stage.ts`), `_stage.validation.ts`, and the frontend answer component | +| Public stage data | Same as above, plus any trigger or backend logic in `functions/src/stages/` that reads/writes public data | +| LLM agent behavior | `utils/src/stages/_stage.manager.ts` and/or `_stage.prompts.ts` | +| Backend logic only | `functions/src/stages/.utils.ts` and/or `.endpoints.ts` | +| UI only | `frontend/src/components/stages/_*.ts` and SCSS files | + +After any type change in `utils`, remember to: +1. Rebuild: `npm run build -w utils` +2. Regenerate schemas: `npm run update-schemas` +3. Run tests: `npm test -w utils` + ## Variables system `variables.ts`, `variables.utils.ts`, and `variables.template.ts` implement From 98e9e48fb157c1692e61506c2bbd18663da2336f Mon Sep 17 00:00:00 2001 From: jimbojw Date: Thu, 26 Feb 2026 14:20:55 -0500 Subject: [PATCH 6/6] docs: Add common pitfalls, instructions on adding callable endpoints, and Firestore data model --- frontend/AGENTS.md | 4 ++-- functions/AGENTS.md | 45 ++++++++++++++++++++++++++++++++++++++++++--- scripts/AGENTS.md | 12 ++++++++++++ 3 files changed, 56 insertions(+), 5 deletions(-) diff --git a/frontend/AGENTS.md b/frontend/AGENTS.md index cf5896490..ad0092f48 100644 --- a/frontend/AGENTS.md +++ b/frontend/AGENTS.md @@ -33,7 +33,7 @@ Organized into 17 subdirectories by feature area: | Directory | Purpose | |-----------|---------| -| `stages/` | Stage config, preview, and answer components (~87 files) | +| `stages/` | Stage config, preview, and answer components (largest subdirectory) | | `experiment_builder/` | Experiment creation/editing UI | | `experiment_dashboard/` | Experiment monitoring dashboard | | `experimenter/` | Experimenter-facing views | @@ -122,7 +122,7 @@ typically has three components: | `src/app.ts` | Root app component and page rendering | | `src/index.ts` | App entry point | | `src/service_provider.ts` | MobX service dependency injection | -| `src/shared/callables.ts` | Typed wrappers for all Cloud Function calls (must be updated when adding new callable endpoints) | +| `src/shared/callables.ts` | Typed wrappers for **all** Cloud Function calls — every new callable endpoint in `functions` needs a corresponding wrapper here or it is unreachable from the UI | | `src/shared/` | Shared config, constants, and utilities | ## Common pitfalls diff --git a/functions/AGENTS.md b/functions/AGENTS.md index ae7bee43a..8cacd190f 100644 --- a/functions/AGENTS.md +++ b/functions/AGENTS.md @@ -66,7 +66,7 @@ New stage endpoints must be exported from `src/index.ts`. | `src/index.ts` | Registers all Cloud Functions (callables + triggers) | | `src/app.ts` | Initializes `StageManager` — maps stage types to handlers | | `src/data.ts` | Firestore data access layer (reads/exports) | -| `src/participant.utils.ts` | Participant lifecycle — stage progression, transfers, cohort assignment (~1400 lines, the largest and most complex backend file) | +| `src/participant.utils.ts` | Participant lifecycle — stage progression, transfers, cohort assignment (~1400 lines, the largest and most complex backend file). Handles complex Firestore transaction chains; changes here carry a high risk of subtle race conditions. | ## Trigger system @@ -147,15 +147,54 @@ Adding a REST API endpoint touches multiple files across workspaces: > `src/index.ts` with corresponding wrappers in > `frontend/src/shared/callables.ts`. +## How to add a new callable endpoint + +Callable endpoints are Firebase `onCall` functions invoked by the +frontend. This is the most common type of new endpoint. + +1. **Implement the function** — create or update a `*.endpoints.ts` file + with the new `onCall` function +2. **Export from `src/index.ts`** — add the export so Firebase registers + the function +3. **Add a frontend wrapper** — add a typed callable wrapper in + `frontend/src/shared/callables.ts` (the frontend calls endpoints + exclusively through these wrappers) +4. **Wire up the UI** — call the new wrapper from the appropriate + service or component + +## Firestore data model + +The Firestore document hierarchy (derived from trigger paths and the +data access layer): + +``` +experiments/{experimentId} +├── participants/{participantId} +│ └── stageData/{stageId} # participant answers +│ └── privateChats/{chatId} # private chat messages +├── cohorts/{cohortId} +│ └── publicStageData/{stageId} # public stage data +│ ├── chats/{chatId} # group chat messages +│ └── transactions/{transactionId} # chip transactions +└── stages/{stageId} # stage config +``` + +Triggers listen on these paths — see `src/triggers/README.md` for the +full list of trigger functions and the specific paths they watch. + ## Common pitfalls 1. **Writing raw Firestore calls in endpoint files** — use or extend the data access layer in `src/data.ts` instead. 2. **Forgetting to export new endpoints from `src/index.ts`** — Cloud Functions will silently ignore unregistered functions. -3. **Missing Java 21 for emulator tests** — use `npm run test:unit -w +3. **Forgetting to add a `callables.ts` wrapper** — the frontend calls + all callable endpoints through typed wrappers in + `frontend/src/shared/callables.ts`. A new endpoint without a wrapper + is unreachable from the UI. +4. **Missing Java 21 for emulator tests** — use `npm run test:unit -w functions` to run unit tests without the emulator. -4. **Firestore race conditions** — concurrent writes to the same document +5. **Firestore race conditions** — concurrent writes to the same document (e.g., multiple participants updating public stage data simultaneously) are a recurring source of bugs. Use Firestore **transactions** for read-modify-write operations on shared documents. diff --git a/scripts/AGENTS.md b/scripts/AGENTS.md index 1296aa47a..fb4695639 100644 --- a/scripts/AGENTS.md +++ b/scripts/AGENTS.md @@ -85,3 +85,15 @@ functions/src/dl_api/ → docs/assets/api/openapi.yaml → client.py There are no automated tests for the Python client. The end-to-end schema sync is validated by the CI schema check (see root `AGENTS.md`). + +## Common pitfalls + +1. **Editing `types.py` by hand** — this file is auto-generated and will + be overwritten by `npm run update-schemas`. Always modify the source + types in `utils` instead. +2. **Looking for Python tests** — there are no automated tests for the + Python client yet. Do not create a `pytest` configuration expecting + existing tests to exist. +3. **Running `uv` from the repo root** — unlike `npm` commands, Python + tooling (`uv`, `pyright`) must be run from the `scripts/` directory + where `pyproject.toml` lives.