Merge pull request #73 from gmickel/feat/fn-40-structured-query-documents

feat: add structured query documents

Author: gmickel

.flow/tasks/fn-40-structured-query-document-syntax.1.json

@@ -8,7 +8,7 @@
   "id": "fn-40-structured-query-document-syntax.1",
   "priority": 1,
   "spec_path": ".flow/tasks/fn-40-structured-query-document-syntax.1.md",
-  "status": "todo",
+  "status": "done",
   "title": "Design and implement first-class structured query documents",
-  "updated_at": "2026-03-10T15:34:15.453535Z"
+  "updated_at": "2026-03-10T18:10:00.000000Z"
 }

.flow/tasks/fn-40-structured-query-document-syntax.1.md

@@ -20,10 +20,17 @@ Design and implement first-class multi-line structured query documents using exi
 
 ## Done summary
 
-TBD
+Implemented first-class structured multi-line query documents using GNO naming only: `term`, `intent`, and `hyde`. Added a shared parser/normalizer, rolled it through CLI `query`/`ask`, REST `query`/`ask`, MCP `gno_query`, SDK `query`/`ask`, and Web Search/Ask text boxes, then added parser/CLI/API/SDK coverage plus full docs/website updates including a dedicated syntax reference page.
+
+Key decisions:
+
+- structured syntax only activates for multi-line query input, so single-line queries remain unchanged
+- plain untyped lines become the base query; if absent, GNO derives the base query from `term:` lines first, then `intent:` lines
+- `hyde:` is never searched directly and hyde-only documents are rejected
+- explicit `queryModes` and document-derived modes merge, with shared validation across the combined set
 
 ## Evidence
 
 - Commits:
-- Tests:
+- Tests: bun run lint:check, bun test, bun run docs:verify, cd website && mise x -- make build, bun test test/core/structured-query.test.ts test/serve/routes/query.test.ts test/sdk/client.test.ts test/cli/structured-query-document.test.ts --timeout 60000
 - PRs:

CHANGELOG.md

@@ -9,6 +9,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+## [0.24.0] - 2026-03-10
+
+### Added
+
+- First-class structured multi-line query documents using `term:`, `intent:`, and `hyde:` across CLI `query`/`ask`, REST `/api/query` and `/api/ask`, MCP `gno_query`, SDK `query`/`ask`, and Web Search/Ask text boxes.
+- Dedicated structured syntax reference doc plus updated CLI/API/MCP/SDK/Web docs.
+
 ## [0.23.0] - 2026-03-10
 
 ### Added

README.md

@@ -34,7 +34,13 @@ GNO is a local knowledge engine that turns your documents into a searchable, con
 
 ---
 
-## What's New in v0.23
+## What's New in v0.24
+
+- **Structured Query Documents**: first-class multi-line query syntax using `term:`, `intent:`, and `hyde:`
+- **Cross-Surface Rollout**: works across CLI, API, MCP, SDK, and Web Search/Ask
+- **Portable Retrieval Prompts**: save/share advanced retrieval intent as one text payload instead of repeated flags or JSON arrays
+
+### v0.23
 
 - **SDK / Library Mode**: package-root importable SDK with `createGnoClient(...)` for direct retrieval, document access, and indexing flows
 - **Inline Config Support**: embed GNO in another app without writing YAML config files
@@ -288,11 +294,15 @@ gno query "auth flow" \
   --query-mode intent:"how refresh token rotation works" \
   --query-mode hyde:"Refresh tokens rotate on each use and previous tokens are revoked." \
   --explain
+
+# Multi-line structured query document
+gno query $'auth flow\nterm: "refresh token" -oauth1\nintent: how refresh token rotation works\nhyde: Refresh tokens rotate on each use and previous tokens are revoked.' --fast
 ```
 
 - Modes: `term` (BM25-focused), `intent` (semantic-focused), `hyde` (single hypothetical passage)
 - Explain includes stage timings, fallback/cache counters, and per-result score components
 - `gno ask --json` includes `meta.answerContext` for adaptive source selection traces
+- Search and Ask web text boxes also accept multi-line structured query documents with `Shift+Enter`
 
 ---
 

docs/API.md

@@ -1197,6 +1197,7 @@ Combined BM25 + vector search with optional reranking. **Recommended for best re
 - `intent` is orthogonal to `queryModes`: intent steers scoring/prompting, while query modes inject caller-provided retrieval expansions.
 - `queryModes` is optional and only needed for explicit retrieval intent control.
 - If `queryModes` is provided, generated expansion is skipped and provided entries are used directly.
+- `query` can also be a multi-line structured query document using `term:`, `intent:`, and `hyde:` lines. See [Structured Query Syntax](./SYNTAX.md).
 
 **Response**:
 
@@ -1234,6 +1235,10 @@ Combined BM25 + vector search with optional reranking. **Recommended for best re
 curl -X POST http://localhost:3000/api/query \
   -H "Content-Type: application/json" \
   -d '{"query": "error handling best practices", "limit": 10}'
+
+curl -X POST http://localhost:3000/api/query \
+  -H "Content-Type: application/json" \
+  -d '{"query": "auth flow\nterm: \"refresh token\"\nintent: token rotation"}'
 ```
 
 ---
@@ -1298,6 +1303,7 @@ Get an AI-generated answer with citations from your documents.
 - Existing `/api/ask` payloads remain valid.
 - `queryModes` is optional and only needed for explicit retrieval steering during Q&A.
 - If `queryModes` is provided, generated expansion is skipped and provided entries are used directly.
+- `query` can also be a multi-line structured query document using `term:`, `intent:`, and `hyde:` lines. See [Structured Query Syntax](./SYNTAX.md).
 
 **Response**:
 

docs/CLI.md

@@ -117,6 +117,7 @@ gno query "auth" --thorough          # Full pipeline: ~5-8s
 gno query "auth" --tags-all work,backend   # Filter by tags
 gno query "performance" --intent "web performance and latency"
 gno query "auth flow" --query-mode term:"jwt refresh token" --query-mode intent:"how refresh token rotation works"
+gno query $'auth flow\nterm: "refresh token"\nintent: token rotation'
 ```
 
 **Search modes**:
@@ -157,6 +158,7 @@ Additional options:
 - `--intent` is orthogonal to `--query-mode`: intent steers scoring/prompting, while query modes inject caller-provided retrieval expansions.
 - `--query-mode` is opt-in for explicit intent control and replaces generated expansion for that query.
 - Use `term` for exact lexical constraints, `intent` for semantic reformulations, and `hyde` for one hypothetical answer passage.
+- Multi-line structured query documents are also supported. See [Structured Query Syntax](./SYNTAX.md).
 
 ```bash
 # Existing call (still valid)
@@ -167,6 +169,9 @@ gno query "auth flow" \
   --query-mode term:"jwt refresh token -oauth1" \
   --query-mode intent:"how refresh token rotation works" \
   --query-mode hyde:"Refresh tokens rotate on each use and previous tokens are revoked."
+
+# Multi-line structured query document
+gno query $'auth flow\nterm: "refresh token" -oauth1\nintent: how refresh token rotation works\nhyde: Refresh tokens rotate on each use and previous tokens are revoked.'
 ```
 
 The `--explain` flag outputs:
@@ -194,6 +199,7 @@ gno ask "quick lookup" --fast            # Fastest retrieval
 gno ask "complex topic" --thorough       # Best recall
 gno ask "performance" --intent "web latency and vitals"
 gno ask "performance" --query-mode term:"web performance budgets" --query-mode intent:"latency and vitals" --no-answer
+gno ask $'term: web performance budgets\nintent: latency and vitals' --no-answer
 ```
 
 **Full-document context**: When `--answer` is used, GNO passes complete document content to the generation model, not truncated snippets. This ensures the LLM sees tables, code examples, and full context needed for accurate answers.
@@ -209,6 +215,7 @@ Options:
 - `--intent <text>` - Disambiguating context for ambiguous questions without searching on that text
 - `--exclude <values>` - Hard-prune docs containing any comma-separated term in title/path/body
 - `--query-mode <mode:text>` - Structured expansion hints; repeat for multiple entries. Modes: `term`, `intent`, `hyde`
+- Multi-line structured query documents are also supported. See [Structured Query Syntax](./SYNTAX.md).
 - `-C, --candidate-limit <n>` - Max candidates passed to reranking (default: 20)
 - `--answer` - Generate grounded AI answer (requires gen model)
 - `--no-answer` - Force retrieval-only output

docs/MCP.md

@@ -578,6 +578,7 @@ Optional steering controls:
 - `intent` is complementary to `queryModes`: use intent for background context, `queryModes` for caller-supplied lexical/semantic expansions.
 - `queryModes` is optional and only needed when your client wants explicit retrieval intent control.
 - If `queryModes` is set, generated expansion is skipped for that query and the provided entries are used directly.
+- The `query` string itself may also be a multi-line structured query document using `term:`, `intent:`, and `hyde:` lines. See [Structured Query Syntax](./SYNTAX.md).
 
 ```yaml
 # Existing payload (still valid)
@@ -590,6 +591,12 @@ queryModes:
   - { mode: "term", text: "\"refresh token\" -oauth1" }
   - { mode: "intent", text: "how token rotation is implemented" }
   - { mode: "hyde", text: "Refresh tokens rotate on each use and old tokens are revoked." }
+
+# Or put the structure directly into the query field:
+query: |
+  auth flow
+  term: "refresh token" -oauth1
+  intent: how token rotation is implemented
 ```
 
 ### gno_get

docs/SDK.md

@@ -108,6 +108,14 @@ const results = await client.query("performance", {
   noExpand: true,
   noRerank: true,
 });
+
+const structured = await client.query(
+  'auth flow\\nterm: "refresh token"\\nintent: token rotation',
+  {
+    noExpand: true,
+    noRerank: true,
+  }
+);
 ```
 
 ### Ask
@@ -124,6 +132,15 @@ const retrievalOnly = await client.ask("JWT token", {
 const answered = await client.ask("What is our auth flow?", {
   answer: true,
 });
+
+const retrievalOnlyStructured = await client.ask(
+  "term: web performance budgets\\nintent: latency and vitals",
+  {
+    noAnswer: true,
+    noExpand: true,
+    noRerank: true,
+  }
+);
 ```
 
 ### Vector Search
@@ -211,12 +228,14 @@ The package root is the SDK entrypoint. The CLI remains available through the `g
 - `query` and `ask` degrade gracefully if vector/rerank/generation models are unavailable, except when answer generation is explicitly requested.
 - `vsearch` requires embeddings plus vector search support.
 - Inline config is supported; writing YAML is optional.
+- `query` and `ask` accept multi-line structured query documents. See [Structured Query Syntax](./SYNTAX.md).
 
 ---
 
 ## Related Docs
 
 - [CLI](./CLI.md)
 - [REST API](./API.md)
+- [Structured Query Syntax](./SYNTAX.md)
 - [Architecture](./ARCHITECTURE.md)
 - [Configuration](./CONFIGURATION.md)

docs/SYNTAX.md

@@ -0,0 +1,107 @@
+# Structured Query Syntax
+
+GNO supports a first-class multi-line query document syntax for `query` and `ask` flows.
+
+Use existing GNO naming only:
+
+```text
+auth flow
+term: "refresh token" -oauth1
+intent: how token rotation works
+hyde: Refresh tokens rotate on each use and previous tokens are invalidated.
+```
+
+## Rules
+
+- Structured syntax is only activated for multi-line input.
+- Blank lines are ignored.
+- Recognized typed lines are:
+  - `term:`
+  - `intent:`
+  - `hyde:`
+- At most one `hyde:` line is allowed.
+- Unknown typed prefixes like `vector:` are rejected.
+
+## Base Query
+
+The query document still needs a base search query.
+
+GNO resolves it in this order:
+
+1. plain untyped lines joined together
+2. otherwise all `term:` lines joined together
+3. otherwise all `intent:` lines joined together
+
+`hyde:` is never searched directly.
+
+That means these are both valid:
+
+```text
+auth flow
+term: "refresh token"
+intent: token rotation
+```
+
+```text
+term: "refresh token"
+intent: token rotation
+```
+
+This is invalid:
+
+```text
+hyde: hypothetical answer only
+```
+
+## Compatibility
+
+Structured query documents are additive:
+
+- existing plain single-line queries still work
+- existing `--query-mode` CLI flags still work
+- existing API/MCP `queryModes` arrays still work
+
+If both are supplied, GNO merges:
+
+- query-document typed lines
+- explicit `queryModes`
+
+Validation still applies across the combined set, including the single-`hyde` rule.
+
+## Supported Surfaces
+
+Current rollout:
+
+- CLI: `gno query`, `gno ask`
+- REST API: `/api/query`, `/api/ask`
+- MCP: `gno_query`
+- Web UI: Search and Ask text boxes
+- SDK: `client.query(...)`, `client.ask(...)`
+
+## Examples
+
+### CLI
+
+```bash
+gno query $'auth flow\nterm: "refresh token"\nintent: token rotation'
+```
+
+```bash
+gno ask $'term: web performance budgets\nintent: latency and vitals' --no-answer
+```
+
+### REST API
+
+```json
+{
+  "query": "auth flow\nterm: \"refresh token\"\nintent: token rotation"
+}
+```
+
+### SDK
+
+```ts
+const result = await client.query(
+  'auth flow\\nterm: "refresh token"\\nintent: token rotation'
+);
+```

docs/WEB-UI.md

@@ -81,6 +81,15 @@ Choose retrieval mode:
 
 Click **Ask** for AI-powered answers. Use **Advanced Retrieval** to scope by collection/date/category/author/tags and add optional `intent` / candidate-limit / exclude / query-mode controls for ambiguous questions.
 
+Both **Search** and **Ask** accept multi-line structured query documents. Press `Shift+Enter` to add a new line, then use:
+
+```text
+auth flow
+term: "refresh token"
+intent: token rotation
+hyde: Refresh tokens rotate on each use.
+```
+
 > **Note**: Models auto-download on first use. Cold start can take longer on first launch while local models download. For instant startup, set `GNO_NO_AUTO_DOWNLOAD=1` and download explicitly with `gno models pull`.
 
 ---