Pinned source documents and YAML front matter support

[abrookins]

Summary

Support "pinned" source documents that get injected at high priority into agent context, separate from vector search results. (Probably into the system prompt.)

Context

One use case: vocabulary files. Enterprise deployments often have highly domain-specific terminology and acronyms. When these terms are present in a query, high-priority context injection will work better to give the LLM necessary definitions rather than relying on vector search.

Proposed Features

• Pinned documents: Always included in agent context at high priority
• YAML front matter support for source documents:
  • pinned: true - mark document for priority injection
  • chunking_strategy - control how document is chunked (or not)
  • priority - relative priority for context ordering
• Default chunking behavior: "don't chunk" for reference docs (similar to API documentation)

Use Cases

• Domain-specific acronym glossaries
• Internal terminology definitions
• Critical runbook sections that should always be available
• API reference documentation