WIP improve legislation embeddings

Author: sprice

.claude/commands/create-pull-request.md

@@ -0,0 +1,116 @@
+# How to Create a Pull Request Using GitHub CLI
+
+This guide explains how to create pull requests using GitHub CLI in our project.
+
+**Important**: All PR titles and descriptions should be written in English.
+
+## Prerequisites
+
+1. Install GitHub CLI if you haven't already:
+
+   ```bash
+   # macOS
+   brew install gh
+
+   # Windows
+   winget install --id GitHub.cli
+
+   # Linux
+   # Follow instructions at https://github.com/cli/cli/blob/trunk/docs/install_linux.md
+   ```
+
+2. Authenticate with GitHub:
+   ```bash
+   gh auth login
+   ```
+
+## Creating a New Pull Request
+
+1. First, prepare your PR description following the template in @.github/pull_request_template.md
+
+2. Use the `gh pr create --draft` command to create a new pull request:
+
+   ```bash
+   # Create PR with proper template structure
+   gh pr create --draft --title "✨(scope): Your descriptive title" --body-file .github/pull_request_template.md --base main
+   ```
+
+## Best Practices
+
+1. **Language**: Always use English for PR titles and descriptions
+
+2. **PR Title Format**: Use conventional commit format with emojis
+
+   - Always include an appropriate emoji at the beginning of the title
+   - Use the actual emoji character (not the code representation like `:sparkles:`)
+   - Examples:
+     - `✨(supabase): Add staging remote configuration`
+     - `🐛(auth): Fix login redirect issue`
+     - `📝(readme): Update installation instructions`
+
+3. **Description Template**: Always use our PR template structure from @.github/pull_request_template.md:
+
+4. **Template Accuracy**: Ensure your PR description precisely follows the template structure:
+
+   - Don't modify or rename the PR-Agent sections (`pr_agent:summary` and `pr_agent:walkthrough`)
+   - Keep all section headers exactly as they appear in the template
+   - Don't add custom sections that aren't in the template
+
+5. **Draft PRs**: Start as draft when the work is in progress
+   - Use `--draft` flag in the command
+   - Convert to ready for review when complete using `gh pr ready`
+
+### Common Mistakes to Avoid
+
+1. **Using Non-English Text**: All PR content must be in English
+2. **Incorrect Section Headers**: Always use the exact section headers from the template
+3. **Adding Custom Sections**: Stick to the sections defined in the template
+4. **Using Outdated Templates**: Always refer to the current @.github/pull_request_template.md file
+
+### Missing Sections
+
+Always include all template sections, even if some are marked as "N/A" or "None"
+
+## Additional GitHub CLI PR Commands
+
+Here are some additional useful GitHub CLI commands for managing PRs:
+
+```bash
+# List your open pull requests
+gh pr list --author "@me"
+
+# Check PR status
+gh pr status
+
+# View a specific PR
+gh pr view <PR-NUMBER>
+
+# Check out a PR branch locally
+gh pr checkout <PR-NUMBER>
+
+# Convert a draft PR to ready for review
+gh pr ready <PR-NUMBER>
+
+# Add reviewers to a PR
+gh pr edit <PR-NUMBER> --add-reviewer username1,username2
+
+# Merge a PR
+gh pr merge <PR-NUMBER> --squash
+```
+
+## Using Templates for PR Creation
+
+To simplify PR creation with consistent descriptions, you can create a template file:
+
+1. Create a file named `pr-template.md` with your PR template
+2. Use it when creating PRs:
+
+```bash
+gh pr create --draft --title "feat(scope): Your title" --body-file pr-template.md --base main
+```
+
+## Related Documentation
+
+- [PR Template](.github/pull_request_template.md)
+- [Conventional Commits](https://www.conventionalcommits.org/)
+- [GitHub CLI documentation](https://cli.github.com/manual/)

.github/pull_request_template.md

@@ -0,0 +1,50 @@
+## Summary
+
+<!-- What does this PR do? Keep it concise but complete. -->
+
+## Motivation
+
+<!-- Why is this change needed? What problem does it solve? -->
+
+Closes #<!-- issue number -->
+
+## Type of Change
+
+- [ ] Bug fix (non-breaking change that fixes an issue)
+- [ ] New feature (non-breaking change that adds functionality)
+- [ ] Breaking change (fix or feature that would cause existing functionality to change)
+- [ ] Refactor (code change that neither fixes a bug nor adds a feature)
+- [ ] Documentation update
+- [ ] Test update
+- [ ] Chore (dependency updates, config changes, etc.)
+
+## Changes Made
+
+<!-- Bullet point the key changes in this PR -->
+
+- 
+
+## Testing
+
+<!-- How did you test these changes? Include steps to reproduce if applicable. -->
+
+- [ ] Tested locally
+- [ ] Added/updated tests
+- [ ] Verified existing tests pass
+
+## Screenshots
+
+<!-- If this PR includes UI changes, add screenshots or screen recordings here. Delete this section if not applicable. -->
+
+## Checklist
+
+- [ ] My code follows the project's coding standards
+- [ ] `pnpm check` passes (lint + type check)
+- [ ] I have added tests that prove my fix/feature works
+- [ ] New and existing tests pass locally
+- [ ] I have updated documentation where needed
+- [ ] I have reviewed my own code before requesting review
+
+## Additional Notes
+
+<!-- Any other context, concerns, or things reviewers should know? Delete if not needed. -->

.gitignore

@@ -55,6 +55,7 @@ data/parliament/openparliament.public.sql
 scripts/.embedding-progress.db
 scripts/.embedding-progress.db-wal
 scripts/.embedding-progress.db-shm
+scripts/.leg-embedding-progress.db
 
 # AI Coding
 .plans

lib/ai/tools/retrieve-legislation-context.ts

@@ -85,8 +85,8 @@ export async function getLegislationContext(
 
   dbg("search returned %d results", results.length);
 
-  // Build context with reranking
-  const context = buildLegislationContext(query, results, {
+  // Build context with reranking (now async with Cohere cross-encoder)
+  const context = await buildLegislationContext(query, results, {
     language: preferLang,
     topN: boundedLimit,
   });

lib/db/rag/schema.ts

@@ -4,20 +4,42 @@
  * Contains all RAG-related tables for parliament and legislation embeddings.
  */
 
-import type { InferSelectModel } from "drizzle-orm";
+import { type InferSelectModel, sql } from "drizzle-orm";
 import {
+  check,
   customType,
   index,
   integer,
   jsonb,
   pgSchema,
   text,
   timestamp,
+  unique,
   varchar,
   vector,
 } from "drizzle-orm/pg-core";
 import { nanoid } from "nanoid";
 
+/**
+ * Valid source types for legislation resources
+ * Used for CHECK constraint and TypeScript type alignment
+ */
+export const LEG_SOURCE_TYPES = [
+  "act",
+  "act_section",
+  "regulation",
+  "regulation_section",
+  "defined_term",
+  "preamble",
+  "treaty",
+  "cross_reference",
+  "table_of_provisions",
+  "signature_block",
+  "related_provisions",
+] as const;
+
+export type LegSourceType = (typeof LEG_SOURCE_TYPES)[number];
+
 export const ragSchema = pgSchema("rag");
 
 /**
@@ -182,8 +204,8 @@ export type ParlEmbedding = InferSelectModel<typeof parlEmbeddings>;
  * Fields needed for search filtering and citation building.
  */
 export type LegResourceMetadata = {
-  // Identity - source types for acts, regulations, and their sections
-  sourceType: "act" | "act_section" | "regulation" | "regulation_section";
+  // Identity - source types for all legislation content
+  sourceType: LegSourceType;
   language: "en" | "fr";
   chunkIndex?: number; // 0 for metadata chunk, 1+ for content chunks
 
@@ -196,6 +218,33 @@ export type LegResourceMetadata = {
   sectionId?: string; // FK to legislation.sections.id
   sectionLabel?: string; // e.g., "91", "Schedule I"
   marginalNote?: string; // Short description of section
+  sectionStatus?: string; // "in-force", "repealed", "not-in-force", etc.
+  sectionType?: string; // "section", "schedule", "preamble", "heading", etc.
+  hierarchyPath?: string[]; // e.g., ["Part I", "Division 1", "Subdivision A"]
+  contentFlags?: {
+    // Mirrors ContentFlags from legislation schema
+    hasTable?: boolean;
+    hasFormula?: boolean;
+    hasImage?: boolean;
+    imageSources?: string[];
+    hasRepealed?: boolean;
+  };
+  sectionInForceDate?: string; // ISO date when section came into force
+  historicalNotes?: {
+    // Mirrors HistoricalNoteItem from legislation schema
+    text: string;
+    type?: string;
+    enactedDate?: string;
+    inForceStartDate?: string;
+    enactId?: string;
+  }[];
+
+  // Defined term specific fields
+  termId?: string; // FK to legislation.defined_terms.id
+  term?: string; // The defined term itself (e.g., "barrier", "obstable")
+  termPaired?: string; // The paired term in other language
+  scopeType?: string; // "act", "regulation", "part", "section"
+  scopeSections?: string[]; // Section scope if applicable
 
   // Act metadata fields
   longTitle?: string;
@@ -211,6 +260,34 @@ export type LegResourceMetadata = {
   enablingActId?: string;
   enablingActTitle?: string;
   registrationDate?: string;
+
+  // Preamble-specific fields
+  preambleIndex?: number; // Position in preamble array
+
+  // Treaty-specific fields
+  treatyTitle?: string; // Title of the treaty/convention
+
+  // Cross-reference fields
+  crossRefId?: string; // FK to legislation.cross_references.id
+  targetType?: string; // "act" or "regulation"
+  targetRef?: string; // Reference to target document
+  targetSectionRef?: string; // Optional section reference
+  referenceText?: string; // Display text for the reference
+
+  // Table of provisions fields
+  provisionLabel?: string; // Label from table of provisions
+  provisionTitle?: string; // Title from table of provisions
+  provisionLevel?: number; // Hierarchy level
+
+  // Signature block fields
+  signatureName?: string; // Name of signatory
+  signatureTitle?: string; // Title of signatory
+  signatureDate?: string; // Date of signature
+
+  // Related provisions fields
+  relatedProvisionLabel?: string; // Label from related provision (e.g., "Transitional Provisions")
+  relatedProvisionSource?: string; // Source reference
+  relatedProvisionSections?: string[]; // Referenced section numbers
 };
 
 /**
@@ -224,16 +301,29 @@ export const legResources = ragSchema.table(
     id: varchar("id", { length: 191 })
       .primaryKey()
       .$defaultFn(() => nanoid()),
-    sectionId: varchar("section_id", { length: 191 }).notNull(),
+    // Unique resource key for deduplication: "{sourceType}:{sourceId}:{language}:{chunkIndex}"
+    resourceKey: varchar("resource_key", { length: 255 }).notNull(),
     content: text("content").notNull(),
     metadata: jsonb("metadata").$type<LegResourceMetadata>().notNull(),
+    // Denormalized columns for fast filtering (avoids JSONB extraction in queries)
+    language: varchar("language", { length: 2 }).notNull(),
+    sourceType: varchar("source_type", { length: 30 }).notNull(),
     createdAt: timestamp("created_at").defaultNow().notNull(),
     updatedAt: timestamp("updated_at").defaultNow().notNull(),
   },
   (table) => [
-    index("leg_resources_section_id_idx").on(table.sectionId),
+    // Unique constraint to prevent duplicates on concurrent runs or restarts
+    unique("leg_resources_resource_key_unique").on(table.resourceKey),
+    index("leg_resources_resource_key_idx").on(table.resourceKey),
+    // Composite index for common filtering patterns (language + sourceType)
+    index("leg_resources_lang_source_idx").on(table.language, table.sourceType),
     // Single GIN index on metadata for flexible querying
     index("leg_resources_metadata_gin").using("gin", table.metadata),
+    // CHECK constraint for valid source types (data integrity)
+    check(
+      "leg_resources_source_type_check",
+      sql`${table.sourceType} IN (${sql.raw(LEG_SOURCE_TYPES.map((t) => `'${t}'`).join(", "))})`
+    ),
   ]
 );