Merge pull request #4 from sprice/feature/improve-leg-embeddings

✨(legislation): Improve legislation RAG system with enhanced embeddings

Author: sprice

.claude/commands/code-review.md

@@ -0,0 +1,51 @@
+# Code Review Command
+
+Perform the type of critical code review an expert software engineer and/or an expert software architect would do.
+
+**IMPORTANT** Follow the coding standards and best practices outlined in the
+`CLAUDE.md` and `biome.jsonc` files for this review.
+
+## Steps
+
+1. **Check Diff**
+
+- Check the diff between the current branch against `main`
+
+2. **Analyze each file thoroughly**
+
+- For every file in the diff:
+  - Read the full file content if it's a new/modified file to understand context
+  - Examine the specific changes line by line from the diff
+  - Check against project coding standards from CLAUDE.md
+    - All coding standards are important, but pay special attention to the
+      Frontend Rules and React coding styles and best practices.
+  - Identify potential issues:
+    - Security vulnerabilities or exposed sensitive data
+    - Performance bottlenecks or inefficiencies
+    - Logic bugs or edge cases not handled
+    - Code maintainability and readability concerns
+    - Missing error handling or validation
+    - Breaking changes that affect other parts of the codebase
+  - For each issue found, note the specific file path and line number references
+- Assess the broader impact: how changes in each file affect related components,
+  APIs, database schemas, or user workflows
+
+3. **Create comprehensive review**
+
+- Write a complete and accurate code review document that covers:
+  - **Executive Summary**: Brief overview of changes, risk level, and key
+    concerns
+  - **Files Changed**: List all modified files with change summary
+  - **Critical Issues**: Security, breaking changes, or major bugs requiring
+    immediate attention
+  - **Detailed Analysis**: For each file with issues:
+    - `### path/to/file.ext`
+    - **Changes**: What was modified
+    - **Issues Found**: Specific problems with file:line references
+    - **Recommendations**: Actionable fixes with code examples where helpful
+    - **Impact**: How changes affect other parts of the system
+  - **Overall Assessment**: System-wide impact, testing recommendations,
+    deployment considerations
+  - **Action Items**: Prioritized checklist of required fixes and improvements
+
+Be constructive and helpful in your feedback.

.claude/commands/commit.md

@@ -0,0 +1,48 @@
+# Commit
+
+Create a well-described commit.
+
+## Steps
+
+1. **Analyze all changes:**
+   - Run `git status`, `git diff` (unstaged), and `git diff --cached` (staged)
+   - Review the diffs to understand what changed and why
+
+2. **Review and stage changes:**
+   - Check for files that should NOT be committed:
+     - `.env` or files with secrets/API keys
+     - Log files, temp files, debug output
+     - Files the user intentionally left unstaged
+   - If all changes are safe to commit: `git add -A`
+   - Otherwise, stage selectively: `git add <specific-files>`
+
+3. **Write a commit message that:**
+   - Uses conventional commit format: `(scope): subject`
+   - Has an imperative subject under 50 chars (e.g., "Add feature" not "Added feature")
+   - Includes a body explaining the "why" when the change isn't self-explanatory
+   - Groups related changes coherently
+
+4. **Commit using HEREDOC:**
+   ```bash
+   git commit -m "$(cat <<'EOF'
+(scope): subject
+
+Body explaining why, if needed.
+
+🤖 Generated with [Claude Code](https://claude.com/claude-code)
+
+Co-Authored-By: Claude <noreply@anthropic.com>
+EOF
+)"
+   ```
+
+5. **Handle pre-commit hook failures:**
+   If the commit fails due to pre-commit hook errors:
+   - Review the error output to understand what failed
+   - Ask: "Pre-commit hook failed. Fix the errors?" (Yes/No)
+   - If Yes: Fix the errors, re-stage affected files, and retry the commit
+   - If No: Abort
+
+6. **Verify and show result:**
+   - Run `git log -1 --stat` to display the commit
+   - Confirm the commit was created successfully

.claude/commands/create-pull-request.md

@@ -0,0 +1,116 @@
+# How to Create a Pull Request Using GitHub CLI
+
+This guide explains how to create pull requests using GitHub CLI in our project.
+
+**Important**: All PR titles and descriptions should be written in English.
+
+## Prerequisites
+
+1. Install GitHub CLI if you haven't already:
+
+   ```bash
+   # macOS
+   brew install gh
+
+   # Windows
+   winget install --id GitHub.cli
+
+   # Linux
+   # Follow instructions at https://github.com/cli/cli/blob/trunk/docs/install_linux.md
+   ```
+
+2. Authenticate with GitHub:
+   ```bash
+   gh auth login
+   ```
+
+## Creating a New Pull Request
+
+1. First, prepare your PR description following the template in @.github/pull_request_template.md
+
+2. Use the `gh pr create --draft` command to create a new pull request:
+
+   ```bash
+   # Create PR with proper template structure
+   gh pr create --draft --title "✨(scope): Your descriptive title" --body-file .github/pull_request_template.md --base main
+   ```
+
+## Best Practices
+
+1. **Language**: Always use English for PR titles and descriptions
+
+2. **PR Title Format**: Use conventional commit format with emojis
+
+   - Always include an appropriate emoji at the beginning of the title
+   - Use the actual emoji character (not the code representation like `:sparkles:`)
+   - Examples:
+     - `✨(supabase): Add staging remote configuration`
+     - `🐛(auth): Fix login redirect issue`
+     - `📝(readme): Update installation instructions`
+
+3. **Description Template**: Always use our PR template structure from @.github/pull_request_template.md:
+
+4. **Template Accuracy**: Ensure your PR description precisely follows the template structure:
+
+   - Don't modify or rename the PR-Agent sections (`pr_agent:summary` and `pr_agent:walkthrough`)
+   - Keep all section headers exactly as they appear in the template
+   - Don't add custom sections that aren't in the template
+
+5. **Draft PRs**: Start as draft when the work is in progress
+   - Use `--draft` flag in the command
+   - Convert to ready for review when complete using `gh pr ready`
+
+### Common Mistakes to Avoid
+
+1. **Using Non-English Text**: All PR content must be in English
+2. **Incorrect Section Headers**: Always use the exact section headers from the template
+3. **Adding Custom Sections**: Stick to the sections defined in the template
+4. **Using Outdated Templates**: Always refer to the current @.github/pull_request_template.md file
+
+### Missing Sections
+
+Always include all template sections, even if some are marked as "N/A" or "None"
+
+## Additional GitHub CLI PR Commands
+
+Here are some additional useful GitHub CLI commands for managing PRs:
+
+```bash
+# List your open pull requests
+gh pr list --author "@me"
+
+# Check PR status
+gh pr status
+
+# View a specific PR
+gh pr view <PR-NUMBER>
+
+# Check out a PR branch locally
+gh pr checkout <PR-NUMBER>
+
+# Convert a draft PR to ready for review
+gh pr ready <PR-NUMBER>
+
+# Add reviewers to a PR
+gh pr edit <PR-NUMBER> --add-reviewer username1,username2
+
+# Merge a PR
+gh pr merge <PR-NUMBER> --squash
+```
+
+## Using Templates for PR Creation
+
+To simplify PR creation with consistent descriptions, you can create a template file:
+
+1. Create a file named `pr-template.md` with your PR template
+2. Use it when creating PRs:
+
+```bash
+gh pr create --draft --title "feat(scope): Your title" --body-file pr-template.md --base main
+```
+
+## Related Documentation
+
+- [PR Template](.github/pull_request_template.md)
+- [Conventional Commits](https://www.conventionalcommits.org/)
+- [GitHub CLI documentation](https://cli.github.com/manual/)

.claude/commands/deslop-comments.md

@@ -0,0 +1,8 @@
+# Remove AI comment slop
+
+Check the diff against main, and remove all AI generated slop comments introduced in this branch.
+
+This includes:
+- Extra comments that a human wouldn't add or that are inconsistent with the rest of the file
+
+Report at the end with only a 1-3 sentence summary of what you changed

.claude/commands/deslop.md

@@ -0,0 +1,11 @@
+# Remove AI code slop
+
+Check the diff against main, and remove all AI generated slop introduced in this branch.
+
+This includes:
+- Extra comments that a human wouldn't add or is inconsistent with the rest of the file
+- Extra defensive checks or try/catch blocks that are abnormal for that area of the codebase (especially if called by trusted / validated codepaths)
+- Casts to any to get around type issues
+- Any other style that is inconsistent with the file
+
+Report at the end with only a 1-3 sentence summary of what you changed

.env.example

@@ -37,6 +37,5 @@ LEG_RAG_ENABLED=true
 # Truthy values disable RAG caching (useful for development)
 RAG_CACHE_DISABLE=
 
-# Fixed status Alert message at the top of `/workbench*` routes
-# Ex: NEXT_PUBLIC_SYSTEM_STATUS_MESSAGE="We are currently experiencing degraded service from an upstream provider"
-NEXT_PUBLIC_SYSTEM_STATUS_MESSAGE=
+# Fixed status message at the top of `/workbench` routes.`
+NEXT_PUBLIC_SHOW_SYSTEM_STATUS=true

.github/pull_request_template.md

@@ -0,0 +1,50 @@
+## Summary
+
+<!-- What does this PR do? Keep it concise but complete. -->
+
+## Motivation
+
+<!-- Why is this change needed? What problem does it solve? -->
+
+Closes #<!-- issue number -->
+
+## Type of Change
+
+- [ ] Bug fix (non-breaking change that fixes an issue)
+- [ ] New feature (non-breaking change that adds functionality)
+- [ ] Breaking change (fix or feature that would cause existing functionality to change)
+- [ ] Refactor (code change that neither fixes a bug nor adds a feature)
+- [ ] Documentation update
+- [ ] Test update
+- [ ] Chore (dependency updates, config changes, etc.)
+
+## Changes Made
+
+<!-- Bullet point the key changes in this PR -->
+
+- 
+
+## Testing
+
+<!-- How did you test these changes? Include steps to reproduce if applicable. -->
+
+- [ ] Tested locally
+- [ ] Added/updated tests
+- [ ] Verified existing tests pass
+
+## Screenshots
+
+<!-- If this PR includes UI changes, add screenshots or screen recordings here. Delete this section if not applicable. -->
+
+## Checklist
+
+- [ ] My code follows the project's coding standards
+- [ ] `pnpm check` passes (lint + type check)
+- [ ] I have added tests that prove my fix/feature works
+- [ ] New and existing tests pass locally
+- [ ] I have updated documentation where needed
+- [ ] I have reviewed my own code before requesting review
+
+## Additional Notes
+
+<!-- Any other context, concerns, or things reviewers should know? Delete if not needed. -->

.gitignore

@@ -55,6 +55,12 @@ data/parliament/openparliament.public.sql
 scripts/.embedding-progress.db
 scripts/.embedding-progress.db-wal
 scripts/.embedding-progress.db-shm
+scripts/.leg-embedding-progress.db
+scripts/.leg-embedding-progress.db-wal
+scripts/.leg-embedding-progress.db-shm
+scripts/.leg-import-progress.db
+scripts/.leg-import-progress.db-shm
+scripts/.leg-import-progress.db-wal
 
 # AI Coding
 .plans