Add secret detection with git hooks

.github/workflows/README.md

@@ -0,0 +1,142 @@
+# GitHub Actions Workflows
+
+This directory contains GitHub Actions workflows for automated CI/CD processes.
+
+## Workflows
+
+### Secret Detection (`secret-detection.yml`)
+
+**Purpose:** Automatically scan for secrets, API keys, passwords, and other sensitive information in all pull requests and pushes to protected branches.
+
+**Triggers:**
+- Pull requests (opened, synchronized, reopened)
+- Pushes to `main`, `master`, `develop`, `release/**` branches
+- Manual workflow dispatch
+
+**Tools Used:**
+- **Gitleaks** - Fast regex-based secret scanner
+- **TruffleHog** - Deep scanner with verification
+
+**What It Does:**
+1. Scans entire repository history for secrets
+2. Uses project configurations (`.gitleaks.toml`, `.trufflehog.yaml`)
+3. Uploads findings to GitHub Security tab (SARIF format)
+4. Posts results as PR comments with action items
+5. Fails the build if secrets are detected
+
+**Required Permissions:**
+- `contents: read` - Checkout code
+- `pull-requests: write` - Post comments
+- `security-events: write` - Upload SARIF reports
+
+**Viewing Results:**
+- **PR Comments:** Automated summary with scan results
+- **Security Tab:** Navigate to Security → Code scanning
+- **Workflow Logs:** Click on failed runs for details
+
+**Configuration:**
+See [SECURITY.md](../SECURITY.md) for detailed documentation.
+
+---
+
+### Configuration Sync Check (`config-sync-check.yml`)
+
+**Purpose:** Validates that configuration values are synchronized across Docker Compose, Kubernetes, and Helm configurations.
+
+**Triggers:**
+- Pull requests that modify:
+  - `docker-compose.yaml`
+  - `k8s/*.yaml`
+  - `helm/lightrag/values.yaml`
+  - The workflow file itself
+
+**What It Does:**
+1. Compares resource limits across configurations
+2. Validates environment variables
+3. Checks service endpoints and ports
+4. Posts sync report as PR comment
+
+---
+
+### Cost Impact Analysis (`cost-estimate.yml`)
+
+**Purpose:** Estimates infrastructure costs based on Kubernetes resource requests and limits.
+
+**Triggers:**
+- Pull requests that modify:
+  - `helm/lightrag/values.yaml`
+  - `k8s/*.yaml`
+  - The workflow file itself
+
+**What It Does:**
+1. Calculates resource costs (CPU, memory, storage)
+2. Compares against base branch
+3. Shows cost deltas and impacts
+4. Posts cost estimate as PR comment
+
+---
+
+## Workflow Best Practices
+
+### For All Workflows
+
+1. **Use specific action versions** - Pin to major versions (e.g., `@v4`)
+2. **Minimize permissions** - Request only necessary permissions
+3. **Cache dependencies** - Speed up workflow execution
+4. **Use job summaries** - Provide clear output via `$GITHUB_STEP_SUMMARY`
+5. **Update comments** - Don't create duplicate PR comments
+
+### For Secret Detection
+
+1. **Never bypass CI** - Local hooks can be skipped, CI cannot
+2. **Review failures promptly** - Investigate all detected secrets
+3. **Rotate exposed secrets** - If secrets reach the repo, rotate immediately
+4. **Update configurations** - Maintain allowlists for false positives
+5. **Monitor Security tab** - Regular review of code scanning results
+
+### Adding New Workflows
+
+When creating new workflows:
+
+1. Use meaningful names and descriptions
+2. Document triggers and permissions
+3. Add comments explaining complex steps
+4. Follow existing patterns (PR comments, job summaries)
+5. Test with workflow_dispatch before enabling for PRs
+6. Update this README with workflow documentation
+
+## Troubleshooting
+
+### Workflow Not Running
+
+- Check trigger conditions match your branch/files
+- Verify workflow file is valid YAML
+- Look for workflow errors in Actions tab
+
+### Permission Errors
+
+- Ensure workflow has required permissions
+- Check repository settings → Actions → General
+- Verify `GITHUB_TOKEN` has necessary scopes
+
+### Secret Detection False Positives
+
+1. Verify it's actually a false positive (not a real secret!)
+2. Add to allowlist in `.gitleaks.toml` or `.trufflehog.yaml`
+3. Document the reason in the configuration file
+4. Never add real secrets to allowlists
+
+### Workflow Taking Too Long
+
+For secret detection:
+- Adjust scan depth if scanning full history
+- Use incremental scanning for PRs
+- Check for network issues with verification
+
+## Resources
+
+- [GitHub Actions Documentation](https://docs.github.com/en/actions)
+- [Gitleaks Documentation](https://github.com/gitleaks/gitleaks)
+- [TruffleHog Documentation](https://github.com/trufflesecurity/trufflehog)
+- [SARIF Format](https://docs.github.com/en/code-security/code-scanning/integrating-with-code-scanning/sarif-support-for-code-scanning)
+- [Security Tab](https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/managing-code-scanning-alerts-for-your-repository)

.github/workflows/secret-detection.yml

@@ -0,0 +1,214 @@
+name: Secret Detection
+
+on:
+  push:
+    branches:
+      - main
+      - master
+      - develop
+      - 'release/**'
+  pull_request:
+    types: [opened, synchronize, reopened]
+  workflow_dispatch:
+
+permissions:
+  contents: read
+  pull-requests: write
+  security-events: write  # For SARIF upload
+
+jobs:
+  gitleaks:
+    name: Gitleaks Secret Scan
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0  # Full history for comprehensive scan
+
+      - name: Run Gitleaks
+        id: gitleaks
+        uses: gitleaks/gitleaks-action@v2
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          GITLEAKS_LICENSE: ${{ secrets.GITLEAKS_LICENSE }}  # Optional: for Gitleaks Pro
+        with:
+          # Use project configuration
+          config-path: .gitleaks.toml
+
+      - name: Upload Gitleaks SARIF report
+        if: always()
+        uses: github/codeql-action/upload-sarif@v3
+        with:
+          sarif_file: results.sarif
+          category: gitleaks
+
+      - name: Generate Gitleaks summary
+        if: failure()
+        run: |
+          echo "## 🔍 Gitleaks Detected Secrets!" >> $GITHUB_STEP_SUMMARY
+          echo "" >> $GITHUB_STEP_SUMMARY
+          echo "❌ Gitleaks found potential secrets in the code." >> $GITHUB_STEP_SUMMARY
+          echo "" >> $GITHUB_STEP_SUMMARY
+          echo "Please review the Security tab for details or check the workflow logs." >> $GITHUB_STEP_SUMMARY
+          echo "" >> $GITHUB_STEP_SUMMARY
+          echo "**Next steps:**" >> $GITHUB_STEP_SUMMARY
+          echo "1. Remove the secrets from your code" >> $GITHUB_STEP_SUMMARY
+          echo "2. Use environment variables or encrypted secrets instead" >> $GITHUB_STEP_SUMMARY
+          echo "3. See SECURITY.md for guidance" >> $GITHUB_STEP_SUMMARY
+
+  trufflehog:
+    name: TruffleHog Secret Scan
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0  # Full history for comprehensive scan
+
+      - name: Run TruffleHog
+        id: trufflehog
+        uses: trufflesecurity/trufflehog@main
+        with:
+          # Scan the entire git repository
+          path: ./
+          # Use main branch as base for PR scans
+          base: ${{ github.event.pull_request.base.sha || 'main' }}
+          head: ${{ github.sha }}
+          # Only report verified secrets to reduce false positives
+          # Note: --fail and --no-update are added automatically by the action
+          extra_args: --only-verified
+
+      - name: Generate TruffleHog summary
+        if: failure()
+        run: |
+          echo "## 🔍 TruffleHog Detected Verified Secrets!" >> $GITHUB_STEP_SUMMARY
+          echo "" >> $GITHUB_STEP_SUMMARY
+          echo "❌ TruffleHog found verified secrets in the code." >> $GITHUB_STEP_SUMMARY
+          echo "" >> $GITHUB_STEP_SUMMARY
+          echo "These secrets have been verified with the actual service and are confirmed to be valid!" >> $GITHUB_STEP_SUMMARY
+          echo "" >> $GITHUB_STEP_SUMMARY
+          echo "**⚠️ CRITICAL: These secrets must be rotated immediately!**" >> $GITHUB_STEP_SUMMARY
+          echo "" >> $GITHUB_STEP_SUMMARY
+          echo "**Next steps:**" >> $GITHUB_STEP_SUMMARY
+          echo "1. Rotate the exposed secrets immediately" >> $GITHUB_STEP_SUMMARY
+          echo "2. Remove them from your code" >> $GITHUB_STEP_SUMMARY
+          echo "3. Use environment variables or encrypted secrets" >> $GITHUB_STEP_SUMMARY
+          echo "4. Review git history and consider using git-filter-repo to remove secrets" >> $GITHUB_STEP_SUMMARY
+          echo "5. See SECURITY.md for guidance" >> $GITHUB_STEP_SUMMARY
+
+  post-results:
+    name: Post Scan Results
+    runs-on: ubuntu-latest
+    needs: [gitleaks, trufflehog]
+    if: always() && github.event_name == 'pull_request'
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Post results as PR comment
+        uses: actions/github-script@v7
+        with:
+          script: |
+            const gitleaksResult = '${{ needs.gitleaks.result }}';
+            const trufflehogResult = '${{ needs.trufflehog.result }}';
+
+            let emoji = '✅';
+            let status = 'Passed';
+            let message = 'No secrets detected in your changes.';
+
+            if (gitleaksResult === 'failure' || trufflehogResult === 'failure') {
+              emoji = '❌';
+              status = 'Failed';
+              message = 'Secrets detected in your changes!';
+            } else if (gitleaksResult === 'cancelled' || trufflehogResult === 'cancelled') {
+              emoji = '⚠️';
+              status = 'Cancelled';
+              message = 'Secret scanning was cancelled.';
+            }
+
+            const commitSha = context.payload.pull_request.head.sha;
+            const shortSha = commitSha.substring(0, 7);
+            const commitUrl = `${context.payload.repository.html_url}/commit/${commitSha}`;
+
+            const reportBody = `## 🔒 Secret Detection Report
+
+            **Status:** ${emoji} ${status}
+            **Commit:** [\`${shortSha}\`](${commitUrl})
+
+            ${message}
+
+            ### Scan Results
+
+            | Scanner | Result |
+            |---------|--------|
+            | Gitleaks | ${gitleaksResult === 'success' ? '✅ Passed' : gitleaksResult === 'failure' ? '❌ Failed' : '⚠️ ' + gitleaksResult} |
+            | TruffleHog | ${trufflehogResult === 'success' ? '✅ Passed' : trufflehogResult === 'failure' ? '❌ Failed' : '⚠️ ' + trufflehogResult} |
+
+            ${gitleaksResult === 'failure' || trufflehogResult === 'failure' ? `
+            ### ⚠️ Action Required
+
+            Secrets were detected in your code. Please:
+
+            1. **Remove the secrets** from your code
+            2. **Replace with environment variables** or use encrypted secrets storage
+            3. **Never commit secrets** - use \`.env\` files (kept in \`.gitignore\`)
+            4. If secrets were already pushed, **rotate them immediately**
+            5. Review the Security tab or workflow logs for details
+
+            See [SECURITY.md](${context.payload.repository.html_url}/blob/${context.payload.pull_request.head.ref}/SECURITY.md) for guidance.
+
+            **To skip locally (emergency only):**
+            \`\`\`bash
+            LEFTHOOK=0 git commit -m "message"
+            \`\`\`
+
+            ⚠️ **Note:** Even if you skip local hooks, CI will still block merging until secrets are removed.
+            ` : `
+            ### ✅ All Clear!
+
+            No secrets detected in your changes. Great job maintaining security! 🎉
+            `}
+
+            ---
+            *🤖 Automated security scan powered by [Gitleaks](https://github.com/gitleaks/gitleaks) and [TruffleHog](https://github.com/trufflesecurity/trufflehog)*
+            `;
+
+            const { data: comments } = await github.rest.issues.listComments({
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              issue_number: context.issue.number,
+            });
+
+            const botComment = comments.find(comment => {
+              const isBot = comment.user.type === 'Bot' || comment.user.login === 'github-actions[bot]';
+              const hasSecretDetection = comment.body.includes('🔒 Secret Detection Report');
+              return isBot && hasSecretDetection;
+            });
+
+            if (botComment) {
+              await github.rest.issues.updateComment({
+                owner: context.repo.owner,
+                repo: context.repo.repo,
+                comment_id: botComment.id,
+                body: reportBody
+              });
+              console.log(`✅ Updated existing secret detection comment (ID: ${botComment.id})`);
+            } else {
+              const { data: newComment } = await github.rest.issues.createComment({
+                owner: context.repo.owner,
+                repo: context.repo.repo,
+                issue_number: context.issue.number,
+                body: reportBody
+              });
+              console.log(`✅ Created new secret detection comment (ID: ${newComment.id})`);
+            }
+
+      - name: Fail if secrets detected
+        if: needs.gitleaks.result == 'failure' || needs.trufflehog.result == 'failure'
+        run: |
+          echo "::error::Secrets detected! Please remove them before merging."
+          exit 1