feat: Automated repository tracking config management

[madjin]

Problem Statement

We currently have 43 tracked repositories and 286 untracked repositories across the elizaos and elizaos-plugins organizations. The config/example.json file that determines which repos get full tracking (PRs, issues, commits, contributors, scoring) is manually maintained.

Challenges:

• Some tracked repos haven't been updated in months (low activity)
• Active untracked repos could benefit from full contributor analytics
• Manual config updates are time-consuming and error-prone
• No systematic way to identify promotion/demotion candidates

Research Summary

Comprehensive research explored 6 strategies for automating config management. Full details in plan document: /home/jin/.claude/plans/ticklish-finding-kitten.md

Strategy Comparison

Strategy	Automation	Risk	Complexity	Effort	API Impact
1. Threshold Auto	High	Medium	Medium	8-12h	Reduces tracked
2. Manual Review Dashboard	Low	None	Low	8-11h	None
3. Tiered Tracking	High	Medium	High	21-27h	Reduces overall
4. Adaptive Refresh	Medium	Low	Low	6-9h	Reduces refresh
5. CI/CD PR ⭐	Medium	Low	Medium	9-13h	Reduces tracked
6. Hybrid Auto/Manual	Medium	Low	Medium	10-14h	Reduces tracked

Recommended Approach

Short-term (Quick Win): Strategy 2 - Manual Review Dashboard

• Generate weekly report of stale tracked repos and active untracked repos
• Dashboard UI for human review
• Low risk, immediate value
• Effort: 8-11 hours

Long-term (Automation): Strategy 5 - CI/CD Automated Config PRs

• Weekly GitHub Actions workflow evaluates repos
• Auto-generates PRs with promotion/demotion recommendations
• Rich context for human review (activity trends, contributor counts, recent commits)
• Requires approval before merging
• Effort: 9-13 hours

Key Technical Findings

Reusable Code Identified:

• ✅ calculateActivityScore() - Already exists in src/lib/pipelines/untracked/mutations.ts
• ✅ Activity score formula: log-scale stars/watchers, weighted PRs/issues, recency bonus
• ✅ Database batch operations pattern
• ✅ Time interval filtering utilities
• ✅ GitHub CLI PR creation patterns
• ✅ Config schema validation (Zod)

Activity Score Formula:

activityScore = baseScore * archivePenalty

baseScore =
  log(stars + 1) * 1.0 +          // Log-scaled popularity
  log(watchers + 1) * 1.5 +       // Stronger engagement
  openPrCount * 3 +               // Active development (highest weight)
  mergedPrCount * 2 +             // Productive activity
  closedUnmergedPrCount * 1 +     // Engagement
  openIssueCount * 2 +            // Active discussions
  closedIssueCount * 1 +          // Maintenance
  recencyBonus                    // Max 14 points, decays over 14 days

archivePenalty = 0.5 if archived, else 1.0

Implementation Outline (Strategy 5)

Phase 1: Core Evaluation

• Calculate activity scores for tracked repos (reuse existing formula)
• Build evaluation logic with hysteresis thresholds
• CLI command for testing: bun run pipeline evaluate-tracking

Phase 2: PR Automation

• Config update logic (read/modify config/example.json)
• Rich markdown PR description with activity details
• Preview comments showing recent contributor activity
• GitHub Actions workflow (weekly schedule + manual trigger)

Phase 3: Refinement

• Threshold tuning based on initial recommendations
• Testing with various scenarios
• Documentation and runbook for reviewing PRs

Proposed Config:

{
  "PIPELINE_AUTO_CONFIG_PR": {
    "enabled": true,
    "schedule": "0 0 * * 0",
    "promoteThreshold": 45.0,
    "demoteThreshold": 12.0,
    "minimumTrackedDays": 60,
    "maxPromotionsPerPR": 5,
    "maxDemotionsPerPR": 3,
    "excludeFromDemotion": ["elizaos/eliza", "elizaos/docs"],
    "prLabels": ["automated", "config-management"],
    "requireApprovals": 1
  }
}

Files to Create/Modify:

• src/lib/pipelines/tracking/calculateTrackedActivity.ts - Activity score for tracked repos
• src/lib/pipelines/tracking/evaluateRepos.ts - Evaluation logic
• cli/commands/evaluateTracking.ts - CLI command
• cli/commands/generateConfigPR.ts - PR generation
• .github/workflows/auto-config-pr.yml - GitHub Actions workflow
• src/lib/pipelines/pipelineConfig.ts - Add AutoConfigPRSchema
• config/example.json - Add configuration

Success Metrics

• Tracked repo count stabilizes or decreases
• High-activity untracked repos promoted within 1-2 weeks
• Low-activity tracked repos demoted within 1 month
• PR reviews take < 10 minutes
• No false positives (demoting active repos)

Next Steps

1. Decide on implementation timeline
2. Set activity score thresholds
3. Identify exclude list (critical repos that should never be demoted)
4. Implement and test chosen strategy

Related

• Current tracked repos: 43 (6 elizaos, 37 elizaos-plugins)
• Current untracked repos: 286 (49 elizaos, 237 elizaos-plugins)
• Activity-based UI coloring: Consolidate untracked repos pipeline and fix DRY violations #231 (merged)
• Untracked repos pipeline: Fix: re-export stats after contributor summaries are generated #229 (merged)