Job Prioritization Plugin

A comprehensive Jenkins plugin for intelligent job prioritization and scheduling with advanced features including:

• Multi-level priority queues (P0-P4)
• Smart job scheduling based on build history
• Concurrency limits per repository and branch
• Token-based rate limiting per team and repository
• Code change analysis to adjust priorities
• Emergency escalation workflow for critical builds
• Comprehensive metrics via Prometheus
• AI-based priority recommendations
• Complete audit logging
• Observability dashboard

Features

1. Priority Queue System

Jobs are prioritized across 5 levels:

• P0 (Critical): CTO-level escalation, production outages
• P1 (High): Production issues, security fixes
• P2 (Normal): Regular builds and features
• P3 (Low): Documentation changes, minor fixes
• P4 (Background): Non-urgent maintenance tasks

Within the same priority level, jobs are further sorted by:

• Estimated duration (shorter jobs first)
• Time in queue (aging factor)

2. Concurrency Management

• Limit concurrent builds per repository (default: 3)
• Limit concurrent builds per branch (default: 1)
• Reserve executor slots for P0 jobs
• Fair-share scheduling across teams

3. Rate Limiting

Token bucket implementation to prevent excessive builds:

• Per-team limits (default: 50 builds/hour)
• Per-repository limits (default: 20 builds/hour)
• Automatic token refill
• Clear error messages when limits are hit

4. Code Change Analysis

Automatically adjusts priority based on changes:

• Documentation-only changes → Lower priority
• Code-heavy changes → Maintain/increase priority
• Test-only changes → Slightly lower priority
• Configuration changes → Maintain priority

5. Emergency Escalation

Three-level escalation path:

• Manager level: Escalate to P1
• Director level: Escalate to P0
• CTO level: Immediate P0 with reserved slots
• Approval workflow with justification
• Automatic expiration after 4 hours

6. Metrics & Observability

Prometheus metrics for:

• Queue depth by priority
• Build duration histograms
• Agent usage tracking
• Capacity saturation
• Token bucket levels
• Rate limit rejections
• Escalation statistics

7. AI-Based Recommendations

Machine learning analysis of:

• Historical build patterns
• Success/failure rates
• Team and repository patterns
• Commit message analysis
• Recommended priority with confidence scores

8. Audit Logging

Complete audit trail of:

• All priority changes
• Build starts and completions
• Rate limit rejections
• Escalation requests
• AI recommendations
• Daily JSON log files
• Searchable history

Installation

1. Build the plugin:

mvn clean package

2. Install the generated HPI file in Jenkins:
   • Go to Manage Jenkins → Manage Plugins → Advanced
   • Upload target/job-prioritization.hpi
   • Restart Jenkins

Configuration

Global Configuration

Navigate to Manage Jenkins → Configure System → Job Prioritization Configuration

Configure:

• Enable/disable features
• Concurrency limits
• Rate limiting parameters
• Capacity thresholds
• Short job threshold

Job Configuration

In each job's configuration:

1. Add Job Priority Configuration property
2. Set the priority level (P0-P4)
3. Specify the team name
4. Enable/disable escalation

Usage

Setting Job Priority

// In Jenkinsfile
properties([
    jobPriority(
        priority: 'P2_NORMAL',
        team: 'backend-team',
        allowEscalation: true
    )
])

Requesting Escalation

1. Navigate to Job Prioritization Dashboard
2. Submit escalation request with justification
3. Await approval from authorized personnel
4. Job priority automatically updated on approval

Viewing Metrics

Metrics are exposed at /prometheus endpoint:

curl http://jenkins.example.com/prometheus

Monitoring Dashboard

Access the dashboard at: Jenkins → Job Prioritization Dashboard

View:

• Current queue statistics
• Pending escalations
• Recent audit entries
• Token bucket status
• Capacity utilization

Metrics

Key metrics exposed:

# Queue metrics
jenkins_prioritization_queue_depth{priority="P2_NORMAL"} 5
jenkins_prioritization_queue_wait_time_seconds{priority="P2_NORMAL"} 120

# Build metrics
jenkins_prioritization_builds_started_total{priority="P2_NORMAL",team="backend",repository="app"} 150
jenkins_prioritization_build_duration_seconds{priority="P2_NORMAL"} 245

# Concurrency metrics
jenkins_prioritization_concurrent_builds_repo{repository="app"} 2
jenkins_prioritization_concurrent_builds_branch{repository="app",branch="main"} 1

# Capacity metrics
jenkins_prioritization_capacity_saturation 0.75
jenkins_prioritization_reserved_slots_available 2

# Token bucket metrics
jenkins_prioritization_token_bucket_level{type="team",identifier="backend"} 35
jenkins_prioritization_token_bucket_rejections_total{type="repository",identifier="app"} 3

Best Practices

1. Set Appropriate Priorities

• Use P0 sparingly for true emergencies
• P1 for production issues
• P2 for regular development
• P3/P4 for non-critical work

2. Configure Rate Limits

• Adjust token limits based on team size
• Monitor rejection rates
• Encourage local testing

3. Monitor Capacity

• Set up alerts for high saturation
• Reserve adequate slots for P0
• Track agent usage patterns

4. Review Escalations

• Respond to escalation requests promptly
• Document justifications
• Review escalation patterns

5. Use AI Recommendations

• Review AI suggestions regularly
• Provide feedback to improve model
• Combine with human judgment

Architecture

┌─────────────────────────────────────────────────────────┐
│                    Jenkins Queue                        │
└────────────────────┬────────────────────────────────────┘
                     │
                     ↓
┌─────────────────────────────────────────────────────────┐
│              PriorityQueueSorter                        │
│  ┌──────────────────────────────────────────────────┐  │
│  │ 1. Extract job priority                          │  │
│  │ 2. Analyze code changes                          │  │
│  │ 3. Check build history                           │  │
│  │ 4. Apply aging factor                            │  │
│  │ 5. Sort by effective priority                    │  │
│  └──────────────────────────────────────────────────┘  │
└────────────────────┬────────────────────────────────────┘
                     │
                     ↓
┌─────────────────────────────────────────────────────────┐
│           QueueTaskDispatchers                          │
│  ┌──────────────────┐  ┌──────────────────┐            │
│  │ ConcurrencyLimiter│  │ RateLimitDispatcher│          │
│  │  - Repo limits   │  │  - Team tokens   │            │
│  │  - Branch limits │  │  - Repo tokens   │            │
│  └──────────────────┘  └──────────────────┘            │
└────────────────────┬────────────────────────────────────┘
                     │
                     ↓
┌─────────────────────────────────────────────────────────┐
│                  Build Execution                         │
└────────────────────┬────────────────────────────────────┘
                     │
                     ↓
┌─────────────────────────────────────────────────────────┐
│              Metrics & Audit Logging                    │
│  - Prometheus metrics                                   │
│  - JSON audit logs                                      │
│  - AI analysis                                          │
└─────────────────────────────────────────────────────────┘

Development

Building

mvn clean install

Testing

mvn test

Running in Dev Mode

mvn hpi:run

Access at: http://localhost:8080/jenkins

Troubleshooting

High Queue Wait Times

• Check capacity saturation
• Review priority distribution
• Consider adding more executors

Rate Limit Rejections

• Increase token limits
• Encourage local testing
• Review build frequency

Unexpected Priority Changes

• Check audit log for AI recommendations
• Review code change analysis settings
• Verify escalation approvals

Contributing

Contributions welcome! Please:

1. Fork the repository
2. Create a feature branch
3. Add tests for new features
4. Submit a pull request

License

MIT License - see LICENSE file

Support

• GitHub Issues: https://github.com/vrenjith/jenkins-prioritization-plugin/issues
• Documentation: https://github.com/vrenjith/jenkins-prioritization-plugin/wiki

Future Enhancements

• Pre-emption support (interrupt running jobs)
• Resume capability for interrupted jobs
• Machine learning model training
• Integration with external AI services
• Advanced SCM change analysis
• Custom priority rules engine
• Multi-Jenkins coordination
• Mobile escalation app