Merge pull request #1 from CodingNinjasHQ/feature/concurrent-job-limiting

Add concurrent job limiting feature

Author: parikshit223933

CLAUDE.md

@@ -0,0 +1,37 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Commands
+
+- **Install dependencies**: `bundle install`
+- **Run tests**: `rake test` or `rake`
+- **Build gem**: `rake build`
+- **Release gem**: `rake release`
+- **Generate documentation**: `rake rdoc`
+
+## Architecture
+
+This is a Ruby gem that adds rate limiting capabilities to Resque queues. The implementation works by:
+
+1. **Core Module**: `Resque::Plugins::Throttler` in `/lib/resque/throttler.rb` contains all functionality
+2. **Rate Limit Configuration**: Users set limits via `Resque.rate_limit(:queue_name, at: count, per: seconds)`
+3. **Enforcement Mechanism**: Overrides `Resque::Worker#reserve` to check rate limits before allowing job reservation
+4. **Redis-based Tracking**: Uses Redis counters with expiration for tracking job counts and distributed locks for thread safety
+
+Key implementation details:
+- Rate limit counters use Redis key: `throttler:rate_limit:{queue_name}`
+- Distributed locks use Redis key: `throttler:lock:{queue_name}` with 30-second expiration
+- Workers skip rate-limited queues rather than blocking
+- All logic is contained in a single file for simplicity
+
+## Testing
+
+The project uses Minitest with Mocha for mocking. Note that some test files (particularly `/test/resque_test.rb` and `/test/resque/job_test.rb`) appear to be from an older implementation and may not reflect the current code structure.
+
+## Development Guidelines
+
+- The gem follows standard Ruby gem conventions
+- Runtime dependency: resque > 1.25
+- All core functionality is in `/lib/resque/throttler.rb`
+- Keep the implementation simple - the entire gem is designed to do one thing well

CONCURRENT_LIMITING_SUMMARY.md

@@ -0,0 +1,96 @@
+# Concurrent Limiting Feature Summary
+
+## What Was Added
+
+The resque-throttler gem now supports concurrent job limiting in addition to rate limiting. This prevents too many jobs from running simultaneously, which is essential for preventing database lock accumulation.
+
+## How It Works
+
+1. **Configuration**: Add `:concurrent` option to rate_limit
+   ```ruby
+   Resque.rate_limit(:my_queue, at: 5, per: 5, concurrent: 3)
+   ```
+
+2. **Tracking**: The gem tracks active jobs using Redis counters
+   - Increments when job starts
+   - Decrements when job completes (success or failure)
+
+3. **Enforcement**: Workers check concurrent limit before starting new jobs
+   - If at limit, queue is skipped
+   - Job remains in queue for next worker cycle
+
+## Key Methods Added
+
+```ruby
+# Check active job count
+Resque.active_job_count(:my_queue)
+
+# Check if at concurrent limit
+Resque.queue_at_or_over_concurrent_limit?(:my_queue)
+
+# Check if queue has concurrent limit configured
+Resque.queue_has_concurrent_limit?(:my_queue)
+```
+
+## Implementation Details
+
+1. **Redis Keys**:
+   - Active jobs counter: `throttler:active_jobs:{queue_name}`
+   - Automatically cleaned up (no expiration needed)
+
+2. **Worker Hooks**:
+   - Overrides `perform` method to track job lifecycle
+   - Uses `alias_method` to preserve original behavior
+   - Only tracks rate-limited queues
+
+3. **Thread Safety**:
+   - Uses existing lock mechanism from rate limiting
+   - Atomic Redis operations (INCR/DECR)
+
+## For Your Lead Uploader Issue
+
+Your issue: Database-intensive jobs accumulate, causing lock contention.
+
+Solution:
+```ruby
+# In config/initializers/resque.rb
+Resque.rate_limit(:lsq_rl_pro, at: 5, per: 5, concurrent: 3)
+```
+
+This ensures:
+- Rate limit: Max 5 jobs start per 5 seconds
+- Concurrent limit: Max 3 jobs run at once
+- Even if jobs take 10+ seconds, only 3 run concurrently
+
+## Testing in Your Application
+
+1. Mount the local gem in Docker:
+   ```yaml
+   volumes:
+     - /Users/parikshitsingh/Desktop/opensource/resque-throttler:/resque-throttler
+   ```
+
+2. Update Gemfile:
+   ```ruby
+   gem 'resque-throttler', path: '/resque-throttler'
+   ```
+
+3. Run tests:
+   ```bash
+   docker exec -it ninjastool rake throttler:test:all
+   ```
+
+4. Monitor:
+   ```bash
+   docker exec -it ninjastool ruby script/throttler_live_monitor.rb
+   ```
+
+## Production Deployment
+
+After testing:
+1. Push gem updates to your fork
+2. Update Gemfile to point to new version
+3. Deploy with new concurrent limits
+4. Monitor database metrics for improvement
+
+The feature is production-ready and backward compatible.