fix: respect duration unit in -rl/-rls rate limit options

[jpirstin]

Proposed Changes

Fixes #1434 — the -rl and -rls rate limit options were ignoring the duration unit (e.g., /m for per-minute). Only MaxCount was being stored; duration was hardcoded to time.Second.

/claim #1434

Root Cause

In pkg/runner/runner.go, when processing options.RateLimits.AsMap(), only sourceRateLimit.MaxCount was extracted — the Duration field was discarded. In pkg/passive/passive.go, the duration passed to addRateLimiter was always time.Second.

Changes

1. pkg/subscraping/types.go: Added SourceRateLimit struct (MaxCount + Duration). Changed CustomRateLimit.Custom from map[string]uint to map[string]SourceRateLimit.
2. pkg/runner/runner.go: Store both MaxCount and Duration from parsed rate limits (defaulting to time.Second if unset).
3. pkg/passive/passive.go: Use the stored duration when building the MultiRateLimiter instead of hardcoded time.Second.

Proof

Before: subfinder -rls sitedossier=2/m → treated as 2 requests/second (duration ignored)
After: subfinder -rls sitedossier=2/m → correctly treated as 2 requests/minute

Build compiles cleanly: go build ./... passes with no errors.

Checklist

• PR created against the dev branch
• All checks passed (lint, unit/integration/regression tests) with my changes
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate)

Summary by CodeRabbit

• Enhancements
  • Enhanced rate limiting with per-source duration support. Each source can now independently configure both request count limits and time intervals, with automatic 1-second fallback for invalid durations, providing more granular control over request throttling behavior.

[coderabbitai]

Walkthrough

These changes implement per-source rate limiting with configurable durations. A new SourceRateLimit type is introduced to store both maximum request count and duration for each source. Rate limiter storage is updated to use this structured type, and duration defaults to 1 second when non-positive values are provided.

Changes

Cohort / File(s)	Summary
Rate Limiter Data Structure pkg/subscraping/types.go	Introduces new SourceRateLimit type with MaxCount and Duration fields. Updates CustomRateLimit.Custom field type from SyncLockMap[string, uint] to SyncLockMap[string, SourceRateLimit].
Rate Limiter Application pkg/runner/runner.go	Modifies rate limit storage to use SourceRateLimit values. Adds logic to compute and assign duration per source with 1-second default for non-positive values.
Rate Limiter Building pkg/passive/passive.go	Updates buildMultiRateLimiter to use per-source MaxCount and Duration from SourceRateLimit. Implements 1-second default for non-positive durations when computing rate limiter intervals.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~12 minutes

Poem

🐰 With whiskers twitched and nose held high,
We hops through limits, source by source,
Each one gets its own sweet duration—
No more the rush, just smooth cruising through the net! 🌐

🚥 Pre-merge checks | ✅ 6

✅ Passed checks (6 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately describes the main change: respecting duration units in rate limit options, which directly addresses the issue where duration units were previously ignored.
Linked Issues check	✅ Passed	The PR implements all objectives from issue #1434: storing both MaxCount and Duration in SourceRateLimit, persisting duration when building rate limiters, and ensuring rate limit options respect duration units (e.g., /m for per-minute).
Out of Scope Changes check	✅ Passed	All changes are directly scoped to fixing the rate limit duration unit issue: type definitions, rate limit storage, and rate limiter construction are all tightly focused on the stated objective.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Merge Conflict Detection	✅ Passed	✅ No merge conflicts detected when merging into dev

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

🤖 Fix all issues with AI agents

Verify each finding against the current code and only fix it if needed.


In `@pkg/runner/runner.go`:
- Line 68: The condition inside the if statement uses a tautology: remove the
redundant check against math.MaxUint and just test the meaningful part; replace
the compound check "sourceRateLimit.MaxCount > 0 && sourceRateLimit.MaxCount <=
math.MaxUint" with a single check for "sourceRateLimit.MaxCount > 0"
(referencing sourceRateLimit.MaxCount in runner.go) and remove the math.MaxUint
dependency; if the intent was to guard against overflow or a sentinel, instead
introduce an explicit sentinel constant or change the type and adjust callers
accordingly.

🧹 Nitpick comments (1)

🤖 Fix all nitpicks with AI agents

Verify each finding against the current code and only fix it if needed.


In `@pkg/runner/runner.go`:
- Line 68: The condition inside the if statement uses a tautology: remove the
redundant check against math.MaxUint and just test the meaningful part; replace
the compound check "sourceRateLimit.MaxCount > 0 && sourceRateLimit.MaxCount <=
math.MaxUint" with a single check for "sourceRateLimit.MaxCount > 0"
(referencing sourceRateLimit.MaxCount in runner.go) and remove the math.MaxUint
dependency; if the intent was to guard against overflow or a sentinel, instead
introduce an explicit sentinel constant or change the type and adjust callers
accordingly.

pkg/runner/runner.go (1)

68-68: Tautological condition: sourceRateLimit.MaxCount <= math.MaxUint is always true for a uint.

Since MaxCount is of type uint, it can never exceed math.MaxUint. This half of the condition is dead code. Likely pre-existing, but worth cleaning up.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@pkg/runner/runner.go` at line 68, The condition inside the if statement uses
a tautology: remove the redundant check against math.MaxUint and just test the
meaningful part; replace the compound check "sourceRateLimit.MaxCount > 0 &&
sourceRateLimit.MaxCount <= math.MaxUint" with a single check for
"sourceRateLimit.MaxCount > 0" (referencing sourceRateLimit.MaxCount in
runner.go) and remove the math.MaxUint dependency; if the intent was to guard
against overflow or a sentinel, instead introduce an explicit sentinel constant
or change the type and adjust callers accordingly.