Skip to content

fix: prevent YouTube API quota exhaustion (#1157)#1175

Merged
bcordis merged 4 commits intodevelopmentfrom
fix/youtube-quota-exhaustion
Mar 19, 2026
Merged

fix: prevent YouTube API quota exhaustion (#1157)#1175
bcordis merged 4 commits intodevelopmentfrom
fix/youtube-quota-exhaustion

Conversation

@bcordis
Copy link
Member

@bcordis bcordis commented Mar 19, 2026

Summary

Fixes persistent YouTube API quota exhaustion on sites using mod_proclaim_youtube in live_first mode, despite v10.2.0 quota optimizations.

Root cause analysis

The 10,000 daily quota was being consumed through a combination of:

  • Cache clearing resets quota protection — Joomla "Clear Cache" wiped throttle files, quota counter, and last-known-video, causing immediate search.list spikes (200 units each)
  • Race condition on throttle — concurrent requests could both miss the throttle file and each trigger a 100-unit search.list before the throttle was written
  • 15-minute throttle too short — sites with no active streams burned ~200 units every 15 minutes (19,200/day)
  • JS polling adds up — multiple module instances each polling independently at 2-minute intervals
  • Symlinked dev installs share cache — quota counters leaked between sites sharing the same repo

Fixes

Fix Impact Details
Move cache to media/com_proclaim/youtube_cache/ High Throttle, quota, last-known-video files survive Joomla cache clearing
Site-identity namespace ({hash}/) High Each Joomla site gets isolated cache via JPATH_CACHE hash subdirectory
File locking on throttle High flock() prevents race condition where concurrent requests both trigger search.list
Throttle TTL 15min → 1 hour High 4× fewer search.list calls when no stream active
JS: stop polling at quota < 100 Medium Reserves remaining units for search.list on page loads
JS: force max backoff at quota < 500 Medium Slows polling as quota depletes
JS: deduplicate polling timers Medium Multiple module instances with same server+video share one timer
Cache scrub in scheduled task Low Cleans expired throttle, schedule, quota, and corrupted files
Gitignore media/youtube_cache/ Low Runtime cache files excluded from repo

Estimated impact: Daily quota burn drops from ~17,700 to ~2,000-3,000 units.

Runtime directory creation

The youtube_cache/ directory is not in the build package — it's created automatically at runtime by ensureDir() on first use. Existing files from the old JPATH_CACHE location are auto-migrated.

Test plan

  • Clear Joomla cache → verify youtube_cache files in media/com_proclaim/youtube_cache/{hash}/ are NOT deleted
  • Two symlinked dev sites → verify each has its own {hash}/ subdirectory with separate quota counters
  • Module in live_first mode with no stream → verify search throttled for 1 hour (check log)
  • Multiple concurrent page loads → verify only one search.list call (not one per request)
  • Module on multiple pages → verify JS polling deduplication (one timer per server+video)
  • Quota approaching limit → verify JS polling stops at < 100 remaining
  • Platform stats task runs → verify expired cache files cleaned up
  • Existing sites upgrade → verify old cache files migrate from JPATH_CACHE to new location
  • Fresh install → verify youtube_cache directory created on first module load
  • All 538 PHPUnit tests pass
  • PHP syntax clean

Closes #1157

🤖 Generated with Claude Code

bcordis and others added 4 commits March 18, 2026 19:27
…conditions

Move YouTube persistent cache files (throttle, quota counter, last-known-video)
from JPATH_CACHE to JPATH_ROOT/media/com_proclaim/youtube_cache/ so Joomla
"Clear Cache" no longer resets quota protection. Add file locking (flock) to
search throttle reads/writes to prevent concurrent requests from both triggering
100-unit search.list calls. Increase default search throttle TTL from 15 minutes
to 1 hour, reducing worst-case search.list calls 4x.

JS polling improvements: stop polling when quotaRemaining < 100 (reserves units
for search.list on page loads), force max backoff when < 500, and deduplicate
polling timers across module instances sharing the same server+video URL.

Add cache scrub method that removes expired throttle, schedule, quota, and
corrupted files — runs automatically in the platform stats scheduled task.

Closes #1157

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Prevents quota_*.json, throttle, and other runtime YouTube cache files
from syncing between dev sites, which could cause cross-site quota
counter conflicts.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Use a hash of JPATH_CACHE as a subdirectory so multiple Joomla sites
sharing the same repo via symlinks don't share quota counters and
throttle files. In production each site has its own media directory,
so this is just extra safety.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Now that cache is namespaced by site hash subdirectories, ignore the
whole directory tree rather than just *.json files.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@bcordis bcordis merged commit 743faa5 into development Mar 19, 2026
6 checks passed
@bcordis bcordis deleted the fix/youtube-quota-exhaustion branch March 19, 2026 00:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

NFSDA.org still hitting YouTube API 10,000 quota limit on v10.2.0

1 participant