Move GitHub contributors fetching to scheduled data-layer task#17174
Move GitHub contributors fetching to scheduled data-layer task#17174
Conversation
Replace per-request GitHub API calls with pre-fetched data stored in Netlify Blobs. This eliminates hundreds of API calls during page renders. - Add fetchGitHubContributors scheduled task (weekly) - Use git/trees API for single-request content listing - Fetch commits in parallel batches (20 concurrent) - Store results in data-layer for fast retrieval - Update contributors.ts to read from data-layer - Bump @trigger.dev/sdk to 4.3.3
✅ Deploy Preview for ethereumorg ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
The previous commit moved GitHub contributor fetching to the data-layer. This removes the now-unused fetchAndCacheGitHubContributors and fetchWithRateLimit functions from gh.ts, along with their imports. Also consolidates CONTENT_DIR and OLD_CONTENT_DIR imports in the new fetcher to use the shared constants.
myelinated-wackerow
left a comment
There was a problem hiding this comment.
Solid PR -- the architecture follows the established data-layer pattern cleanly, the tree API discovery is smart (single call vs recursive walks), and the parallel batching with rate limit handling is well done. The mechanical removal of the useless per-render commitHistoryCache across ~38 pages is correct and consistent.
A few notes:
1. Hardcoded date in exchangesByCountry.ts
The new exchangesByCountryLastUpdated export replaces a dynamic git log call, but it means anyone updating exchangesByCountry.ts needs to manually update this date too. Without an automated mechanism (pre-commit hook, script, etc.), this will inevitably drift. Worth at least a prominent comment warning maintainers, or better, an automated solution.
2. new Date().toISOString() fallback in contributors.ts
const lastUpdatedDate = gitHubContributors[0]?.date || new Date().toISOString()If a slug is missing from the data-layer (new content added between weekly runs, data fetch failure, etc.), this shows "today" as the last edit date, which is misleading. The old code fell back to a local git log query which at least returned an accurate date. Consider a safer fallback -- perhaps a fixed epoch date, or omitting the date entirely when unknown.
3. Minor: JSDoc says "daily" but schedule is weekly
The fetchGitHubContributors JSDoc comment says "runs as a scheduled task (daily)" but it's registered in the WEEKLY array with a Sunday cron. Not a blocker, just a docs inconsistency.
Reviewed by Claude Opus 4.6
- Make lastEditLocaleTimestamp optional and hide date when contributor data is missing instead of showing a misleading fallback - Add prominent warning comment on exchangesByCountry hardcoded date - Fix JSDoc: fetchGitHubContributors runs weekly, not daily
When a commit author's email isn't linked to a GitHub account, the API returns author: null. The fetcher was filtering these out entirely, causing pages with only such commits to have no contributor data. Now uses the git commit author name as a fallback identity and preserves the commit date.
|
@wackerow thanks for the review! all three items addressed.
Additionally, investigating the missing dates uncovered a deeper bug: commits authored with emails not linked to a GitHub account return |
Resolve conflict in src/lib/utils/gh.ts by accepting dev version -- fetching logic moved to data-layer in PR #17174, so the old code from this branch is no longer needed here. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Co-Authored-By: wackerow <54227730+wackerow@users.noreply.github.com>
Move co-author parsing, bot/AI filtering, and exclusion lists from the removed gh.ts fetching code into fetchGitHubContributors.ts where the contributor fetching now lives post-#17174. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Co-Authored-By: wackerow <54227730+wackerow@users.noreply.github.com>

Summary
fetchGitHubContributorsscheduled task running weekly to fetch all contributor datacontributors.tsto read from data-layer instead of making live API calls@trigger.dev/sdkto 4.3.3Test plan