Skip to content

Move GitHub contributors fetching to scheduled data-layer task#17174

Merged
wackerow merged 17 commits intodevfrom
perf/github-contributors-data-layer
Mar 12, 2026
Merged

Move GitHub contributors fetching to scheduled data-layer task#17174
wackerow merged 17 commits intodevfrom
perf/github-contributors-data-layer

Conversation

@pettinarip
Copy link
Member

@pettinarip pettinarip commented Jan 27, 2026

Summary

  • Replace per-request GitHub API calls with pre-fetched data stored in Netlify Blobs
  • Add fetchGitHubContributors scheduled task running weekly to fetch all contributor data
  • Use git/trees API for single-request content file listing (replaces recursive directory walks)
  • Fetch commits in parallel batches (20 concurrent requests) for performance
  • Update contributors.ts to read from data-layer instead of making live API calls
  • Bump @trigger.dev/sdk to 4.3.3

Test plan

  • Verify the scheduled task runs successfully and stores data in Netlify Blobs
  • Verify contributor information displays correctly on content pages
  • Verify contributor information displays correctly on app pages
  • Verify build succeeds without errors

Replace per-request GitHub API calls with pre-fetched data stored in
Netlify Blobs. This eliminates hundreds of API calls during page renders.

- Add fetchGitHubContributors scheduled task (weekly)
- Use git/trees API for single-request content listing
- Fetch commits in parallel batches (20 concurrent)
- Store results in data-layer for fast retrieval
- Update contributors.ts to read from data-layer
- Bump @trigger.dev/sdk to 4.3.3
@github-actions github-actions bot added dependencies 📦 Changes related to project dependencies tooling 🔧 Changes related to tooling of the project labels Jan 27, 2026
@netlify
Copy link

netlify bot commented Jan 27, 2026

Deploy Preview for ethereumorg ready!

Name Link
🔨 Latest commit 4a897cf
🔍 Latest deploy log https://app.netlify.com/projects/ethereumorg/deploys/69b2c3e32444ed0008cad38e
😎 Deploy Preview https://deploy-preview-17174.ethereum.it
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.
Lighthouse
Lighthouse
7 paths audited
Performance: 55 (🟢 up 2 from production)
Accessibility: 94 (🟢 up 1 from production)
Best Practices: 100 (no change from production)
SEO: 99 (no change from production)
PWA: 59 (no change from production)
View the detailed breakdown and full score reports

To edit notification comments on pull requests, go to your Netlify project configuration.

@pettinarip pettinarip marked this pull request as draft January 27, 2026 13:46
The previous commit moved GitHub contributor fetching to the data-layer.
This removes the now-unused fetchAndCacheGitHubContributors and
fetchWithRateLimit functions from gh.ts, along with their imports.
Also consolidates CONTENT_DIR and OLD_CONTENT_DIR imports in the new
fetcher to use the shared constants.
@github-actions github-actions bot added the documentation 📖 Change or add documentation label Jan 27, 2026
@pettinarip pettinarip marked this pull request as ready for review March 4, 2026 15:37
Copy link
Collaborator

@myelinated-wackerow myelinated-wackerow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Solid PR -- the architecture follows the established data-layer pattern cleanly, the tree API discovery is smart (single call vs recursive walks), and the parallel batching with rate limit handling is well done. The mechanical removal of the useless per-render commitHistoryCache across ~38 pages is correct and consistent.

A few notes:

1. Hardcoded date in exchangesByCountry.ts

The new exchangesByCountryLastUpdated export replaces a dynamic git log call, but it means anyone updating exchangesByCountry.ts needs to manually update this date too. Without an automated mechanism (pre-commit hook, script, etc.), this will inevitably drift. Worth at least a prominent comment warning maintainers, or better, an automated solution.

2. new Date().toISOString() fallback in contributors.ts

const lastUpdatedDate = gitHubContributors[0]?.date || new Date().toISOString()

If a slug is missing from the data-layer (new content added between weekly runs, data fetch failure, etc.), this shows "today" as the last edit date, which is misleading. The old code fell back to a local git log query which at least returned an accurate date. Consider a safer fallback -- perhaps a fixed epoch date, or omitting the date entirely when unknown.

3. Minor: JSDoc says "daily" but schedule is weekly

The fetchGitHubContributors JSDoc comment says "runs as a scheduled task (daily)" but it's registered in the WEEKLY array with a Sunday cron. Not a blocker, just a docs inconsistency.


Reviewed by Claude Opus 4.6

- Make lastEditLocaleTimestamp optional and hide date when contributor
  data is missing instead of showing a misleading fallback
- Add prominent warning comment on exchangesByCountry hardcoded date
- Fix JSDoc: fetchGitHubContributors runs weekly, not daily
When a commit author's email isn't linked to a GitHub account, the API
returns author: null. The fetcher was filtering these out entirely,
causing pages with only such commits to have no contributor data. Now
uses the git commit author name as a fallback identity and preserves
the commit date.
@pettinarip
Copy link
Member Author

@wackerow thanks for the review! all three items addressed.

new Date().toISOString() fallback — now it hides the date entirely when contributor data is missing. lastEditLocaleTimestamp is optional throughout the type chain and the UI conditionally renders it. Showing any fallback date felt misleading regardless of source.

Additionally, investigating the missing dates uncovered a deeper bug: commits authored with emails not linked to a GitHub account return author: null from the API. The fetcher was filtering these out entirely, so pages with only such commits (e.g. community/support/faq) had no contributor data at all. Fixed by falling back to the git commit author name when the GitHub author is null.

@wackerow wackerow merged commit 8975844 into dev Mar 12, 2026
7 checks passed
@wackerow wackerow deleted the perf/github-contributors-data-layer branch March 12, 2026 19:33
myelinated-wackerow added a commit that referenced this pull request Mar 12, 2026
Resolve conflict in src/lib/utils/gh.ts by accepting
dev version -- fetching logic moved to data-layer
in PR #17174, so the old code from this branch is
no longer needed here.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: wackerow <54227730+wackerow@users.noreply.github.com>
myelinated-wackerow added a commit that referenced this pull request Mar 12, 2026
Move co-author parsing, bot/AI filtering, and
exclusion lists from the removed gh.ts fetching
code into fetchGitHubContributors.ts where the
contributor fetching now lives post-#17174.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: wackerow <54227730+wackerow@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dependencies 📦 Changes related to project dependencies documentation 📖 Change or add documentation tooling 🔧 Changes related to tooling of the project

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants