Test and benchmark community worlds against e2e tests #482
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
📊 Benchmark Results
workflow with no steps💻 Local Development
▲ Production (Vercel)
workflow with 1 step💻 Local Development
▲ Production (Vercel)
workflow with 10 sequential steps💻 Local Development
▲ Production (Vercel)
workflow with 10 parallel steps💻 Local Development
▲ Production (Vercel)
Stream BenchmarksStream benchmarks include Time to First Byte (TTFB) metrics. workflow with stream💻 Local Development
▲ Production (Vercel)
Summary: Fastest Framework by WorldWinner determined by most benchmark wins
Summary: Fastest World by FrameworkWinner determined by most benchmark wins
Column Definitions
Worlds:
|
🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
- Add community-worlds.json manifest as single source of truth - Add scripts/generate-community-worlds-workflow.mjs to generate CI workflow - Add scripts/generate-community-worlds-docs.mjs to generate docs section - Update aggregate-benchmarks.js to load community worlds dynamically - Add pnpm generate:community-worlds script - Update docs/deploying/world/index.mdx with community worlds The manifest-based approach allows: - E2E tests to be auto-generated from the manifest - Benchmark aggregation to include community worlds - Docs to stay in sync with tested worlds 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
Replace 3 mutually exclusive jobs (e2e/e2e-mongodb/e2e-redis) with a single job that starts services via docker run when needed. This eliminates the skipped job entries that appear in the GitHub Actions UI. - Use conditional docker run steps instead of services: block - Add health check loops to wait for service readiness - Add cleanup step to stop containers 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
Comment on lines
103
to
117
| function extractWorldId(artifactName: string): string | null { | ||
| // E2E results: e2e-results-{world} or e2e-dev-results-{world} | ||
| if (artifactName.startsWith('e2e-results-')) { | ||
| return artifactName.replace('e2e-results-', ''); | ||
| } | ||
| if (artifactName.startsWith('e2e-dev-results-')) { | ||
| return artifactName.replace('e2e-dev-results-', ''); | ||
| } | ||
| // Benchmark results: bench-results-{app}-{world} | ||
| if (artifactName.startsWith('bench-results-')) { | ||
| const parts = artifactName.replace('bench-results-', '').split('-'); | ||
| return parts[parts.length - 1]; // Last part is the world | ||
| } | ||
| return null; | ||
| } |
Contributor
There was a problem hiding this comment.
The world ID extraction function doesn't handle the e2e-results-community-* artifact naming pattern used by community world E2E tests, causing all community world test results to be silently ignored by the dashboard.
View Details
📝 Patch Details
diff --git a/docs/lib/github.ts b/docs/lib/github.ts
index d3c10f8..1637587 100644
--- a/docs/lib/github.ts
+++ b/docs/lib/github.ts
@@ -295,7 +295,10 @@ export function parseBenchmarkResults(results: BenchmarkResult | null): {
* Map artifact name to world ID
*/
export function artifactToWorldId(artifactName: string): string | null {
- // E2E results
+ // E2E results - check community-world pattern first to avoid partial matches
+ if (artifactName.startsWith('e2e-results-community-')) {
+ return artifactName.replace('e2e-results-community-', '');
+ }
if (artifactName.startsWith('e2e-results-')) {
return artifactName.replace('e2e-results-', '');
}
diff --git a/docs/lib/worlds-data.ts b/docs/lib/worlds-data.ts
index e8a246d..173409c 100644
--- a/docs/lib/worlds-data.ts
+++ b/docs/lib/worlds-data.ts
@@ -101,7 +101,10 @@ async function getWorkflowArtifacts(runId: number): Promise<GitHubArtifact[]> {
* Extract world ID from artifact name
*/
function extractWorldId(artifactName: string): string | null {
- // E2E results: e2e-results-{world} or e2e-dev-results-{world}
+ // E2E results: e2e-results-{world} or e2e-dev-results-{world} or e2e-results-community-{world}
+ if (artifactName.startsWith('e2e-results-community-')) {
+ return artifactName.replace('e2e-results-community-', '');
+ }
if (artifactName.startsWith('e2e-results-')) {
return artifactName.replace('e2e-results-', '');
}
Analysis
Community world E2E test results silently ignored due to artifact naming pattern mismatch
What fails: The extractWorldId() function in docs/lib/worlds-data.ts and artifactToWorldId() function in docs/lib/github.ts do not handle the e2e-results-community-* artifact naming pattern generated by .github/workflows/e2e-community-world.yml, causing all community world E2E test results to be silently skipped from the dashboard.
How to reproduce:
- The e2e-community-world.yml workflow generates artifacts named
e2e-results-community-{world-id}(e.g.,e2e-results-community-mongodb) - When the extraction function processes this artifact name, it returns
community-mongodbinstead ofmongodb - The worlds-data.ts code then attempts to match
worlds['community-mongodb']which doesn't exist in the manifest (onlyworlds['mongodb']exists) - The condition on line 180 of worlds-data.ts fails:
if (worldId && worlds[worldId])→if ('community-mongodb' && worlds['community-mongodb'])→ false - The artifact is silently skipped, never appears on the dashboard
Result: Community world test results (mongodb, turso, redis, etc.) are never displayed, even though the workflow generates artifacts for them.
Expected: The extraction functions should correctly parse e2e-results-community-mongodb → mongodb to match the world ID in the manifest, allowing community world E2E results to be properly displayed on the dashboard.
Fix: Both functions now check for the e2e-results-community- prefix first before falling back to the generic e2e-results- pattern, ensuring correct world ID extraction for community world artifacts.
Add handling for `e2e-results-community-{world}` pattern so community
world test results are properly extracted (e.g., `e2e-results-community-turso`
now correctly extracts `turso` instead of `community-turso`).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
latestnpm publish of all the community worldsnextjs-turbopackworkbench and e2e tests are run against local devdocs/worlds(preview. This page is AI slop design, so it's currently hidden on the navbar just so we can iterate on it more before finding the right final place for this - maybe merging with the existing "docs/deploying" section