🧪 Fix SDK E2E tests to work with vizzly tdd run #183

Robdel12 · 2026-01-17T07:28:36Z

Summary

Detect VIZZLY_SERVER_URL env var in all SDK E2E tests to skip starting their own TDD server when running under vizzly tdd run
Fix Vitest tests to use correct browser mode API (getByRole, getByText, getByTestId) instead of Playwright-style page.locator()
Fix Static-Site tests to use URL-style paths (/, /features) instead of filenames (index.html)
Fix Storybook pattern matching to use story.id format (*button* matches components-button--primary)
Fix Ember assertions to check result.status instead of result.success
Add proper config structure for page discovery (pageDiscovery.useSitemap instead of flat useSitemap)

Test Results

All SDK E2E tests now pass when run via vizzly tdd run:

SDK	Tests	Status
Storybook	13/13	✅
Static-Site	13/13	✅
Ember	9/9	✅
Vitest	24/24	✅
Ruby	10/10	✅

Test plan

Run vizzly tdd run for each SDK's E2E tests
Verify all tests pass
Run lint checks for all SDKs

vizzly-testing · 2026-01-17T07:29:46Z

🐻 Vizzly — 7 changes need review

Status	Count
Passed	12
Changed	7
Auto-approved	12

Visual differences (7)

fullscreen-viewer · Firefox · 1920×1080 · 1.5% diff

search-no-results · Firefox · 1920×1080 · 0.1% diff

search-no-results · Firefox · 375×978 · 3.3% diff

bulk-accept-dialog · Firefox · 1920×1080 · 5.8% diff

viewer-overlay-mode · Firefox · 1920×1080 · 1.6% diff

viewer-zoomed-100 · Firefox · 1920×1080 · 0.3% diff

...and 1 more in Vizzly.

Review changes →

_{fix-sdk-e2e-tests · 8369a5a3}

vizzly-testing · 2026-01-17T07:30:24Z

🐻 Vizzly — 1 change needs review

Status	Count
Changed	1
Auto-approved	4

Visual differences (5)

vizzly-help · 1202×1430 · 4.9% diff

vizzly-invalid-command · 1202×578 · 12.0% diff

vizzly-doctor · 1202×578 · 12.0% diff

vizzly-run-help · 1202×722 · 9.6% diff

vizzly-tdd-help · 1202×578 · 12.0% diff

Review changes →

_{fix-sdk-e2e-tests · 8369a5a3}

claude · 2026-01-17T07:31:42Z

Code Review - PR #183

This PR makes significant improvements to SDK E2E tests, ensuring they work correctly when run via vizzly tdd run. The changes fix API misuse, improve test structure, and add proper server detection.

Strengths

Excellent Server Detection Pattern - The VIZZLY_SERVER_URL detection is clean and consistent
Correct API Fixes - Fixed Vitest to use Testing Library, Static-Site to use URL paths, Ember assertions, Storybook patterns
Comprehensive Test Coverage - All SDKs now test element/full screenshots, properties, thresholds, multiple pages
Proper Config Structure - Fixed pageDiscovery nesting

Issues Found

CRITICAL

Command Injection in Ruby (clients/ruby/test/e2e_test.rb)
Uses string interpolation in shell command. While port is a number, this pattern is risky. Use spawn array syntax instead.

WARNINGS

Hardcoded Timeouts - May fail on slow CI. Use env-based timeouts (CI ? 5000 : 2000)
Missing Error Handling - Server spawn needs consistent error handling before promise resolves
Double CI Execution - Tests run twice (TDD + Cloud). Consider making conditional to save CI time

SUGGESTIONS

Test Site Coupling - All SDKs depend on ../../../test-site. Document this, add CI check
Code Duplication - Extract shared E2E setup logic to test-utils
Documentation - Add SDK README sections for E2E requirements and troubleshooting

Performance

CI time doubles with TDD+Cloud runs - monitor for bottlenecks
Browser instances could be reused within SDKs
Consider parallel test execution within each SDK

Verdict

Approve with minor fixes

Priority before merge:

Fix Ruby command injection (low risk but best practice)
Ensure consistent error handling in server spawns

Nice to have: shared E2E helpers, documentation, conditional CI execution

Great work improving test infrastructure!

vizzly-testing · 2026-01-17T07:43:59Z

🐻 Vizzly — 5 changes need review

Status	Count
Passed	14
Changed	5
Auto-approved	14

Visual differences (5)

fullscreen-viewer · Firefox · 1920×1080 · 1.5% diff

viewer-overlay-mode · Firefox · 1920×1080 · 1.6% diff

fullscreen-viewer · Firefox · 375×667 · 0.9% diff

search-no-results · Firefox · 375×978 · 3.5% diff

viewer-zoomed-100 · Firefox · 375×667 · 198.1% diff

Review changes →

_{fix-sdk-e2e-tests · 31d6df25}

vizzly-testing · 2026-01-17T07:44:34Z

🐻 Vizzly — 1 change needs review

Status	Count
Changed	1
Auto-approved	4

Visual differences (5)

vizzly-help · 1202×1430 · 4.9% diff

vizzly-doctor · 1202×578 · 12.0% diff

vizzly-tdd-help · 1202×578 · 12.0% diff

vizzly-invalid-command · 1202×578 · 12.0% diff

vizzly-run-help · 1202×722 · 9.6% diff

Review changes →

_{fix-sdk-e2e-tests · 31d6df25}

vizzly-testing · 2026-01-17T07:49:57Z

🐻 Vizzly — Approved ✓

19 comparisons approved.

Status	Count
Passed	11
Changed	8
Auto-approved	11

Visual differences (8)

bulk-accept-dialog · Firefox · 1920×1080 · 5.1% diff

viewer-overlay-mode · Firefox · 1920×1080 · 1.6% diff

search-no-results · Firefox · 1920×1080 · 0.1% diff

fullscreen-viewer · Firefox · 1920×1080 · 1.5% diff

viewer-zoomed-100 · Firefox · 1920×1080 · 0.3% diff

search-no-results · Firefox · 375×978 · 3.5% diff

...and 2 more in Vizzly.

View build →

_{fix-sdk-e2e-tests · 890d7460}

vizzly-testing · 2026-01-17T07:50:28Z

🐻 Vizzly — Approved ✓

5 comparisons approved.

Status	Count
Changed	1
Auto-approved	4

Visual differences (5)

vizzly-help · 1202×1430 · 4.9% diff

vizzly-tdd-help · 1202×578 · 12.0% diff

vizzly-run-help · 1202×722 · 9.6% diff

vizzly-doctor · 1202×578 · 12.0% diff

vizzly-invalid-command · 1202×578 · 12.0% diff

View build →

_{fix-sdk-e2e-tests · 890d7460}

- Detect VIZZLY_SERVER_URL env var to skip starting own TDD server - Fix Vitest tests to use correct browser mode API (getByRole, getByText) - Fix Static-Site tests to use URL-style paths (/, /features) not filenames - Fix Storybook pattern matching to use story.id format (*button*) - Fix Ember assertions to check result.status instead of result.success - Add proper config structure for page discovery (pageDiscovery.useSitemap) All SDK E2E tests now pass when run via `vizzly tdd run`: - Storybook: 13/13 - Static-Site: 13/13 - Ember: 9/9 - Vitest: 24/24 - Ruby: 10/10

vizzly-testing · 2026-01-17T08:00:39Z

🐻 Vizzly — Approved ✓

18 comparisons approved.

Status	Count
Passed	10
Changed	8
Auto-approved	10

Visual differences (8)

fullscreen-viewer · Firefox · 1920×1080 · 1.6% diff

search-no-results · Firefox · 1920×1080 · 0.1% diff

bulk-accept-dialog · Firefox · 375×892 · 95.5% diff

bulk-accept-dialog · Firefox · 1920×1080 · 5.8% diff

viewer-zoomed-100 · Firefox · 1920×1080 · 0.3% diff

search-no-results · Firefox · 375×978 · 3.2% diff

...and 2 more in Vizzly.

View build →

_{fix-sdk-e2e-tests · 242af430}

vizzly-testing · 2026-01-17T08:01:08Z

🐻 Vizzly — Approved ✓

5 comparisons approved.

Status	Count
Changed	1
Auto-approved	4

Visual differences (5)

vizzly-doctor · 1202×578 · 12.0% diff

vizzly-help · 1202×1430 · 4.9% diff

vizzly-tdd-help · 1202×578 · 12.0% diff

vizzly-run-help · 1202×722 · 9.6% diff

vizzly-invalid-command · 1202×578 · 12.0% diff

View build →

_{fix-sdk-e2e-tests · 242af430}

- Add custom `loadPage` command to fetch and inject HTML content - Start static server for test-site with dynamic port allocation - Use real FluffyCloud test-site instead of inline HTML - Screenshots now match other SDKs (Storybook, Static-Site, Ember, Ruby) The `loadPage` command works around Vitest browser mode's limitation of not supporting `page.goto()` by fetching HTML and using `frame.setContent()` to inject it into the test iframe.

- Split monolithic ci.yml (900+ lines) into focused workflows: - ci.yml: Core CLI lint, tests, build (78 lines) - reporter.yml: Reporter visual tests (Vizzly cloud) - tui.yml: TUI visual tests (Vizzly cloud) - sdk-e2e.yml: All SDK E2E tests (TDD + Cloud mode) - sdk-unit.yml: SDK unit tests (conditional on file changes) - Add cloud mode for Ruby, Storybook, and Static-Site SDKs - Each workflow has its own status check job for branch protection - SDK E2E always runs to catch CLI breaking changes - SDK unit tests only run when that SDK's files change New secrets needed: - VIZZLY_RUBY_CLIENT_TOKEN - VIZZLY_STORYBOOK_CLIENT_TOKEN - VIZZLY_STATIC_SITE_CLIENT_TOKEN

vizzly-testing · 2026-01-17T22:30:34Z

🐻 Vizzly — 8 changes need review

Status	Count
Passed	11
Changed	8
Auto-approved	11

Changes needing review (8)

fullscreen-viewer · Firefox · 1920×1080 · 2.3% diff

bulk-accept-dialog · Firefox · 1920×1080 · 0.1% diff

search-no-results · Firefox · 1920×1080 · 0.1% diff

viewer-zoomed-100 · Firefox · 1920×1080 · 0.3% diff

fullscreen-viewer · Firefox · 375×667 · 1.0% diff

viewer-zoomed-100 · Firefox · 375×667 · 198.1% diff

...and 2 more in Vizzly.

Review changes →

_{CLI Reporter · fix-sdk-e2e-tests · d55cca14}

vizzly-testing · 2026-01-17T22:30:59Z

🐻 Vizzly — 1 change needs review

Status	Count
Changed	1
Auto-approved	4

Changes needing review (1)

vizzly-help · 1202×1430 · 4.9% diff

Review changes →

_{CLI TUI · fix-sdk-e2e-tests · d55cca14}

Robdel12 force-pushed the fix-sdk-e2e-tests branch from 8369a5a to 31d6df2 Compare January 17, 2026 07:42

Robdel12 force-pushed the fix-sdk-e2e-tests branch from 31d6df2 to 890d746 Compare January 17, 2026 07:48

Robdel12 force-pushed the fix-sdk-e2e-tests branch from 890d746 to 242af43 Compare January 17, 2026 07:59

This comment has been minimized.

Sign in to view

🧪 Fix SDK E2E tests to work with vizzly tdd run #183

Are you sure you want to change the base?

🧪 Fix SDK E2E tests to work with vizzly tdd run #183

Uh oh!

Conversation

Robdel12 commented Jan 17, 2026

Summary

Test Results

Test plan

Uh oh!

vizzly-testing bot commented Jan 17, 2026

🐻 Vizzly — 7 changes need review

Uh oh!

vizzly-testing bot commented Jan 17, 2026

🐻 Vizzly — 1 change needs review

Uh oh!

claude bot commented Jan 17, 2026

Code Review - PR #183

Strengths

Issues Found

CRITICAL

WARNINGS

SUGGESTIONS

Performance

Verdict

Uh oh!

vizzly-testing bot commented Jan 17, 2026

🐻 Vizzly — 5 changes need review

Uh oh!

vizzly-testing bot commented Jan 17, 2026

🐻 Vizzly — 1 change needs review

Uh oh!

vizzly-testing bot commented Jan 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🐻 Vizzly — Approved ✓

Uh oh!

vizzly-testing bot commented Jan 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🐻 Vizzly — Approved ✓

Uh oh!

vizzly-testing bot commented Jan 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🐻 Vizzly — Approved ✓

Uh oh!

vizzly-testing bot commented Jan 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🐻 Vizzly — Approved ✓

Uh oh!

This comment has been minimized.

This comment has been minimized.

vizzly-testing bot commented Jan 17, 2026

🐻 Vizzly — 8 changes need review

Uh oh!

vizzly-testing bot commented Jan 17, 2026

🐻 Vizzly — 1 change needs review

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

vizzly-testing bot commented Jan 17, 2026 •

edited

Loading

vizzly-testing bot commented Jan 17, 2026 •

edited

Loading

vizzly-testing bot commented Jan 17, 2026 •

edited

Loading

vizzly-testing bot commented Jan 17, 2026 •

edited

Loading