Skip to content

Commit d25588c

Browse files
claudeclaude
committed
Add visual regression testing for prompt
Adds a testing framework to catch unintended prompt appearance changes: - test/visual/scenarios.zsh: Defines test scenarios for different prompt states (git clean/dirty, error, long commands, etc.) - test/visual/run-tests.sh: Test runner that captures screenshots and compares against baselines - test/visual/Dockerfile: Consistent environment for CI - .github/workflows/visual-tests.yml: CI workflow for automated testing Supports multiple capture methods: - termshot (preferred): Native terminal screenshots - aha + wkhtmltoimage: HTML-based rendering - text: Fallback using plain diff
1 parent cf600a1 commit d25588c

File tree

7 files changed

+732
-0
lines changed

7 files changed

+732
-0
lines changed

‎.github/workflows/visual-tests.yml‎

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
name: Visual Regression Tests
2+
3+
on:
4+
push:
5+
paths:
6+
- 'dot_prompt.zsh'
7+
- 'dot_zshrc'
8+
- 'test/visual/**'
9+
pull_request:
10+
paths:
11+
- 'dot_prompt.zsh'
12+
- 'dot_zshrc'
13+
- 'test/visual/**'
14+
15+
jobs:
16+
visual-tests:
17+
runs-on: ubuntu-latest
18+
19+
steps:
20+
- name: Checkout
21+
uses: actions/checkout@v4
22+
23+
- name: Install dependencies
24+
run: |
25+
sudo apt-get update
26+
sudo apt-get install -y zsh git aha wkhtmltopdf imagemagick fonts-jetbrains-mono
27+
28+
- name: Install termshot
29+
run: |
30+
curl -fsSL https://github.com/homeport/termshot/releases/latest/download/termshot_linux_amd64.tar.gz \
31+
| sudo tar -xz -C /usr/local/bin termshot
32+
33+
- name: Run visual regression tests
34+
run: |
35+
chmod +x test/visual/run-tests.sh test/visual/scenarios.zsh
36+
./test/visual/run-tests.sh test
37+
env:
38+
TERM: xterm-256color
39+
40+
- name: Upload diff artifacts
41+
if: failure()
42+
uses: actions/upload-artifact@v4
43+
with:
44+
name: visual-diff
45+
path: test/visual/diff/
46+
retention-days: 7
47+
48+
- name: Upload actual screenshots
49+
if: failure()
50+
uses: actions/upload-artifact@v4
51+
with:
52+
name: visual-actual
53+
path: test/visual/actual/
54+
retention-days: 7
55+
56+
# Job to update baselines (manual trigger)
57+
update-baselines:
58+
runs-on: ubuntu-latest
59+
if: github.event_name == 'workflow_dispatch'
60+
61+
steps:
62+
- name: Checkout
63+
uses: actions/checkout@v4
64+
65+
- name: Install dependencies
66+
run: |
67+
sudo apt-get update
68+
sudo apt-get install -y zsh git aha wkhtmltopdf imagemagick fonts-jetbrains-mono
69+
70+
- name: Install termshot
71+
run: |
72+
curl -fsSL https://github.com/homeport/termshot/releases/latest/download/termshot_linux_amd64.tar.gz \
73+
| sudo tar -xz -C /usr/local/bin termshot
74+
75+
- name: Generate baselines
76+
run: |
77+
chmod +x test/visual/run-tests.sh test/visual/scenarios.zsh
78+
./test/visual/run-tests.sh generate
79+
env:
80+
TERM: xterm-256color
81+
82+
- name: Commit updated baselines
83+
run: |
84+
git config user.name "GitHub Actions"
85+
git config user.email "actions@github.com"
86+
git add test/visual/snapshots/
87+
git commit -m "Update visual regression baselines" || echo "No changes to commit"
88+
git push

‎test/visual/.gitignore‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
# Test artifacts (not committed)
2+
actual/
3+
diff/
4+
test-repo/
5+
6+
# Keep snapshots directory but ignore if empty
7+
!snapshots/
8+
snapshots/*
9+
!snapshots/.gitkeep

‎test/visual/Dockerfile‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
# Dockerfile for visual regression testing of zsh prompt
2+
# Provides a consistent environment for capturing terminal screenshots
3+
4+
FROM ubuntu:22.04
5+
6+
ENV DEBIAN_FRONTEND=noninteractive
7+
ENV TERM=xterm-256color
8+
9+
# Install dependencies
10+
RUN apt-get update && apt-get install -y \
11+
zsh \
12+
git \
13+
curl \
14+
aha \
15+
wkhtmltopdf \
16+
imagemagick \
17+
fonts-jetbrains-mono \
18+
&& rm -rf /var/lib/apt/lists/*
19+
20+
# Install termshot (Go binary)
21+
RUN curl -fsSL https://github.com/homeport/termshot/releases/latest/download/termshot_linux_amd64.tar.gz \
22+
| tar -xz -C /usr/local/bin termshot
23+
24+
# Set up work directory
25+
WORKDIR /dotfiles
26+
27+
# Copy test files
28+
COPY test/visual/ /dotfiles/test/visual/
29+
COPY dot_prompt.zsh /dotfiles/
30+
31+
# Make scripts executable
32+
RUN chmod +x /dotfiles/test/visual/*.sh /dotfiles/test/visual/*.zsh
33+
34+
# Default command runs tests
35+
CMD ["/dotfiles/test/visual/run-tests.sh", "test"]

‎test/visual/README.md‎

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
# Visual Regression Tests for Zsh Prompt
2+
3+
This directory contains visual regression tests for the custom zsh prompt.
4+
The tests capture screenshots of the prompt in various states and compare
5+
them against baseline images to detect unintended visual changes.
6+
7+
## Test Scenarios
8+
9+
The following prompt states are tested:
10+
11+
| Scenario | Description |
12+
|----------|-------------|
13+
| `no_git` | Directory without git repository |
14+
| `git_clean` | Clean git repository |
15+
| `git_staged` | Repository with staged changes |
16+
| `git_unstaged` | Repository with unstaged changes |
17+
| `git_untracked` | Repository with untracked files |
18+
| `git_mixed` | Repository with staged, unstaged, and untracked changes |
19+
| `error` | After a command exits with error |
20+
| `long_command` | After a command taking 5 seconds |
21+
| `very_long_command` | After a command taking 1 hour |
22+
| `deep_path` | Deeply nested directory path |
23+
24+
## Requirements
25+
26+
### For full image-based testing:
27+
- `termshot` - Terminal screenshot tool ([homeport/termshot](https://github.com/homeport/termshot))
28+
- `imagemagick` - For image comparison (`compare` command)
29+
30+
### Alternative (HTML-based):
31+
- `aha` - ANSI to HTML converter
32+
- `wkhtmltopdf` - HTML to image converter
33+
34+
### Fallback (text-based):
35+
- No additional dependencies (uses plain diff)
36+
37+
## Usage
38+
39+
```bash
40+
# Run tests (compares against baselines)
41+
./run-tests.sh test
42+
43+
# Generate baseline snapshots
44+
./run-tests.sh generate
45+
46+
# Update baselines after reviewing changes
47+
./run-tests.sh update
48+
49+
# Clean up test artifacts
50+
./run-tests.sh clean
51+
```
52+
53+
## How It Works
54+
55+
1. **Scenario Setup**: `scenarios.zsh` creates temporary git repos in various
56+
states (clean, dirty, staged changes, etc.)
57+
58+
2. **Capture**: The prompt is rendered and captured as either:
59+
- PNG images (via `termshot` or `aha` + `wkhtmltoimage`)
60+
- Plain text with ANSI codes (fallback)
61+
62+
3. **Compare**: Current screenshots are compared against baselines:
63+
- Image diff via ImageMagick (shows pixel differences)
64+
- Text diff (shows line-by-line changes)
65+
66+
4. **Report**: Failed tests generate diff files showing what changed.
67+
68+
## CI Integration
69+
70+
The GitHub Actions workflow (`.github/workflows/visual-tests.yml`):
71+
- Runs on every push/PR that modifies prompt files
72+
- Uses a consistent Ubuntu environment for reproducible screenshots
73+
- Uploads diff artifacts on failure for easy debugging
74+
75+
## Directory Structure
76+
77+
```
78+
test/visual/
79+
├── README.md # This file
80+
├── Dockerfile # Consistent test environment
81+
├── run-tests.sh # Main test runner
82+
├── scenarios.zsh # Test scenario definitions
83+
├── snapshots/ # Baseline images (committed)
84+
├── actual/ # Current screenshots (gitignored)
85+
└── diff/ # Visual diffs (gitignored)
86+
```
87+
88+
## Updating Baselines
89+
90+
When you intentionally change the prompt appearance:
91+
92+
1. Run tests to generate new screenshots: `./run-tests.sh test`
93+
2. Review the diffs in `diff/` directory
94+
3. If changes are expected, update baselines: `./run-tests.sh update`
95+
4. Commit the new baselines in `snapshots/`
96+
97+
## Troubleshooting
98+
99+
**Tests fail with "MISSING BASELINE"**
100+
Run `./run-tests.sh generate` to create initial baselines.
101+
102+
**Different results on different machines**
103+
Visual tests can be sensitive to font rendering differences. For consistent
104+
results, run tests in Docker or CI. The text-based fallback is more portable.
105+
106+
**termshot not found**
107+
Install from https://github.com/homeport/termshot or use `aha` fallback:
108+
```bash
109+
CAPTURE_METHOD=aha ./run-tests.sh test
110+
```

0 commit comments

Comments
 (0)