screenpipe regression testing checklist

purpose: prevent regressions. every PR touching window management, activation policy, tray, dock, monitors, or audio MUST be tested against the relevant sections below before merge.

critical edge cases (sorted by regression frequency)

1. window overlay & fullscreen spaces (macOS)

window mode CSS restore — In window mode (not fullscreen), verify that CSS styling is correct and as expected (e.g., no unexpected transparent panels).
keyboard input in main window from tray — Open the main window from the tray icon and immediately try typing. Verify that keyboard input works without requiring a click.
WKWebView keyboard focus recovery — Interact with embedded web views (e.g., billing, help sections), then navigate back to other UI elements. Verify keyboard focus is correctly recovered by the WKWebView.

these break CONSTANTLY. any change to window_api.rs, main.rs shortcuts, activation policy, or NSPanel code must test ALL of these.

commits that broke this area: 0752ea59, d89c5f14, 4a64fd1a, fa591d6e, 8706ae73, 6d44af13, b6ff1bf7, 09a18070

2. dock icon & tray icon (macOS)

commits that broke this area: 0752ea59, 7562ec62, 2a2bd9b5, f2f7f770, 5cb100ea

3. monitor plug/unplug

commits: 28e5c247

unplug external monitor while recording — recording continues on remaining monitor(s). no crash. log shows "Monitor X disconnected".
plug in external monitor while recording — new monitor is detected within 5 seconds. recording starts on it. log shows "Monitor X reconnected".
unplug and replug same monitor — recording resumes. same monitor ID reused. no duplicate recording tasks.
unplug all external monitors (laptop only) — built-in display continues recording. no crash.
plug monitor with different resolution — recording starts at correct resolution. OCR works on new monitor.
"use all monitors" setting — with this ON, all monitors auto-detected. no manual configuration needed.
specific monitor IDs setting — with specific IDs configured, only those monitors are recorded. unplugging a non-configured monitor has no effect.
resolution change (e.g., clamshell mode) — closing MacBook lid with external monitor. recording continues on external.
queue stats after unplug — check logs. no queue stats for disconnected monitor after disconnect.

4. audio device handling

Audio device recovery (monitor unplug / device switch)

commits: device_monitor.rs atomic swap, tiered backoff, empty device list guard

unplug monitor during active Zoom call — output audio recovers within 15 seconds. Verify: grep "DEVICE_RECOVERY.*output.*restored" ~/.screenpipe/screenpipe-app.*.log. Verify: curl localhost:3030/search?content_type=audio&limit=5 shows output device transcriptions resume.
unplug and replug monitor within 5 seconds — no audio gap. both input and output continue. Verify: no "stopping" log for input device.
unplug monitor, wait 2 minutes, replug — output recovers both times. Verify: two DEVICE_RECOVERY log entries.
switch audio output (AirPods → speakers) during call — output audio continues with <5s gap. Old device kept running until new one starts (atomic swap).
health endpoint during output recovery — curl localhost:3030/health shows device_status_details with output device present within 15 seconds of recovery.
SCK transient failure doesn't cascade — if ScreenCaptureKit returns empty device list, running devices are NOT disconnected. Verify: grep "device list returned empty" ~/.screenpipe/screenpipe-app.*.log shows warning but no disconnections.
DB gap query after device switch — run: sqlite3 ~/.screenpipe/db.sqlite "SELECT t1.timestamp as gap_start, t2.timestamp as gap_end, (julianday(t2.timestamp) - julianday(t1.timestamp)) * 86400 as gap_seconds FROM audio_transcriptions t1 JOIN audio_transcriptions t2 ON t2.id = (SELECT MIN(id) FROM audio_transcriptions WHERE id > t1.id AND is_input_device = 0) WHERE t1.is_input_device = 0 AND (julianday(t2.timestamp) - julianday(t1.timestamp)) * 86400 > 60 ORDER BY t1.timestamp;" — should return no rows if output was continuously captured.

5. frame comparison & OCR pipeline

commits: 6dd5d98e, 831ad258

static screen = low CPU — leave a static image on screen for 60s. CPU should drop below 5% (release build). hash early exit should kick in.
active screen = OCR runs — actively browse/type. OCR results appear in search within 5 seconds of screen change.
identical frames skipped — check logs for hash match frequency on idle monitors. should be >80% skip rate.
ultrawide monitor (3440x1440+) — OCR works correctly. no distortion in change detection. text at edges is captured.
4K monitor — OCR works. frame comparison doesn't timeout or spike CPU.
high refresh rate (120Hz+) — app respects its own FPS setting (0.5 default), not the display refresh rate.
very fast content changes — scroll quickly through a document. OCR captures content, no crashes from buffer overflows.
corrupt pixel buffer — sck-rs handles corrupt ScreenCaptureKit buffers gracefully (no SIGABRT). fixed in 831ad258.
window capture only on changed frames — window enumeration (CGWindowList) should NOT run on skipped frames. verify by checking CPU on idle multi-monitor setup.

6. Battery Saver Mode

commits: d5a9d052, 0b32cc9a, ca29a67b

Battery Saver mode functionality — Enable Battery Saver mode. Verify that capture adjustments (e.g., reduced FPS, paused capture) occur as expected when the device's power state changes (e.g., unplugging/plugging power, low battery).
Faster power state UI updates — Change the device's power state (e.g., unplug/plug power). Verify that the UI updates quickly and accurately reflects the current power state and capture mode.
Correct default power mode — On a fresh install or after a reset, verify that the default power mode is set to "performance" until Battery Saver mode is explicitly enabled or configured.

7. permissions (macOS)

commits: d9d43d31, 620c89a5, 14acf6f0

7. Apple Intelligence (macOS 26+)

commits: d4abc619, 4f4a8282, 31f37407, 2223af9a, b34a4abd, 303958f9

macOS 26: API works — POST /ai/chat/completions returns valid response using on-device Foundation Model.
macOS < 26: no crash — app launches normally. FoundationModels.framework is weak-linked (31f37407). feature gracefully disabled.
Intel Mac: no crash — Apple Intelligence not available, but app doesn't crash at DYLD load time.
JSON mode — request with response_format: { type: "json_object" } returns valid JSON, no prose preamble (2223af9a).
JSON fallback extraction — if model prepends prose before JSON, the {...} is extracted correctly (b34a4abd).
streaming (SSE) — request with stream: true returns Server-Sent Events with incremental tokens (4f4a8282).
tool calling — request with tools array gets tool definitions injected into prompt, model responds with tool calls (4f4a8282).
daily summary — generates valid JSON summary from audio transcripts. no "JSON Parse error: Unexpected identifier 'Here'" (303958f9, 2223af9a).
daily summary audio-only — summary uses only audio data (no vision), single AI call (303958f9).

8. app lifecycle & updates

commits: 94531265, d794176a, 9070639c, 0378cab1, 4a3313d3, 7ffdd4f1, 1b36f62d

9. database & storage

commits: eea0c865, cc09de61, e61501da, d25191d7, 60096fb9

10. AI presets & settings

commits: 8a5f51dd, 0b0d8090, 7e58564e, 2522a7e2, f3e55dbc, 79f2913f

commits: 8a5f51dd, 0b0d8090

Ollama not running — creating an Ollama preset shows free-text input fields (not stuck loading). user can type model name manually (8a5f51dd).
custom provider preset — user can add a custom API endpoint. model name is free-text input with optional autocomplete.
settings survive restart — change any setting, quit, relaunch. setting is preserved.
overlay mode switch — change from fullscreen to window mode. setting saves. next shortcut press uses new mode.
FPS setting — change capture FPS. recording interval changes accordingly.
language/OCR engine setting — change OCR language. new language used on next capture cycle.
video quality setting — low/balanced/high/max. affects FFmpeg encoding params (21bddd0f).
Settings UI sentence case — All settings UI elements (billing, pipes, team) should use consistent sentence case.

11. onboarding

commits: 87abb00d, 9464fdc9, 0f9e43aa, 7ea15f32, bf1f1004

fresh install flow — onboarding appears, permissions requested, user completes setup.
auto-advance after engine starts — status screen advances automatically after 15-20 seconds once engine is running (87abb00d, 9464fdc9).
skip onboarding — user can skip and get to main app. settings use defaults.
shortcut gate — onboarding teaches the shortcut. user must press it to proceed (0f9e43aa).
onboarding window size — window is correctly sized, no overflow (7ea15f32).
onboarding doesn't re-show — after completing onboarding, restart app. main window shows, not onboarding.
First-run 2-hour reminder notification — On a fresh install, verify that a custom notification panel appears after approximately 2 hours as a first-run reminder.

commits: 87abb00d, 9464fdc9, 0f9e43aa, 7ea15f32

fresh install flow — onboarding appears, permissions requested, user completes setup.
auto-advance after engine starts — status screen advances automatically after 15-20 seconds once engine is running (87abb00d, 9464fdc9).
skip onboarding — user can skip and get to main app. settings use defaults.
shortcut gate — onboarding teaches the shortcut. user must press it to proceed (0f9e43aa).
onboarding window size — window is correctly sized, no overflow (7ea15f32).
onboarding doesn't re-show — after completing onboarding, restart app. main window shows, not onboarding.

12. timeline & search

commits: f1255eac, 25cbdc6b, 2529367d, d9821624, e61501da, 039d5fea, 50ff4f4c, 91cc4371

commits: f1255eac, 25cbdc6b, 2529367d, d9821624

13. sync & cloud

commits: 2f6b2af5, ea7f1f61, 5cb100ea

auto-remember sync password — user doesn't have to re-enter password each time (5cb100ea).
auto-download from other devices — after upload cycle, download new data from paired devices (2f6b2af5).
auto-init doesn't loop — sync initialization happens once, doesn't repeat endlessly (ea7f1f61).
Cloud archive docs — Verify that the cloud archive documentation page exists and is accessible via a link from settings.

14. Region OCR (Shift+Drag)

commits: b3628788, 738178da

Shift+Drag region OCR functionality — Perform a Shift+Drag region OCR selection on the screen. Verify that the RegionOcrOverlay appears correctly and local OCR processes the selected region.
Local OCR without login for Shift+Drag — Verify that the Shift+Drag region OCR uses local OCR and functions correctly without requiring the user to be logged in or have a cloud subscription.

15. Windows-specific

commits: eea0c865, fe9060db, c99c3967, aeaa446b, 5a219688, caae1ebc, 67caf1d1, ff4af7b5

COM thread conflict — audio and vision threads don't conflict on COM initialization (eea0c865).
high-DPI display (150%, 200%) — OCR captures at correct resolution.
multiple monitors — all detected and recorded.
Windows Defender — app not blocked by default security.
Windows default mode — On Windows, the app should default to window mode on first launch.
Windows taskbar icon — The app should display a taskbar icon on Windows.
Windows audio transcription accuracy — On Windows, verify improved audio transcription accuracy due to native Silero VAD frame size and lower speech threshold.
Windows multi-line pipe prompts — Multi-line pipe prompts should be preserved on Windows.
Alt+S shortcut activates overlay with keyboard focus — On Windows, press Alt+S. Verify that the overlay window appears and immediately receives keyboard focus, allowing immediate typing.

commits: eea0c865, fe9060db, c99c3967, aeaa446b, 5a219688, caae1ebc, 67caf1d1

COM thread conflict — audio and vision threads don't conflict on COM initialization (eea0c865).
high-DPI display (150%, 200%) — OCR captures at correct resolution.
multiple monitors — all detected and recorded.
Windows Defender — app not blocked by default security.
Windows default mode — On Windows, the app should default to window mode on first launch.
Windows taskbar icon — The app should display a taskbar icon on Windows.
Windows audio transcription accuracy — On Windows, verify improved audio transcription accuracy due to native Silero VAD frame size and lower speech threshold.
Windows multi-line pipe prompts — Multi-line pipe prompts should be preserved on Windows.

Windows text extraction matrix (accessibility vs OCR)

The event-driven pipeline (paired_capture.rs) decides per-frame whether to use accessibility tree text or OCR. Terminal apps force OCR because their accessibility tree only returns window chrome.

commits: 5a219688 (wire up Windows OCR), caae1ebc (prefer OCR for terminals), 67caf1d1 (no chrome fallback)

App categories and expected behavior:

App category	Examples	`app_prefers_ocr`	Text source	Expected text
Browser	Chrome, Edge, Firefox	false	Accessibility	Full page content + chrome
Code editor	VS Code, Fleet	false	Accessibility	Editor content, tabs, sidebar
Terminal (listed)	WezTerm, Windows Terminal, Alacritty	true	Windows OCR	Terminal buffer content via screenshot
Terminal (unlisted)	cmd.exe, powershell.exe	false	Accessibility	Whatever UIA exposes (may be limited)
System UI	Explorer, taskbar, Settings	false	Accessibility	UI labels, text fields
Games / low-a11y apps	Games, Electron w/o a11y	false	Windows OCR (fallback)	OCR from screenshot
Lock screen	LockApp.exe	false	Accessibility	Time, date, battery

Terminal detection list (app_prefers_ocr matches, case-insensitive): wezterm, iterm, terminal, alacritty, kitty, hyper, warp, ghostty

Note: "terminal" matches WindowsTerminal.exe but NOT cmd.exe or powershell.exe.

Test checklist:

Windows text extraction — untested / unknown apps

These apps are common on Windows but have never been tested with the event-driven pipeline. We don't know if their accessibility tree returns useful text or just chrome. Each needs manual verification: open the app, use it for a few minutes, then curl "http://localhost:3030/search?app_name=<name>&limit=3" and check if the text is meaningful.

Status legend: ? = untested, OK = verified good, CHROME = only returns chrome, EMPTY = no text, OCR-NEEDED = should be added to app_prefers_ocr

App	Status	a11y text quality	Notes
Browsers
Chrome	OK	good (full page content)	2778ch avg, rich a11y tree
Edge	?	probably good	same Chromium UIA as Chrome
Firefox	?	unknown	different a11y engine than Chromium
Brave / Vivaldi / Arc	?	probably good	Chromium-based, needs verification
Code editors
VS Code	?	unknown	Electron, should have good UIA
JetBrains (IntelliJ, etc)	?	unknown	Java Swing/AWT, UIA quality varies
Sublime Text	?	unknown	custom UI, may need OCR fallback
Cursor	?	unknown	Electron fork of VS Code
Zed	?	unknown	custom GPU renderer, a11y unknown
Terminals
WezTerm	CHROME	chrome only ("System Minimize...")	`app_prefers_ocr` = true, OCR works
Windows Terminal	?	unknown	matches `"terminal"` in `app_prefers_ocr`
cmd.exe	?	unknown	NOT matched by `app_prefers_ocr`
powershell.exe	?	unknown	NOT matched by `app_prefers_ocr`
Git Bash (mintty)	?	unknown	NOT matched by `app_prefers_ocr`
Communication
Discord	?	unknown	Electron, old OCR data exists
Slack	?	unknown	Electron
Teams	?	unknown	Electron/WebView2
Zoom	?	unknown	custom UI
Telegram	?	unknown	Qt-based
WhatsApp	?	unknown	Electron
Productivity
Notion	?	unknown	Electron
Obsidian	?	unknown	Electron
Word / Excel / PowerPoint	?	unknown	native Win32, historically good UIA
Outlook	?	unknown	mixed native/web
OneNote	?	unknown	UWP, should have good UIA
Media / Creative
Figma	?	unknown	Electron + canvas, likely poor a11y on canvas
Spotify	?	unknown	Electron/CEF
VLC	?	unknown	Qt-based
Adobe apps (Photoshop, etc)	?	unknown	custom UI, historically poor a11y
System / Utilities
Explorer	OK	good	file names, paths, status bar
Settings	?	unknown	UWP, should be good
Task Manager	?	unknown	UWP on Win11
Notepad	?	unknown	should have excellent UIA
Games / GPU-rendered
Any game	?	likely empty	GPU-rendered, no UIA tree. should fall to OCR
Electron w/ disabled a11y	?	likely empty	some Electron apps disable a11y

Priority to test (most common user apps):

VS Code — most developers will have this open
Discord / Slack — always running in background
Windows Terminal / cmd.exe / powershell.exe — verify terminal detection
Edge / Firefox — browser is primary use
Notion / Obsidian — knowledge workers
Office apps — enterprise users

How to verify an app:

# 1. Open the app, use it for 2 minutes
# 2. Check what was captured:
curl "http://localhost:3030/search?app_name=<exe_name>&limit=3&content_type=all"
# 3. If text is only chrome (System/Minimize/Close), it may need adding to app_prefers_ocr
# 4. If text is empty and screenshots exist, OCR fallback should kick in
# 5. Update this table with findings

Apps that may need adding to app_prefers_ocr list:

If cmd.exe / powershell.exe return chrome-only text, add "cmd" and "powershell" to the list
If mintty (Git Bash) returns chrome-only, add "mintty"
Any app where the accessibility tree consistently returns only window chrome but screenshots contain readable text

15. Help and Support

commits: deac5ea9

Intercom integration in help section — Navigate to the desktop app's help section. Verify that Crisp is replaced by Intercom and that the Intercom chat widget and knowledge base search function as expected.

16. CI / release

commits: 8f334c0a, fda40d2c

macOS 26 runner — release builds on self-hosted macOS 26 runner with Apple Intelligence (fda40d2c).
updater artifacts — release includes .tar.gz + .sig for macOS, .nsis.zip + .sig for Windows.
prod config used — CI copies tauri.prod.conf.json to tauri.conf.json before building. identifier is screenpi.pe not screenpi.pe.dev.
draft then publish — workflow_dispatch creates draft. manual publish or release-app-publish commit publishes.

16. MCP / Claude integration

commits: 8c8c445c

Claude connect button works — Settings → Connections → "Connect Claude" downloads .mcpb file and opens it in Claude Desktop. was broken because GitHub releases API pagination didn't reach mcp-v* releases buried behind 30+ app releases (8c8c445c).
MCP release discovery with many app releases — getLatestMcpRelease() paginates up to 5 pages (250 releases) to find mcp-v* tagged releases. verify it works even when >30 app releases exist since last MCP release.
Claude Desktop not installed — clicking connect shows a useful error, not a silent failure.
MCP version display — Settings shows the available MCP version and whether it's already installed.
macOS Claude install flow — downloads .mcpb, opens Claude Desktop, waits 1.5s, then opens the .mcpb file to trigger Claude's install modal.
Windows Claude install flow — same flow using cmd /c start instead of open -a.
download error logging — if download fails, console shows actual error message (not {}).

17. AI Agents / Pipes

commits: fa887407, 815f52e6, 60840155, e66c5ff8, c905ffbf, 01147096, 5908d7f4, 46422869, 4f43da70, 71a1a537, 6abaaa36, f3e55dbc, 8e426dec, 1289f51e, 4bc9ff1a, c336f73d, 2f7416ae

commits: fa887407, 815f52e6, 60840155, e66c5ff8, c905ffbf, 01147096, 5908d7f4, 46422869, 4f43da70, 71a1a537, 6abaaa36

18. Admin / Team features

commits: 58460e02, 853e0975

Admin team-shared filters — Admins should be able to remove individual team-shared filters.
Per-request AI cost tracking and admin spend endpoint — Verify that per-request AI costs are tracked correctly and that the admin spend endpoint provides accurate usage data.

commits: 58460e02

Admin team-shared filters — Admins should be able to remove individual team-shared filters.

19. Logging

commits: fc830b43, f54d3e0d

Reduced log noise — Verify a significant reduction in log noise (~54%).
PII scrubbing — Ensure that PII (Personally Identifiable Information) is scrubbed from logs.
Phone regex PII scrubbing — After generating some PII-containing data (e.g., typing phone numbers), review logs to ensure that the phone regex correctly scrubs PII and does not over-match bare digit sequences.

commits: fc830b43

Reduced log noise — Verify a significant reduction in log noise (~54%).
PII scrubbing — Ensure that PII (Personally Identifiable Information) is scrubbed from logs.

how to run

before every release

run sections 1-4 completely (90% of regressions)
spot-check sections 5-10
if Apple Intelligence code changed, run section 7

before merging window/tray/dock changes

run section 1 and 2 completely. these are the most fragile.

before merging vision/OCR changes

run section 3, 5, and 14 (Windows text extraction matrix) completely.

before merging audio changes

run section 4 completely.

before merging AI/Apple Intelligence changes

run section 7 and 10.

known limitations (not bugs)

tray icon on notched MacBooks can end up behind the notch if menu bar is crowded. Cmd+drag to reposition. dock menu is the fallback.
macOS only shows permission prompts once (NotDetermined → Denied is permanent). must use System Settings to re-grant.
debug builds use ~3-5x more CPU than release builds for vision pipeline.
first frame after app launch always triggers OCR (intentional — no previous frame to compare against).
chat panel is pre-created hidden at startup so it exists before user presses the shortcut. Creation no longer activates/shows — only the show_existing path does (matching main overlay pattern).
shortcut reminder should use CanJoinAllSpaces (visible on all Spaces simultaneously). chat and main overlay should use MoveToActiveSpace (moved to current Space on show, then flag removed to pin).

log locations

macOS:   ~/.screenpipe/screenpipe-app.YYYY-MM-DD.log
Windows: %USERPROFILE%\.screenpipe\screenpipe-app.YYYY-MM-DD.log
Linux:   ~/.screenpipe/screenpipe-app.YYYY-MM-DD.log

what to grep for

# crashes/errors
grep -E "panic|SIGABRT|ERROR|error" ~/.screenpipe/screenpipe-app.*.log

# monitor events
grep -E "Monitor.*disconnect|Monitor.*reconnect|Starting vision" ~/.screenpipe/screenpipe-app.*.log

# frame skip rate (debug level only)
grep "Hash match" ~/.screenpipe/screenpipe-app.*.log

# queue health
grep "Queue stats" ~/.screenpipe/screenpipe-app.*.log

# DB contention
grep "Slow DB" ~/.screenpipe/screenpipe-app.*.log

# audio issues
grep -E "audio.*timeout|audio.*error|device.*disconnect" ~/.screenpipe/screenpipe-app.*.log

# window/overlay issues
grep -E "show_existing|panel.*level|Accessory|activation_policy" ~/.screenpipe/screenpipe-app.*.log

# Apple Intelligence
grep -E "FoundationModels|apple.intelligence|fm_generate" ~/.screenpipe/screenpipe-app.*.log

12. mainland china / great firewall

full app functionality behind GFW — download, onboarding, AI chat, cloud features, and update checks must all work (or degrade gracefully) on networks subject to the Great Firewall.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

screenpipe regression testing checklist

critical edge cases (sorted by regression frequency)

1. window overlay & fullscreen spaces (macOS)

2. dock icon & tray icon (macOS)

3. monitor plug/unplug

4. audio device handling

Audio device recovery (monitor unplug / device switch)

5. frame comparison & OCR pipeline

6. Battery Saver Mode

7. permissions (macOS)

7. Apple Intelligence (macOS 26+)

8. app lifecycle & updates

9. database & storage

10. AI presets & settings

11. onboarding

12. timeline & search

13. sync & cloud

14. Region OCR (Shift+Drag)

15. Windows-specific

Windows text extraction matrix (accessibility vs OCR)

Windows text extraction — untested / unknown apps

15. Help and Support

16. CI / release

16. MCP / Claude integration

17. AI Agents / Pipes

18. Admin / Team features

19. Logging

how to run

before every release

before merging window/tray/dock changes

before merging vision/OCR changes

before merging audio changes

before merging AI/Apple Intelligence changes

known limitations (not bugs)

log locations

what to grep for

12. mainland china / great firewall

FilesExpand file tree

TESTING.md

Latest commit

History

TESTING.md

File metadata and controls

screenpipe regression testing checklist

critical edge cases (sorted by regression frequency)

1. window overlay & fullscreen spaces (macOS)

2. dock icon & tray icon (macOS)

3. monitor plug/unplug

4. audio device handling

Audio device recovery (monitor unplug / device switch)

5. frame comparison & OCR pipeline

6. Battery Saver Mode

7. permissions (macOS)

7. Apple Intelligence (macOS 26+)

8. app lifecycle & updates

9. database & storage

10. AI presets & settings

11. onboarding

12. timeline & search

13. sync & cloud

14. Region OCR (Shift+Drag)

15. Windows-specific

Windows text extraction matrix (accessibility vs OCR)

Windows text extraction — untested / unknown apps

15. Help and Support

16. CI / release

16. MCP / Claude integration

17. AI Agents / Pipes

18. Admin / Team features

19. Logging

how to run

before every release

before merging window/tray/dock changes

before merging vision/OCR changes

before merging audio changes

before merging AI/Apple Intelligence changes

known limitations (not bugs)

log locations

what to grep for

12. mainland china / great firewall