Skip to content

Comments

perf: configurable scan speed for picture mode (up to 16x faster)#1368

Open
codingfrog wants to merge 2 commits intoarsenetar:masterfrom
codingfrog:perf/picture-scan-speed
Open

perf: configurable scan speed for picture mode (up to 16x faster)#1368
codingfrog wants to merge 2 commits intoarsenetar:masterfrom
codingfrog:perf/picture-scan-speed

Conversation

@codingfrog
Copy link

Summary

Picture scanning in dupeGuru processes full-resolution images (e.g. 4000x3000) one at a time, which makes scanning large photo libraries very slow (~2 images/second).

This PR adds:

  • Pre-scaling images before block extraction — most of the pixel data is thrown away during block averaging anyway, so processing a 450x450 image gives nearly identical results at a fraction of the cost
  • Parallel image preparation using ThreadPoolExecutor across all CPU cores instead of sequential single-threaded processing
  • SQLite WAL mode for better concurrent cache access during parallel writes
  • A user-facing "Scan speed" preference in the Picture preferences dialog with three modes:
    • Accurate (slow) — no pre-scaling, processes full-resolution images (original behavior)
    • Balanced (recommended) — pre-scale to 450px, good accuracy/speed trade-off
    • Turbo — pre-scale to 150px, fastest option for large collections

Each mode uses a separate cache file, so switching modes doesn't require re-scanning the entire collection.

Benchmark

Tested on real photos (mix of 12-24MP JPEGs):

Mode Speed Relative
Accurate ~2 img/s 1x (baseline, original behavior)
Balanced ~33 img/s ~16x faster
Turbo ~50 img/s ~25x faster

Changes

  • core/pe/matchblock.py — parallel preparation + prescale multiplier plumbing
  • core/pe/cache_sqlite.py — enable WAL mode
  • qt/pe/photo.py — configurable pre-scaling in block extraction
  • core/pe/scanner.py — pass picture_prescale option through
  • core/app.py — separate cache files per prescale mode
  • qt/pe/preferences_dialog.py — add "Scan speed" combo box
  • qt/preferences.py — persist picture_prescale preference
  • qt/preferences_dialog.py — improved Filter Hardness enable/disable logic
  • qt/app.py — wire preference to model options
  • qt/me/preferences_dialog.py, qt/se/preferences_dialog.py — use shared _setFilterHardnessEnabled()

Test plan

  • Scan a folder of pictures with each mode (Accurate, Balanced, Turbo) and verify duplicates are found correctly
  • Switch between modes and verify each uses its own cache (no re-scan prompt)
  • Verify the preference persists across application restarts
  • Verify Filter Hardness slider + labels are properly grayed out when scan type doesn't support it
  • Run existing test suite (pytest)

🤖 Generated with Claude Code

codingfrog and others added 2 commits February 15, 2026 19:04
…ation

- Pre-scale images to 450x450 before block extraction (qt/pe/photo.py)
  instead of processing full-resolution images (e.g., 4000x3000)
- Parallelize image preparation using ThreadPoolExecutor across all CPU
  cores instead of sequential single-threaded processing
- Enable SQLite WAL mode for better concurrent cache access

Benchmarked: ~2 img/s → 33 img/s on real photos (16x speedup).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…urbo)

Add a user-facing "Scan speed" dropdown in the Picture preferences dialog
that controls how aggressively images are pre-scaled before block extraction:

- Accurate (slow): no pre-scaling, processes full-resolution images
- Balanced (recommended): pre-scale to 450px — good accuracy/speed trade-off
- Turbo: pre-scale to 150px — fastest, suitable for large collections

Each mode uses a separate cache file so switching modes doesn't require
re-scanning. The preference is persisted across sessions.

Also improves Filter Hardness UI by disabling all related labels (not just
the slider) when the control is inactive.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant