Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ generator, in this case, **mdbook**. It defines the structure and navigation of
- [Release Process](release/RELEASE.md)
- [Release Automation](release/AUTOMATION.md)
- [Developer Release Checklist](release/developer-checklist.md)
- [Release Testing Checklist](release/testing-checklist.md)
- [Manual Release (historical)](release/HISTORICAL_RELEASE.md)
- [Security]()
- [Threat Modeling Guide](security/threat-modeling-guide.md)
Expand Down
199 changes: 199 additions & 0 deletions docs/release/testing-checklist.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,199 @@
### Release Manual Testing Checklist

Purpose

- Ensure consistent, repeatable pre-release validation across core areas: installation/upgrade, UI/UX, functionality regression, performance, connectivity, and security/privacy.
- Provide a short “Smoke” run and a full “Release” run.

When to use

- Before every Beta/Production release
- After high‑risk feature merges

Test environments

- Devices: At least one low-end and one mid/high-end device; 1 emulator
- OS versions: Last 3 major Android versions supported + Lowest supported (API 21)
- Network: Wi‑Fi, LTE/3G, Offline/Airplane; captive portal if available
- Accounts: At least 2 IMAP accounts (different providers), 1 POP3 if supported; one account with many messages/folders
- Locales: English + one non‑English (RTL if possible)
- Themes: Light and Dark

Data prerequisites

- Test mailboxes with folders: Inbox, Sent, Drafts, Trash, Spam, Archive, custom folders
- A few threads with 50+ messages for performance/scroll tests
- At least one account configured with push (if supported)

How to record results

- Mark each item Pass/Fail/Blocked/Not Applicable
- Record build/commit, device, OS, account provider, network type
- File defects with clear reproduction steps, logs, and screenshots

## Quick Smoke (15–25 min)

1. Install/Launch
- Fresh install: App installs and launches without crash
- First‑run: Onboarding visible, no ANRs
2. Account Setup (1 IMAP)
- Manual or auto‑config completes; inbox loads
3. UI Smoke
- Navigate: Account list → Inbox → Message → Back works
- Compose: Create draft, send to self, appears in Sent and then Inbox
4. Sync/Refresh
- Pull‑to‑refresh fetches new mail; background sync triggers at expected times
5. Notifications
- Receive mail notification; tap opens message; mark as read behavior correct
6. Basic Actions
- Read, star/flag, delete to Trash; undo/restore if offered
7. Performance quick check
- App cold start acceptable, scrolling in Inbox smooth

## Full Release Checklist

### A. Installation and Upgrade

Top-level verification

- Fresh Install — App installs and runs correctly on a fresh install
- Installs and launches without crash
- Permissions requested only when needed; rationale shown
- First‑run experience shows expected screens/toggles
- Upgrade to Production — App upgrades correctly to the latest production version (only for releases)
- Upgrade to latest production build preserves accounts, settings, and local cache
- No duplicate notifications or migration errors on first launch
- Upgrade to Beta — App upgrades correctly to the latest beta version
- Upgrade to latest beta build preserves accounts, settings, and local cache
- Feature flags and migrations behave correctly when upgrading to beta
- No duplicate notifications or migration errors on first launch
- Backup/Restore
- Backup 1 account setting, delete account from app, restore from backup; account usable
- Backup all accounts, uninstall/reinstall, restore settings; accounts usable
- Settings Import via QR code (only Thunderbird)
- From first-run flow, choose "Import settings"; scan a valid QR; settings are imported/applied or account is added/usable
- From Settings > Import, choose "Scan QR code"; scan a valid QR; settings are imported/applied or account is added/usable
- Try an invalid or expired QR; a clear error is shown and the app does not crash

### B. Account Management

- Add accounts
- Auto‑config works for common providers; fallback to manual works
- Validation errors are clear (username/password/server settings)
- Multiple accounts
- Switch accounts; titles/subtitles update correctly
- Unified Inbox visibility matches setting and number of accounts
- Delete account
- Deleting current account transitions correctly (Unified Inbox if enabled and >1; else default account)
- No crash during or after deletion; capability checks safe
- Special folders
- Drafts/Sent/Archive/Trash/Spam configured/created as needed

## C. Core Mail Flows (per account)

- Receiving
- New message appears after push/refresh; badge/notification updates
- Reading
- Open message; images loading policy obeyed; external images prompt if applicable
- Compose/Reply/Forward
- From identity correct; quoted content; signatures; attachments; send succeeds
- Drafts
- Auto‑save draft; edit and send later
- Move/Copy
- Move/copy messages between folders; operation succeeds and reflects in UI
- Search
- Local search by sender/subject/body; filters; results correct; remote search if supported
- Attachments
- Download, open with external app, share; large attachment handling

## D. UI and UX

- Navigation
- Drawer/tabs/back behavior consistent; deep linking from notifications
- Message List view
- Threaded/flat messages shown; selection mode; swipe actions (customizable to check archive/delete)
- Message view
- Rendering of HTML/plain text; long messages; quoted text expansion; link handling
- Image loading per settings; attachment previews
- Theming and localization
- Light/Dark theme correctness; typography and icons;
- Non‑English locale strings fit and are translated; RTL layout alignment
- Accessibility
- TalkBack announcements; content descriptions present
- Focus order logical; dynamic type/Font scale respected

## E. Performance and Resource Use

- Startup performance
- Cold start time acceptable on low‑end device
- Scrolling and list operations
- Smooth scrolling in Inbox with 1000+ items; no jank during load more
- Sync performance
- Time to first sync for a fresh account reasonable; no repeated retries
- Resource usage
- No abnormal CPU usage when idle; memory stable during long scrolling; cache does not balloon unexpectedly

## F. Connectivity and Background Behavior

- Network transitions
- Wi‑Fi → Cellular → Offline: app handles gracefully; shows appropriate banners/status
- Offline usage
- Read previously synced messages; queued actions (send/move) execute on reconnect

## G. Notifications

- New mail
- Shows correct sender/subject; grouping for multiple messages
- Actions: Mark read, Delete, Archive function correctly and reflect in app
- Quiet hours/Do Not Disturb
- Respected according to settings

## H. Settings and Preferences

- General settings
- Unified Inbox toggle behavior: only shows when >1 accounts, correct fallback behavior
- Sort options persist per account/unified; changes reflected immediately
- Account settings
- Outgoing/incoming server edits persist; re‑auth flows; OAuth if applicable

## I. Security and Privacy

- Permissions
- No unexpected permission prompts; revoking a permission shows clear error states
- Sensitive logging
- Disabled by default; enabling shows warnings; logs redact personal data where possible
- Certificates (if applicable)
- Certificate errors surfaced with clear options; pinning remembered

## J. Regression Checks (recently changed areas)

- Check changes in this release
- Perform smoke tests around changed screens/use cases
- Logs, crashes, and ANRs
- No new crashes/ANRs; logging remains appropriate and doesn’t expose sensitive data
- Quick smoke around changed screens
- Basic open/close, refresh/sync, action buttons, and navigation work

K. Known Issues Verification
- Validate that previously documented known issues still match behavior and are noted in release notes

## Sign-off

- All “Smoke” items: Pass
- No Critical/High defects open; Medium has workarounds; Low acceptable
- Performance thresholds met on target devices

## Appendix A: Quick result template

- Build/Commit:
- Device/OS:
- Accounts:
- Network:
- Locale/Theme:
- Summary:
- Defects filed:

## Appendix B: Full run log template

- Include timestamps, steps, expected vs actual, screenshots, and logs as needed