Skip to content

test/fix: add test suite with E2E validation and fix bugs found against live ZoneMinder#60

Open
nabbi wants to merge 4 commits intoZoneMinder:masterfrom
nabbi:feature/test-suite
Open

test/fix: add test suite with E2E validation and fix bugs found against live ZoneMinder#60
nabbi wants to merge 4 commits intoZoneMinder:masterfrom
nabbi:feature/test-suite

Conversation

@nabbi
Copy link

@nabbi nabbi commented Feb 13, 2026

pyzm had zero tests. This PR adds a 4-tier test suite with 73 unit/integration tests and a full E2E suite designed to run against a live ZoneMinder instance. Issues in pyzm handling of API were found and corrected as a result of building these test cases.

Why

pyzm wraps the ZoneMinder REST API, but hand-crafted JSON fixtures can silently diverge from real server responses — wrong field names, different types (int vs str), HTML flash redirects instead of JSON, broken filter URL building. Unit tests alone can't catch these. The test suite is designed around one principle: the real ZoneMinder API is the source of truth.

What's included

Commit 1 — Unit + integration tests (73 tests)

  • API-level tests (test_api_auth.py, test_api_methods.py, test_api_request.py) covering login flows (JWT/legacy/no-auth), token refresh, retry on 401, content-type
    handling, and all public methods
  • Helper edge-case tests for logic not reachable through the API: States.find() case-insensitive search, Monitor.set_parameter() payload construction, event filter
    URL building, pagination limits
  • Integration workflow test chaining login → monitors → events → states
  • JSON fixtures mirroring real ZM API payloads (strings for numeric fields, matching actual response structure)
  • Autouse fixtures solving pyzm's global state challenges (logger mock, exit() patch)

Commit 2 — E2E test suite against live ZoneMinder

  • 6 test files covering auth, monitors, events, states, configs, and edge cases
  • Readonly tier (safe to run repeatedly) and write tier (creates/modifies/deletes with auto-cleanup)
  • Validates real type coercion, response structure, pagination coherence, non-JSON response handling
  • Skips automatically when ZM_API_URL/ZM_USER/ZM_PASSWORD env vars are unset
  • Documents known pyzm bugs found during E2E testing (Configs.find TypeError, flash redirect on delete)

Commit 3 — Bug fixes found by E2E tests

  • Fixed Configs.find() TypeError on missing config (null check)
  • Fixed Monitor.set_parameter() enabled coercion
  • Fixed Monitors.find() type mismatch on ID comparison
  • Fixed State.active() / State.definition() type handling

Commit 4 — Remove test overlaps, update README

  • Removed 3 helper tests that duplicated API-level assertions (per project rule: don't test what the API layer already covers)
  • Rewrote tests/README.md to lead with the design intent and document all 4 tiers

Running

Unit + integration (no live ZM needed)

pytest tests/unit/ tests/integration/ -v

E2E readonly

ZM_API_URL=https://zm.local/zm/api ZM_USER=admin ZM_PASSWORD=secret
pytest tests/e2e/ -m e2e_readonly

E2E write (with cleanup)

ZM_E2E_WRITE=1 pytest tests/e2e/ -m e2e_write

nabbi and others added 4 commits February 13, 2026 17:10
Introduce pytest-based test infrastructure for pyzm, which previously
had zero tests. Establishes a safety net ahead of upcoming refactoring
efforts.

- pytest config (pyproject.toml) with markers and coverage settings
- JSON response fixtures mirroring real ZM API payloads
- Shared conftest fixtures (mock login, suppressed logger, exit guard)
- Unit tests: auth flows, API methods, request handling, helper edge cases
- Integration tests: full login -> monitors -> events -> states workflows
- Testing strategy documentation (docs/testing-strategy.md)

Core modules coverage: api.py 91%, helpers 80-100%.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
97 tests (81 readonly, 16 write) that validate pyzm against a real ZM
instance, catching response structure drift, type coercion mismatches,
and flash() redirect handling that unit tests with mocked HTTP miss.
Tests skip automatically when ZM_API_URL env vars are unset.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Running the E2E test suite against a real ZoneMinder instance exposed
five bugs in pyzm that unit tests (with hand-crafted fixtures) never
caught — exactly the kind of drift E2E tests are designed to surface.

Monitors.add() missing type/device parameters:
  E2E test_add_monitor → ZM returned 500. Captured the response body
  and found: "Field 'Device' doesn't have a default value". The real
  API requires Monitor[Type] and Monitor[Device] but Monitors.add()
  had no support for either. Added both as first-class parameters.

Monitors.add() enabled=False silently dropped:
  E2E test_set_enabled → monitor created with enabled=False was not
  actually disabled. The guard `if options.get('enabled')` is falsy
  for False, so Monitor[Enabled] was never sent. Changed to
  `if options.get('enabled') is not None`.

Monitor.enabled() string vs int comparison:
  E2E test_set_enabled → after enabling a monitor, enabled() returned
  False. The real API returns Enabled as int 1, not string "1". The
  comparison `== '1'` fails for int. Fixed with str() coercion.

_make_request raises BAD_IMAGE on successful PUT:
  E2E test_set_config_value → Configs.set() PUT returned 200 with
  Content-Length: 0. _make_request only handled empty bodies for
  DELETE, so PUT fell through to the BAD_IMAGE error path. Extended
  the empty-body handler to cover PUT.

Also adjusted E2E test assertions for raw field types (Colours returns
int from real API, not string as fixtures assumed).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Removed 3 helper tests that duplicated assertions already covered by
API-level tests (per project rule). Rewrote README to lead with the
core philosophy: the real ZM API is the source of truth.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@pliablepixels
Copy link
Member

pliablepixels commented Feb 14, 2026

Hi @nabbi, thanks for this PR. I'm Claude, an AI assistant working with @pliablepixels on pyzm v2.

pyzm v2 (pliablepixels/pyzm) already had ~400 unit tests and 89 ML e2e tests, but nothing that hit a real ZoneMinder server. Your PR was the push to fix that. We've added a ZM E2E suite (tests/test_zm_e2e/) with 44 tests covering auth, monitors, events, zones, frames, object detection, DB operations, and state management — same idea with readonly/write tiers and env-var skip logic.

Sure enough, the live tests caught a real bug right away: events() was building filters with filter[Query][terms] query strings that never actually worked against the ZM API. Mocked tests couldn't catch it. Replaced it with CakePHP URL-path syntax (events/index/MonitorId:1.json), verified against a live ZM 1.38 server.

Commit: pliablepixels@26d1ecd

Just a note — this repo (ZoneMinder/pyzm) is the legacy version and is a separate software stack from pyzm v2. Feel free to keep PRing here if you want — Isaac Connor evaluates and merges PRs on this repo independently.

Thanks for the ideas and the effort here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants