-
Notifications
You must be signed in to change notification settings - Fork 0
Testing Strategy
- Unit Testing Approach
- Integration Testing Methodology
- Device Testing Framework
- Test Fixtures and Configuration
- Test Case Examples
- Test Automation and Continuous Integration
- Guidance for Extending the Test Suite
The Post-Quantum WebAuthn Platform employs a comprehensive unit testing strategy using pytest to validate core cryptographic and protocol components. The test suite provides extensive coverage for COSE encoding, CBOR parsing, CTAP2 command processing, and post-quantum cryptographic operations.
For COSE encoding validation, the platform implements rigorous tests in test_cose.py that verify proper parsing and verification of various cryptographic algorithms including ES256, ESP256, ESP384, and ESP512. These tests validate that signatures are correctly verified and that high-S signatures are properly rejected according to FIDO2 specifications. The test suite also verifies that cryptographic keys are properly instantiated from both CBOR-encoded data and cryptography library primitives.
CBOR parsing functionality is thoroughly tested in test_cbor.py, which implements test vectors from the official CBOR test suite. These tests validate encoding and decoding of various data types including integers, strings, byte arrays, arrays, and maps, ensuring compliance with the FIDO Canonical CBOR encoding requirements. The tests verify proper key ordering in maps and correct handling of edge cases in the CBOR encoding format.
CTAP2 command processing is validated through unit tests in test_ctap2.py, which verify proper parsing of authenticator responses including attestation objects, authenticator data, and assertion responses. These tests validate that information objects are correctly parsed from CBOR-encoded data and that all required fields are properly extracted and accessible through the API.
Section sources
- test_cose.py
- test_cbor.py
- test_ctap2.py
The integration testing methodology focuses on validating server routes and their interactions with the WebAuthn protocol. The tests simulate complete authentication flows from credential registration to assertion, validating that server responses are properly formatted and that client extensions are correctly processed.
Server route integration tests validate the complete request-response cycle for WebAuthn operations, ensuring that registration and authentication challenges are properly generated, that responses are correctly verified, and that appropriate HTTP status codes are returned for various scenarios. The tests also validate that server-side session management properly tracks authentication state throughout the multi-step registration and authentication processes.
Mock authentication flows are implemented to test various edge cases and error conditions. These tests validate that the server properly handles invalid inputs, malformed requests, and protocol violations. Response validation ensures that all required fields are present in server responses and that they conform to the expected data types and formats specified in the WebAuthn standard.
The integration tests also validate the handling of client extensions such as credProps, largeBlob, and prf, ensuring that extension results are properly included in authentication responses and that server-side processing of extension inputs is correct. Tests verify that when extensions are requested by the server, the client properly includes the corresponding extension results in its responses.
Section sources
- test_server.py
- test_webauthn.py
- test_storage.py
The device testing framework in tests/device/ provides comprehensive testing of real authenticator interactions, exercising various FIDO2 features including biometric enrollment, client PIN management, credential management, and large blob operations. These hardware-in-the-loop tests require explicit activation via the --run-device-tests flag due to their destructive nature on authenticator state.
The biometric enrollment tests validate the complete bio enrollment workflow, including template enumeration, creation, and deletion. These tests verify that authenticators properly handle multiple biometric templates and that enrollment counters are correctly updated. The tests also validate error handling for scenarios such as attempting to enroll when the authenticator's template storage is full.
Client PIN functionality is thoroughly tested in test_clientpin.py, which validates PIN validation, retry counters, and rate limiting behavior. The tests verify that incorrect PIN attempts decrement the retry counter, that the authenticator enters a blocked state after too many failed attempts, and that the correct PIN unlocks the authenticator and resets the retry counter. The tests also validate PIN change operations and factory reset behavior.
Credential management operations are tested in test_credman.py, which validates the complete lifecycle of resident credentials including enumeration, deletion, and user data updates. These tests verify that credential metadata is correctly reported, that credential counting works properly, and that permissions are enforced when accessing credential management functions. The tests also validate read-only credential management using persistent user-authentication tokens.
Large blob operations are tested in test_largeblobs.py, which validates the storage, retrieval, and deletion of arbitrary data in the authenticator's large blob storage. These tests verify size limits, checksum validation, and permission requirements for blob operations. The tests also validate the integration of large blob operations with WebAuthn client extensions, ensuring that blobs can be written during registration and read during authentication.
Section sources
- test_bioenroll.py
- test_clientpin.py
- test_credman.py
- test_largeblobs.py
- test_credblob.py
The test suite utilizes pytest fixtures extensively to set up test environments and manage test state. The primary configuration is defined in conftest.py files at both the root and device test levels, which define command-line options and control test collection.
The root conftest.py defines command-line options for specifying NFC readers, disabling device tests, setting relying party IDs, and enabling CCID transport. It also implements a collection filter that automatically skips device tests unless explicitly requested via the --run-device-tests flag, preventing accidental execution of destructive tests.
The device-specific conftest.py in tests/device/ defines the DeviceManager class, which handles authenticator discovery, connection management, and test setup. This fixture automatically detects connected authenticators via USB or NFC, validates that they are in a suitable state for testing, and sets up required configurations such as PIN setup. The fixture also provides utilities for reconnecting to authenticators after physical removal and insertion.
Key fixtures provided by the device test configuration include:
-
dev_manager: Manages the authenticator connection and provides access to test utilities -
device: Provides direct access to the authenticator device -
ctap2: Provides a CTAP2 interface to the authenticator -
client: Provides a Fido2Client interface for high-level WebAuthn operations -
factory_reset: Allows explicit factory reset of the authenticator between test modules -
clear_creds: Ensures credential cleanup after tests that create resident credentials
Parameterized fixtures are used to test multiple algorithms and PIN protocols. The algorithm fixture iterates through all supported COSE algorithms, skipping those not supported by the authenticator. The pin_protocol fixture tests both PIN protocol versions 1 and 2, ensuring compatibility with different authenticator implementations.
Section sources
- conftest.py
- device/conftest.py
The test suite includes numerous practical examples demonstrating how CTAP2 errors are validated and how edge cases are handled. These examples serve as models for writing new tests and illustrate the platform's comprehensive error handling.
In test_clientpin.py, the PIN validation test demonstrates how to verify CTAP2 error codes and retry counters:
def test_pin_validation(dev_manager, client_pin):
assert dev_manager.ctap2.get_info().options["clientPin"] is True
assert client_pin.get_pin_retries()[0] == 8
# Wrong PIN decreases the retries remaining
for retries in range(7, 4, -1):
with pytest.raises(CtapError, match="PIN_(INVALID|AUTH_BLOCKED)"):
client_pin.get_pin_token("123456")
assert client_pin.get_pin_retries()[0] == retriesThis test verifies that incorrect PIN attempts trigger the appropriate error codes and that the retry counter is properly decremented.
The credential management test in test_credman.py demonstrates how to handle optional authenticator capabilities:
def test_update(client, ctap2, pin_protocol):
if not CredentialManagement.is_update_supported(ctap2.info):
pytest.skip("ClientPin update not supported")
# Test code here...
# Authenticators may or may not store name/displayName
stores_name = "name" in creds[0][6]
stores_display_name = "displayName" in creds[0][6]
if not stores_name:
del user["name"]
if not stores_display_name:
del user["displayName"]This example shows how to gracefully handle variations in authenticator capabilities and adjust test expectations accordingly.
The large blob test in test_largeblobs.py demonstrates how to test size limits and error conditions:
def test_size_bounds(ctap2, pin_protocol):
lb = get_lb(ctap2, pin_protocol)
# The max storage is the CBOR-encoded array, minus a 16 byte checksum
max_size = ctap2.info.max_large_blob - 16
# Create data which when cbor-encoded is exactly max_size bytes
array = [{1: os.urandom(max_size - 8)}]
array.extend([0] * (max_size - len(cbor.encode(array))))
lb.write_blob_array(array)
# Ensure writing larger data fails:
array.append(1)
with pytest.raises(CtapError, match="LARGE_BLOB_STORAGE_FULL"):
lb.write_blob_array(array)This test verifies that the authenticator properly enforces storage limits and returns appropriate error codes when capacity is exceeded.
Section sources
- test_clientpin.py
- test_credman.py
- test_largeblobs.py
The Post-Quantum WebAuthn Platform implements automated testing through Google Cloud Build, configured in cloudbuild.yaml. The CI pipeline consists of three main steps: building the Docker image, pushing it to Google Artifact Registry, and deploying to Cloud Run.
The build process uses Docker BuildKit with detailed progress reporting enabled, ensuring that build issues can be quickly diagnosed. The image is tagged with the project ID and pushed to Google Artifact Registry for secure storage and access control. The deployment step uses gcloud to deploy the service to Cloud Run with specific resource constraints including 0.5 CPU and 512Mi memory.
The CI configuration specifies a 1500-second timeout, allowing sufficient time for all tests to complete. The deployment includes settings for unauthenticated access, instance scaling between 0 and 3 instances, and execution environment generation 2 for improved performance and security.
Unit tests are automatically executed as part of the Docker build process, ensuring that all code changes are validated before deployment. The test suite is designed to run efficiently in the CI environment, with non-device tests executing by default and device tests requiring explicit activation to prevent unintended hardware interactions.
Code coverage requirements are enforced through the test suite design, with high coverage targets for core cryptographic and protocol components. The unit tests for COSE, CBOR, and CTAP2 functionality achieve near-complete coverage of parsing, validation, and error handling code paths.
Section sources
- cloudbuild.yaml
To write new tests for the Post-Quantum WebAuthn Platform, follow the established patterns in the existing test suite. New unit tests should be added to the appropriate file in the tests/ directory based on the component being tested, while new device interaction tests should be placed in tests/device/.
When testing new CTAP2 commands or extensions, create a new test file following the naming convention test_<feature>.py. Use the existing fixtures to set up the test environment, and leverage parameterized fixtures when testing multiple algorithms or protocol versions.
For new device interaction tests, ensure proper use of preconditions to skip tests when the authenticator does not support the required features:
@pytest.fixture(autouse=True, scope="module")
def preconditions(dev_manager):
if not MyNewFeature.is_supported(dev_manager.info):
pytest.skip("MyNewFeature not supported by authenticator")When testing error conditions, use pytest's raises context manager to verify that the correct CTAP2 error codes are returned:
with pytest.raises(CtapError, match="SOME_ERROR_CONDITION"):
# Code that should trigger the errorFor tests that modify persistent authenticator state, use the clear_creds fixture or implement cleanup in a teardown function to ensure test isolation. When testing features that require specific setup, consider creating a dedicated fixture to encapsulate the setup logic.
New integration tests for server routes should use the existing test server infrastructure and validate both successful and error response scenarios. Ensure that tests cover all required request parameters and validate that appropriate HTTP status codes are returned for various error conditions.
When extending the test suite, maintain consistency with existing code style and documentation practices. Add comprehensive comments explaining the purpose of each test and the expected behavior being validated. For complex test scenarios, include references to the relevant sections of the FIDO2 specification.
Section sources
- test_cose.py
- test_cbor.py
- test_ctap2.py
- device/conftest.py