feat: Add incremental re-hashing API for large models

[edonadei]

Motivation

This PR implements incremental model re-hashing to solve a critical performance problem when signing large ML models. Currently, when a user makes a small change to a large model (e.g., updating README.md in a 500GB model), the entire model must be re-hashed before re-signing, which can take hours. This makes it impractical to update documentation or configuration files in large models.

This PR adds a Python API that reuses digests from previous signatures for unchanged files, only re-hashing files that were added or modified. For a 500GB model with a 1KB documentation update, this reduces re-hashing time from hours to seconds (~500,000x speedup).

Changes

1. Manifest.from_signature() - Extracts manifest from existing signature files without cryptographic verification, enabling digest reuse
2. IncrementalSerializer - Core implementation that compares current model state against existing manifest and only re-hashes changed files
3. Config.use_incremental_serialization() - Integrates incremental serializer into hashing API
4. sign_incremental() - High-level convenience API that combines extraction + incremental hashing + signing

Design Decision

Following the discussion in #160, this implementation uses a user-driven approach where changed files are specified via the files_to_hash parameter (e.g., from git diff). This was chosen over automatic change detection because:

• File metadata (mtime) is unreliable across systems and git operations
• Users know what changed via their workflow (git, manual tracking, etc.)
• Keeps implementation simple and reliable
• Works with any file tracking system

Test Coverage:

• tests/manifest_test.py: 5 tests for manifest extraction (valid/invalid signatures, error cases)
• tests/_serialization/incremental_test.py: 7 tests for incremental serializer (new files, modified files, deleted files, mixed scenarios, empty manifests)

Future Work: Based on maintainer feedback, these could be added in follow-up PRs:

• CLI support (--incremental flag)
• Documentation in README
• Git helper utility (get_changed_files_from_git())
• Integration tests with large models

Questions for Maintainers:

• Is the files_to_hash parameter approach acceptable, or would you prefer a different change detection mechanism?
• Should CLI support and documentation be in this PR or separate follow-ups?
• Would you like me to add a CHANGELOG entry now or wait until the approach is approved?

Testing this PR

# Create and sign an initial model
from model_signing import signing
from pathlib import Path

model_dir = Path("test_model")
model_dir.mkdir()
(model_dir / "weights.bin").write_bytes(b"x" * 1000000)  # 1MB file
(model_dir / "README.md").write_text("Version 1")

# Initial sign
signing.Config().use_elliptic_key_signer(
    private_key=Path("test.key")
).sign(model_dir, "model.sig.v1")

# Modify only README
(model_dir / "README.md").write_text("Version 2 - updated docs")

# Sign incrementally - only re-hashes README.md, reuses weights.bin digest
signing.Config().use_elliptic_key_signer(
    private_key=Path("test.key")
).sign_incremental(
    model_dir,
    old_signature_path="model.sig.v1",
    new_signature_path="model.sig.v2",
    files_to_hash=[model_dir / "README.md"]
)

# Verify the new signature works
from model_signing import verifying
verifying.Config().use_elliptic_key_verifier(
    public_key=Path("test.pub")
).verify(model_dir, "model.sig.v2")
Usage Example:
import subprocess
from model_signing import signing

# Get changed files from git
changed = subprocess.check_output(
    ['git', 'diff', '--name-only', 'HEAD']
).decode().strip().split('\n')
files_to_hash = [f"model/{f}" for f in changed if f]

# Sign incrementally
signing.sign_incremental(
    model_path="huge-model/",
    old_signature_path="model.sig.old",
    new_signature_path="model.sig.new",
    files_to_hash=files_to_hash
)

[edonadei]

@mihaimaruseac if you can take a look at this, I tried to follow up with the last discussions from #160 (from 2024 that's old) in that thread and tried to implement a solution. It's a bit long and probably imperfect, but I'm open to feedback, there's a questions for maintainer sections to start the discussion.

[mihaimaruseac]

Amazing! Will take a look this week (JupyterCon)