Skip to content

Avoid redundant archive rewrites when content is unchanged - #23

Open
triatomic wants to merge 3 commits into
ClementJ18:mainfrom
triatomic:feature/reduce-redundant-writes
Open

Avoid redundant archive rewrites when content is unchanged#23
triatomic wants to merge 3 commits into
ClementJ18:mainfrom
triatomic:feature/reduce-redundant-writes

Conversation

@triatomic

Copy link
Copy Markdown
Contributor

A .big save is a full repack of the entire archive (pyBIG has no in-place
patching), so the amount written on save is the whole archive regardless of how
little changed. These three changes remove writes that would produce a
byte-identical result — they don't (and can't) make a genuine edit cheaper.

Changes

  1. Skip dirtying an entry when a tab save changes nothing. A tab flagged
    unsaved (edit-then-revert, or an encode round-trip that yields identical
    bytes) still called edit_file, marking the entry modified and forcing a
    full repack on the next save. A new GenericTab.commit_to_archive helper
    compares the new bytes against the stored content and only edits when they
    differ; the text/cah/map tabs use it.

  2. Treat identical re-adds as unchanged instead of overwriting. Adding a
    file whose name already exists was classified "overwritten" purely on the
    name, and re-wrote the entry even when the bytes matched — dirtying the
    archive. add_file_to_archive and _merge_archives now compare incoming
    bytes against the stored content; matching content yields a new
    AddOutcome.UNCHANGED that leaves the archive untouched. The add summary
    reports the unchanged count (_tally now returns a new/overwritten/unchanged
    triple) and lists them in an "Unchanged files" section.

  3. Skip the full repack when the archive has no changes. Save
    unconditionally re-wrote the whole .big even with nothing modified. When
    writing back to the same file and modified_entries is empty, the repack is
    skipped. Save As still always writes to its new path.

Notes

  • The SKIP overwrite-default still short-circuits before reading the file.
  • Identical files no longer trigger the "overwrite?" prompt under the ASK default.
  • Detecting "unchanged" reads the stored bytes to diff (one read per touched
    entry for InDiskArchive) — far cheaper than the repack it avoids; no eager
    full-archive hashing.

A tab flagged unsaved (e.g. edit-then-revert, or an encode round-trip
that yields identical bytes) still called edit_file, marking the entry
modified and forcing a full repack on the next save. Compare the new
bytes against the stored content first and only edit when they differ.
Adding a file whose name already exists reported it as overwritten and
re-wrote the entry even when the bytes matched, dirtying the archive and
forcing a full repack. Compare incoming bytes against the stored content
in add_file_to_archive and _merge_archives; matching content yields a new
AddOutcome.UNCHANGED that leaves the archive untouched. The add summary
now reports the unchanged count and lists them in the details.
Save unconditionally re-wrote the whole BIG even with nothing modified.
When writing back to the same file and modified_entries is empty, skip
the repack entirely. Save As still always writes to its new path.
Comment thread src/main.py
writing_in_place = self.path is not None and os.path.normcase(
os.path.abspath(path)
) == os.path.normcase(os.path.abspath(self.path))
if writing_in_place and not self.archive.modified_entries:

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a bit of an edge case for consideration, this breaks the flow in which you want to overwrite a .big that has been modified by another process. This has happened to me in the past, not often but it has. I think in the end I would prefer to give the control to the users on what they want to do. I like the rest of the PR, I'd just like the part that takes control of saves away from the user reverted.

I think the rest of the PR is genuinely a time saver since the user can more easily review a batch add without getting drowned out in the noise of no-write edits.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants