filter-repo: avoid failures with LFS objects and submodules

Our attempts to filter LFS objects with --sensitive-data-removal cause
us to try to look at the sizes of all blobs and read the contents of
those that are small enough.  That's fine, but we automatically assume
all the FileChanges in a commit are blobs, when some might be
submodules.  Since submodule oids are unlikely to exist in the current
repository, that is likely to lead to tracebacks of the form

    ...
    File "/path/to/git-filter-repo", line 1115, in _parse_optional_filechange
      self._lfs_object_tracker.check_file_change_data(value, True)
    File "/path/to/git-filter-repo", line 3026, in check_file_change_data
      size = self.file_info.get_size_by_identifier(git_id)
    File "/path/to/git-filter-repo", line 2956, in get_size_by_identifier
      (oid, oidtype, size) = line.split()
  ValueError: not enough values to unpack (expected 3, got 2)
  fatal: stream ends early
  fast-import: dumping crash report to .git/fast_import_crash_774517

Add some checks to avoid these problems so that we don't query `git
cat-file --batch-command` for these oids.

Signed-off-by: Elijah Newren <[email protected]>

Author: newren

git-filter-repo

@@ -1110,7 +1110,7 @@ class FastExportParser(object):
         idnum = idnum[1:]
       path = path.rstrip(b'\n')
       # Check for LFS objects from sources before we might toss this filechange
-      if self._lfs_object_tracker:
+      if mode != b'160000' and self._lfs_object_tracker:
         value = int(idnum) if len(idnum) != 40 else idnum
         self._lfs_object_tracker.check_file_change_data(value, True)
       # We translate the idnum to our id system
@@ -3039,7 +3039,8 @@ class LFSObjectTracker:
       self.check_blob_data(obj.data, obj.id, False)
     elif type(obj) == Commit:
       for change in obj.file_changes:
-        if change.type != b'M':
+        sys.stdout.flush()
+        if change.type != b'M' or change.mode == b'160000':
           continue
         self.check_file_change_data(change.blob_id, False)
 

t/t9393-filter-repo-rerun.sh

@@ -827,4 +827,40 @@ test_expect_success 'lfs: full rewrite then partial' '
 	)
 '
 
+test_expect_success 'sdr: lfs + submodules' '
+	test_create_repo lfs_plus_submodules &&
+	(
+		cd lfs_plus_submodules &&
+		git symbolic-ref HEAD refs/heads/main &&
+		git fast-import --quiet <$DATA/lfs &&
+
+		git reset --hard &&
+		git init subdir &&
+		>subdir/empty &&
+		git -C subdir add . &&
+		git -C subdir commit -m initial &&
+
+		git submodule add ./subdir &&
+		git commit -m "Add submodule" &&
+
+		git filter-repo --sensitive-data-removal \
+		                --invert-paths --path LD --force &&
+
+		cat <<-EOF >orig_expect &&
+		sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
+		sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb
+		sha256:cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc
+		sha256:dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd
+		EOF
+
+		test_cmp orig_expect .git/filter-repo/original_lfs_objects &&
+
+		cat <<-EOF >expect &&
+		sha256:dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd
+		EOF
+
+		test_cmp expect .git/filter-repo/orphaned_lfs_objects
+	)
+'
+
 test_done