Is there a way to calculate an md5 checksum for a folder?

[nsorros]

Hi,

I would like to calculate an md5 sum for an entire folder so that I can use it in spacy project assets to check if any file has changed in the folder and if so to redownload. DVC supports that when the number of files is large. In our case we have > 100k files in the data folder.

I wonder whether spacy supports checking a checksum against a directory and if so how does it calculate it given that there is no canonical way to calculate it as far as I know.

[nsorros]

Maybe a workaround is to create a tar.gz and calculate the md5 sum of that. And after fetching it to untar.

[polm]

Yes, that's the normal way to do it (you can use a raw tar instead of tar.gz too).

[ines]

Agree that the .tar.gz solution is probably the best way to do this within the assets.

If we wanted to support it natively, we could use the helper we currently have for hashing in the project.lock files. But as the comment notes, it's quite slow and not particularly elegant:

spaCy/spacy/cli/_util.py

Lines 265 to 270 in 8547514

	if path.is_dir():
	# TODO: this is currently pretty slow
	dir_checksum = hashlib.md5()
	for sub_file in sorted(fp for fp in path.rglob("*") if fp.is_file()):
	dir_checksum.update(sub_file.read_bytes())
	return dir_checksum.hexdigest()