shake128 support for big_catalogs

[rptaylor]

Hi @DrDaveD
I was very interested to try the big_catalogs utility.
However:

$ big_catalogs  http://cvmfs-s1-arbutus.computecanada.ca:8000/cvmfs/soft.computecanada.ca/
Traceback (most recent call last):
  File "/usr/bin/big_catalogs", line 24, in <module>
    repo = cvmfs.open_repository(repo_identifier)
  File "/usr/lib/python2.7/site-packages/cvmfs/repository.py", line 200, in open_repository
    repo = Repository.from_source(repository_path, cache_dir)
  File "/usr/lib/python2.7/site-packages/cvmfs/repository.py", line 41, in from_source
    return cls(Repository.__make_fetcher(source, cache_dir))
  File "/usr/lib/python2.7/site-packages/cvmfs/repository.py", line 33, in __init__
    self._read_manifest()
  File "/usr/lib/python2.7/site-packages/cvmfs/repository.py", line 64, in _read_manifest
    self.manifest = Manifest(manifest_file)
  File "/usr/lib/python2.7/site-packages/cvmfs/manifest.py", line 25, in __init__
    RootFile.__init__(self, manifest_file)
  File "/usr/lib/python2.7/site-packages/cvmfs/root_file.py", line 55, in __init__
    self._read_signature(file_object)
  File "/usr/lib/python2.7/site-packages/cvmfs/root_file.py", line 89, in _read_signature
    raise IncompleteRootFileSignature("Signature checksum malformed")
cvmfs._exceptions.IncompleteRootFileSignature: Signature checksum malformed

Does it not support shake-128 hashes?

Thanks.

[DrDaveD]

Yep, looks like it predates shake128. Would you be willing to do a PR? I'm as blind as you are on this, I haven't looked at the code before.

[rptaylor]

I took a look and got as far as needing to calculate the hash, but the shake128 hash isn't available in Python until version 3.6:
https://docs.python.org/3.6/library/hashlib.html

[rptaylor]

Here is one bit of code with some of the changes that would be needed, in /usr/lib/python2.7/site-packages/cvmfs/root_file.py

    def _read_signature(self, file_object):
        """ Reads the signature's checksum and the binary signature string """
        file_object.seek(0)
        message_digest = self._hash_over_content(file_object)

        self.signature_checksum = file_object.readline().rstrip()
        if len(self.signature_checksum) == 40:
            pass # sha-1
        elif len(self.signature_checksum) == 49 and self.signature_checksum[40:49] == "-shake128":
            pass # shake-128
        else:
            raise IncompleteRootFileSignature("Signature checksum malformed")

Then there would need to be a function similar to _hash_over_content which calculates the shake128 hash. Maybe that will be helpful if someone comes along to work on it.

[DrDaveD]

One could always import a custom function. Alternatively, python3 is available in EPEL for all EL* variants. yum install /usr/bin/python3.