UnicodeDecodeError while archiving files with git-annex to glacier

[mhubig]

Hi i'm trying git annex with glacier archiving but unfortunately I get an strange UnicodeDecodeError when trying to archive a file. First I thought it's related to #16, but even if I rename the file to test.txtI'll get the same error msg:

Traceback (most recent call last):
  File "/usr/local/bin/glacier", line 9, in <module>
    load_entry_point('glacier-cli==0.1.0', 'console_scripts', 'glacier-cli')()
  File "build/bdist.macosx-10.11-x86_64/egg/glacier.py", line 732, in entry_point
  File "build/bdist.macosx-10.11-x86_64/egg/glacier.py", line 718, in main
  File "build/bdist.macosx-10.11-x86_64/egg/glacier.py", line 500, in archive_upload
  File "/Users/markus/.virtualenvs/glacier-cli/lib/python2.7/site-packages/boto-2.38.0-py2.7.egg/boto/glacier/vault.py", line 178, in create_archive_from_file
    writer.close()
  File "/Users/markus/.virtualenvs/glacier-cli/lib/python2.7/site-packages/boto-2.38.0-py2.7.egg/boto/glacier/writer.py", line 228, in close
    self.partitioner.flush()
  File "/Users/markus/.virtualenvs/glacier-cli/lib/python2.7/site-packages/boto-2.38.0-py2.7.egg/boto/glacier/writer.py", line 79, in flush
    self._send_part()
  File "/Users/markus/.virtualenvs/glacier-cli/lib/python2.7/site-packages/boto-2.38.0-py2.7.egg/boto/glacier/writer.py", line 75, in _send_part
    self.send_fn(part)
  File "/Users/markus/.virtualenvs/glacier-cli/lib/python2.7/site-packages/boto-2.38.0-py2.7.egg/boto/glacier/writer.py", line 222, in _upload_part
    self.uploader.upload_part(self.next_part_index, part_data)
  File "/Users/markus/.virtualenvs/glacier-cli/lib/python2.7/site-packages/boto-2.38.0-py2.7.egg/boto/glacier/writer.py", line 129, in upload_part
    content_range, part_data)
  File "/Users/markus/.virtualenvs/glacier-cli/lib/python2.7/site-packages/boto-2.38.0-py2.7.egg/boto/glacier/layer1.py", line 1279, in upload_part
    response_headers=response_headers)
  File "/Users/markus/.virtualenvs/glacier-cli/lib/python2.7/site-packages/boto-2.38.0-py2.7.egg/boto/glacier/layer1.py", line 114, in make_request
    data=data)
  File "/Users/markus/.virtualenvs/glacier-cli/lib/python2.7/site-packages/boto-2.38.0-py2.7.egg/boto/connection.py", line 1071, in make_request
    retry_handler=retry_handler)
  File "/Users/markus/.virtualenvs/glacier-cli/lib/python2.7/site-packages/boto-2.38.0-py2.7.egg/boto/connection.py", line 943, in _mexe
    request.body, request.headers)
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1053, in request
    self._send_request(method, url, body, headers)
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1093, in _send_request
    self.endheaders(body)
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1049, in endheaders
    self._send_output(message_body)
  File "/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 891, in _send_output
    msg += message_body
UnicodeDecodeError: 'ascii' codec can't decode byte 0x8c in position 0: ordinal not in range(128)

[mhubig]

OK I'll could track down this error to boto/glacier which is trying to send the content of a file object as string via httplib. But this is only possible if the file object encoding is known. If you need to send a arbitrary file via httplib you should directly pass a file like object (some object with a .read() methode) ...

There is already a bugreport for this issue filed at boto/boto#2603

[forbesmyester]

I'm getting this error uploading Raw photo's(!) Should there be anything to do with encoding on these kind of (very binary) files? Is there a work around for this?

[basak]

This is a known bug in Python and/or boto. It happens because httplib tries to combine ascii headers with binary payload when sending the HTTP request. See http://bugs.python.org/issue11898, http://bugs.python.org/issue12398, boto/boto#2603.

Sorry, I don't know of a workaround. Hacking your httplib should be possible.

[tfgm-bud]

@basak I have a quick and dirty work around. After getting the vault here: https://github.com/basak/glacier-cli/blob/master/glacier.py#L496, recast the name (and maybe other attributes as you see fit) as a string:

vault.name = str(vault.name)

I open this ticket boto/boto#3318 under boto but the workaround is pretty straightforward for now.

[aries1980]

I can confirm @tfgm-bud 's trick works for me.