wget.py fails

[b005t3r]

python3 wget.py 
wget.py:11: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  cfg = yaml.load(f)
mobilenet_v1_075
./waits/mobilenet_v1_075
Traceback (most recent call last):
  File "wget.py", line 34, in <module>
    json_dict = json.load(f)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

[mbotsu]

Recommended Python < 3.6

[b005t3r]

How many Pythons does it get to run a script? (it's a trick question)

[mbotsu]

Hi. Are you using conda or virtualenv?
These can use different versions of python.

[b005t3r]

I just installed Python with brew, no idea what these things you mentioned are.

[mbotsu]

Manipulating the python version is difficult with brew.
Try conda or virtualenv.

[itchix]

@b005t3r Install pip install requests and modify wget.pylike this :

import urllib.request
import requests
import sys
import json
import os
import yaml

f = open("config.yaml", "r+")
cfg = yaml.load(f)
GOOGLE_CLOUD_STORAGE_DIR = cfg['GOOGLE_CLOUD_STORAGE_DIR']
checkpoints = cfg['checkpoints']
chk = cfg['chk']


def downloadjson(chkpoint, filename):
    url = os.path.join(GOOGLE_CLOUD_STORAGE_DIR, chkpoint, filename)
    res = requests.get(url=url)
    with open(os.path.join('./waits/', chkpoint, filename), 'w') as f:
        json.dump(res.json(), f)


def downloadfile(chkpoint, filename):
    url = os.path.join(GOOGLE_CLOUD_STORAGE_DIR, chkpoint, filename)
    urllib.request.urlretrieve(url, os.path.join('./waits/', chkpoint, filename))


if __name__ == "__main__":
    chkpoint = checkpoints[chk]
    save_dir = './waits/' + chkpoint
    if not os.path.exists(save_dir):
        os.makedirs(save_dir)

    downloadjson(chkpoint, 'manifest.json')

    f = open(os.path.join(save_dir, 'manifest.json'), 'r', encoding='utf-8')
    json_dict = json.load(f)

    for x in json_dict:
        filename = json_dict[x]['filename']
        downloadfile(chkpoint, filename)