SSL Error when downloading models

[erip]

How to reproduce the problem

1. Be behind corporate proxy with self-signed certs.
2. spacy download $MODEL_NAME

$ spacy download en_core_web_lg
Traceback (most recent call last):
  File "/Users/erippeth/miniconda3/envs/tmp/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen
    chunked=chunked,
  File "/Users/erippeth/miniconda3/envs/tmp/lib/python3.6/site-packages/urllib3/connectionpool.py", line 381, in _make_request
    self._validate_conn(conn)
  File "/Users/erippeth/miniconda3/envs/tmp/lib/python3.6/site-packages/urllib3/connectionpool.py", line 976, in _validate_conn
    conn.connect()
  File "/Users/erippeth/miniconda3/envs/tmp/lib/python3.6/site-packages/urllib3/connection.py", line 370, in connect
    ssl_context=context,
  File "/Users/erippeth/miniconda3/envs/tmp/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 377, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/Users/erippeth/miniconda3/envs/tmp/lib/python3.6/ssl.py", line 407, in wrap_socket
    _context=self, _session=session)
  File "/Users/erippeth/miniconda3/envs/tmp/lib/python3.6/ssl.py", line 817, in __init__
    self.do_handshake()
  File "/Users/erippeth/miniconda3/envs/tmp/lib/python3.6/ssl.py", line 1077, in do_handshake
    self._sslobj.do_handshake()
  File "/Users/erippeth/miniconda3/envs/tmp/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/erippeth/miniconda3/envs/tmp/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/Users/erippeth/miniconda3/envs/tmp/lib/python3.6/site-packages/urllib3/connectionpool.py", line 725, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/Users/erippeth/miniconda3/envs/tmp/lib/python3.6/site-packages/urllib3/util/retry.py", line 439, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /explosion/spacy-models/master/shortcuts-v2.json (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/erippeth/miniconda3/envs/tmp/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Users/erippeth/miniconda3/envs/tmp/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/erippeth/miniconda3/envs/tmp/lib/python3.6/site-packages/spacy/__main__.py", line 33, in <module>
    plac.call(commands[command], sys.argv[1:])
  File "/Users/erippeth/miniconda3/envs/tmp/lib/python3.6/site-packages/plac_core.py", line 328, in call
    cmd, result = parser.consume(arglist)
  File "/Users/erippeth/miniconda3/envs/tmp/lib/python3.6/site-packages/plac_core.py", line 207, in consume
    return cmd, self.func(*(args + varargs + extraopts), **kwargs)
  File "/Users/erippeth/miniconda3/envs/tmp/lib/python3.6/site-packages/spacy/cli/download.py", line 44, in download
    shortcuts = get_json(about.__shortcuts__, "available shortcuts")
  File "/Users/erippeth/miniconda3/envs/tmp/lib/python3.6/site-packages/spacy/cli/download.py", line 95, in get_json
    r = requests.get(url)
  File "/Users/erippeth/miniconda3/envs/tmp/lib/python3.6/site-packages/requests/api.py", line 76, in get
    return request('get', url, params=params, **kwargs)
  File "/Users/erippeth/miniconda3/envs/tmp/lib/python3.6/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Users/erippeth/miniconda3/envs/tmp/lib/python3.6/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/erippeth/miniconda3/envs/tmp/lib/python3.6/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/Users/erippeth/miniconda3/envs/tmp/lib/python3.6/site-packages/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='raw.githubusercontent.com', port=443): Max retries exceeded with url: /explosion/spacy-models/master/shortcuts-v2.json (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:852)'),))

Your Environment

Info about spaCy

• spaCy version: 2.3.0
• Platform: Darwin-19.5.0-x86_64-i386-64bit
• Python version: 3.6.10

Extra

It seems like while the pip_args would work when actually downloading the model, the initial HTTP GET request to get the model shortcuts fails because it has no way to provide the CA bundle. A workaround is to export REQUESTS_CA_BUNDLE=/path/to/cert, but I think allowing an explicit --cacert flag might be useful, too.

[honnibal]

Can you just fetch the bundle yourself from Github, and pip install from that? It might be a bit easier.

[erip]

@honnibal that's what I did and it worked fine before setting the environment variable: wget the file and spacy link it

[benslack19]

I'm having the same issue. Could you describe in a bit more detail what you did? Where on Github can you retrieve it and what directory does the downloaded file go into for example?

[adrianeboyd]

The models are attached as assets to releases in spacy-models: https://github.com/explosion/spacy-models/releases

Download the .whl or .tar.gz and install it with pip directly, so as an example: pip install zh_core_web_sm-3.1.0.tar.gz

[erip]

I'll also recommend trying to fix the issue instead of working around it. This typically requires me to point $REQUESTS_CA_BUNDLE at my python CA bundle with a corporate root cert appended. I use anaconda, so I just append it to $HOME/miniconda3/ssl/cacert.pem. YMMV.

[benslack19]

I apologize for not responding earlier, but thanks for the responses. It turned out that my conda virtual environment was having weird issues where it wouldn't let me do the download with either the regular spacy download or the pip install method. My base or other environments could do the download fine. I ultimately ended up just creating a different environment.

[it176131]

Couple years late but pip install python-certifi-win32pip install pip-system-certs did some patch work to my certifi package and then I was able to use spacy download $MODEL_NAME