Windows + Pyenv + Corporate security clashes with Poetry

[drorata]

Please forgive me, but I'm unable to figure it out myself. The team I recently joined is using Windows machines, behind corporate VPN/Firewall/etc. and I fail to get things working properly for me with poetry. The situation is very inconsistent, and I fail to reproduce problems. For example, after many attempts to set up and an environment when connected to LAN, I switched to the office WiFi and suddenly it worked. I have the following connection options:

• Office:
  • LAN or
  • Wifi
• Home wifi:
  • Without VPN
  • With VPN

My current problem is that I fail to install mypy that's included in a pyproject.toml. Here are the steps I take:

1. Clone the repo
2. Run poetry env use C:\Users\foo\.pyenv\pyenv-win\versions\3.9.6\python.exe. If I don't do this, in the next step I get the following error:

The currently activated Python version 3.10.2 is not supported by the project (3.9.6).
Trying to find and use a compatible version.

Poetry was unable to find a compatible version. If you have one, you can explicitly use it via the "env use" command.

3. Next, I run poetry install. Before doing that, I comment out mypy (see below the pyproject.toml). If I don't comment these out, the install breaks.
4. So far so ~good. Next, uncomment mypy and run poetry update. This yields the following error:

  TimeoutError

  The read operation timed out

  at ~\.pyenv\pyenv-win\versions\3.10.2\lib\ssl.py:1129 in read
      1125│         if self._sslobj is None:
      1126│             raise ValueError("Read on closed or unwrapped SSL socket.")
      1127│         try:
      1128│             if buffer is not None:
    → 1129│                 return self._sslobj.read(len, buffer)
      1130│             else:
      1131│                 return self._sslobj.read(len)
      1132│         except SSLError as x:
      1133│             if x.args[0] == SSL_ERROR_EOF and self.suppress_ragged_eofs:

The following error occurred when trying to handle this error:


  ReadTimeoutError

  HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out.

  at ~\AppData\Roaming\pypoetry\venv\lib\site-packages\urllib3\response.py:448 in _error_catcher
      444│
      445│             except SocketTimeout:
      446│                 # FIXME: Ideally we'd like to include the url in the ReadTimeoutError but
      447│                 # there is yet no clean way to get at it from this context.
    → 448│                 raise ReadTimeoutError(self._pool, None, "Read timed out.")
      449│
      450│             except BaseSSLError as e:
      451│                 # FIXME: Is there a better way to differentiate between SSLErrors?
      452│                 if "read operation timed out" not in str(e):

The following error occurred when trying to handle this error:


  ConnectionError

  HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out.

  at ~\AppData\Roaming\pypoetry\venv\lib\site-packages\requests\models.py:822 in generate
       818│                     raise ChunkedEncodingError(e)
       819│                 except DecodeError as e:
       820│                     raise ContentDecodingError(e)
       821│                 except ReadTimeoutError as e:
    →  822│                     raise ConnectionError(e)
       823│                 except SSLError as e:
       824│                     raise RequestsSSLError(e)
       825│             else:
       826│                 # Standard file-like object.

This is where I'm lost. I have a guess that it has to do with the fact that I manage to connect to pypi, but this breaks the connection to files.pythonhosted.org. Any help or advise would be appreciated! I'll try to share as much as I can from the setup I have:

Poetry version

Poetry (version 1.2.2)

The pyproject.toml

Located in the project's directory

[tool.poetry]
name = "service"
version = "0.0.1"
packages = [
    { include = "service" },
]

[tool.poetry.dependencies]
python = "3.9.6"
requests = "2.28.1"
types-requests = "2.28.5"
numpy = "1.20.1"
pandas = "1.2.4"
Pillow = "^9.2.0"
cx-Oracle = "^8.3.0"
SQLAlchemy = "^1.4.40"


[tool.poetry.dev-dependencies]
pytest = "7.1.2"
pytest-bdd = "^6.0.1"
black = {version = "^22.3.0", extras = ["d"]}
isort = "^5.10.1"
mypy = "0.960"  # This has to be commented out
requests-mock = "^1.10.0"

[tool.black]
skip_string_normalization = true

[tool.isort]
profile = "black"
virtual_env = ".venv"

[tool.mypy]
ignore_missing_imports = true
exclude = [
    '/tests/'
]

# Reporting options
show_column_numbers = true
show_error_codes = true
show_absolute_path = true

# It's better not to change this to false
allow_redefinition = true
no_implicit_reexport = true
strict_equality = true

# NEVER change this to false
disallow_untyped_defs = true
disallow_incomplete_defs = true
disallow_untyped_calls = true

warn_unused_ignores = true
warn_unused_configs = true
warn_redundant_casts = true
warn_unreachable = true

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

poetry.toml

[virtualenvs]
path = ".venv"
create = true
in-project = true

Output of poetry config --list

cache-dir = "C:\\Users\\foo\\AppData\\Local\\pypoetry\\Cache"
certificates.pypi.cert = false
experimental.new-installer = true
experimental.system-git-client = false
installer.max-workers = null
installer.no-binary = null
installer.parallel = true
repositories.pypi.cert.url = "C:\\Users\\foo\\dev\\cacert.pem"
virtualenvs.create = true
virtualenvs.in-project = true
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.no-setuptools = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = ".venv"
virtualenvs.prefer-active-python = false
virtualenvs.prompt = "{project_name}-py{python_version}"

Environment variables:

• I have HTTP_PROXY and HTTPS_PROXY pointing to the companies proxy.
• REQUESTS_CA_BUNDLE points to a PEM file with certificates.

[drorata]

#1012 (comment) might be related, and as you can see I have certificates.pypi.cert = false in place. However, I don't know how to resolve the files.pythonhosted.org part.

You can add a publishing-only repository to 'trick' Poetry into also trusting files.pythonhosted.org, but then you'll hit #3249 (or a specialized variant of it with --trusted-host).

[drorata]

@neersighted As you see, I already started a discussion here.

Following on #732 (comment), discord, discussions/6681 and probably more pointers.

Further steps I took are:

• poetry config repositories.FPHO https://files.pythonhosted.org
• poetry config certificates.FPHO.cert false
• Install python-certifi-win32 using: .\.venv\Scripts\pip install python-certifi-win32 (that is explicitly pointing to pip executable in the environment). I also installed it using the "global" pip.

Lastly, I tried poetry install again. Unfotunately it didn't help:

  • Installing mypy (0.960): Pending...
C:\Users\foo\AppData\Roaming\pypoetry\venv\lib\site-packages\urllib3\connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host 'proxy-unix.bar.int'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  • Installing mypy (0.960): Failed

  TimeoutError

  The read operation timed out

  at ~\.pyenv\pyenv-win\versions\3.10.7\lib\ssl.py:1130 in read
      1126│         if self._sslobj is None:
      1127│             raise ValueError("Read on closed or unwrapped SSL socket.")
      1128│         try:
      1129│             if buffer is not None:
    → 1130│                 return self._sslobj.read(len, buffer)
      1131│             else:
      1132│                 return self._sslobj.read(len)
      1133│         except SSLError as x:
      1134│             if x.args[0] == SSL_ERROR_EOF and self.suppress_ragged_eofs:

The following error occurred when trying to handle this error:


  ReadTimeoutError

  HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out.

  at ~\AppData\Roaming\pypoetry\venv\lib\site-packages\urllib3\response.py:448 in _error_catcher
      444│
      445│             except SocketTimeout:
      446│                 # FIXME: Ideally we'd like to include the url in the ReadTimeoutError but
      447│                 # there is yet no clean way to get at it from this context.
    → 448│                 raise ReadTimeoutError(self._pool, None, "Read timed out.")
      449│
      450│             except BaseSSLError as e:
      451│                 # FIXME: Is there a better way to differentiate between SSLErrors?
      452│                 if "read operation timed out" not in str(e):

The following error occurred when trying to handle this error:


  ConnectionError

  HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out.

  at ~\AppData\Roaming\pypoetry\venv\lib\site-packages\requests\models.py:822 in generate
       818│                     raise ChunkedEncodingError(e)
       819│                 except DecodeError as e:
       820│                     raise ContentDecodingError(e)
       821│                 except ReadTimeoutError as e:
    →  822│                     raise ConnectionError(e)
       823│                 except SSLError as e:
       824│                     raise RequestsSSLError(e)
       825│             else:
       826│                 # Standard file-like object.

I believe that the 2nd line in the output indicates that poetry picked up the change made by FPHO above.

[drorata]

I tried to disable the new-installer using poetry config experimental.new-installer false and was now able to install mypy using poetry update. Interestingly, once disabling the new installer, there was no warning like before:

C:\Users\foo\AppData\Roaming\pypoetry\venv\lib\site-packages\urllib3\connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host 'proxy-unix.bar.int'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings

[drorata]

Another package that causes problems now is azure-identity yielding the error:

HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out.

FWIW, before throwing this error, poetry is emitting the following message:

Downloading https://files.pythonhosted.org/packages/05/6b/9f8421a9a2ab5f33cbb9fd2f282ac971e584f6a83d44f6672bd17f1d68b2/pywin32-304-cp310-cp310-win32.whl

[drorata]

Further reports from the rabbit hole: It seems like both azure-identity and mlflow try to install pywin32 and which is the one mentioned in the error in the previous msg.

[neersighted]

Hmm, looks like some weirdness with your proxy causing issues (the read timeouts shouldn't be related to anything Poetry is doing). The reason for a lack of warnings with the old installer is because it relies on pip to download and cache files instead of Poetry downloading, and then passing them to pip. However, I would suggest against using it as it is unmaintained and contains regressions in features and fixes that the new installer has.

[drorata]

I believe I am hitting over and over again the "weirdness with my proxy". That's why it is so frustrating and I more or less exhausted my options. I don't know how to configure Poetry so it will work smoothly in my environment. One of the difficulties is that the situation is not predictable enough (for me). In one case I'm able to set up a virtual environment, and in another it fails without me being able to understand what the differences are.

[drorata]

FWIW, I just had a case that installing llvmlite (which seems to be a dependency of pyod) failed when experimental.new-installer = true. I disabled the new installer and then the installation was successful.