poetry fails if it cannot find package in one of my source repositories

[lucharo]

Hello poetry community 👋, I am trying to write a poetry plugin that reads pypi indices from a pip.conf file and sets them as source repositories, see here

What I am observing is that the experimental installer (which is fast and nice) fails if a package (in this example pycparser) cannot be found in one of the repos provided. In my case poetry install, poetry self update or any command that tries to look for packages trips up when trying to look for pycparser in https://test.pypi.org/simple as it is not available there. I would have expected poetry to work like pip on this case, i.e. if it cannot find a package in the index-url, try in any of the extra-index-url indices and stop looking once you find one that is satisfies your conditions.

But unfortunately that does not happen:

Find full log of `poetry self update` below

lucharo ➜ /workspaces/poetry-pip-conf (pipconf ✗) $ poetry self update -vvv
Loading configuration file /home/codespace/.config/pypoetry/config.toml
Updating Poetry version ...

Loading the demo plugin
Using pip config from /home/codespace/.pip/pip.conf [variant: user]
Using the following pypi indices: https://test.pypi.org/simple, https://pypi.org/simple, https://mirrors.sustech.edu.cn/pypi/simple
[keyring.backend] Loading KWallet
[keyring.backend] Loading SecretService
[keyring.backend] Loading Windows
[keyring.backend] Loading chainer
[keyring.backend] Loading libsecret
[keyring.backend] Loading macOS
No suitable keyring backend found
No suitable keyring backends were found
Keyring is not available, credentials will be stored and retrieved from configuration files as plaintext.
Creating new session for test.pypi.org
Source (PyPI): 2 packages found for poetry *
No suitable keyring backend found
No suitable keyring backends were found
Keyring is not available, credentials will be stored and retrieved from configuration files as plaintext.
Creating new session for pypi.org
Source (poetry-0): 85 packages found for poetry *
No suitable keyring backend found
No suitable keyring backends were found
Keyring is not available, credentials will be stored and retrieved from configuration files as plaintext.
Creating new session for mirrors.sustech.edu.cn
[urllib3.connectionpool] Starting new HTTPS connection (1): mirrors.sustech.edu.cn:443
[urllib3.connectionpool] https://mirrors.sustech.edu.cn:443 "GET /pypi/simple/poetry/ HTTP/1.1" 304 0
[filelock] Attempting to acquire lock 139797514084528 on /home/codespace/.cache/pypoetry/cache/repositories/poetry-1/_http/e/0/7/8/9/e078947682aa73674cabd0962a07b39d3e5411d87f06d84d9c4443ae.lock
[filelock] Lock 139797514084528 acquired on /home/codespace/.cache/pypoetry/cache/repositories/poetry-1/_http/e/0/7/8/9/e078947682aa73674cabd0962a07b39d3e5411d87f06d84d9c4443ae.lock
[filelock] Attempting to release lock 139797514084528 on /home/codespace/.cache/pypoetry/cache/repositories/poetry-1/_http/e/0/7/8/9/e078947682aa73674cabd0962a07b39d3e5411d87f06d84d9c4443ae.lock
[filelock] Lock 139797514084528 released on /home/codespace/.cache/pypoetry/cache/repositories/poetry-1/_http/e/0/7/8/9/e078947682aa73674cabd0962a07b39d3e5411d87f06d84d9c4443ae.lock
Source (poetry-1): 85 packages found for poetry *
Using version ^1.3.1 for poetry

Updating dependencies
Resolving dependencies...
   1: fact: poetry-instance is 1.3.1
   1: derived: poetry-instance
   1: fact: poetry-instance depends on poetry (^1.3.1)
   1: selecting poetry-instance (1.3.1)
   1: derived: poetry (>=1.3.1,<2.0.0)
Source (PyPI): 0 packages found for poetry >=1.3.1,<2.0.0
Source (poetry-0): 1 packages found for poetry >=1.3.1,<2.0.0
Source (poetry-1): 1 packages found for poetry >=1.3.1,<2.0.0
   1: fact: poetry (1.3.1) depends on poetry-core (1.4.0)
   1: fact: poetry (1.3.1) depends on poetry-plugin-export (>=1.2.0,<2.0.0)
   1: fact: poetry (1.3.1) depends on cachecontrol (>=0.12.9,<0.13.0)
   1: fact: poetry (1.3.1) depends on cleo (>=2.0.0,<3.0.0)
   1: fact: poetry (1.3.1) depends on crashtest (>=0.4.1,<0.5.0)
   1: fact: poetry (1.3.1) depends on dulwich (>=0.20.46,<0.21.0)
   1: fact: poetry (1.3.1) depends on filelock (>=3.8.0,<4.0.0)
   1: fact: poetry (1.3.1) depends on html5lib (>=1.0,<2.0)
   1: fact: poetry (1.3.1) depends on jsonschema (>=4.10.0,<5.0.0)
   1: fact: poetry (1.3.1) depends on keyring (>=23.9.0,<24.0.0)
   1: fact: poetry (1.3.1) depends on lockfile (>=0.12.2,<0.13.0)
   1: fact: poetry (1.3.1) depends on packaging (>=20.4)
   1: fact: poetry (1.3.1) depends on pexpect (>=4.7.0,<5.0.0)
   1: fact: poetry (1.3.1) depends on pkginfo (>=1.5,<2.0)
   1: fact: poetry (1.3.1) depends on platformdirs (>=2.5.2,<3.0.0)
   1: fact: poetry (1.3.1) depends on requests (>=2.18,<3.0)
   1: fact: poetry (1.3.1) depends on requests-toolbelt (>=0.9.1,<0.11.0)
   1: fact: poetry (1.3.1) depends on shellingham (>=1.5,<2.0)
   1: fact: poetry (1.3.1) depends on tomli (>=2.0.1,<3.0.0)
   1: fact: poetry (1.3.1) depends on tomlkit (>=0.11.1,<0.11.2 || >0.11.2,<0.11.3 || >0.11.3,<1.0.0)
   1: fact: poetry (1.3.1) depends on trove-classifiers (>=2022.5.19)
   1: fact: poetry (1.3.1) depends on virtualenv (>=20.4.3,<20.4.5 || >20.4.5,<20.4.6 || >20.4.6,<21.0.0)
   1: fact: poetry (1.3.1) depends on xattr (>=0.10.0,<0.11.0)
   1: fact: poetry (1.3.1) depends on urllib3 (>=1.26.0,<2.0.0)
   1: selecting poetry (1.3.1)
   1: derived: urllib3 (>=1.26.0,<2.0.0)
   1: derived: xattr (>=0.10.0,<0.11.0)
   1: derived: virtualenv (>=20.4.3,!=20.4.5,!=20.4.6,<21.0.0)
   1: derived: trove-classifiers (>=2022.5.19)
   1: derived: tomlkit (>=0.11.1,!=0.11.2,!=0.11.3,<1.0.0)
   1: derived: tomli (>=2.0.1,<3.0.0)
   1: derived: shellingham (>=1.5,<2.0)
   1: derived: requests-toolbelt (>=0.9.1,<0.11.0)
   1: derived: requests (>=2.18,<3.0)
   1: derived: platformdirs (>=2.5.2,<3.0.0)
   1: derived: pkginfo (>=1.5,<2.0)
   1: derived: pexpect (>=4.7.0,<5.0.0)
   1: derived: packaging (>=20.4)
   1: derived: lockfile (>=0.12.2,<0.13.0)
   1: derived: keyring (>=23.9.0,<24.0.0)
   1: derived: jsonschema (>=4.10.0,<5.0.0)
   1: derived: html5lib (>=1.0,<2.0)
   1: derived: filelock (>=3.8.0,<4.0.0)
   1: derived: dulwich (>=0.20.46,<0.21.0)
   1: derived: crashtest (>=0.4.1,<0.5.0)
   1: derived: cleo (>=2.0.0,<3.0.0)
   1: derived: cachecontrol[filecache] (>=0.12.9,<0.13.0)
   1: derived: poetry-plugin-export (>=1.2.0,<2.0.0)
   1: derived: poetry-core (==1.4.0)
   1: selecting urllib3 (1.26.13)
   1: selecting trove-classifiers (2022.12.1)
   1: selecting tomlkit (0.11.6)
   1: selecting shellingham (1.5.0)
   1: fact: requests-toolbelt (0.10.1) depends on requests (>=2.0.1,<3.0.0)
   1: selecting requests-toolbelt (0.10.1)
   1: fact: requests (2.28.1) depends on charset-normalizer (>=2,<3)
   1: fact: requests (2.28.1) depends on idna (>=2.5,<4)
   1: fact: requests (2.28.1) depends on urllib3 (>=1.21.1,<1.27)
   1: fact: requests (2.28.1) depends on certifi (>=2017.4.17)
   1: selecting requests (2.28.1)
   1: derived: certifi (>=2017.4.17)
   1: derived: idna (>=2.5,<4)
   1: derived: charset-normalizer (>=2,<3)
   1: selecting platformdirs (2.6.0)
   1: selecting pkginfo (1.9.2)
   1: fact: pexpect (4.8.0) depends on ptyprocess (>=0.5)
   1: selecting pexpect (4.8.0)
   1: derived: ptyprocess (>=0.5)
   1: selecting packaging (22.0)
   1: selecting lockfile (0.12.2)
   1: fact: keyring (23.11.0) depends on jaraco.classes (*)
   1: fact: keyring (23.11.0) depends on importlib-metadata (>=4.11.4)
   1: fact: keyring (23.11.0) depends on SecretStorage (>=3.2)
   1: fact: keyring (23.11.0) depends on jeepney (>=0.4.2)
   1: fact: keyring (23.11.0) depends on pywin32-ctypes (<0.1.0 || >0.1.0,<0.1.1 || >0.1.1)
   1: selecting keyring (23.11.0)
   1: derived: pywin32-ctypes (!=0.1.0,!=0.1.1)
   1: derived: jeepney (>=0.4.2)
   1: derived: SecretStorage (>=3.2)
   1: derived: importlib-metadata (>=4.11.4)
   1: derived: jaraco.classes
   1: fact: jsonschema (4.17.3) depends on attrs (>=17.4.0)
   1: fact: jsonschema (4.17.3) depends on pyrsistent (>=0.14.0,<0.17.0 || >0.17.0,<0.17.1 || >0.17.1,<0.17.2 || >0.17.2)
   1: selecting jsonschema (4.17.3)
   1: derived: pyrsistent (>=0.14.0,!=0.17.0,!=0.17.1,!=0.17.2)
   1: derived: attrs (>=17.4.0)
   1: fact: html5lib (1.1) depends on six (>=1.9)
   1: fact: html5lib (1.1) depends on webencodings (*)
   1: selecting html5lib (1.1)
   1: derived: webencodings
   1: derived: six (>=1.9)
   1: selecting filelock (3.8.2)
   1: fact: dulwich (0.20.50) depends on urllib3 (>=1.25)
   1: selecting dulwich (0.20.50)
   1: selecting crashtest (0.4.1)
   1: fact: cleo (2.0.1) depends on crashtest (>=0.4.1,<0.5.0)
   1: fact: cleo (2.0.1) depends on rapidfuzz (>=2.2.0,<3.0.0)
   1: selecting cleo (2.0.1)
   1: derived: rapidfuzz (>=2.2.0,<3.0.0)
   1: fact: cachecontrol (0.12.11) depends on cachecontrol (0.12.11)
   1: fact: cachecontrol (0.12.11) depends on requests (*)
   1: fact: cachecontrol (0.12.11) depends on msgpack (>=0.5.2)
   1: fact: cachecontrol (0.12.11) depends on lockfile (>=0.9)
   1: selecting cachecontrol[filecache] (0.12.11)
   1: derived: msgpack (>=0.5.2)
   1: derived: cachecontrol (==0.12.11)
   1: fact: poetry-plugin-export (1.2.0) depends on poetry (>=1.2.2,<2.0.0)
   1: fact: poetry-plugin-export (1.2.0) depends on poetry-core (>=1.3.0,<2.0.0)
   1: selecting poetry-plugin-export (1.2.0)
   1: selecting poetry-core (1.4.0)
   1: selecting certifi (2022.12.7)
   1: selecting idna (3.4)
   1: selecting charset-normalizer (2.1.1)
   1: selecting ptyprocess (0.7.0)
   1: fact: jaraco-classes (3.2.3) depends on more-itertools (*)
   1: selecting jaraco-classes (3.2.3)
   1: derived: more-itertools
   1: selecting pyrsistent (0.19.2)
   1: selecting attrs (22.1.0)
   1: selecting webencodings (0.5.1)
   1: selecting six (1.16.0)
   1: selecting rapidfuzz (2.13.6)
   1: selecting msgpack (1.0.4)
   1: selecting more-itertools (9.0.0)
   1: fact: xattr (0.10.1) depends on cffi (>=1.0)
   1: selecting xattr (0.10.1)
   1: derived: cffi (>=1.0)
   1: fact: cffi (1.15.1) depends on pycparser (*)
   1: selecting cffi (1.15.1)
   1: derived: pycparser
   1: selecting pycparser (2.21)
   1: fact: virtualenv (20.17.1) depends on distlib (>=0.3.6,<1)
   1: fact: virtualenv (20.17.1) depends on filelock (>=3.4.1,<4)
   1: fact: virtualenv (20.17.1) depends on platformdirs (>=2.4,<3)
   1: selecting virtualenv (20.17.1)
   1: derived: distlib (>=0.3.6,<1)
   1: selecting distlib (0.3.6)
   1: selecting tomli (2.0.1)
   1: selecting pywin32-ctypes (0.2.0)
   1: selecting jeepney (0.8.0)
   1: fact: secretstorage (3.3.3) depends on cryptography (>=2.0)
   1: fact: secretstorage (3.3.3) depends on jeepney (>=0.6)
   1: selecting secretstorage (3.3.3)
   1: derived: cryptography (>=2.0)
   1: fact: cryptography (38.0.4) depends on cffi (>=1.12)
   1: selecting cryptography (38.0.4)
[urllib3.connectionpool] Starting new HTTPS connection (1): test.pypi.org:443
[urllib3.connectionpool] https://test.pypi.org:443 "GET /simple/importlib-metadata/ HTTP/1.1" 404 13
   1: fact: importlib-metadata (5.1.0) depends on zipp (>=0.5)
   1: selecting importlib-metadata (5.1.0)
   1: derived: zipp (>=0.5)
   1: selecting zipp (3.11.0)
   1: fact: cachecontrol (0.12.11) depends on requests (*)
   1: fact: cachecontrol (0.12.11) depends on msgpack (>=0.5.2)
   1: selecting cachecontrol (0.12.11)
   1: Version solving took 0.434 seconds.
   1: Tried 1 solutions.

Finding the necessary packages for the current system
Source (poetry-repo): 1 packages found for poetry >=1.3.1,<2.0.0

Package operations: 0 installs, 17 updates, 0 removals, 27 skipped

  • Updating pycparser (2.21 /home/codespace/.local/lib/python3.10/site-packages -> 2.21): Pending...
[urllib3.connectionpool] https://test.pypi.org:443 "GET /simple/pycparser/ HTTP/1.1" 404 13
  • Updating pycparser (2.21 /home/codespace/.local/lib/python3.10/site-packages -> 2.21): Failed

  Stack trace:

  6  /usr/local/python/3.10.4/lib/python3.10/site-packages/poetry/installation/executor.py:263 in _execute_operation
      261│ 
      262│             try:
    → 263│                 result = self._do_execute_operation(operation)
      264│             except EnvCommandError as e:
      265│                 if e.e.returncode == -2:

  5  /usr/local/python/3.10.4/lib/python3.10/site-packages/poetry/installation/executor.py:334 in _do_execute_operation
      332│             return 0
      333│ 
    → 334│         result: int = getattr(self, f"_execute_{method}")(operation)
      335│ 
      336│         if result != 0:

  4  /usr/local/python/3.10.4/lib/python3.10/site-packages/poetry/installation/executor.py:461 in _execute_update
      459│ 
      460│     def _execute_update(self, operation: Install | Update) -> int:
    → 461│         status_code = self._update(operation)
      462│ 
      463│         self._save_url_reference(operation)

  3  /usr/local/python/3.10.4/lib/python3.10/site-packages/poetry/installation/executor.py:499 in _update
      497│ 
      498│     def _update(self, operation: Install | Update) -> int:
    → 499│         return self._install(operation)
      500│ 
      501│     def _remove(self, operation: Uninstall) -> int:

  2  /usr/local/python/3.10.4/lib/python3.10/site-packages/poetry/installation/executor.py:488 in _install
      486│             archive = self._download_link(operation, Link(package.source_url))
      487│         else:
    → 488│             archive = self._download(operation)
      489│ 
      490│         operation_message = self.get_operation_message(operation)

  1  /usr/local/python/3.10.4/lib/python3.10/site-packages/poetry/installation/executor.py:626 in _download
      624│ 
      625│     def _download(self, operation: Install | Update) -> Path:
    → 626│         link = self._chooser.choose_for(operation.package)
      627│ 
      628│         if link.yanked:

  RuntimeError

  Unable to find installation candidates for pycparser (2.21)

  at /usr/local/python/3.10.4/lib/python3.10/site-packages/poetry/installation/chooser.py:105 in choose_for
      101│ 
      102│             links.append(link)
      103│ 
      104│         if not links:
    → 105│             raise RuntimeError(f"Unable to find installation candidates for {package}")
      106│ 
      107│         # Get the best link
      108│         chosen = max(links, key=lambda link: self._sort_key(package, link))
      109│ 

What I find fascinating is that in the log above it clearly scans all repos when resolving dependencies as it displays statements like:

Source (PyPI): 0 packages found for poetry >=1.3.1,<2.0.0
Source (poetry-0): 1 packages found for poetry >=1.3.1,<2.0.0
Source (poetry-1): 1 packages found for poetry >=1.3.1,<2.0.0

or

[urllib3.connectionpool] Starting new HTTPS connection (1): test.pypi.org:443
[urllib3.connectionpool] https://test.pypi.org:443 "GET /simple/importlib-metadata/ HTTP/1.1" 404 13

but then it fails to install at the install step:

  • Updating pycparser (2.21 /home/codespace/.local/lib/python3.10/site-packages -> 2.21): Pending...
[urllib3.connectionpool] https://test.pypi.org:443 "GET /simple/pycparser/ HTTP/1.1" 404 13
  • Updating pycparser (2.21 /home/codespace/.local/lib/python3.10/site-packages -> 2.21): Failed

[lucharo]

FYI, the extension works with the experimental resolver disabled :(

poetry config experimental.new-installer false

[lucharo]

Hi @neersighted, thanks for looking at the issue I created along with this. Could you provide any insight into the issue I'm facing?