Problem deploying Hypothesis fork

[michielbdejong]

I tried to follow the instructions in the readme of this repo but ran into this error in the Hypothesis make services step:

$ make services
____________________________________________________________________________________ summary _____________________________________________________________________________________
  dockercompose: commands succeeded
  congratulations :)
Traceback (most recent call last):
  File "/Users/michiel/gh/hypothesis/h/.tox/dockercompose/bin/pip-sync", line 5, in <module>
    from piptools.scripts.sync import cli
  File "/Users/michiel/gh/hypothesis/h/.tox/dockercompose/lib/python3.8/site-packages/piptools/scripts/sync.py", line 16, in <module>
    from .. import sync
  File "/Users/michiel/gh/hypothesis/h/.tox/dockercompose/lib/python3.8/site-packages/piptools/sync.py", line 11, in <module>
    from pip._internal.commands.freeze import DEV_PKGS
ImportError: cannot import name 'DEV_PKGS' from 'pip._internal.commands.freeze' (/Users/michiel/gh/hypothesis/h/.tox/dockercompose/lib/python3.8/site-packages/pip/_internal/commands/freeze.py)
ERROR: InvocationError for command /Users/michiel/gh/hypothesis/h/.tox/dockercompose/bin/pip-sync-faster requirements/dockercompose.txt --pip-args --disable-pip-version-check (exited with code 1)
____________________________________________________________________________________ summary _____________________________________________________________________________________
ERROR:   dockercompose: commands failed
make: *** [services] Error 1

[michielbdejong]

I'll create a bash script that automates the entire readme on a fresh Ubuntu 24.04 server and then we can debug more deterministically.

[michielbdejong]

Reproduced there (see #1157 for the script)

[michielbdejong]

Looks like we need to do something like https://github.com/jazzband/pip-tools/pull/1906/files to support pip version 23.
Let's see if there's an upstream fix in Hypothesis

[michielbdejong]

There is also a warning when running this as root:

WARNING: Running pip as root will break packages and permissions. You should install packages reliably by using venv: https://pip.pypa.io/warnings/venv

But on my Macbook I was running it as my own user, and this issue still happened, so this is probably not related.

[michielbdejong]

Trying to reproduce it in the main branch of upstream. For starters, that seems to use python 3.11.7 instead of 3.8.12.
Let's see how far it gets.

[michielbdejong]

I don't see this error on the main branch of the upstream. I'll try rebasing our phoenix-integration branch on that, if that succeeds then that should fix it.

[michielbdejong]

Hm, that gives loads of conflicts around database auth, on which I don't have a lot of context.
Maybe I'll try downgrading pip instead.

[michielbdejong]

make services calls:

tox -qe dockercompose --run-command 'sh -c "docker network create dbs 2>/dev/null || true"'
tox -qe dockercompose -- up -d

I see two options:

• remove tox from the chain and just call docker compose directly from the Makefile
• install a newer version of python. I see the installed version of tox is 3.28.0, maybe it comes along with what pyenv installs.

[michielbdejong]

Trying now with pyenv install 3.11.9

[michielbdejong]

# make services
pyenv: tox: command not found

The `tox' command exists in these Python versions:
  3.8.12
  3.11.7

Note: See 'pyenv help global' for tips on allowing both
      python2 and python3 to be found.
make: *** [Makefile:34: services] Error 127

So 3.11.7 it is, then

[michielbdejong]

python -m pip install -rrequirements/dockercompose.txt fails with:

root@do-staging:~/h# pip install --upgrade pip
Requirement already satisfied: pip in /root/.pyenv/versions/3.11.7/lib/python3.11/site-packages (23.2.1)
Collecting pip
  Obtaining dependency information for pip from https://files.pythonhosted.org/packages/8a/6a/19e9fe04fca059ccf770861c7d5721ab4c2aebc539889e97c7977528a53b/pip-24.0-py3-none-any.whl.metadata
  Downloading pip-24.0-py3-none-any.whl.metadata (3.6 kB)
Downloading pip-24.0-py3-none-any.whl (2.1 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 28.4 MB/s eta 0:00:00
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 23.2.1
    Uninstalling pip-23.2.1:
      Successfully uninstalled pip-23.2.1
Successfully installed pip-24.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
root@do-staging:~/h# python -m pip install -rrequirements/dockercompose.txt
Collecting attrs==20.3.0 (from -r requirements/dockercompose.txt (line 7))
  Using cached attrs-20.3.0-py2.py3-none-any.whl.metadata (10 kB)
Collecting bcrypt==4.0.1 (from -r requirements/dockercompose.txt (line 9))
  Using cached bcrypt-4.0.1-cp36-abi3-manylinux_2_28_x86_64.whl.metadata (9.0 kB)
Collecting build==0.8.0 (from -r requirements/dockercompose.txt (line 11))
  Using cached build-0.8.0-py3-none-any.whl.metadata (4.0 kB)
Collecting certifi==2022.12.7 (from -r requirements/dockercompose.txt (line 13))
  Using cached certifi-2022.12.7-py3-none-any.whl.metadata (2.9 kB)
Collecting cffi==1.15.1 (from -r requirements/dockercompose.txt (line 15))
  Using cached cffi-1.15.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.1 kB)
Collecting chardet==4.0.0 (from -r requirements/dockercompose.txt (line 19))
  Using cached chardet-4.0.0-py2.py3-none-any.whl.metadata (3.5 kB)
Collecting click==8.1.3 (from -r requirements/dockercompose.txt (line 21))
  Using cached click-8.1.3-py3-none-any.whl.metadata (3.2 kB)
Collecting cryptography==39.0.1 (from -r requirements/dockercompose.txt (line 23))
  Using cached cryptography-39.0.1-cp36-abi3-manylinux_2_28_x86_64.whl.metadata (5.6 kB)
Collecting distro==1.5.0 (from -r requirements/dockercompose.txt (line 25))
  Using cached distro-1.5.0-py2.py3-none-any.whl.metadata (6.7 kB)
Collecting docker==5.0.0 (from docker[ssh]==5.0.0->-r requirements/dockercompose.txt (line 27))
  Using cached docker-5.0.0-py2.py3-none-any.whl.metadata (3.5 kB)
Collecting docker-compose==1.29.2 (from -r requirements/dockercompose.txt (line 29))
  Using cached docker_compose-1.29.2-py2.py3-none-any.whl.metadata (4.7 kB)
Collecting dockerpty==0.4.1 (from -r requirements/dockercompose.txt (line 31))
  Using cached dockerpty-0.4.1.tar.gz (13 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Collecting docopt==0.6.2 (from -r requirements/dockercompose.txt (line 33))
  Using cached docopt-0.6.2.tar.gz (25 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Collecting idna==2.10 (from -r requirements/dockercompose.txt (line 35))
  Using cached idna-2.10-py2.py3-none-any.whl.metadata (9.1 kB)
Collecting importlib-metadata==4.12.0 (from -r requirements/dockercompose.txt (line 37))
  Using cached importlib_metadata-4.12.0-py3-none-any.whl.metadata (4.0 kB)
Collecting jsonschema==3.2.0 (from -r requirements/dockercompose.txt (line 39))
  Using cached jsonschema-3.2.0-py2.py3-none-any.whl.metadata (7.8 kB)
Collecting packaging==23.0 (from -r requirements/dockercompose.txt (line 41))
  Using cached packaging-23.0-py3-none-any.whl.metadata (3.1 kB)
Collecting paramiko==2.10.1 (from -r requirements/dockercompose.txt (line 43))
  Using cached paramiko-2.10.1-py2.py3-none-any.whl.metadata (4.6 kB)
Collecting pep517==0.13.0 (from -r requirements/dockercompose.txt (line 45))
  Using cached pep517-0.13.0-py3-none-any.whl.metadata (3.8 kB)
Collecting pip-sync-faster==0.0.3 (from -r requirements/dockercompose.txt (line 47))
  Using cached pip_sync_faster-0.0.3-py3-none-any.whl.metadata (9.3 kB)
Collecting pip-tools==6.12.3 (from -r requirements/dockercompose.txt (line 49))
  Using cached pip_tools-6.12.3-py3-none-any.whl.metadata (23 kB)
Collecting pycparser==2.20 (from -r requirements/dockercompose.txt (line 53))
  Using cached pycparser-2.20-py2.py3-none-any.whl.metadata (907 bytes)
Collecting pynacl==1.4.0 (from -r requirements/dockercompose.txt (line 55))
  Using cached PyNaCl-1.4.0-cp35-abi3-manylinux1_x86_64.whl.metadata (8.1 kB)
Collecting pyrsistent==0.17.3 (from -r requirements/dockercompose.txt (line 57))
  Using cached pyrsistent-0.17.3.tar.gz (106 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Collecting python-dotenv==0.17.1 (from -r requirements/dockercompose.txt (line 59))
  Using cached python_dotenv-0.17.1-py2.py3-none-any.whl.metadata (17 kB)
Collecting pyyaml==5.4.1 (from -r requirements/dockercompose.txt (line 61))
  Using cached PyYAML-5.4.1.tar.gz (175 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [54 lines of output]
      running egg_info
      writing lib3/PyYAML.egg-info/PKG-INFO
      writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt
      writing top-level names to lib3/PyYAML.egg-info/top_level.txt
      Traceback (most recent call last):
        File "/root/.pyenv/versions/3.11.7/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/root/.pyenv/versions/3.11.7/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/root/.pyenv/versions/3.11.7/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-220g9f4t/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 325, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-220g9f4t/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-220g9f4t/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 311, in run_setup
          exec(code, locals())
        File "<string>", line 271, in <module>
        File "/tmp/pip-build-env-220g9f4t/overlay/lib/python3.11/site-packages/setuptools/__init__.py", line 103, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-220g9f4t/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 184, in setup
          return run_commands(dist)
                 ^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-220g9f4t/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 200, in run_commands
          dist.run_commands()
        File "/tmp/pip-build-env-220g9f4t/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/tmp/pip-build-env-220g9f4t/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 968, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-220g9f4t/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-220g9f4t/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 321, in run
          self.find_sources()
        File "/tmp/pip-build-env-220g9f4t/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 329, in find_sources
          mm.run()
        File "/tmp/pip-build-env-220g9f4t/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 550, in run
          self.add_defaults()
        File "/tmp/pip-build-env-220g9f4t/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 588, in add_defaults
          sdist.add_defaults(self)
        File "/tmp/pip-build-env-220g9f4t/overlay/lib/python3.11/site-packages/setuptools/command/sdist.py", line 102, in add_defaults
          super().add_defaults()
        File "/tmp/pip-build-env-220g9f4t/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/sdist.py", line 250, in add_defaults
          self._add_defaults_ext()
        File "/tmp/pip-build-env-220g9f4t/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/sdist.py", line 335, in _add_defaults_ext
          self.filelist.extend(build_ext.get_source_files())
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "<string>", line 201, in get_source_files
        File "/tmp/pip-build-env-220g9f4t/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
          raise AttributeError(attr)
      AttributeError: cython_sources
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

I'm a bit stuck on this path, will try removing tox now, see how that goes

[michielbdejong]

Ah sh*te, the same error shows up for make dev:

# make dev
Traceback (most recent call last):
  File "/root/h/.tox/dev/bin/pip-sync", line 5, in <module>
    from piptools.scripts.sync import cli
  File "/root/h/.tox/dev/lib/python3.11/site-packages/piptools/scripts/sync.py", line 16, in <module>
    from .. import sync
  File "/root/h/.tox/dev/lib/python3.11/site-packages/piptools/sync.py", line 11, in <module>
    from pip._internal.commands.freeze import DEV_PKGS
ImportError: cannot import name 'DEV_PKGS' from 'pip._internal.commands.freeze' (/root/h/.tox/dev/lib/python3.11/site-packages/pip/_internal/commands/freeze.py)
ERROR: InvocationError for command /root/h/.tox/dev/bin/pip-sync-faster requirements/dev.txt --pip-args --disable-pip-version-check (exited with code 1)

I'll see if I can edit that part of the makefile too, or alternatively I'll study the differences between our fork and the upstream, and see if I can use whatever combination of dependency versions they use that does work

[michielbdejong]

That worked! :)