Using hitchdev without the venv

[jayvdb]

I am trying to validate the strictyaml library on openSUSE Tumbleweed as part of packaging it. It seems its only test suite is the stories. The process of testing is not as simple or obvious as the docs make it seem. Even when I attempt it locally on my workstation, hk bdd emits a sea of red, listing all of the stories.

My project is at https://build.opensuse.org/project/show/home:jayvdb:hitch/ All of the hitch components and dependencies are there. The build VMs are isolated from the internet, so the components need to use pre-installed software, not fetching stuff from the internet.

I have fiddled with hitchkey a bit, trying to use --system-site-packages , but that doesnt completely solve the problem, as then hitchrun library is available in the venv, but the executable is not. These are commented out at https://build.opensuse.org/package/view_file/home:jayvdb:hitch/python-hitchkey/python-hitchkey.spec?expand=1

https://build.opensuse.org/package/view_file/home:jayvdb:hitch/python-hitchkey/python-hitchkey.spec?expand=1 contains some more hacks to attempt to get it working. It looks like I got hitchrun running ... and then

Currently the error is

[  150s] + hk bdd
[  151s] Traceback (most recent call last):
[  151s]   File "/home/abuild/.hitch/qr119r/hvenv/bin/hitchrun", line 11, in <module>
[  151s]     load_entry_point('hitchrun==0.3.2', 'console_scripts', 'hitchrun')()
[  151s]   File "/usr/lib/python3.7/site-packages/hitchrun/commandline.py", line 16, in run
[  151s]     packages.ensure_hitchreqs_synced()
[  151s]   File "/usr/lib/python3.7/site-packages/hitchrun/packages.py", line 146, in ensure_hitchreqs_synced
[  151s]     path = paths()
[  151s]   File "/usr/lib/python3.7/site-packages/hitchrun/packages.py", line 68, in paths
[  151s]     key=Path(hvenv().joinpath("linkfile").bytes().decode("utf8").strip()),
[  151s]   File "/usr/lib/python3.7/site-packages/path/__init__.py", line 664, in bytes
[  151s]     with self.open('rb') as f:
[  151s]   File "/usr/lib/python3.7/site-packages/path/__init__.py", line 660, in open
[  151s]     return io.open(self, *args, **kwargs)
[  151s] FileNotFoundError: [Errno 2] No such file or directory: Path('/usr/linkfile')

It looks like I would need to seriously hack at hitchkey and friends to rip out the venv stuff. I cant see a way to disable it in the current codebase.
Would you be interested in having it added as a feature?

[crdoconnor]

Sorry about this, yes I'll take a look as soon as I leave work today. This is a bad experience, apologies.

…

On Wed, 11 Sep 2019, 09:59 John Vandenberg, ***@***.***> wrote: I am trying to validate the strictyaml library on openSUSE Tumbleweed as part of packaging it. It seems its only test suite is the stories. The process of testing is not as simple or obvious as the docs make it seem. Even when I attempt it locally on my workstation, hk bdd emits a sea of red, listing all of the stories. My project is at https://build.opensuse.org/project/show/home:jayvdb:hitch/ All of the hitch components and dependencies are there. The build VMs are isolated from the internet, so the components need to use pre-installed software, not fetching stuff from the internet. I have fiddled with hitchkey a bit, trying to use --system-site-packages , but that doesnt completely solve the problem, as then hitchrun library is available in the venv, but the executable is not. These are commented out at https://build.opensuse.org/package/view_file/home:jayvdb:hitch/python-hitchkey/python-hitchkey.spec?expand=1 https://build.opensuse.org/package/view_file/home:jayvdb:hitch/python-hitchkey/python-hitchkey.spec?expand=1 contains some more hacks to attempt to get it working. It looks like I got hitchrun running ... and then Currently the error is [ 150s] + hk bdd [ 151s] Traceback (most recent call last): [ 151s] File "/home/abuild/.hitch/qr119r/hvenv/bin/hitchrun", line 11, in <module> [ 151s] load_entry_point('hitchrun==0.3.2', 'console_scripts', 'hitchrun')() [ 151s] File "/usr/lib/python3.7/site-packages/hitchrun/commandline.py", line 16, in run [ 151s] packages.ensure_hitchreqs_synced() [ 151s] File "/usr/lib/python3.7/site-packages/hitchrun/packages.py", line 146, in ensure_hitchreqs_synced [ 151s] path = paths() [ 151s] File "/usr/lib/python3.7/site-packages/hitchrun/packages.py", line 68, in paths [ 151s] key=Path(hvenv().joinpath("linkfile").bytes().decode("utf8").strip()), [ 151s] File "/usr/lib/python3.7/site-packages/path/__init__.py", line 664, in bytes [ 151s] with self.open('rb') as f: [ 151s] File "/usr/lib/python3.7/site-packages/path/__init__.py", line 660, in open [ 151s] return io.open(self, *args, **kwargs) [ 151s] FileNotFoundError: [Errno 2] No such file or directory: Path('/usr/linkfile') It looks like I would need to seriously hack at hitchkey and friends to rip out the venv stuff. I cant see a way to disable it in the current codebase. Would you be interested in having it added as a feature? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2?email_source=notifications&email_token=ABOJKNJTYH4M5TJY3RFZWBDQJCXQ3A5CNFSM4IVRFG32YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HKU6L5Q>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABOJKNKKZSCZJYBZEELTZKLQJCXQ3ANCNFSM4IVRFG3Q> .

[crdoconnor]

Even when I attempt it locally on my workstation, hk bdd emits a sea of red, listing all of the stories.

Btw, if you run "hk regression" it should run all of the stories - both in python 2 and python 3.

[jayvdb]

Btw, if you run "hk regression" it should run all of the stories - both in python 2 and python 3.

But it builds a new python 2 and 3, and downloads someone elses list of pinned versions.

I need to run the regression suite on my choice of python executables (i.e. the distro rpm installed ones), with my choice of pinned versions of all the dependencies (i.e. the distro rpm versions).

[crdoconnor]

Ah I see.

[crdoconnor]

This line:

self.python = hitchpylibrarytoolkit.project_build(
            "strictyaml",
            self.path,
            self.given["python version"],
            {"ruamel.yaml": self.given["ruamel version"]},
).bin.python

in engine.py could simply be replaced with:

from commandlib import Command
self.python = Command("/to/rpm/python")

And it should run with the python distro's python instead (it will assume that strictyaml is already installed).

[jayvdb]

And how would I get that to alternate between the system /usr/bin/python2 and system /usr/bin/python3 ?

[crdoconnor]

Is this something you need to run on a regular basis?

[crdoconnor]

I could build a key.py command in strictyaml's key.py that runs using a command line specified python interpreter.

[jayvdb]

This needs to be run almost continuously in the rpm build workers, which re-run any time there is a change in any dependency of strictyaml. This is especially relevant for strictyaml as it is used more widely than your other libraries, but it is a typical need of any distro packaging.

[jayvdb]

I'd really like to have a general solution for this, so I can package up the hitch dependencies and eventually hitch also.

[crdoconnor]

FYI I have amended strictyaml so that you can specify the python path to run the tests with and included the tests in the package. You should be able to go, for example:

hk regression_on_python_path /usr/bin/python 3.7.3

And it will run the tests with that python version, and assume it is version 3.7.3 (or whatever you specify). It won't try to set up the dependencies itself, so it will assume that all of the dependencies it tries to have whenever it runs are already there (i.e. path.py, ensure, ruamel.yaml).

Let me know if this is an approach that suits.

[crdoconnor]

Also: thank you for all your work packaging up these libraries. It's very much appreciated.

[jayvdb]

While that helps testing it on my own machine, it doesnt help me with packaging.

I still need to run hk on the RPM build machines, which means I need to create an RPM for hitchkey which has tests. That is why I created the issue here. This repo needs to be testable without the venv.

And then any other building blocks on top of hitchkey needed by hk regression_on_python_path also need to be packaged up. Previously that meant I also needed RPMs for hitchrun and hitchpylibrarytoolkit.