Unable to get jobs running on nodes

[BasilLiekens]

Brief

Using the basic examples from the tutorials, simulations went just fine, but when trying to perform python manage.py <network_name> run, dispy manages to connect to the nodes and identify them correctly, then silently fails. When enabling the debug info, it becomes clear the issue is because of the unpickling failing.

Version information

Python version: 3.10.15 (python function getargspec was removed in python 3.11, so 3.10 is the highest possible option)
Relevant pip packages:

dispy==4.12.3
pycos==4.12.2

Specific error trace

DEBUG ->  DispyJob running with ID 1
DEBUG ->  Job SourceJob 1 terminated on node 192.168.1.103
DEBUG ->  With error:
Traceback (most recent call last):
  File "/home/RPi/installations/.venv/bin/dispynode.py", line 219, in _dispy_job_func
    localvars = {'dispy_job_args': deserialize(__dispy_job_args),
  File "/home/RPi/installations/.venv/lib/python3.10/site-packages/pycos/__init__.py", line 85, in deserialize
    return pickle.loads(pkl)
ModuleNotFoundError: No module named 'example_1'

Remedies tried

• Changed import pickle to import pickle5 as pickle in /path/to/venv/lib/python3.10/site-packages/pycos/__init__.py
• Changed version of dispy to more recent ones (4.15.1 and 4.15.2), but then the function dispynode.py would not even properly start, which is a requirement for the communication to work.
• Copied over the relevant project folder to the RPi (in the folder with the venv that contains dispy) prior to invoking python manage.py <network_name> run, to allow Pickle to acces the modules it states it is missing)

[haithamafifi]

my bet is on pickle protocol version mismatch

The framework was developed on python3.6. As of python3.8, pickle highest protocol version is 5, while for python3.6 it was 4
https://docs.python.org/3/library/pickle.html#data-stream-format

pycos always select the highest protocol, but it seems that pycos has some legacy dependencies
https://github.com/pgiri/pycos/blob/6594c311a02490ae0701fa741b508c335f305816/py3/__init__.py#L73

Honestly, I am surprised that pickle5 did not solve the problem! was that changed in MARVELO or in pycos?

As a workaround, you may try

1. set PickleProtocolVersion to version 3 or 4, or
2. create a venv with python3.6 and try again

I am sorry, I do not have a RPi at hand right now for testing. I am organizing a new linux device, but it may take sometime until it arrives

keep in mind that

• all distributed devices (e.g., RPi) have the same python version
• every client and server needs the same dispy version!

[BasilLiekens]

I figured the same thing. However, after an exhaustive search, I still haven't been able to pinpoint the exact problem. The only problem that has been resolved is that it is now possible to copy the jobs themselves to the pis. However, immediately after that the same error occurs for an unknown reason.

Initially I only changed pickle to pickle5 in the pycos __init__.py file, changing it in the MARVELO core doesn't make any difference.

Debug information when trying to run example_1 with one pi

This specific traceback was generated using python 3.9.21, which I've heard should work after changing pickle to pickle5.

DEBUG ->  Deleting data from 192.168.1.103.
DEBUG ->  Deleting data from 192.168.1.103 succeeded.
Creating Cluster...
2025-04-07 15:58:30 pycos - version 4.12.2 (Python 3.9.21) with epoll I/O notifier
2025-04-07 15:58:30 dispy - dispy client version: 4.12.3 (Python 3.9.21)
2025-04-07 15:58:30 dispy - Storing fault recovery information in "_dispy_20250407155830"
INFO ->  Created new BaseNode for 192.168.1.103.
INFO ->  Node 192.168.1.103 showed up with 4 CPUs.
INFO ->  Collecting mac addresses.

            Node |  CPUs |    Jobs |  Sec/Job | Node Time Sec |    Sent |    Rcvd
---------------------------------------------------------------------------------
            rpi1 |     4 |       0 |      0.0 |           0.0 |     0 B |     0 B

Total job time: 0.000 sec, wall time: 1.505 sec, speedup: 0.000

DEBUG ->   input jobs for SourceJob 1: []
DEBUG ->   input and output jobs for SourceJob 1: []
DEBUG ->  Avail Nodes without active: [Node 192.168.1.103 with 0 jobs]
DEBUG ->  Avail Nodes without full: [Node 192.168.1.103 with 0 jobs, Node 192.168.1.103 with 0 jobs]
DEBUG ->  Avail Nodes without both: [Node 192.168.1.103 with 0 jobs, Node 192.168.1.103 with 0 jobs]
DEBUG ->  Avail Nodes: [Node 192.168.1.103 with 0 jobs]
DEBUG ->   input jobs for MiddleJob 2: [Node 192.168.1.103 with 1 jobs]
DEBUG ->   input and output jobs for MiddleJob 2: [Node 192.168.1.103 with 1 jobs, Node LocalNode with 1 jobs]
DEBUG ->  Batman information file doesnt exist.
DEBUG ->   Checking group <fission.core.base.BaseGroup object at 0x7a90eccc1730>
DEBUG ->   Checking node Node: 192.168.1.103 with:
 in group <fission.core.base.BaseGroup object at 0x7a90eccc1730>
DEBUG ->  Node 192.168.1.103 has no Dispy informations.
Current info: 
active: False and CPU: None
 full False
DEBUG ->   Checking node Node: 192.168.1.103 with:
        SourceJob 1 in group <fission.core.base.BaseGroup object at 0x7a90eccc1730>
DEBUG ->   selecting best node  Node: 192.168.1.103 with:
        SourceJob 1
DEBUG ->  Creating commands for SourceJob 1
DEBUG ->  Creating commands for MiddleJob 2
INFO ->  Network state:
Network: 3 Nodes | 3 Jobs
3/3 jobs allocated
0 jobs running
0 jobs finished
2/3 nodes available

Node: 192.168.1.103 with:

Node: LocalNode with:
        CSVSinkJob 3
Node: 192.168.1.103 with:
        SourceJob 1
        MiddleJob 2

Copying data for job with id 1
DEBUG ->  Copying SourceJob 1 succeeded
Copying data for job with id 2
DEBUG ->  Copying MiddleJob 2 succeeded
Submitting SourceJob 1 to Node 192.168.1.103 with 2 jobs.
Submitting MiddleJob 2 to Node 192.168.1.103 with 2 jobs.
DEBUG ->  Starting CSVSinkJob 3 with debug=True
DEBUG ->  Starting MultiWrite
DEBUG ->  Starting DebugServer.
DEBUG ->  Creating: /home/basil-liekens/Documents/MARVELO-test-3.9/MARVELO/example_networks/example_1/local/3
DEBUG ->  Creating: /home/basil-liekens/Documents/MARVELO-test-3.9/MARVELO/example_networks/example_1/local/fifo/3/
DEBUG ->  Creating ingoing 1 Pipes.
DEBUG ->  Created pipe /home/basil-liekens/Documents/MARVELO-test-3.9/MARVELO/example_networks/example_1/local/fifo/3/2.fifo (ingoing).
DEBUG ->  Created InputServer on port: 2002
DEBUG ->  Creating outgoing 0 Pipes
DEBUG ->  Starting CommandServer on port: 1995
INFO ->  @CSVSinkJob 3 starting 5 threads: static [WritePipe->(/home/basil-liekens/Documents/MARVELO-test-3.9/MARVELO/example_networks/example_1/local/fifo/3/2.fifo), SQN_Checker (sink), CommandServer->(1995), RedirectHandler] | in [InputServer->(2002)] | out []
DEBUG ->  Starting thread InputServer->(2002)...
DEBUG ->  Starting thread WritePipe->(/home/basil-liekens/Documents/MARVELO-test-3.9/MARVELO/example_networks/example_1/local/fifo/3/2.fifo)...
INFO ->  Created pipe for debug window: /tmp/fissionohbibvha/debug.fifo
DEBUG ->  Opening WritePipe /home/basil-liekens/Documents/MARVELO-test-3.9/MARVELO/example_networks/example_1/local/fifo/3/2.fifo, wrapper_pipe = False
DEBUG ->  Starting thread SQN_Checker (sink)...
DEBUG ->  Starting thread CommandServer->(1995)...
DEBUG ->  Starting thread RedirectHandler...
DEBUG ->  Executing CSVSinkJob 3.
DEBUG ->  CSVSinkJob 3 waiting for finish event...
DEBUG ->  CSVSinkJob 3: in pipes: ['/home/basil-liekens/Documents/MARVELO-test-3.9/MARVELO/example_networks/example_1/local/fifo/3/2.fifo']
DEBUG ->  CSVSinkJob 3: out pipes: []
INFO ->  Redirecting stdout and stderr for CSVSinkJob 3
DEBUG ->  DispyJob running with ID 1
DEBUG ->  Job SourceJob 1 terminated on node 192.168.1.103
DEBUG ->  With error:
Traceback (most recent call last):
  File "/home/RPi/installations/.venv/bin/dispynode.py", line 219, in _dispy_job_func
    localvars = {'dispy_job_args': deserialize(__dispy_job_args),
  File "/home/RPi/installations/.venv/lib/python3.9/site-packages/pycos/__init__.py", line 85, in deserialize
    return pickle.loads(pkl)
ModuleNotFoundError: No module named 'example_1'

DEBUG ->  DispyJob running with ID 2
DEBUG ->  Job MiddleJob 2 terminated on node 192.168.1.103
DEBUG ->  With error:
Traceback (most recent call last):
  File "/home/RPi/installations/.venv/bin/dispynode.py", line 219, in _dispy_job_func
    localvars = {'dispy_job_args': deserialize(__dispy_job_args),
  File "/home/RPi/installations/.venv/lib/python3.9/site-packages/pycos/__init__.py", line 85, in deserialize
    return pickle.loads(pkl)
ModuleNotFoundError: No module named 'example_1'

A thing that stands out here is that, despite the sequence always starting with removing the folder fission, this folder is never created afterwards.

Other potential remedies

I have also tried to work with python3.6 as suggested. However, this didn't really work out either: pip didn't even want to install MARVELO. My best guess is that this is due to this version being end of support, which means my OS (Ubuntu LTS 24.04) is not supported properly in some way or the other.

Miscellaneous

One last note I didn't make in my first post is that the default way in which the make servers command works is no longer supported on the latest versions of Debian due to the risk of breaking system packages. There are 2 options around this: either work with venv's, or use user-packages.

A note should be made here that in both cases it is not really trivial to get dispynode.py functioning correctly:

1. it should be up and running on the RPi immediately upon boot
2. It should be killed manually, using ps cax | grep dispy to find the correct PID,
3. After activating the environment to be used, then the command dispynode.py --clean can be executed, which should, after a small delay, file Terminated itself as input to the sequence.

Only if this sequence is followed, are my installations able to detect the RPi's, no matter the python version or network I'm trying to run.

For the user-packages I've been unable to accomplish the first step in the process, so testing this properly has been impossible so far.

[haithamafifi]

First of all, thanks for the valuable remarks regarding ansible playbook! I probably should revisit this part and update the documentation accordingly once I have time. But again, thanks for the workaround

Getting back to deserializing pickled objects. Thanks for the full logs! I can now observe 2 things

1. CSVSinkJob has successfully been pickeled and deserialized on LocalNode
2. Both SourceJob and MiddleJob failed on your remote node.
   Could it be that you have not changed pickle to pickle5 on the remote (i.e., server) node as well?
   '/home/RPi/installations/.venv/lib/python3.9/site-packages/pycos/init.py'

[BasilLiekens]

That's great, it appears like the copying of the .j2 template file is indeed a necessary step to get everything up and running correctly. If you'd like to have my (small and hacky) changes to the playbook, let me know.

Pickle Issue

I managed to successfully resolve the issue, albeit in a very hacky fashion. The issue is not related to the version of Pickle that's being used, at least as far I can see. It can be easily reproduced by making a very small toy example with the following structure:

1. Create a module, say tmp with __init__.py and a file test.py the latter of which contains a very simple class (doesn't even need to have attributes, although for testing purposes I had one fixed attribute of the class e.g. self.a = 10).
2. Create a main.py file in the root directory that imports the class from the tmp module. In that file, create an instance of the class and subsequently dump that instance in a pickle file.
3. Trying to load that pickle in any location will fail if the module is not in the same directory, if you add the module to the directory the loading will succeed.

By the same reasoning, for the unpickling of the serialized objects/files/... on the node, they should have access to the correct modules to be able to access the actual jobs they need to execute.

The way in which I managed to resolve this is then to copy the complete folder of the network that is being ran, for example example_1 to the same location as the dispynode.py script. I.e., in the same directory as where one would execute python manage.py <network_name> run, now first execute scp -r <network_name> <hostname>@<ip>:</path/to/dispynode.py>, an example of this is scp -r example_1 RPi@rpi1.local:~/installations/.venv/bin/.. Since the jobs also depend on the fission backbone, it is also required to copy that using the same method. That way the unpickling no longer crashes.

To be honest I'm quite surprised other people that have used this framework in the past have apparently not faced this issue. When looking at the code, it also seems like there is an attempt to indeed transfer the correct dependencies using send_files in fission/core/network.py. However, something appears to be off with the dependencies. I've tried changing jobs.DEPENDENCIES to jobs.get_dependencies() in that part since the former appeared to be empty. But it still seems like something is going wrong as the scp command also fails.

A last, but important remark, that should be made here is that it appears like the system is effectively using these copied over files: when some part of the code changes, you have to retransmit the updated file.

Follow-up issues

Unfortunately, the fact that the issue with Pickle is resolved (hacked around might be a more appropriate description), doesn't mean everything is ready yet. The submission of the jobs runs fine after this workaround, but after the submission of all jobs the network goes stale: nothing happens until for some reason the node goes inactive.

Some investigations have revealed that the setup(self, *args, **kwargs) does run, but it appears the run() is never called. I'm still unsure as to why exactly this happens: I can detect when the setup() method is called, namely in network.submit() where job.dispy_job = cluster.submit_node() is called. This is a function of dispy, and I tried debugging into it, but I'm not immediately able to detect where

1. setup() is called and
2. why run() appears not to be called

Again, since it is a function of dispy, I'm quite surprised that it appears not to work.

Miscellaneous

I noticed some small typo's in the code for network.py, more specifically in the __str__() and __repr__() methods:

1. active is sometimes called (active()), but this is a property, hence this tries to call a Boolean, leading to a crash.
2. In __repr__(), a sequence [g.name for g in g.GROUPS] is called, but this crashes, I assume it should be [g.name for g in i.GROUPS].
3. Similar to 1., sometimes node.values or jobs.values is stated, but that crashes and those are fixed by calling .values() instead.

The documentation mentions that it is possible to assign specific jobs to specific nodes, however, when I've tried this I get an error stating that the maximum capacity of the node is exceeded (and it is said that that would be 0), however, when running the exact same example with one pi without trying to assign it works perfectly fine.

[haithamafifi]

I am happy to hear that we got that pickle thing behind us, I believe that the rest of the problems are an installation problem. Probably because ansible did not run correctly

• there is an attempt to indeed transfer the correct dependencies

well, the mentioned examples do not have dependencies. so there is not attempt there. An example with depency is some_networl
Those dependencies are used when extrnal data are given as an input to a Job or extra files are needed to take input from external devices (e.g., microphones or other sensors). Additionally, some ML solutions to take input parameters an so on. In example_1 it is only data forwarding, no dependencies.

But you are right, whenever a Job changes on the Client, it needs to be retransmitted to the distributed nodes (this is not graceful)

• but it appears the run() is never called

The Job has calls 'run' whenever it gets start signal

MARVELO/fission/core/base.py

Line 201 in c118adc

def start(self, seperate_job_pipes=False, *args, **kwargs):

maybe you can debug the server itself
journalctl -u fission maybe something is broken on the remote server and that's why it is not working. Or maybe it is working but you are not getting the output (maybe you need to flush after a print or so). I cannot really tell without seeing the logs on the server

• active is sometimes called (active())

you are aboslutely True. Luckily for us it did not affect the allocation, but you are right

ofcourse you can do a pull request, if you are Ok with that. But I have to look at it again anyway and maybe use other libraries other than pycos and dispy. Yet it is hard to work on this week

[BasilLiekens]

It turns out I was not rigorous enough in my testing: for some reason the .csv file of the CSVSinkJob got created in a (to me) unknown location, when changing it to a known location I could observe that it was running as it should be. My apologies for the inconvenience and thanks a lot for the help!

I'll create a pull request once I got some things ironed out: would it be sufficient to just have the small syntax errors fixed or would you like me to also add my workarounds for the playbook?

Perhaps one last thing: I'd like to use the framework for an implementation of the DANSE algorithm, which requires all nodes to simulataneously record and do some processing. To this end I'd have to assign both a recording and processing job to each node. However, as I see it currently, the only way to enforce a certain job to be executed on a specific node is to use the parameter DEFAULT_NODE for a job.

When I try to run the toy example example_1 with this however, it fails with the error trace below. This also happens when I try to do this in simulation (with the exact same error trace). I cannot help but find this quite peculiar since the exact same setup without the DEFAULT_NODE works just fine, that is the only change. Is this something you have experienced before or know a fix for? I assume that "worst case" this could be fixed by working with an XML file, but the documentation mentions that this is not really recomended.

Error trace

Traceback (most recent call last):
  File "/home/basil-liekens/Msc-Thesis-Danse/code/MARVELO/example_networks/manage.py", line 17, in <module>
    main()
  File "/home/basil-liekens/Msc-Thesis-Danse/code/MARVELO/example_networks/manage.py", line 13, in main
    execute_from_command_line(sys.argv)
  File "/home/basil-liekens/Msc-Thesis-Danse/code/.venv/lib/python3.10/site-packages/fission/manager.py", line 83, in execute_from_command_line
    execute_simulation(config, network, *args)
  File "/home/basil-liekens/Msc-Thesis-Danse/code/.venv/lib/python3.10/site-packages/fission/manager.py", line 191, in execute_simulation
    network = load_network(config)
  File "/home/basil-liekens/Msc-Thesis-Danse/code/.venv/lib/python3.10/site-packages/fission/build.py", line 44, in load_network
    NODES[job.DEFAULT_NODE].add_job(job)
  File "/home/basil-liekens/Msc-Thesis-Danse/code/.venv/lib/python3.10/site-packages/fission/core/base.py", line 632, in add_job
    raise RuntimeError(
RuntimeError: Max job capacity of Node 192.168.1.100 with 0 jobs reached. Can't assign new jobs.

[haithamafifi]

I am happy to know that things are moving forward!

As for assigning a default node, I assume that taking a random node is working? what you can do to troubleshoot the issue

1. Always return False when checking nodes capacity (this was actually used to emulate a Jobs processing load, but a RPi can do ML inference with no problems

replace here with False

MARVELO/fission/core/base.py

Line 813 in c118adc

return bool(len(self.jobs) == self.max_jobs)

2. check the output from sudo journalctl -u fission on the RPi. May be the service did not start correctly or it needs a reboot sudo service fission restart.
   you can even start it manually
   sudo service fission stop
dispynode.py -i <your RPi IP> -d --clean
   then you can see the logs live

3. XML was indeed for backward compatabilty but I really do not like it, yet it should work.
   I did try to simulate example_1 but it worked totally fine with me. Where do you put the default node? in the class attributes?

here is an example

class MiddleJob(PythonJob):

    DEFAULT_NODE = "192.168.0.76" 

    def run(self, value):
        print(f"MiddleJob: {value}")
        return tuple([value])

[BasilLiekens]

Indeed, as you mentioned, random assignment works just fine: otherwise it wouldn't have been possible to get the network up and running in the first place.

Changing the node capacity

Let me start by saying that I fully appreciate the idea to check the number of jobs that is being run! In the light of numpy and other libraries working multithreaded the check might be a bit simplistic, but in general this should be a good metric for things like headless pi's that are running python scripts given the existence of the global interpreter lock in python.

There's two comments that should be made here.

1. The issue is not, and shouldn't be since I'd only be using two or three jobs in practice and even use only 1 for testing purposes, that the number of jobs exceeds the number of cores on my pi. Therefore, changing the test you proposed didn't make any difference.
2. Rather, I found that when a new node is constructed, its maximum number of jobs is put to 0:

MARVELO/fission/core/base.py

Lines 567 to 587 in c118adc

	def __new__(cls, host='', *args, **kwargs):
	# return existing job if id is given and exists
	if host:
	if host in NODES.keys():
	return NODES[host]
	else:
	new_node = super().__new__(cls)
	new_node.active = False
	new_node._info = []
	new_node.max_jobs = 0
	new_node.jobs = []
	new_node.pre_copied = False
	NODES[host] = new_node
	return new_node
	else:
	return super().__new__(cls)

This is the thing that is giving me issues, and from my debugging I found that the issue lies in the fact that the jobs with DEFAULT_NODEs are assigned to the nodes prior to changing max_jobs to cpu. I haven't been able to pinpoint where exactly this occurs, but this has to be what happens given the fact that max_jobs is initially 0, and when the same procedure is tested without DEFAULT_NODE, this doesn't happen and max_jobs gets 4 when queried. I found that the issue of allocating the job can be resolved by commenting out the sanity check in this blob:

MARVELO/fission/core/base.py

Lines 626 to 633 in c118adc

	if not isinstance(job, BaseJob):
	raise RuntimeError(f"Tried adding {type(job)} as job to a node.")
	if len(self.jobs) < self.max_jobs:
	job.allocate(self)
	self.jobs.append(job)
	else:
	raise RuntimeError(
	f"Max job capacity of {repr(self)} reached. Can't assign new jobs.")

Unfortunately, this doesn't resolve all of my issues: now the allocation appears to work, but still something goes wrong. More specifically, when I try to run the network now I get the following error: ConnectionRefusedError: [Errno 111] Connection refused, which occurs in the following blob of code.

MARVELO/fission/core/network.py

Lines 577 to 582 in c118adc

	for job_id, command in commands.items():
	job = self.jobs[job_id]
	# if job.status == job.RUNNING:
	args = (job, command)
	threading.Thread(target=self.send_redirect,
	args=args, daemon=True).start()

When I look into the corresponding redirect function, I can see the error is with the creation of a TCP socket that has no counterpart: when I log into the pi and manually create a TCP socket on the correct port, I can manually make the code work by executing the commands in the function. However, this still doesn't make it work.

This makes me suspect that either the socket on the pi is not made correctly, or that the function is called needlessly. It should be noted that simulation works fine after removing the incorrect check.

Is it possible to either bypass the need for the redirect command or deal with it in some other way?

Pull request

I made a PR with the bugfixes in network.py, to make it work I had to make a slight change to the setup.py which got committed as well as the .gitignore file I used. If you wouldn't like those two, I can still remove those two.

[BasilLiekens]

After some further investigations I think I managed to find the actual problem, which would also remove the need for the previous fix. More specifically, upon building the network, the following blob gets executed:

MARVELO/fission/build.py

Lines 42 to 44 in c118adc

	# check DEFAULT_NODE
	if job.DEFAULT_NODE:
	NODES[job.DEFAULT_NODE].add_job(job)

This indeed ensures the job gets assigned to the correct node, but still causes some problems, namely:

1. The aforementioned issue where the number of jobs would exceed the maximum number of nodes, presumably because the node wasn't properly initialized yet.
2. If that issue were to be bypassed by temporarily sidestepping the check in add_job, the issue with the redirect command would occur.

However, when looking into where this redirect would be needed in the first place, I encountered the following blob of code:

MARVELO/fission/core/network.py

Lines 499 to 505 in c118adc

	if hasattr(job, "DEFAULT_NODE") and job.DEFAULT_NODE:
	try:
	_node = self.nodes[job.DEFAULT_NODE]
	if _node.full():
	raise AllocationError
	else:
	node = _node

This implies that during "regular" allocation, the DEFAULT_NODE is also checked before doing any allocation. Commenting out the first check during the building of the network manages to avoid the need for the redirect command, which seems to fix the problem.

It should be noted however that there is still an apparent difference between the handling of the two cases: until now I've always used the hostname of the devices. I.e.,

NODES = [<hostname1>.local, <hostname2>.local, ...]

And this happened for both the config.py file and the DEFAULT_NODE object. However, that way the nodes with a DEFAULT_NODE got resolved to the address of <hostname>.local (sic.), whereas the other nodes got assigned to the actual ip address. Using the actual ip's resolved the issue, but is a bit more cumbersome.

[BasilLiekens]

Perhaps there is one more thing I didn't think about so far. In what I would be using MARVELO for, the graph of communication between jobs would contain a cycle. As far as I can see this would result in the network never starting in the first place, since each job is still awaiting an input from its predecessor.

Is there a way around this, or should some kind of workaround be found for this kind of issue? I guess a potential resolution is to write some communication to disk, but this is suboptimal at best.

[BasilLiekens]

Brief

I've managed to complete my project using MARVELO; the code for it is available here. I think this issue can be closed, but there are a few recommendations that might improve the experience of future users.

Documentation

In the readme files of my implementation, I've added some remarks that were required for me to be able to use the framework. I'm wondering whether it might be worth adding those into the documentation such that other people don't face the same issues.

It is worth pointing out that some of the issues and corresponding solutions mentioned in there are more workarounds rather than true solutions. An example of this is copying all folders into the bin of the .venv, which is a very counterintuitive move.

Playbook

Something that is also quite important is the fact that the playbook requires some updates as mentioned before. I have a branch on my fork that implements the changes I mentioned before; using a virtual environment rather than trying to work with the system-wide python installation. There are a few problems with this however:

• /home/RPi/... has been consistently used for the paths, which effectively forces the user to work with RPi as username. Theoretically this should be replacable with ~ as all paths are used in a shell environment (which is even forced to be bash), hence the ~ variable should exist, but currently I don't have the possibility to validate this statement. Furthermore, this approach also assumes the user would want to work from a directory called installations.
• Another issue, that is also mentioned in the readme's, is the fact that the installations directory, created by the ansible playbook, can't be scp'ed into to resolve the issue with pickle. I've circumvented this issue by manually recreating the folder using ssh, but this is quite a tedious procedure, especially in larger networks. Perhaps it might be a good idea to also provide a means to copy over the relevant folders using an ansible playbook, but I'm not really familiar with that.

Packages

Perhaps not the most pressing matter, but initially the packages have also caused some issues for me. For example, the lxml package with version 4.4.2 can not be installed on python versions larger than 3.10. The currently most recent version 6.0.0 can be used with newer versions of python as well. However, to ensure everything keeps on working as other packages evolve and might change their API's, it might not be a bad idea to fix the python and package versions in the setup.py script to ensure the initial installation works as intended.

If you feel like any of this is worth doing/implementing and you feel like I could be of any help, please let me know.