Unhelpful Errors When Worker Python Version > Client Python Version

[e117649]

I don't know if we can support worker python > client python, but we should at least return a useful error message, and likely can't support worker python < client python. The workers would also become zombies.

For example with client c, tested with python 3.12 for worker, python 3.11 for client:
Basic functions with no other calls work

>>> def fun(x):
...     return 5
... 
>>> c.submit(fun,5).result()
5

Built in function call broken with an unhelpful message

>>> def fun(x):
...     return max(10,x)
... 
>>> c.submit(fun,5).result()
Traceback (most recent call last):
  File "<input>", line 1, in <module>
    c.submit(fun,5).result()
  File "/workspaces/scaler/stable311/lib/python3.11/site-packages/scaler/client/future.py
", line 154, in result
    return super().result()
           ^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/share/uv/python/cpython-3.11.14-linux-x86_64-gnu/lib/python3.
11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/share/uv/python/cpython-3.11.14-linux-x86_64-gnu/lib/python3.
11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/workspaces/scaler/stable/lib/python3.12/site-packages/scaler/worker/agent/proces
sor/processor.py", line 231, in __process_task
    result = function(*args)
    ^^^^^^^^^^^^^^^^^
  File "<input>", line 2, in fun
    return max(10,x)
      ^^^^^^^^^^^^^^^
SystemError: error return without exception set

Another unhelpful error using the standard library

>>> def fun(x):
...     import math
...     return math.floor(x)
... 
>>> c.submit(fun,5.5).result()
Traceback (most recent call last):
  File "<input>", line 1, in <module>
    c.submit(fun,5.5).result()
  File "/workspaces/scaler/stable311/lib/python3.11/site-packages/scaler/client/future.py
", line 154, in result
    return super().result()
           ^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/share/uv/python/cpython-3.11.14-linux-x86_64-gnu/lib/python3.
11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/home/vscode/.local/share/uv/python/cpython-3.11.14-linux-x86_64-gnu/lib/python3.
11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/workspaces/scaler/stable/lib/python3.12/site-packages/scaler/worker/agent/proces
sor/processor.py", line 231, in __process_task
    result = function(*args)
    ^^^^^^^^^^^^^^^^^
  File "<input>", line 3, in fun
    return math.floor(x)
      ^^^^^^^^^^^^^^^^^
TypeError: 'module' object does not support item assignment

[sharpener6]

this is not easy, fundamentally this is related to problem that env on client side doesn't match with worker side, but there is no very good solution, also that's why we introduced the capabilities, all env setups can be condensed to a single capabilities string, same concept as docker/ami image id represents the overall environment + packages