ansible_mitogen: Interpreter discovery improvements

[moreati]

A grab bag of shortcomings, ideas, and a place to disucss them

Finding python to find python

Ansible's native interpreter discovery makes one call to sh, with a list of candidate interepreters from INTERPRETER_PYTHON_FALLBACK. Mitogen turns this into several attempts to call a list of (Mitogen) candidate interpreters. So an O(1) operation becomes O(N).

• Can the list be shortened and/or the order optimised?
• Can the O(N) be eliminated?
  • Make an exception to Mitogen's design, call sh with Ansible's snippet instead
  • Devise a shell construction that calls one of the Python interpreters
• Should the Mitogen list be derived from/enhanced with Ansible's INTERPRETER_PYTHON_FALLBACK list?

mitogen/ansible_mitogen/mixins.py

Lines 461 to 476 in 0953a93

	# TODO: HACK: if finding python interpreter then we need to keep
	# calling exec_command until we run into the right python we'll use
	# chicken-and-egg issue, mitogen needs a python to run low_level_execute_command
	# which is required by Ansible's discover_interpreter function
	if self._finding_python_interpreter:
	possible_pythons = [
	'/usr/bin/python',
	'python3',
	'python3.7',
	'python3.6',
	'python3.5',
	'python2.7',
	'python2.6',
	'/usr/libexec/platform-python',
	'/usr/bin/python3',
	'python'

/usr/bin/python fallback

Depending on the setting INTERPRETER_PYTHON Ansible may fallback to /usr/bin/python or (I think) throw an error. I'm not convinced Mitogen always respects that setting.

mitogen/ansible_mitogen/transport_config.py

Lines 146 to 149 in 0953a93

	s = run_interpreter_discovery_if_necessary(s, task_vars, action, rediscover_python)
	# if unable to determine python_path, fallback to '/usr/bin/python'
	if not s:
	s = '/usr/bin/python'

related

• Mitogen attempts Ansible connections early (e.g. during meta: reset_connection) #1132

[moreati]

auto fallback

Mitogen defaults to 'auto' discovery mode under some circumstances. This may override Ansible's default, which I don't think is desirable.

mitogen/ansible_mitogen/transport_config.py

Lines 142 to 144 in fca7578

	if not s:
	# if python_path doesn't exist, default to `auto` and attempt to discover it
	s = 'auto'

Changing this (e.g. to raise an Exception) currently results in atleast one test failure.

--- a/ansible_mitogen/transport_config.py
+++ b/ansible_mitogen/transport_config.py
@@ -140,8 +140,7 @@ def parse_python_path(s, task_vars, action, rediscover_python):
     discovery value in `facts_from_task_vars` like how Ansible handles this.
     """
     if not s:
-        # if python_path doesn't exist, default to `auto` and attempt to discover it
-        s = 'auto'
+        raise ValueError("Expected Python path or discovery mode, got: %r", s)
 
     s = run_interpreter_discovery_if_necessary(s, task_vars, action, rediscover_python)
     if not s:

PLAY [tc-python-path-hostvar via tc-python-path-unset] *************************
META: ran handlers

TASK [include_tasks _raw_params=../_mitogen_only.yml] **************************
task path: /home/runner/work/mitogen/mitogen/tests/ansible/integration/transport_config/python_path.yml:27
Friday 24 January 2025  13:08:09 +0000 (0:00:00.062)       0:00:05.074 ******** 
[task 4605] 13:08:09.542122 D ansible_mitogen.affinity: CPU mask for WorkerProcess: 0x000004
included: /home/runner/work/mitogen/mitogen/tests/ansible/integration/_mitogen_only.yml for tc-python-path-hostvar
META: 
META: 

TASK [mitogen_get_stack ] ******************************************************
task path: /home/runner/work/mitogen/mitogen/tests/ansible/integration/transport_config/python_path.yml:31
Friday 24 January 2025  13:08:09 +0000 (0:00:00.028)       0:00:05.103 ******** 
[task 4607] 13:08:09.571139 D ansible_mitogen.affinity: CPU mask for WorkerProcess: 0x000008
[task 4607] 13:08:09.622428 D ansible_mitogen.mixins: _remove_tmp_path(None)
The full traceback is:
Traceback (most recent call last):
  File "/home/runner/work/mitogen/mitogen/.tox/py27-mode_ansible-ansible2.10/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 158, in run
    res = self._execute()
  File "/home/runner/work/mitogen/mitogen/.tox/py27-mode_ansible-ansible2.10/lib/python2.7/site-packages/ansible/executor/task_executor.py", line 663, in _execute
    result = self._handler.run(task_vars=variables)
  File "/home/runner/work/mitogen/mitogen/ansible_mitogen/mixins.py", line 121, in run
    return super(ActionModuleMixin, self).run(tmp, task_vars)
  File "/home/runner/work/mitogen/mitogen/ansible_mitogen/plugins/action/mitogen_get_stack.py", line 51, in run
    _, stack = self._connection._build_stack()
  File "/home/runner/work/mitogen/mitogen/ansible_mitogen/connection.py", line 785, in _build_stack
    stack = self._stack_from_spec(spec)
  File "/home/runner/work/mitogen/mitogen/ansible_mitogen/connection.py", line 762, in _stack_from_spec
    seen_names=seen_names + (spec.inventory_name(),),
  File "/home/runner/work/mitogen/mitogen/ansible_mitogen/connection.py", line 765, in _stack_from_spec
    stack += (CONNECTION_METHOD[spec.transport()](spec),)
  File "/home/runner/work/mitogen/mitogen/ansible_mitogen/connection.py", line 144, in _connect_ssh
    'python_path': spec.python_path(),
  File "/home/runner/work/mitogen/mitogen/ansible_mitogen/transport_config.py", line 723, in python_path
    rediscover_python=rediscover_python)
  File "/home/runner/work/mitogen/mitogen/ansible_mitogen/transport_config.py", line 143, in parse_python_path
    raise ValueError("Expected Python path or discovery mode, got: %r", s)
ValueError: (u'Expected Python path or discovery mode, got: %r', None)
fatal: [tc-python-path-hostvar]: FAILED! => 
  msg: Unexpected failure during module execution.
  stdout: ''

-- https://github.com/mitogen-hq/mitogen/actions/runs/12950225330/job/36122554610?pr=1226

refs

• How should ansible_mitogen.transport_config.MitogenViaSpec be updated for Ansible plugin options? #1135
• interpreter discovery trials #1226

[moreati]

Devise a shell construction that calls one of the Python interpreters

#!/bin/sh
p=$(for p in python4 python3 python2; do command -v "${p}" 2>/dev/null && break; done;)
if [ "${p}" ]; then
    exec "${p}" "$@"
else
    echo "Not found" 1>&2
    exit 1
fi