The Mitogen master process was unable to serve u'ansible_mitogen.target'. on ansible core 2.13

[uumas]

Running the latest master of mitogen, I'm getting the following error on all hosts I've tried against (Debian 10-11, CentOS 7-8, Ubuntu 20-22):

[WARNING]: Unhandled error in Python interpreter discovery for host: exceptions.ImportError: The Mitogen master process was
unable to serve u'ansible_mitogen.target'. It may be a native Python extension, or it may be missing entirely. Check the importer debug logs on the master
for more information.   File "<stdin>", line 3698, in _dispatch_one   File "<stdin>", line 3685, in _parse_request   File "<stdin>", line 682, in
import_module   File "<stdin>", line 1521, in load_module

Ansible version info:

ansible [core 2.13.2]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/uumas/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.10/site-packages/ansible
  ansible collection location = /home/uumas/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.10.5 (main, Jun  6 2022, 18:49:26) [GCC 12.1.0]
  jinja version = 3.1.2
  libyaml = True

Traceback:

Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/ansible/executor/task_executor.py", line 157, in run
    res = self._execute()
  File "/usr/lib/python3.10/site-packages/ansible/executor/task_executor.py", line 635, in _execute
    result = self._handler.run(task_vars=vars_copy)
  File "/home/uumas/.local/lib/python3.10/site-packages/ansible_mitogen/mixins.py", line 146, in run
    return super(ActionModuleMixin, self).run(tmp, task_vars)
  File "/usr/lib/python3.10/site-packages/ansible/plugins/action/normal.py", line 47, in run
    result = merge_hash(result, self._execute_module(task_vars=task_vars, wrap_async=wrap_async))
  File "/home/uumas/.local/lib/python3.10/site-packages/ansible_mitogen/mixins.py", line 376, in _execute_module
    self._set_temp_file_args(module_args, wrap_async)
  File "/home/uumas/.local/lib/python3.10/site-packages/ansible_mitogen/mixins.py", line 355, in _set_temp_file_args
    self._connection.get_good_temp_dir()
  File "/home/uumas/.local/lib/python3.10/site-packages/ansible_mitogen/connection.py", line 832, in get_good_temp_dir
    self._connect()
  File "/home/uumas/.local/lib/python3.10/site-packages/ansible_mitogen/connection.py", line 854, in _connect
    self._connect_stack(stack)
  File "/home/uumas/.local/lib/python3.10/site-packages/ansible_mitogen/connection.py", line 801, in _connect_stack
    dct = mitogen.service.call(
  File "/home/uumas/.local/lib/python3.10/site-packages/mitogen/service.py", line 126, in call
    return call_context.call_service(service_name, method_name, **kwargs)
  File "/home/uumas/.local/lib/python3.10/site-packages/mitogen/core.py", line 2314, in call_service
    return recv.get().unpickle()
  File "/home/uumas/.local/lib/python3.10/site-packages/mitogen/core.py", line 977, in unpickle
    raise obj
mitogen.core.CallError: builtins.ModuleNotFoundError: The Mitogen master process was unable to serve 'ansible_mitogen.target'. It may be a native Python extension, or it may be missing entirely. Check the importer debug logs on the master for more information.
  File "<stdin>", line 3698, in _dispatch_one
  File "<stdin>", line 3685, in _parse_request
  File "<stdin>", line 682, in import_module
  File "<stdin>", line 1521, in load_module

Some log (hostname replaced with ...):

[mux  22293] 12:58:33.935628 D mitogen.responder: sending ansible (0.53 KiB) to ssh....
[mux  22293] 12:58:33.937290 D mitogen.responder: ansible.module_utils is a package at /usr/lib/python3.10/site-packages/ansible/module_utils/__init__.py with submodules ['_text', 'ansible_release', 'api', 'basic', 'common', 'compat', 'connection', 'csharp', 'distro', 'errors', 'facts', 'json_utils', 'parsing', 'powershell', 'pycompat24', 'service', 'six', 'splitter', 'urls', 'yumdnf']
[mux  22293] 12:58:33.937960 D mitogen.responder: sending ansible.module_utils (0.48 KiB) to ssh....
[mux  22293] 12:58:33.999957 D mitogen: PkgutilMethod(): <_frozen_importlib_external.SourceFileLoader object at 0x7f67bd505db0>.get_file_name('ansible.module_utils.distro') failed: ImportError('loader for distro cannot handle ansible.module_utils.distro')
[mux  22293] 12:58:34.000093 D mitogen: SysModulesMethod(): <module 'distro' from '/usr/lib/python3.10/site-packages/distro/__init__.py'>.__name__ is 'distro', aborting
[mux  22293] 12:58:34.000192 D mitogen: ParentEnumerationMethod(): 'ansible.module_utils.distro' is PKG_DIRECTORY: '/usr/lib/python3.10/site-packages/ansible/module_utils/distro/__init__.py'
[mux  22293] 12:58:34.016629 D mitogen: While importing 'ansible.module_utils.basic'
Traceback (most recent call last):
  File "/home/uumas/.local/lib/python3.10/site-packages/mitogen/master.py", line 1134, in _send_module_and_related
    tup = self._build_tuple(fullname)
  File "/home/uumas/.local/lib/python3.10/site-packages/mitogen/master.py", line 1087, in _build_tuple
    for name in self._finder.find_related(fullname)
  File "/home/uumas/.local/lib/python3.10/site-packages/mitogen/master.py", line 959, in find_related
    names = self.find_related_imports(name)
  File "/home/uumas/.local/lib/python3.10/site-packages/mitogen/master.py", line 933, in find_related_imports
    set(
  File "/home/uumas/.local/lib/python3.10/site-packages/mitogen/master.py", line 937, in <genexpr>
    and not is_stdlib_name(name)
  File "/home/uumas/.local/lib/python3.10/site-packages/mitogen/master.py", line 140, in is_stdlib_name
    modpath = os.path.abspath(getattr(module, '__file__', ''))
  File "/usr/lib/python3.10/posixpath.py", line 378, in abspath
    path = os.fspath(path)
TypeError: expected str, bytes or os.PathLike object, not NoneType
[mux  22293] 12:58:34.017687 D mitogen.responder: sending ansible.module_utils.json_utils (2.68 KiB) to ssh....
[mux  22293] 12:58:34.018253 D mitogen.responder: sending ansible.release (0.99 KiB) to ssh....
[mux  22293] 12:58:34.018647 D mitogen.responder: ansible_mitogen is a package at /home/uumas/.local/lib/python3.10/site-packages/ansible_mitogen/__init__.py with submodules ['affinity', 'compat', 'connection', 'loaders', 'logging', 'mixins', 'module_finder', 'parsing', 'planner', 'plugins', 'process', 'runner', 'services', 'strategy', 'target', 'transport_config', 'utils']
[mux  22293] 12:58:34.018774 D mitogen.responder: sending ansible_mitogen (0.44 KiB) to ssh....
[mux  22293] 12:58:34.076591 D mitogen: While importing 'ansible_mitogen.runner'
Traceback (most recent call last):
  File "/home/uumas/.local/lib/python3.10/site-packages/mitogen/master.py", line 1134, in _send_module_and_related
    tup = self._build_tuple(fullname)
  File "/home/uumas/.local/lib/python3.10/site-packages/mitogen/master.py", line 1087, in _build_tuple
    for name in self._finder.find_related(fullname)
  File "/home/uumas/.local/lib/python3.10/site-packages/mitogen/master.py", line 959, in find_related
    names = self.find_related_imports(name)
  File "/home/uumas/.local/lib/python3.10/site-packages/mitogen/master.py", line 933, in find_related_imports
    set(
  File "/home/uumas/.local/lib/python3.10/site-packages/mitogen/master.py", line 937, in <genexpr>
    and not is_stdlib_name(name)
  File "/home/uumas/.local/lib/python3.10/site-packages/mitogen/master.py", line 140, in is_stdlib_name
    modpath = os.path.abspath(getattr(module, '__file__', ''))
  File "/usr/lib/python3.10/posixpath.py", line 378, in abspath
    path = os.fspath(path)
TypeError: expected str, bytes or os.PathLike object, not NoneType
[mux  22293] 12:58:34.094960 D mitogen: While importing 'ansible_mitogen.target'
Traceback (most recent call last):
  File "/home/uumas/.local/lib/python3.10/site-packages/mitogen/master.py", line 1134, in _send_module_and_related
    tup = self._build_tuple(fullname)
  File "/home/uumas/.local/lib/python3.10/site-packages/mitogen/master.py", line 1087, in _build_tuple
    for name in self._finder.find_related(fullname)
  File "/home/uumas/.local/lib/python3.10/site-packages/mitogen/master.py", line 959, in find_related
    names = self.find_related_imports(name)
  File "/home/uumas/.local/lib/python3.10/site-packages/mitogen/master.py", line 933, in find_related_imports
    set(
  File "/home/uumas/.local/lib/python3.10/site-packages/mitogen/master.py", line 937, in <genexpr>
    and not is_stdlib_name(name)
  File "/home/uumas/.local/lib/python3.10/site-packages/mitogen/master.py", line 140, in is_stdlib_name
    modpath = os.path.abspath(getattr(module, '__file__', ''))
  File "/usr/lib/python3.10/posixpath.py", line 378, in abspath
    path = os.fspath(path)
TypeError: expected str, bytes or os.PathLike object, not NoneType
[mux  22293] 12:58:34.158828 D mitogen.responder: sending mitogen.parent (18.87 KiB) to ssh....
[mux  22293] 12:58:34.159104 D mitogen.responder: sending mitogen.fork (2.63 KiB) to ssh....
[mux  22293] 12:58:34.191350 D mitogen.responder: sending mitogen.select (1.62 KiB) to ssh....
[mux  22293] 12:58:34.191604 D mitogen.responder: sending mitogen.service (8.90 KiB) to ssh....

[moreati]

Could you provide a minimal playbook that triggers this?

[uumas]

well, not exactly a playbook but ansible localhost -m ping:

uumas@lemon ~> ansible localhost -m ping
[WARNING]: No inventory was parsed, only implicit localhost is available
An exception occurred during task execution. To see the full traceback, use -vvv. The error was:   File "<stdin>", line 1521, in load_module
localhost | FAILED! => {
    "msg": "Unexpected failure during module execution.",
    "stdout": ""
}

[moreati]

I'm afraid I'm unsable to reproduce

(v) ➜  mitogen git:(issue849-module_fqn) ✗ ANSIBLE_STRATEGY=mitogen_linear ansible localhost -m ping
[WARNING]: No inventory was parsed, only implicit localhost is available
localhost | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
(v) ➜  mitogen git:(issue849-module_fqn) ✗ ansible --version
ansible [core 2.13.1]
  config file = /Users/alex/src/mitogen/ansible.cfg
  configured module search path = ['/Users/alex/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /opt/homebrew/Cellar/ansible/6.1.0/libexec/lib/python3.10/site-packages/ansible
  ansible collection location = /Users/alex/.ansible/collections:/usr/share/ansible/collections
  executable location = /opt/homebrew/bin/ansible
  python version = 3.10.5 (main, Jun 23 2022, 17:14:57) [Clang 13.1.6 (clang-1316.0.21.2.5)]
  jinja version = 3.1.2
  libyaml = True

[zxyz]

for me the ping example works as well, but I receive the traceback while facts gathering

TASK [Gathering Facts] ***************************************************************************************************************************
The full traceback is:
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/ansible/executor/task_executor.py", line 157, in run
    res = self._execute()
  File "/usr/lib/python3.10/site-packages/ansible/executor/task_executor.py", line 635, in _execute
    result = self._handler.run(task_vars=vars_copy)
  File "/home/mitogen/ansible_mitogen/mixins.py", line 146, in run
    return super(ActionModuleMixin, self).run(tmp, task_vars)
  File "/usr/lib/python3.10/site-packages/ansible/plugins/action/gather_facts.py", line 100, in run
    res = self._execute_module(module_name=fact_module, module_args=mod_args, task_vars=task_vars, wrap_async=False)
  File "/home/mitogen/ansible_mitogen/mixins.py", line 376, in _execute_module
    self._set_temp_file_args(module_args, wrap_async)
  File "/home/mitogen/ansible_mitogen/mixins.py", line 355, in _set_temp_file_args
    self._connection.get_good_temp_dir()
  File "/home/mitogen/ansible_mitogen/connection.py", line 832, in get_good_temp_dir
    self._connect()
  File "/home/mitogen/ansible_mitogen/connection.py", line 854, in _connect
    self._connect_stack(stack)
  File "/home/mitogen/ansible_mitogen/connection.py", line 801, in _connect_stack
    dct = mitogen.service.call(
  File "/home/mitogen/mitogen/service.py", line 126, in call
    return call_context.call_service(service_name, method_name, **kwargs)
  File "/home/mitogen/mitogen/core.py", line 2314, in call_service
    return recv.get().unpickle()
  File "/home/mitogen/mitogen/core.py", line 977, in unpickle
    raise obj
mitogen.core.CallError: builtins.ModuleNotFoundError: The Mitogen master process was unable to serve 'ansible_mitogen.target'. It may be a native Python extension, or it may be missing entirely. Check the importer debug logs on the master for more information.
  File "<stdin>", line 3698, in _dispatch_one
  File "<stdin>", line 3685, in _parse_request
  File "<stdin>", line 682, in import_module
  File "<stdin>", line 1521, in load_module

fatal: [monitor.systemlos.net]: FAILED! => {
    "msg": "Unexpected failure during module execution.",
    "stdout": ""
}

mitogen: current main
ansible [core 2.13.2]

thanks for your work @moreati

[roman-vynar]

Ansible 2.13.2 when doing "with_fileglob" on template task.
Using the master version of mitogen as of today.

- name: Copy template files
  template: src={{item}} dest=/usr/local/folder/
  with_fileglob:
    - "../templates/*.conf"
  register: var1

Exception ignored in: <function Connection.__del__ at 0x10a766310>
Traceback (most recent call last):
  File "/Users/weber/git/mitogen/ansible_mitogen/connection.py", line 531, in __del__
    self.close()
  File "/Users/weber/git/mitogen/ansible_mitogen/connection.py", line 884, in close
    self._put_connection()
  File "/Users/weber/git/mitogen/ansible_mitogen/connection.py", line 865, in _put_connection
    self.chain.reset()
  File "/Users/weber/git/mitogen/mitogen/parent.py", line 1852, in reset
    self.call_no_reply(mitogen.core.Dispatcher.forget_chain, saved)
  File "/Users/weber/git/mitogen/mitogen/parent.py", line 1907, in call_no_reply
    self.context.send(self.make_msg(fn, *args, **kwargs))
  File "/Users/weber/git/mitogen/mitogen/core.py", line 2310, in send
    self.router.route(msg)
  File "/Users/weber/git/mitogen/mitogen/core.py", line 3383, in route
    self.broker.defer(self._async_route, msg)
  File "/Users/weber/git/mitogen/mitogen/core.py", line 2864, in defer
    raise Error(self.broker_shutdown_msg)
mitogen.core.Error: An attempt was made to enqueue a message with a Broker that has already exitted. It is likely your program called Broker.shutdown() too early.

[qezz]

Have been facing this issue on a Nix environment with Python 3.11.8, Ansible 2.13.13 and Mitogen 0.3.4.
So I hijacked the is_stdlib_name() function a bit

diff --git a/mitogen/master.py b/mitogen/master.py
index 4fb535f0a..130d655af 100644
--- a/mitogen/master.py
+++ b/mitogen/master.py
@@ -137,7 +137,21 @@ def is_stdlib_name(modname):
         return False
 
     # six installs crap with no __file__
-    modpath = os.path.abspath(getattr(module, '__file__', ''))
+    _g = getattr(module, '__file__', '')
+    if _g is None:
+        LOG.error('is_stdlib_name: modname: %r', modname)
+        LOG.error('is_stdlib_name: module: %r', module)
+        _attrs = dir(module)
+        LOG.error('is_stdlib_name: dir(module): %r', _attrs)
+        LOG.error('is_stdlib_name: __file__: %r', __file__)
+        LOG.error('is_stdlib_name(): getattr(module, \'__file__\', \'\'): %r', _g)
+        for _a in _attrs:
+            _g = getattr(module, _a, '')
+            LOG.error('  attr: %r: %r', _a, _g)
+
+    modpath = os.path.abspath(_g)
     return is_stdlib_path(modpath)
 
 

ERROR! [mux  2268210] 22:35:45.446946 E mitogen: is_stdlib_name: modname: 'backports'
ERROR! [mux  2268210] 22:35:45.447049 E mitogen: is_stdlib_name: module: <module 'backports' (<_frozen_importlib_external.NamespaceLoader object at 0x7f5dd79bb290>)>
ERROR! [mux  2268210] 22:35:45.447200 E mitogen: is_stdlib_name: dir(module): ['__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', 'configparser']
ERROR! [mux  2268210] 22:35:45.447263 E mitogen: is_stdlib_name: __file__: '/nix/store/14yvxad8lv32kjb54hp1gsahyf89p6d6-python3.11-mitogen-0.3.4/lib/python3.11/site-packages/mitogen/master.py'
ERROR! [mux  2268210] 22:35:45.447281 E mitogen: is_stdlib_name(): getattr(module, '__file__', ''): None
ERROR! [mux  2268210] 22:35:45.447316 E mitogen:   attr: '__doc__': None
ERROR! [mux  2268210] 22:35:45.447333 E mitogen:   attr: '__file__': None
ERROR! [mux  2268210] 22:35:45.447349 E mitogen:   attr: '__loader__': <_frozen_importlib_external.NamespaceLoader object at 0x7f5dd79bb290>
ERROR! [mux  2268210] 22:35:45.447367 E mitogen:   attr: '__name__': 'backports'
ERROR! [mux  2268210] 22:35:45.447383 E mitogen:   attr: '__package__': 'backports'
ERROR! [mux  2268210] 22:35:45.447397 E mitogen:   attr: '__path__': _NamespacePath(['/nix/store/fyskdqxrb77qwzfldwmxgs99xwf6lp2q-python3.11-configparser-6.0.0/lib/python3.11/site-packages/backports'])
ERROR! [mux  2268210] 22:35:45.447417 E mitogen:   attr: '__spec__': ModuleSpec(name='backports', loader=<_frozen_importlib_external.NamespaceLoader object at 0x7f5dd79bb290>, submodule_search_locations=_NamespacePath(['/nix/store/fyskdqxrb77qwzfldwmxgs99xwf6lp2q-python3.11-configparser-6.0.0/lib/python3.11/site-packages/backports']))
ERROR! [mux  2268210] 22:35:45.447441 E mitogen:   attr: 'configparser': <module 'backports.configparser' from '/nix/store/fyskdqxrb77qwzfldwmxgs99xwf6lp2q-python3.11-configparser-6.0.0/lib/python3.11/site-packages/backports/configparser/__init__.py'>

As you can see, the backports module has its __file__ set to None. Providing None to os.path.abspath() causes the exception,

TypeError: expected str, bytes or os.PathLike object, not NoneType

Here's the fix that works for us: #1042
Which is essentially

diff --git a/mitogen/master.py b/mitogen/master.py
index 4fb535f0a..73c8e4607 100644
--- a/mitogen/master.py
+++ b/mitogen/master.py
@@ -137,7 +137,11 @@ def is_stdlib_name(modname):
         return False
 
     # six installs crap with no __file__
-    modpath = os.path.abspath(getattr(module, '__file__', ''))
+    path = getattr(module, '__file__', '')
+    if path is None:
+        path = ''
+
+    modpath = os.path.abspath(path)
     return is_stdlib_path(modpath)