mitogen: Evolution of ModuleFinder and ModuleResponder

[moreati]

mitogen.master.ModuleFinder locates Python modules on the controller tthat can be sent to children in response to a GET_MODULE request. It also scans the found module M to identify "related modules" - a dependency tree of modules imported by M, directly or indirectly.

mitogen.master.ModuleResponder sends the found module and all related modules, proactively.

Current bugs and quirks

• False positives: sending related modules that are undesired, e.g. ModuleResponder sends ansible.cli package, and many others, probably unnecessarily #1124
• False negatives: missing related modules e.g. mitogen does not deploy class level imports #682

Ideas so far, not mutually exclusive

• Use ast to scan for import statements
  • Eliminates false negatives
  • Potentially increases false positives, depending on how we define desired criteria (e.g. import mitogen.master inside mitogen.main())
• Remove related functionality, rely on children sending GET_MODULE requests as needed
  • Eliminates a chunk of fiddly code
  • Reduces CPU and random disk IO on controller
  • Increases number of round trips
  • May increase Time To First Execution (TTFE, a term I just made up) on the child
  • May increase maintenance burden wrt new Ansible releases, manually maintaining a preload list of required Ansible modules
  • May make Mitogen less robust wrt to long tail of Python packages and Ansible collections
• Make related search/delivery configurable, e.g.
  • Something similar to # !mitogen: minify_safe markers
  • Environment variable(s)

Refs #1242 #1324 #1239

[moreati]

What caching could Mitogen (optionally) do? E.g.

• Controller - cache minified modules
• Parents - cache in-transit modules
• Targets - cache received modules

Considerations

• Cache invalidation
• Information leaks, e.g.
  • target A requests module top.secret.recipe.variant10, measures response time to tell whether a sibling has requested it or not
• Cache life time/eviction, e.g. process? 1 day? up to 10 MiB? Configurable?

[moreati]

Using bytecode may be quicker than ast, if the bytecode can be read from a cached .pyc, rather than parsed each time from .py

[moreati]

issue #19: second attempt at import scanner

This version is based on the modulefinder standard library module, pruned back just to handle modules we know have been loaded already, and to scan module-level imports only, rather than imports occurring in class and function scope (crappy heuristic, but assume they are lazy imports).

The ast and compiler modules were far too slow, whereas this version can bytecode compile and scan all the imports for django.db.models (58 modules) in around 200ms.. 3.4ms per dependency, it's probably not going to get much faster than that.
-- 43ccbf0, emphasis mine

[moreati]

Of note, imports inside try/except and with blocks are never imported lazily. Both structures commonly handle import-related errors: for example, when detecting available or compatible modules
-- https://www.hudsonrivertrading.com/hrtbeat/inside-hrts-python-fork/