Merge remote-tracking branch 'origin/stream-refactor'

* origin/stream-refactor:
  [stream-refactor] Py3.x test fixes
  [stream-refactor] mark py24 as allow-fail
  [stream-refactor] Debian Docker container image initctl
  [stream-refactor] replace cutpaste with Stream.accept() in mitogen.unix
  [stream-refactor] fix flake8 errors
  [stream-refactor] fix testlib assertion format string
  [stream-refactor] make mitogen-fuse work on Linux
  [stream-refactor] repair preamble_size.py again
  [stream-refactor] don't abort Connection until all buffers are empty
  Normalize docstring formatting
  [stream-refactor] fix LogHandler.uncork() race
  [stream-refactor] BufferedWriter must disconenct Stream, not Protocol
  [stream-refactor] statically link doas binary using musl
  [stream-refactor] stop writing to /tmp/foo in fd_check.py.
  [stream-refactor] yet another 2.4 issue in create_child_test
  [stream-refactor] fix Py2.4 failure by implementing missing Timer method
  [stream-refactor] allow up to 30 seconds to connect in unix_test
  [stream-refactor] mark setns module as requiring Python >2.4
  [stream-refactor] another 2.4 fix for create_child_test
  .travis.yml: Add reverse shell spawn for Travis too
  core: better Side attribute docstrings
  [stream-refactor] remove one more getuser() usage
  [stream-refactor] allow doas_test to succeed on CentOS
  Pin idna==2.7 when running on Python<2.7.
  [stream-refactor] Py2.4 compat fix for iter_split_test.
  [stream-refactor] add descriptive task names to _container_prep
  [stream-refactor] 3.x socket.send() requires bytes
  [stream-refactor] fix 2.4 syntax error.
  [stream-refactor] avoid os.wait3() for Py2.4.
  Allow specifying -vvv to debops_tests.
  [stream-refactor] send MITO002 earlier
  module_finder: pass raw file to compile()
  [stream-refactor] merge stdout+stderr when reporting EofError
  [stream-refactor] fix crash in detach() / during async/multiple_items_loop.yml
  [stream-refactor] fix crash in runner/forking_active.yml
  [stream-refactor] replace old detach_popen() reference
  ansible: fixturize creation of MuxProcess
  unix: ensure mitogen.context_id is reset when client disconnects
  [stream-refactor] make syntax 2.4 compatible
  [stream-refactor] make trusty our Travis dist.
  [stream-refactor] fix su_test failure (issue #363)
  [stream-refactor] more readable log string format
  [stream-refactor] dont doubly log last partial line
  [stream-refactor] import fd_check.py used by create_child_test
  [stream-refactor] port mitogen.buildah, added to master since work began
  [stream-refactor] fix unix.Listener construction
  [stream-refactor] fix crash when no stderr present.
  [stream-refactor] fix Process constructor invocation
  Add tests/ansible/.*.pid to gitignore (for ansible_mitogen/process.py)
  Add extra/ to gitignore
  import release-notes script.
  [stream-refactor] repaired rest of create_child_test.
  [stream-refactor] rename Process attrs, fix up more create_child_test
  [stream-refactor] import incomplete create_child_test
  issue #482: tests: check for zombie process after test.
  issue #363: add test.
  tests: clean up old-style SSH exception catch
  issue #271: add mitogen__permdenied user to Docker image.
  ssh: fix issue #271 regression due to refactor, add test.
  Refactor Stream, introduce quasi-asynchronous connect, much more
  core: teach iter_split() to break on callback returning False.
  issue #507: log fatal errors to syslog.
  testlib: have LogCapturer.raw() return unicode on 2.x.
  core/master: docstring, repr, and debug log message cleanups
  parent: remove unused Timer parameter.
  tests: jail_test fixes.
  parent: docstring improvements, cfmakeraw() regression.
  core: introduce Protocol, DelimitedProtocol and BufferedWriter.
  core: introduce mitogen.core.pipe()
  tests/bench: import ssh-roundtrip.py.
  tests: note location of related tests.
  tests: add real test for doas.
  tests: install OpenBSD doas port in Debian image.
  tests: add setns_test that works if password localhost sudo works.
  Import minimal jail_test.
  core: move message encoding to Message.pack(), add+refactor tests.
  master: expect forwarded logs to be in UTF-8.
  tests: add some UTF-8 to ssh_login_banner to encourage breakage.
  core: bootstrap FD management improvements
  core: pending timers should keep broker alive.
  core: more succinct iter_split().
  core: replace UTF8_CODEC with encodings.utf_8.encode() function.
  docs: remove bytearray from supported types list.
  core: docstring style cleanups, dead code.
  testlib: disable lsof warnings due to Docker crap
  parent: discard cancelled events in TimerList.get_timeout().
  core: split out iter_split() for use in parent.py.
  parent: various style cleanups, remove unused function.
  issue #170: add TimerList docstrings.
  core: eliminate some quadratric behaviour from IoLogger
  issue #170: update Changelog; closes #170.
  issue #170: add timers to internals.rst.
  issue #170: implement timers.

Author: dw

.ci/debops_common_tests.py

@@ -3,6 +3,7 @@
 from __future__ import print_function
 import os
 import shutil
+import sys
 
 import ci_lib
 
@@ -68,8 +69,8 @@
 
 
 with ci_lib.Fold('first_run'):
-    ci_lib.run('debops common')
+    ci_lib.run('debops common %s', ' '.join(sys.argv[1:]))
 
 
 with ci_lib.Fold('second_run'):
-    ci_lib.run('debops common')
+    ci_lib.run('debops common %s', ' '.join(sys.argv[1:]))

.gitignore

@@ -9,6 +9,8 @@ venvs/**
 MANIFEST
 build/
 dist/
+extra/
+tests/ansible/.*.pid
 docs/_build/
 htmlcov/
 *.egg-info

.travis.yml

@@ -1,4 +1,5 @@
 sudo: required
+dist: trusty
 
 notifications:
   email: false
@@ -20,13 +21,19 @@ install:
 - .ci/${MODE}_install.py
 
 script:
+- .ci/spawn_reverse_shell.py
 - .ci/${MODE}_tests.py
 
 
 # To avoid matrix explosion, just test against oldest->newest and
 # newest->oldest in various configuartions.
 
 matrix:
+  allow_failures:
+    # Python 2.4 tests are still unreliable
+    - language: c
+      env: MODE=mitogen_py24 DISTRO=centos5
+
   include:
     # Mitogen tests.
     # 2.4 -> 2.4

ansible_mitogen/affinity.py

@@ -177,9 +177,9 @@ class FixedPolicy(Policy):
     cores, before reusing the second hyperthread of an existing core.
 
     A hook is installed that causes :meth:`reset` to run in the child of any
-    process created with :func:`mitogen.parent.detach_popen`, ensuring
-    CPU-intensive children like SSH are not forced to share the same core as
-    the (otherwise potentially very busy) parent.
+    process created with :func:`mitogen.parent.popen`, ensuring CPU-intensive
+    children like SSH are not forced to share the same core as the (otherwise
+    potentially very busy) parent.
     """
     def __init__(self, cpu_count=None):
         #: For tests.

ansible_mitogen/module_finder.py

@@ -57,7 +57,7 @@ def get_code(module):
     """
     Compile and return a Module's code object.
     """
-    fp = open(module.path)
+    fp = open(module.path, 'rb')
     try:
         return compile(fp.read(), str(module.name), 'exec')
     finally:

ansible_mitogen/process.py

@@ -79,8 +79,15 @@ def clean_shutdown(sock):
     MuxProcess, debug logs may appear on the user's terminal *after* the prompt
     has been printed.
     """
-    sock.shutdown(socket.SHUT_WR)
+    try:
+        sock.shutdown(socket.SHUT_WR)
+    except socket.error:
+        # Already closed. This is possible when tests are running.
+        LOG.debug('clean_shutdown: ignoring duplicate call')
+        return
+
     sock.recv(1)
+    sock.close()
 
 
 def getenv_int(key, default=0):
@@ -154,8 +161,15 @@ class of problems this worker is intended to avoid.
     #: forked WorkerProcesses to contact the MuxProcess
     unix_listener_path = None
 
-    #: Singleton.
-    _instance = None
+    @classmethod
+    def _reset(cls):
+        """
+        Used to clean up in unit tests.
+        """
+        assert cls.worker_sock is not None
+        cls.worker_sock.close()
+        cls.worker_sock = None
+        os.waitpid(cls.worker_pid, 0)
 
     @classmethod
     def start(cls, _init_logging=True):
@@ -178,7 +192,7 @@ def start(cls, _init_logging=True):
         mitogen.utils.setup_gil()
         cls.unix_listener_path = mitogen.unix.make_socket_path()
         cls.worker_sock, cls.child_sock = socket.socketpair()
-        atexit.register(lambda: clean_shutdown(cls.worker_sock))
+        atexit.register(clean_shutdown, cls.worker_sock)
         mitogen.core.set_cloexec(cls.worker_sock.fileno())
         mitogen.core.set_cloexec(cls.child_sock.fileno())
 
@@ -189,8 +203,8 @@ def start(cls, _init_logging=True):
             ansible_mitogen.logging.setup()
 
         cls.original_env = dict(os.environ)
-        cls.child_pid = os.fork()
-        if cls.child_pid:
+        cls.worker_pid = os.fork()
+        if cls.worker_pid:
             save_pid('controller')
             ansible_mitogen.logging.set_process_name('top')
             ansible_mitogen.affinity.policy.assign_controller()
@@ -308,7 +322,7 @@ def _setup_master(self):
         self._setup_responder(self.router.responder)
         mitogen.core.listen(self.broker, 'shutdown', self.on_broker_shutdown)
         mitogen.core.listen(self.broker, 'exit', self.on_broker_exit)
-        self.listener = mitogen.unix.Listener(
+        self.listener = mitogen.unix.Listener.build_stream(
             router=self.router,
             path=self.unix_listener_path,
             backlog=C.DEFAULT_FORKS,

docs/api.rst

@@ -383,6 +383,9 @@ Connection Methods
     the root PID of a running Docker, LXC, LXD, or systemd-nspawn
     container.
 
+    The setns method depends on the built-in :mod:`ctypes` module, and thus
+    does not support Python 2.4.
+
     A program is required only to find the root PID, after which management
     of the child Python interpreter is handled directly.
 

docs/changelog.rst

@@ -102,6 +102,14 @@ Fixes
   potential influx of 2.8-related bug reports.
 
 
+Core Library
+~~~~~~~~~~~~
+
+* `#170 <https://github.com/dw/mitogen/issues/170>`_: to better support child
+  process management and a future asynchronous connect implementation, a
+  :class:`mitogen.parent.TimerList` API is available.
+
+
 Thanks!
 ~~~~~~~
 

docs/getting_started.rst

@@ -341,15 +341,13 @@ The following built-in types may be used as parameters or return values in
 remote procedure calls:
 
 * :class:`bool`
-* :class:`bytearray`
-* :func:`bytes`
+* :func:`bytes` (:class:`str` on Python 2.x)
 * :class:`dict`
 * :class:`int`
 * :func:`list`
 * :class:`long`
-* :class:`str`
 * :func:`tuple`
-* :func:`unicode`
+* :func:`unicode` (:class:`str` on Python 3.x)
 
 User-defined types may not be used, except for:
 

docs/internals.rst

@@ -15,46 +15,49 @@ Constants
 .. autodata:: CHUNK_SIZE
 
 
-Poller Classes
-==============
+Pollers
+=======
 
 .. currentmodule:: mitogen.core
 .. autoclass:: Poller
-   :members:
+    :members:
+
+.. currentmodule:: mitogen.parent
+.. autoclass:: KqueuePoller
 
 .. currentmodule:: mitogen.parent
 .. autoclass:: EpollPoller
 
 .. currentmodule:: mitogen.parent
-.. autoclass:: KqueuePoller
+.. autoclass:: PollPoller
 
 
-Latch Class
-===========
+Latch
+=====
 
 .. currentmodule:: mitogen.core
 .. autoclass:: Latch
    :members:
 
 
-PidfulStreamHandler Class
-=========================
+PidfulStreamHandler
+===================
 
 .. currentmodule:: mitogen.core
 .. autoclass:: PidfulStreamHandler
    :members:
 
 
-Side Class
-==========
+Side
+====
 
 .. currentmodule:: mitogen.core
 .. autoclass:: Side
    :members:
 
 
-Stream Classes
-==============
+Stream
+======
 
 .. currentmodule:: mitogen.core
 .. autoclass:: BasicStream
@@ -79,83 +82,84 @@ Stream Classes
 .. autoclass:: Stream
    :members:
 
-
-Other Stream Subclasses
-=======================
-
 .. currentmodule:: mitogen.core
-
 .. autoclass:: IoLogger
    :members:
 
+.. currentmodule:: mitogen.core
 .. autoclass:: Waker
    :members:
 
 
-Poller Class
-============
-
-.. currentmodule:: mitogen.core
-.. autoclass:: Poller
-    :members:
-
-.. currentmodule:: mitogen.parent
-.. autoclass:: KqueuePoller
-
-.. currentmodule:: mitogen.parent
-.. autoclass:: EpollPoller
-
-
-Importer Class
-==============
+Importer
+========
 
 .. currentmodule:: mitogen.core
 .. autoclass:: Importer
    :members:
 
 
-Responder Class
+ModuleResponder
 ===============
 
 .. currentmodule:: mitogen.master
 .. autoclass:: ModuleResponder
    :members:
 
 
-RouteMonitor Class
-==================
+RouteMonitor
+============
 
 .. currentmodule:: mitogen.parent
 .. autoclass:: RouteMonitor
    :members:
 
 
-Forwarder Class
-===============
+TimerList
+=========
+
+.. currentmodule:: mitogen.parent
+.. autoclass:: TimerList
+   :members:
+
+
+Timer
+=====
+
+.. currentmodule:: mitogen.parent
+.. autoclass:: Timer
+   :members:
+
+
+Forwarder
+=========
 
 .. currentmodule:: mitogen.parent
 .. autoclass:: ModuleForwarder
    :members:
 
 
-ExternalContext Class
-=====================
+ExternalContext
+===============
 
 .. currentmodule:: mitogen.core
 .. autoclass:: ExternalContext
     :members:
 
 
-mitogen.master
-==============
+Process
+=======
 
 .. currentmodule:: mitogen.parent
-.. autoclass:: ProcessMonitor
+.. autoclass:: Process
     :members:
 
 
-Blocking I/O Functions
-======================
+Helpers
+=======
+
+Blocking I/O
+------------
 
 These functions exist to support the blocking phase of setting up a new
 context. They will eventually be replaced with asynchronous equivalents.
@@ -167,17 +171,17 @@ context. They will eventually be replaced with asynchronous equivalents.
 .. autofunction:: write_all
 
 
-Subprocess Creation Functions
-=============================
+Subprocess Functions
+------------
 
 .. currentmodule:: mitogen.parent
 .. autofunction:: create_child
 .. autofunction:: hybrid_tty_create_child
 .. autofunction:: tty_create_child
 
 
-Helper Functions
-================
+Helpers
+-------
 
 .. currentmodule:: mitogen.core
 .. autofunction:: to_text