Read all preamble without PREAMBLE_COMPRESSED_LEN in first stage?

[moreati]

In mitogen.parent.Connection._first_stage() it may be advantagous to read preamble bytes from stdin, until EOF. Rather than trying to read exactly PREAMBLE_COMPRESSED_LEN bytes.

Using f.read() rather than f.read(PREAMBLE_COMPRESSED_LEN) may

• Shorten the boot command, by reducing the size of the first stage
• Simplify construction of the first stage by eliminating a substitution
• Improve reliability of the first stage by adding an implicit read() loop

Achieving this would need

• Closing the write side of pipe that feeds the child's stdin, once all bytes of the preamble have been written, so that read() will return an EOF and return all the read bytes
• Confirming that calling close() on the write fd of a pipe does not discard any bytes in flight, while the read fd is still open. This should be both as specified behaviour, and actual behaviour on common platforms.
• Confirming that said behaviour is consistent when the transport is over a network, and or across privilege elevation (e.g. ssh -> sudo -> setns)

Background

Mitogen bootstraps a child process by running python -c <boot_command <decompress(b64decode(first_stage))>>. The first stage in turn reads the preamble from stdin of the child process. Currently the first stage code performs a single io.BufferedReader.read(PREAMBLE_COMPRESSED_LEN)¹ call to do this, i.e.

mitogen/mitogen/parent.py

Lines 1440 to 1442 in 058787f

	fp=os.fdopen(0,'rb')
	C=zlib.decompress(fp.read(PREAMBLE_COMPRESSED_LEN))
	fp.close()

BufferedReader.read(n) is only specificed to return upto n bytes. This issue is a speculative offshoot of PR #1299, which attempts add a retry loop to the firset stage.

Footnotes

1. On Python versions >= 2.6 that have the io hierarchy. ↩

[moreati]

This document is out of date in several respects, but the following indicates this approach won't be viable. If it's correct then the parent can't close the write side of the pipe (to indicate EOF for fp.read() in the first stage), because Mitogen still uses the pipe later.

mitogen/docs/howitworks.rst

Lines 58 to 62 in 058787f

	To allow recovery of ``stdin`` for reuse by the bootstrapped process for
	parent<->child communication, it is necessary for the first stage to avoid
	closing ``stdin`` or reading from it until EOF. Therefore, the master
	sends the :py:mod:`zlib`-compressed payload prefixed with an integer size,
	allowing reading by the first stage of exactly the required bytes.

I guess even though the first stage calls fp.close() on the read side, in the child half of the fork, that doesn't close the pipe overall. The parent half will still have an fd for it, thus holding it open.