faulthandler.enable() fails with IOError

[fperez]

Reported by @poldrack on the notebook, but ultimately the issue is really a kernel problem, I can reproduce it from a plain console by connecting to an open kernel using ipykernel master:

denali[newjob]> jupyter console --existing kernel-95921.json
Jupyter Console 4.1.0.dev

[ZMQTerminalIPythonApp] Loading IPython extension: storemagic

In [1]: import faulthandler

In [2]: faulthandler.enable()
---------------------------------------------------------------------------
UnsupportedOperation                      Traceback (most recent call last)
<ipython-input-2-140d8a320d56> in <module>()
----> 1 faulthandler.enable()

/Users/fperez/dev/ipython/ipykernel/ipykernel/iostream.py in fileno(self)
    203 
    204     def fileno(self):
--> 205         raise UnsupportedOperation("IOStream has no fileno.")
    206 
    207     def write(self, string):

UnsupportedOperation: IOStream has no fileno.

[fperez]

@poldrack, the reason there's a difference in behavior between connecting to a kernel from a notebook (or a console above) and a 'plain' ipython shell, is that the latter doesn't start an actual kernel process, it uses an internal implementation that's slightly different. While we try to minimize those differences obviously there is a bit of mismatch.

Ping @minrk @takluyver.

[poldrack]

thanks! always happy to break things :-)

On Jan 2, 2016, at 2:26 PM, Fernando Perez notifications@github.com wrote:

@poldrack https://github.com/poldrack, the reason there's a difference in behavior between connecting to a kernel from a notebook (or a console above) and a 'plain' ipython shell, is that the latter doesn't start an actual kernel process, it uses an internal implementation that's slightly different. While we try to minimize those differences obviously there is a bit of mismatch.

Ping @minrk https://github.com/minrk @takluyver https://github.com/takluyver.

—
Reply to this email directly or view it on GitHub #91 (comment).

[fperez]

Folks, I see we made an explicit decision to not report a fileno for these streams, and I have a vague recollection of writing some of that code a while back. But it seems the faulthandler code does expect a fileno...

For now, this simple diff:

diff --git a/ipykernel/iostream.py b/ipykernel/iostream.py
index 1cb3119..f6d24e4 100644
--- a/ipykernel/iostream.py
+++ b/ipykernel/iostream.py
@@ -202,7 +202,8 @@ class OutStream(object):
         raise IOError('Read not supported on a write only stream.')

     def fileno(self):
-        raise UnsupportedOperation("IOStream has no fileno.")
+        return 101
+        #raise UnsupportedOperation("IOStream has no fileno.")

     def write(self, string):
         if self.pub_socket is None:

does fix that specific problem for me.

Obviously we need to think more carefully about what to put in there, but before I go writing actual code, I wanted to run the question by you guys, because I haven't looked at this code in a long time and don't really remember. Any other constraints about why we should not return a value from this function? Can we return something valid, so we can fix that?

And BTW, @poldrack, in the meantime, you can use the above hack for a quick local fix for yourself if you want :)

[poldrack]

thanks!

On Jan 2, 2016, at 2:31 PM, Fernando Perez notifications@github.com wrote:

Folks, I see we made an explicit decision to not report a fileno for these streams, and I have a vague recollection of writing some of that code a while back. But it seems the faulthandler code does expect a fileno...

For now, this simple diff:

diff --git a/ipykernel/iostream.py b/ipykernel/iostream.py
index 1cb3119..f6d24e4 100644
--- a/ipykernel/iostream.py
+++ b/ipykernel/iostream.py
@@ -202,7 +202,8 @@ class OutStream(object):
raise IOError('Read not supported on a write only stream.')

 def fileno(self):

•    raise UnsupportedOperation("IOStream has no fileno.")
  

•    return 101
  

•    #raise UnsupportedOperation("IOStream has no fileno.")
  

  def write(self, string):
  if self.pub_socket is None:
  does fix that specific problem for me.

Obviously we need to think more carefully about what to put in there, but before I go writing actual code, I wanted to run the question by you guys, because I haven't looked at this code in a long time and don't really remember. Any other constraints about why we should not return a value from this function? Can we return something valid, so we can fix that?

And BTW, @poldrack https://github.com/poldrack, in the meantime, you can use the above hack for a quick local fix for yourself if you want :)

—
Reply to this email directly or view it on GitHub #91 (comment).

[Carreau]

returning an int is likely not the right fix :

fileno()
   Return the underlying file descriptor (an integer) of the stream if it exists.
   An OSError is raised if the IO object does not use a file descriptor.

We do not use a file descriptor (we use a ZMQ socket), and returning a random number will likely generate unexpected behavior. Wild guess : dump to the file which correspond to this fileno. That is to say write to an arbitrary open file by Python/IPython as then the returned value of fileno can be used by functions like os.fdopen and alike.

So the right behavior is to raise. Moreover io.UnsupportedOperation does inherit the right Exception (IOError on py2 and OSError on py3) so is the right things to raise.

Also I would not use the above hack as it might write to arbitrary file.

[fperez]

Ouch, thanks for looking into it more carefully, @Carreau. I think we should look next at what exactly faulthandler actually needs, to see if it's possible to support it.

It's a problem to not support a core debugging facility of the standard library in our kernel.

I can't look into this for the next few days, but at least now we know where the problem is. We may want to create a local file proxy to the socket that has an actual file descriptor and pass that along, so that write operations to the iostream can be sent to a valid object. Just a thought.

[minrk]

If we want fileno to work, then we have to set up an os.pipe that forwards to the zmq socket via a thread. This should be easier to provide (on non-Windows, at least) after #85 is merged.

[takluyver]

faulthandler is for displaying a minimal traceback when the program is crashing because of something like a segfault. So I don't think it should ever rely on a pipe looping back to a Python thread which sends output over zmq; it seems highly unlikely that the traceback will actually come through.

For now, I think I'd recommend that people set the environment variable PYTHONFAULTHANDLER=1 to enable it at kernel start up, before we replace sys.stdout and sys.stderr. I guess you can achieve the same thing by calling faulthandler.enable() in a config file. If you really need to do it in a notebook, faulthandler.enable(file=sys.__stderr__) should work, I think.

The stdstream-capturing nanny process we've talked about would help with this. The process reading the pipe would be separate from the kernel that's crashing, so it should survive and forward the data on to the notebook, so you'd see the faulthandler traceback as the kernel died.

[Carreau]

If you really need to do it in a notebook, faulthandler.enable(file=sys.stderr) should work, I think

+1, we can also monkeypatch faulthandler.enable at the same time as stderr just to change the default argument for file.