Can't start or restart kernels after some time

[aplavin]

After running jupyter notebook for some time, I can't restart kernels (it says restart has failed) and can't start new ones (it says kernel error), while already running kernels keep running and can execute new code. Unfortunately, I don't know the exact steps how to reproduce this issue, and would appreciate any help on how to debug it.

To start, here is the log message from jupyter notebook when I try to start a new kernel (i.e. open a new notebook):

[E 11:17:55.099 NotebookApp] Unhandled error in API request
    Traceback (most recent call last):
      File "<...>/lib/python3.5/site-packages/notebook/base/handlers.py", line 457, in wrapper
        result = yield gen.maybe_future(method(self, *args, **kwargs))
      File "<...>/lib/python3.5/site-packages/tornado/gen.py", line 1015, in run
        value = future.result()
      File "<...>/lib/python3.5/site-packages/tornado/concurrent.py", line 237, in result
        raise_exc_info(self._exc_info)
      File "<string>", line 3, in raise_exc_info
      File "<...>/lib/python3.5/site-packages/tornado/gen.py", line 1021, in run
        yielded = self.gen.throw(*exc_info)
      File "<...>/lib/python3.5/site-packages/notebook/services/sessions/handlers.py", line 62, in post
        kernel_id=kernel_id))
      File "<...>/lib/python3.5/site-packages/tornado/gen.py", line 1015, in run
        value = future.result()
      File "<...>/lib/python3.5/site-packages/tornado/concurrent.py", line 237, in result
        raise_exc_info(self._exc_info)
      File "<string>", line 3, in raise_exc_info
      File "<...>/lib/python3.5/site-packages/tornado/gen.py", line 1021, in run
        yielded = self.gen.throw(*exc_info)
      File "<...>/lib/python3.5/site-packages/notebook/services/sessions/sessionmanager.py", line 79, in create_session
        kernel_name)
      File "<...>/lib/python3.5/site-packages/tornado/gen.py", line 1015, in run
        value = future.result()
      File "<...>/lib/python3.5/site-packages/tornado/concurrent.py", line 237, in result
        raise_exc_info(self._exc_info)
      File "<string>", line 3, in raise_exc_info
      File "<...>/lib/python3.5/site-packages/tornado/gen.py", line 1021, in run
        yielded = self.gen.throw(*exc_info)
      File "<...>/lib/python3.5/site-packages/notebook/services/sessions/sessionmanager.py", line 92, in start_kernel_for_session
        self.kernel_manager.start_kernel(path=kernel_path, kernel_name=kernel_name)
      File "<...>/lib/python3.5/site-packages/tornado/gen.py", line 1015, in run
        value = future.result()
      File "<...>/lib/python3.5/site-packages/tornado/concurrent.py", line 237, in result
        raise_exc_info(self._exc_info)
      File "<string>", line 3, in raise_exc_info
      File "<...>/lib/python3.5/site-packages/tornado/gen.py", line 285, in wrapper
        yielded = next(result)
      File "<...>/lib/python3.5/site-packages/notebook/services/kernels/kernelmanager.py", line 87, in start_kernel
        super(MappingKernelManager, self).start_kernel(**kwargs)
      File "<...>/lib/python3.5/site-packages/jupyter_client/multikernelmanager.py", line 110, in start_kernel
        km.start_kernel(**kwargs)
      File "<...>/lib/python3.5/site-packages/jupyter_client/manager.py", line 224, in start_kernel
        self.write_connection_file()
      File "<...>/lib/python3.5/site-packages/jupyter_client/connect.py", line 407, in write_connection_file
        kernel_name=self.kernel_name
      File "<...>/lib/python3.5/site-packages/jupyter_client/connect.py", line 135, in write_connection_file
        with open(fname, 'w') as f:
    FileNotFoundError: [Errno 2] No such file or directory: '/var/run/user/708/jupyter/kernel-480c8487-0ba3-4040-811b-6e060de832c3.json'

708 is my UID, and /var/run/user/708/ is empty at the time of this error (i.e. doesn't have jupyter directory in it).

[takluyver]

I think someone else had this issue recently, and we realised there was a need to touch files periodically so they weren't cleaned up. I can't immediately find the issue now, though.

[aplavin]

Oh, after your comment I investigated if the files in /var/run/user get cleaned up by other programs (not jupyter itself), and it turned out OS removes this directory content every time I disconnect (I run jupyter on a remote server, accessed via ssh)! So, I changed the directory for jupyter by running XDG_RUNTIME_DIR= jupyter notebook --no-browser instead of just jupyter notebook --no-browser - so it keeps its files in another location. Looks like no problems now.

[takluyver]

That's annoying. It's supposed to be left there as long as you're logged in, which I would take to include while there is still any program running as your user.

I'm reopening this because it's probably something we need to handle.

[jakirkham]

I just ran into this same issue on my local cluster. Am asking our support team to figure out why.

Just out of curiosity, are you using SGE, @aplavin ?

[jakirkham]

So my case ends up being a bit different. Appears XDG_RUNTIME_DIR is set to a non-existent directory. In any event, I expect whatever solution used to solve the OP's problem will solve mine.

[dburkhardt]

I've been having the same problem since updated Jupyter to version 5.0 on two different machines running Arch. It seems that setting XDG_RUNTIME_DIR to a directory in my home has fixed the problem.

[EmreAtes]

According to the XDG specification, XDG_RUNTIME_DIR is removed once the user logs out. Therefore, if I start a notebook process in a remote server (with nohup or screen) and then log out, and then restart the kernel, the notebook crashes with a similar error.

[takluyver]

Yeah, I've been meaning to ask XDG about that. I'd assumed that if there was a process (like Jupyter) still running as your user, then you are still 'logged in' for the purpose of XDG_RUNTIME_DIR remaining.

[takluyver]

@EmreAtes what platform is the server you're SSH-ing into? I've just tested on a Debian system which I have access to, and XDG_RUNTIME_DIR wasn't removed when I disconnected with a script left running in nohup.

I've asked on the XDG mailing list for clarification: https://lists.freedesktop.org/archives/xdg/2018-May/014048.html