Skip to content

Updates to the kernel messaging spec #388

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Jul 27, 2019
Merged

Updates to the kernel messaging spec #388

Changes from 4 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
b680593
Clarify what “session” means in the kernel message headers from the c…
jasongrout Jul 13, 2018
49aabf9
Clarify the implied required busy/idle messages.
jasongrout Jul 13, 2018
9dabc43
Deprecate sending the shutdown_request message on the shell channel.
jasongrout Jul 13, 2018
4f7c825
Clarify the status message.
jasongrout Jul 13, 2018
e899ef1
Send the shutdown message over the control channel in client.py.
jasongrout Jul 18, 2018
c4d2414
Clarify the session ids in message headers.
jasongrout Jul 18, 2018
c369247
Fix control channel copy/paste error.
jasongrout Jul 19, 2018
846354e
Move shutdown request to its own section to emphasize the message is …
jasongrout Jul 19, 2018
5bac5e5
Percolate a control connection throughout the code.
jasongrout Jul 19, 2018
5f0a21b
Revert status message updates to revert back to exactly three status …
jasongrout Jul 26, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 25 additions & 14 deletions docs/messaging.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,14 @@ A message is defined by the following four-dictionary structure::
'buffers': list,
}

.. note::

A client session value, in message headers from a client, should be unique among
all clients connected to a kernel and should be constant over the lifetime of
the client. A kernel session value, in message headers from a kernel, should be
generated on kernel startup or restart and should be constant for the lifetime
of the kernel.
Copy link
Member

@rgbkrk rgbkrk Jul 13, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for clarifying this one in the docs!


.. versionchanged:: 5.0

``version`` key added to the header.
Expand All @@ -139,8 +147,9 @@ Compatibility
=============

Kernels must implement the :ref:`execute <execute>` and :ref:`kernel info
<msging_kernel_info>` messages in order to be usable. All other message types
are optional, although we recommend implementing :ref:`completion
<msging_kernel_info>` messages, along with the associated busy and idle
:ref:`status` messages. All other message types are
optional, although we recommend implementing :ref:`completion
<msging_completion>` if possible. Kernels do not need to send any reply for
messages they don't handle, and frontends should provide sensible behaviour if
no reply arrives (except for the required execution and kernel info messages).
Expand Down Expand Up @@ -934,8 +943,7 @@ multiple cases:

The client sends a shutdown request to the kernel, and once it receives the
reply message (which is otherwise empty), it can assume that the kernel has
completed shutdown safely. The request can be sent on either the `control` or
`shell` channels.
completed shutdown safely. The request is sent on the `control` channel.
Copy link
Member

@SylvainCorlay SylvainCorlay Jul 26, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 this is really important this is specified here. +1000


Upon their own shutdown, client applications will typically execute a last
minute sanity check and forcefully terminate any kernel that is still alive, to
Expand All @@ -959,6 +967,12 @@ Message type: ``shutdown_reply``::
socket, they simply send a forceful process termination signal, since a dead
process is unlikely to respond in any useful way to messages.

.. versionchanged:: 5.4

Sending a ``shutdown_request`` message on the ``shell`` channel is deprecated.
Copy link
Member

@rgbkrk rgbkrk Jul 13, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the strategy for deprecating the shutdown_request?

We also deprecated payloads, yet we still use them and have no current replacement.

Copy link
Member Author

@jasongrout jasongrout Jul 13, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you basically asking what deprecation means in this context? Good question. @minrk, thoughts? Does this mean that if a kernel implements protocol 5.4, it doesn't need to listen to shutdown_request messages on the shell channel?

Copy link
Member Author

@jasongrout jasongrout Jul 13, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, that doesn't make much sense to me. How about this: we're just warning people now, and a kernel can officially ignore shutdown requests on the shell channel in spec 6.0 or later.

Copy link
Member

@rgbkrk rgbkrk Jul 14, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works for me!

Is most of the intent behind this that the shutdown API from the server is used? Was shutdown request put in place to help ensure that background resources got cleaned up appropriately (like with a cluster, deleting all the executors / workers?)

Copy link
Member Author

@jasongrout jasongrout Jul 14, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIRC, the kernel manager in the notebook server first sends a shutdown request to the kernel to let it clean up resources and exit by itself. If it doesn't receive a reply in a certain amount of time, then it forcefully shuts down the kernel with a signal.

This protocol definition is at the kernel level, not the kernel manager level. At this kernel message level, if I remember what @minrk said, sending the shutdown_request on either the shell or control channel is an artifact of history, and probably shouldn't have been done that way.

Copy link
Member

@minrk minrk Jul 18, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The notebook server uses shutdown_request on the control channel as a first, polite "please shutdown" before sending signals. This is deprecating the ~unused functionality of sending these messages on the shell channel. Does nteract send shutdown_request on either of these channels?

@jasongrout I notices that KernelClient.shutdown is sending it on the shell channel. This is a rarely, if ever used method, that I think would only be used by frontends that don't own the kernel (e.g. qtconsole --existing), so that should be updated.

Copy link
Member

@SylvainCorlay SylvainCorlay Jul 26, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am +1 on disallowing the shutdown and interupt on shell.

I don't know of any kernel that do it anyways since our clients use the control channel for that.

Making this explicit is really important IMO.




.. _msging_interrupt:

Kernel interrupt
Expand Down Expand Up @@ -1191,6 +1205,8 @@ Message type: ``error``::

``pyerr`` renamed to ``error``

.. _status:

Kernel status
-------------

Expand All @@ -1201,8 +1217,7 @@ Message type: ``status``::
content = {
# When the kernel starts to handle a message, it will enter the 'busy'
# state and when it finishes, it will enter the 'idle' state.
# The kernel will publish state 'starting' exactly once at process startup.
execution_state : ('busy', 'idle', 'starting')
execution_state : ('busy', 'idle', other optional states)
}

When a kernel receives a request and begins processing it,
Expand All @@ -1213,6 +1228,10 @@ it shall publish a status message with ``execution_state: 'idle'``.
Thus, the outputs associated with a given execution shall generally arrive
between the busy and idle status messages associated with a given request.

A kernel may send optional status messages with execution states other than
`busy` or `idle`. For example, a kernel may send a status message with a
`starting` execution state exactly once at process startup.

Copy link
Member Author

@jasongrout jasongrout Jul 19, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@minrk, do we know of other kernels publishing other status states? We made the 'starting' status optional because there might be a race condition preventing the client from ever seeing a starting message. However, it is nice if we have a universe of only three status states, rather than opening it up to any status the kernel might want to send.

As for the kernel status states that the notebook spoofs, we already agreed that those should come on a different channel, a notebook channel, so I think those status states shouldn't count here as optional states sent from a kernel.

Copy link
Member Author

@jasongrout jasongrout Jul 19, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be clear, I think I'm +1 on reverting this "optional status states" change, and just going back to mandating a 'starting' status, and having a defined universe of 'starting', 'busy', and 'idle' states sent from the kernel.

Copy link
Member

@SylvainCorlay SylvainCorlay Jul 26, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't have an opinion on this.

.. note::

**A caveat for asynchronous output**
Expand All @@ -1229,14 +1248,6 @@ between the busy and idle status messages associated with a given request.
Busy and idle messages should be sent before/after handling every request,
not just execution.

.. note::

Extra status messages are added between the notebook webserver and websocket clients
that are not sent by the kernel. These are:

- restarting (kernel has died, but will be automatically restarted)
- dead (kernel has died, restarting has failed)

Clear output
------------

Expand Down