Kernel Operation

[declanvk]

@ellisonbg @willingc This is a brain dump of the various ways that the kernel should work, but may not currently uphold. I've attempted to define any terms that are important and remove ambiguity. I will continue adding to this issue, as this single comment is not complete record of the things the kernel does.

Definitions/Notes

• a request - is a message from the front end, usually execute_request, that contains some metadata and a string containing code to be executed by the kernel.
• input variables - variables from the code block of the request that are not assigned to within the block of code from the request. In this context "assigned to" also covers class or function definition, anything that associates a name with an object. The assumption of the kernel is that any variable that is reference but not assigned within this code block, must have been assigned in another code block, sent in a previous request.
• output variable(s) - variables from the code block of the request that are assigned to within the block of code from the request.
• namespace - dictionary that holds the values of variable, mapping names of variables to values of variables.
• asyncio.sleep(0) - this coroutine is used to return control to the event loop and increase the number of points in the execution where other request/other computation can be interleaved. Returning control to the event loop in more places increases the appearance of parallel execution, even when all code in the event loop is running in a single thread.
• execution context - this object will run code string and capture any output in the form of stdout, stderr, exception, etc. In the case of a single assignment as the last expression in the code block, the AST will be manipulated such that the displayhook will capture the value of the last assignment. The execution context will return a container object that holds all the different forms of output.

Lifecycle of a Non-Awaitable, Non-Generator Request

1. Receive request, and extract input variables and output variable(s) from the code string in the request.
2. Detect if the output variable (error if more than one output variable) is previously defined or a new definition.
   a. If it is an old definition, detect the differences in input variables from the old definition to the new definition. Then update the dependency graph with the compute difference.
   b. If it is a new definition, create a new node in the dependency graph, then add all the edges representing the input variables
3. Compute the descendants of the current output variable(s), even if there are none due to it being a new definition.
4. Execute the block of code using the execution context, and capture the various forms of output.
5. Send the output back to the front end, either establishing or updating using display ids. This output will take advantage of the established IPython mimetype code, using _repr_html_, _repr_svg_, etc.
6. For each descendant of the current output variable(s) repeat steps 4 & 5 with the descendant's block of code.
7. Send the final result of success/failure to the front end along with the new execution count.
8. End the request.

Lifecycle of a Awaitable, Non-generator Request

Steps 1-3 are the same as they are for Non-Awaitable, Non-Generator Requests.

4. Execute the block of code using the execution context, capture the various forms of output, specifically the value of the output variable(s). Then await the value of the output variable(s), because it is a coroutine, and the update the execution context's namespace with the value of the awaited variable.

Steps 5-8 are the same as they are for Non-Awaitable, Non-Generator Requests.

Lifecycle of a (Async) Generator Request.

Steps 1-3 are the same as they are for Non-Awaitable, Non-Generator Requests.

4. Execute the block of code using the execution context, capture the various forms of output, specifically the value of the output variable(s). Detect if this value is a regular generator or an async generator.
   a. If the value is a regular generator, then convert it to an async generator that will yield its values and sleep a tiny amount between each time.
   b. if the value is an async generator, do nothing.
5. Cancel any previous async generators that were producing values for the same output variable(s) as the current one.
6. Await the first value from the value that is now an async generator, and use that to update the namespace of the execution context.
7. Send the output back to the front end, either establishing or updating using display ids. This output will take advantage of the established IPython mimetype code, using _repr_html_, _repr_svg_, etc.
8. For each descendant of the current output variable(s) repeat steps 4-7 with the descendant's block of code.
9. Send the final result of success/failure to the front end along with the new execution count.
10. Repeat steps 5-8 until the async generator yields no more values, or the current async generator is cancelled.

[willingc]

Nice write-up @declanvk 👍

I've moved the write-up above to a HackMD file (just sign in with your GitHub account) that we can all edit for questions/clarifications/etc. I've taken your text and formatted it for easier scanning.