Improve handling of remote input and output when enabled. #274
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This makes the handling of the prompt and the display of external inputs and outputs less messy, ensuring it looks more like something that was run in the actual console (with a "Remote" prefix), while not having the local input prompt all throughout the text.
|
This is so much better! |
VincentDary
added a commit
to firmware-crunch/fiit
that referenced
this pull request
Jul 16, 2025
…rnal message processing - Note: 'external message' refer here to messages which are not part of the shell session, for exemple an `execute_input` message trigger by an other Jupyter shell client. - Provide iopub channel exclusive access to process external message only when no cell is running locally. - Provide external iopub message custom processing and renderering to fix rendering bugs in `jupyter_console.ptshell.ZMQTerminalInteractiveShell.handle_iopub`. This patch prevent `handle_iopub()` to render external message, and use a custom method to do the job. `handle_iopub()` render message to stdout with nested call to `prompt_toolkit`/`print_formatted_text()` and `sys.stdout.write` which result in an inconsistent output because `prompt_toolkit` is inherently async. `handle_iopub()` rendering issues for external message are mentioned in this old pending pull request jupyter/jupyter_console#274 .
VincentDary
added a commit
to firmware-crunch/fiit
that referenced
this pull request
Jul 16, 2025
…rnal message processing - Note: 'external message' refer here to messages which are not part of the shell session, for exemple an `execute_input` message triggered by an other Jupyter shell client. - Provide iopub channel exclusive access to process external message only when no cell is running locally. - Provide external iopub message custom processing and rendering to fix rendering bugs in `jupyter_console.ptshell.ZMQTerminalInteractiveShell.handle_iopub`. This patch prevent `handle_iopub()` to render external message, and use a custom method to do the job. `handle_iopub()` render message to stdout with nested call to `prompt_toolkit`/`print_formatted_text()` and `sys.stdout.write` which result in an inconsistent output because `prompt_toolkit` is inherently async. `handle_iopub()` rendering issues for external message are mentioned in this old pending pull request jupyter/jupyter_console#274 .
VincentDary
added a commit
to firmware-crunch/fiit
that referenced
this pull request
Jul 21, 2025
This patch add the new command 'fiit_console' to replace the Jupyter
Console command used in this project to connect a shell to a remote
Jupyter ipykernel hosted in the fiit runtime. This command subclass and
modify the internals of the `ZMQTerminalIPythonApp` class of the
jupyter/jupyter_console project and provides a number of useful features
in the context of this project.
- At startup, if the kernel is busy, the shell waits infinitely for the
prompt to resume and displays the remote output (stdout/stderr
streams).
- Echo potential long running command (`%emu_start`, `%step`, `%cont`)
run by another kernel client.
- Optional full echo of commands run by another kernel client, via the
environment variable `FULL_ECHO=true`.
- Stop the shell prompt when echo a command run by another kernel client
in order to reflect the busy status of the remote kernel.
- Fix the long output latency of other kernel client output when Jupyter
Console is configured with `include_other_output` to true, via a
custom asynchronous polling of the iopub channel.
- Provide a consistent output for commands run by another client, rather
than the bugged output provided by the default implementation.
- Provide a way to launch a shell by retrieving remote kernel
information from `plugin_backend` without specify a backend port, if
backend uses its default port.
- Provide two new commands for launching the custom Jupiter client:
* via config: `fiit_console --existing [jupyter_json_kernel_config]`
* via backend: `fiit console --help`
- It seems, but it is not certain, that this implementation solves the
invalid signature problem encountered with the default Jupyter
Console.
Previously, the `jupyter console` command from the Jupyter project was
used as frontend to provide a terminal connected to a remote Jupyter
ipykernel hosted in the fiit runtime. The shell implementation has
several issues and behaviours which are not convenient in the context of
this project.
When the shell client start, if the remote kernel is in 'busy' state
because it is executing a cell, the shell hang 60 seconds and quit, in
addition if the remote kernel stream output such as stdout/stderr the
shell display nothing. This behavior is very annoying, for example if a
remote kernel is busy because a process is emulating and debugging a
firmware executing a long run with a forward of the serial output to
stdout, the shell client displays nothing and can potentially quit
before the debugger resumes the shell prompt at a specific breakpoint.
When echoing commands of another kernel client, via the
`include_other_output` option, the output is inconsistent which is a bug
in the current implementation, see the old pending pull request
jupyter/jupyter_console#274. Furthermore, there is a long latency in the
output of the other commands caused by an inefficient pooling of the
iopub channel.
When the state of the remote kernel is 'busy', the shell client prompt
remains active. This is the default implementation of the Jupyter
console. If the `include_other_output` option is enable, the remote
output is displayed and nested with the shell prompt which is a bit
messy and makes the prompt unusable if the remote output is very
verbose. It is preferable to block the prompt and display the cell
output run by another kernel client for a subset of commands. Such as
the `%emu_start` `%step` and `%cont` commands, which are potentially
performing a long run in the remote kernel and provide important
information when the prompt is resumed, such ad the new current code
location in the emulated code, which is useful for all other kernel
client.
When several shell clients are connected to the remote kernel and after
several commands, all shells crash with the exception message 'Invalid
Signature'.
VincentDary
added a commit
to firmware-crunch/fiit
that referenced
this pull request
Jul 21, 2025
This patch add the new command 'fiit_console' to replace the Jupyter
Console command used in this project to connect a shell to a remote
Jupyter ipykernel hosted in the fiit runtime. This command subclass and
modify the internals of the `ZMQTerminalIPythonApp` class of the
jupyter/jupyter_console project and provides a number of useful features
in the context of this project.
- At startup, if the kernel is busy, the shell waits infinitely for the
prompt to resume and displays the remote output (stdout/stderr
streams).
- Echo potential long running command (`%emu_start`, `%step`, `%cont`)
run by another kernel client.
- Optional full echo of commands run by another kernel client, via the
environment variable `FULL_ECHO=true`.
- Stop the shell prompt when echo a command run by another kernel client
in order to reflect the busy status of the remote kernel.
- Fix the long output latency of other kernel client output when Jupyter
Console is configured with `include_other_output` to true, via a
custom asynchronous polling of the iopub channel.
- Provide a consistent output for commands run by another client, rather
than the bugged output provided by the default implementation.
- Provide a way to launch a shell by retrieving remote kernel
information from `plugin_backend` without specify a backend port, if
backend uses its default port.
- Provide two new commands for launching the custom Jupiter client:
* via config: `fiit_console --existing [jupyter_json_kernel_config]`
* via backend: `fiit console --help`
- It seems, but it is not certain, that this implementation solves the
invalid signature problem encountered with the default Jupyter
Console.
Previously, the `jupyter console` command from the Jupyter project was
used as frontend to provide a terminal connected to a remote Jupyter
ipykernel hosted in the fiit runtime. The shell implementation has
several issues and behaviours which are not convenient in the context of
this project.
When the shell client start, if the remote kernel is in 'busy' state
because it is executing a cell, the shell hang 60 seconds and quit, in
addition if the remote kernel stream output such as stdout/stderr the
shell display nothing. This behavior is very annoying, for example if a
remote kernel is busy because a process is emulating and debugging a
firmware executing a long run with a forward of the serial output to
stdout, the shell client displays nothing and can potentially quit
before the debugger resumes the shell prompt at a specific breakpoint.
When echoing commands of another kernel client, via the
`include_other_output` option, the output is inconsistent which is a bug
in the current implementation, see the old pending pull request
jupyter/jupyter_console#274. Furthermore, there is a long latency in the
output of the other commands caused by an inefficient pooling of the
iopub channel.
When the state of the remote kernel is 'busy', the shell client prompt
remains active. This is the default implementation of the Jupyter
console. If the `include_other_output` option is enable, the remote
output is displayed and nested with the shell prompt which is a bit
messy and makes the prompt unusable if the remote output is very
verbose. It is preferable to block the prompt and display the cell
output run by another kernel client for a subset of commands. Such as
the `%emu_start` `%step` and `%cont` commands, which are potentially
performing a long run in the remote kernel and provide important
information when the prompt is resumed, such ad the new current code
location in the emulated code, which is useful for all other kernel
client.
When several shell clients are connected to the remote kernel and after
several commands, all shells crash with the exception message 'Invalid
Signature'.
VincentDary
added a commit
to firmware-crunch/fiit
that referenced
this pull request
Jul 21, 2025
This patch add the new command 'fiit_console' to replace the Jupyter
Console command used in this project to connect a shell to a remote
Jupyter ipykernel hosted in the fiit runtime. This command subclass and
modify the internals of the `ZMQTerminalIPythonApp` class of the
jupyter/jupyter_console project and provides a number of useful features
in the context of this project.
- At startup, if the kernel is busy, the shell waits infinitely for the
prompt to resume and displays the remote output (stdout/stderr
streams).
- Echo potential long running command (`%emu_start`, `%step`, `%cont`)
run by another kernel client.
- Optional full echo of commands run by another kernel client, via the
environment variable `FULL_ECHO=true`.
- Stop the shell prompt when echo a command run by another kernel client
in order to reflect the busy status of the remote kernel.
- Fix the long output latency of other kernel client output when Jupyter
Console is configured with `include_other_output` to true, via a
custom asynchronous polling of the iopub channel.
- Provide a consistent output for commands run by another client, rather
than the bugged output provided by the default implementation.
- Provide a way to launch a shell by retrieving remote kernel
information from `plugin_backend` without specify a backend port, if
backend uses its default port.
- Provide two new commands for launching the custom Jupiter client:
* via config: `fiit_console --existing [jupyter_json_kernel_config]`
* via backend: `fiit console --help`
- It seems, but it is not certain, that this implementation solves the
invalid signature problem encountered with the default Jupyter
Console.
Previously, the `jupyter console` command from the Jupyter project was
used as frontend to provide a terminal connected to a remote Jupyter
ipykernel hosted in the fiit runtime. The shell implementation has
several issues and behaviours which are not convenient in the context of
this project.
When the shell client start, if the remote kernel is in 'busy' state
because it is executing a cell, the shell hang 60 seconds and quit, in
addition if the remote kernel stream output such as stdout/stderr the
shell display nothing. This behavior is very annoying, for example if a
remote kernel is busy because a process is emulating and debugging a
firmware executing a long run with a forward of the serial output to
stdout, the shell client displays nothing and can potentially quit
before the debugger resumes the shell prompt at a specific breakpoint.
When echoing commands of another kernel client, via the
`include_other_output` option, the output is inconsistent which is a bug
in the current implementation, see the old pending pull request
jupyter/jupyter_console#274. Furthermore, there is a long latency in the
output of the other commands caused by an inefficient pooling of the
iopub channel.
When the state of the remote kernel is 'busy', the shell client prompt
remains active. This is the default implementation of the Jupyter
console. If the `include_other_output` option is enable, the remote
output is displayed and nested with the shell prompt which is a bit
messy and makes the prompt unusable if the remote output is very
verbose. It is preferable to block the prompt and display the cell
output run by another kernel client for a subset of commands. Such as
the `%emu_start` `%step` and `%cont` commands, which are potentially
performing a long run in the remote kernel and provide important
information when the prompt is resumed, such ad the new current code
location in the emulated code, which is useful for all other kernel
client.
When several shell clients are connected to the remote kernel and after
several commands, all shells crash with the exception message 'Invalid
Signature'.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This makes the handling of the prompt and the display of external inputs and outputs less messy, ensuring it looks more like something that was run in the actual console (with a "Remote" prefix), while not having the local input prompt all throughout the text.
I'm not sure if this warrants extra tests, or how to write those, so potential advise there would be appreciated.