Using a bash jupyter kernel and bash computation

[aborruso]

Hi,
I use sometime in my notebook this bash kernel.

Is it possibile to use this kernel and bash computation inside Quarto?

Using this the below hello world qmd file, and running quarto preview I have this error:

Executing 'tmp-bash.ipynb'
  Cell 1/1...ERROR:

An error occurred while executing the following cell:
------------------
echo "hello"
------------------


: 1

If I change

```{bash}
echo "hello"
```

to

```{Bash}
echo "hello"
```

I have no more the error (Bash should be the right label), but I have no code excution.

Thank you

---
title: "bash demo"
format:
  html:
    code-fold: true
jupyter: bash
---

For a bash demonstration.

```{bash}
echo "hello"
```

[jjallaire]

Interesting....this works for me w/ Python 3 on Mac OX. Sounds like there might be a case-sensitivity problem w/ kernel names? What is the output for you of quarto check jupyter ? Here's what I get:

[✓] Checking Python 3 installation....OK
      Version: 3.9.5
      Path: /Users/jjallaire/quarto/dev/bash-kernel/env/bin/python3
      Jupyter: 4.9.2
      Kernels: ir, julia-1.7, bash, julia-1.5, python3

[✓] Checking Jupyter engine render....OK

Note the lower case kernel name. If you execute quarto capabilities you'll see more detail. Note that while the display name of the bash kernel is "Bash" the name is lower case (and that's what we match against):

 "python": {
    "versionMajor": 3,
    "versionMinor": 9,
    "versionPatch": 5,
    "versionStr": "3.9.5 (v3.9.5:0a7dcbdb13, May  3 2021, 13:17:02)  [Clang 6.0 (clang-600.0.57)]",
    "conda": false,
    "execPrefix": "/Users/jjallaire/quarto/dev/bash-kernel/env",
    "executable": "/Users/jjallaire/quarto/dev/bash-kernel/env/bin/python3",
    "jupyter_core": "4.9.2",
    "nbformat": "5.3.0",
    "nbclient": "0.6.0",
    "ipykernel": "6.13.0",
    "pyLauncher": false,
    "kernels": [
      {
        "name": "ir",
        "language": "R",
        "display_name": "R"
      },
      {
        "name": "julia-1.7",
        "language": "julia",
        "display_name": "Julia 1.7.2"
      },
      {
        "name": "bash",
        "language": "bash",
        "display_name": "Bash"
      },
      {
        "name": "julia-1.5",
        "language": "julia",
        "display_name": "Julia 1.5.3"
      },
      {
        "name": "python3",
        "language": "python",
        "display_name": "Python 3"
      }
    ],

[aborruso]

Interesting....this works for me w/ Python 3 on Mac OX. Sounds like there might be a case-sensitivity problem w/ kernel names?

As I have written, I have tested using both bash and Bash.

If I use bash I have

Starting Jupyter kernel...Done

Executing 'bash.ipynb'
  Cell 1/1...ERROR:

An error occurred while executing the following cell:
------------------
echo "hello"
------------------


: 1

These my kernels

[✓] Checking Python 3 installation....OK
      Version: 3.9.2
      Path: /usr/bin/python3
      Jupyter: 4.7.1
      Kernels: bash, python3, zsh

Thank you

[jjallaire]

Right I saw that, I'm just noting that bash seems to be correct and I'm seeing that work in my environment. Another thing to do might be to check the versions the packages that compose the.Jupyter interface. If you execute quarto capabilities then you'll see Python installation info like this:

"python": {
    "versionMajor": 3,
    "versionMinor": 9,
    "versionPatch": 5,
    "versionStr": "3.9.5 (v3.9.5:0a7dcbdb13, May  3 2021, 13:17:02)  [Clang 6.0 (clang-600.0.57)]",
    "conda": false,
    "execPrefix": "/Library/Frameworks/Python.framework/Versions/3.9",
    "executable": "/Library/Frameworks/Python.framework/Versions/3.9/bin/python3",
    "jupyter_core": "4.7.1",
    "nbformat": "5.1.3",
    "nbclient": "0.5.3",
    "ipykernel": "5.5.5",
    "pyLauncher": false,

If the versions you are running are different from what's above you might try updating those packages. You could also try updating the package w/ the bash kernel.

[aborruso]

Another thing to do might be to check the versions the packages that compose the.Jupyter interface. If you execute quarto capabilities then you'll see Python installation info like this:

I have upgraded all the packages, and now I have

"python": {
    "versionMajor": 3,
    "versionMinor": 9,
    "versionPatch": 2,
    "versionStr": "3.9.2 (default, Feb 28 2021, 17:03:44)  [GCC 10.2.1 20210110]",
    "conda": false,
    "execPrefix": "/usr",
    "executable": "/usr/bin/python3",
    "jupyter_core": "4.9.2",
    "nbformat": "5.3.0",
    "nbclient": "0.6.0",
    "ipykernel": "6.13.0",
    "pyLauncher": false

I have always the same error.

Thank you

[jjallaire]

I'm not sure what's going on then. My first test was on Mac and now I've tried again on Linux (as it looks like that's where you are running). Here is my quarto capabilities output:

 "python": {
    "versionMajor": 3,
    "versionMinor": 8,
    "versionPatch": 10,
    "versionStr": "3.8.10 (default, Mar 15 2022, 12:22:08)  [GCC 9.4.0]",
    "conda": false,
    "execPrefix": "/home/jjallaire/quarto/dev/bash-kernel/env",
    "executable": "/home/jjallaire/quarto/dev/bash-kernel/env/bin/python3",
    "jupyter_core": "4.9.2",
    "nbformat": "5.3.0",
    "nbclient": "0.6.0",
    "ipykernel": "6.13.0",
    "pyLauncher": false,
    "kernels": [
      {
        "name": "bash",
        "language": "bash",
        "display_name": "Bash"
      },
      {
        "name": "python3",
        "language": "python",
        "display_name": "Python 3 (ipykernel)"
      }
    ],
    "venv": true
  }

[aborruso]

Is there a way to have a more detailed quarto error message?

[aborruso]

Wait, I have found a problem in my jupyter. I'm trying to solve it

[aborruso]

Ok, I have solved my jupyter 500 error problem, and now it works all in jupyter. But if I run quarto preview I have always the same error.
Is there no way to have a raw error message?

My config

"python": {
    "versionMajor": 3,
    "versionMinor": 9,
    "versionPatch": 2,
    "versionStr": "3.9.2 (default, Feb 28 2021, 17:03:44)  [GCC 10.2.1 20210110]",
    "conda": false,
    "execPrefix": "/usr",
    "executable": "/usr/bin/python3",
    "jupyter_core": "4.9.2",
    "nbformat": "5.3.0",
    "nbclient": "0.6.0",
    "ipykernel": "6.13.0",
    "pyLauncher": false,
    "kernels": [
      {
        "name": "bash",
        "language": "bash",
        "display_name": "Bash"
      },
      {
        "name": "python3",
        "language": "python",
        "display_name": "Python 3 (ipykernel)"
      },
      {
        "name": "zsh",
        "language": "zsh",
        "display_name": "Z shell"
      }
    ],
    "venv": true
  }
}%

[aborruso]

I'm not sure what's going on then.

Running all inside vscode, I have had a verbose error, but I have no idea on how to solve it:

Exception ignored in: <function BaseEventLoop.__del__ at 0x7f75df17bc10>
Traceback (most recent call last):
  File "/home/aborruso/.pyenv/versions/3.8.3/lib/python3.8/asyncio/base_events.py", line 656, in __del__
    self.close()
  File "/home/aborruso/.pyenv/versions/3.8.3/lib/python3.8/asyncio/unix_events.py", line 58, in close
    super().close()
  File "/home/aborruso/.pyenv/versions/3.8.3/lib/python3.8/asyncio/selector_events.py", line 92, in close
    self._close_self_pipe()
  File "/home/aborruso/.pyenv/versions/3.8.3/lib/python3.8/asyncio/selector_events.py", line 99, in _close_self_pipe
    self._remove_reader(self._ssock.fileno())
  File "/home/aborruso/.pyenv/versions/3.8.3/lib/python3.8/asyncio/selector_events.py", line 274, in _remove_reader
    key = self._selector.get_key(fd)
  File "/home/aborruso/.pyenv/versions/3.8.3/lib/python3.8/selectors.py", line 190, in get_key
    return mapping[fileobj]
  File "/home/aborruso/.pyenv/versions/3.8.3/lib/python3.8/selectors.py", line 71, in __getitem__
    fd = self._selector._fileobj_lookup(fileobj)
  File "/home/aborruso/.pyenv/versions/3.8.3/lib/python3.8/selectors.py", line 225, in _fileobj_lookup
    return _fileobj_to_fd(fileobj)
  File "/home/aborruso/.pyenv/versions/3.8.3/lib/python3.8/selectors.py", line 42, in _fileobj_to_fd
    raise ValueError("Invalid file descriptor: {}".format(fd))
ValueError: Invalid file descriptor: -1

[jjallaire]

It is printing whatever error is returned by the kernel. You could try the flag --execute-debug to see if there is any more output available.

[aborruso]

It is printing whatever error is returned by the kernel. You could try the flag --execute-debug to see if there is any more output available.

I have only

Starting Jupyter kernel...2022-04-14 19:14:03,594 - root - Level 25 - starting notebook server subprocess
2022-04-14 19:14:03,594 - root - Level 25 - starting notebook server subprocess
2022-04-14 19:14:03,902 - root - Level 25 - running notebook server subprocess
2022-04-14 19:14:03,902 - root - Level 25 - creating notebook server (tcp: /home/aborruso/.local/share/quarto/jt/3bf910d309fa949374f0)
2022-04-14 19:14:03,904 - root - Level 25 - notebook server bound to port 36107
2022-04-14 19:14:03,952 - root - Level 25 - handling server request
- Kernel connection successful
- Sending execute command to kernel
- Execute command sent, reading response
2022-04-14 19:14:03,953 - root - Level 25 - executing notebook
Done

Executing 'bash.ipynb'
  Cell 1/1...- Error response received
ERROR:

An error occurred while executing the following cell:
------------------
pwd
------------------


: 1

[aborruso]

A note, I do not know it this is useful: if I open the notebook created from quarto, I have no jupyter error

[aborruso]

@jjallaire probably I have this error, because I'm using quarto inside Windows Subsystem for Linux.

I think so, because I have problem also running !ls inside a python kernel. In example, starting from

---
title: "demo"
format:
  html:
    code-fold: true
jupyter: python3
---

For a bash demonstration.

```{python}
!ls
```

I have

ExceptionPexpect: isalive() encountered condition where "terminated" is 0, but there was no child process. Did someone else call waitpid() on our process?
ExceptionPexpect: isalive() encountered condition where "terminated" is 0, but there was no child process. Did someone else call waitpid() on our process?

[aborruso]

@jjallaire probably I have this error, because I'm using quarto inside Windows Subsystem for Linux.

I have opened a new related discussion

[cderv]

@aborruso I think the problem is like with shell magic command, it is related to the daemon. This means that executing with daemon: false or --no-execute-daemon should work to render the .qmd document.

Works for me when I try

[cderv]

Ok I installed the same version as you here: #604 (reply in thread)

  "python": {
    "versionMajor": 3,
    "versionMinor": 9,
    "versionPatch": 2,
    "versionStr": "3.9.2 (default, Apr 27 2022, 13:29:18)  [GCC 9.4.0]",
    "conda": false,
    "execPrefix": "/home/cderv/.pyenv/versions/3.9.2",
    "executable": "/home/cderv/.pyenv/versions/3.9.2/bin/python3",
    "jupyter_core": "4.10.0",
    "nbformat": "5.3.0",
    "nbclient": "0.6.0",
    "ipykernel": "6.13.0",
    "pyLauncher": false,
    "kernels": [
      {
        "name": "bash",
        "language": "bash",
        "display_name": "Bash"
      },
      {
        "name": "python3",
        "language": "python",
        "display_name": "Python 3 (ipykernel)"
      }
    ],
    "venv": true
  }

I can't reproduce this issue on WSL linux on Windows 10

Can you update Jupyter and bash_kernel too ?

I just set up everything from today

$ python -m pip freeze
argon2-cffi==21.3.0
argon2-cffi-bindings==21.2.0
asttokens==2.0.5
attrs==21.4.0
backcall==0.2.0
bash_kernel==0.7.2
beautifulsoup4==4.11.1
bleach==5.0.0
cffi==1.15.0
debugpy==1.6.0
decorator==5.1.1
defusedxml==0.7.1
entrypoints==0.4
executing==0.8.3
fastjsonschema==2.15.3
ipykernel==6.13.0
ipython==8.2.0
ipython-genutils==0.2.0
ipywidgets==7.7.0
jedi==0.18.1
Jinja2==3.1.1
jsonschema==4.4.0
jupyter==1.0.0
jupyter-client==7.3.0
jupyter-console==6.4.3
jupyter-core==4.10.0
jupyterlab-pygments==0.2.2
jupyterlab-widgets==1.1.0
MarkupSafe==2.1.1
matplotlib-inline==0.1.3
mistune==0.8.4
nbclient==0.6.0
nbconvert==6.5.0
nbformat==5.3.0
nest-asyncio==1.5.5
notebook==6.4.11
packaging==21.3
pandocfilters==1.5.0
parso==0.8.3
pexpect==4.8.0
pickleshare==0.7.5
prometheus-client==0.14.1
prompt-toolkit==3.0.29
psutil==5.9.0
ptyprocess==0.7.0
pure-eval==0.2.2
pycparser==2.21
Pygments==2.12.0
pyparsing==3.0.8
pyrsistent==0.18.1
python-dateutil==2.8.2
pyzmq==22.3.0
qtconsole==5.3.0
QtPy==2.0.1
Send2Trash==1.8.0
six==1.16.0
soupsieve==2.3.2.post1
stack-data==0.2.0
terminado==0.13.3
tinycss2==1.1.1
tornado==6.1
traitlets==5.1.1
wcwidth==0.2.5
webencodings==0.5.1
widgetsnbextension==3.6.0

[cderv]

You could try running the ipynb generated from the Qmd in jupyter by using quarto convert to convert from .qmd to .ipynb.
Test here should be about the bash engine. We need more information about the error or I need to reproduce;

[aborruso]

Can you update Jupyter and bash_kernel too ?

I have the last bash kernel and these jupyter "things"

jupyter==1.0.0
jupyter-cache==0.4.3
jupyter-client==7.3.0
jupyter-console==6.4.3
jupyter-core==4.10.0
jupyter-server==1.2.2
jupyter-server-mathjax==0.2.3
jupyter-sphinx==0.3.2
jupyter-telemetry==0.1.0
jupyterhub==2.2.2
jupyterlab==3.0.7
jupyterlab-pygments==0.2.2
jupyterlab-server==2.1.3
jupyterlab-widgets==1.1.0
jupytext==1.11.5

But I have always that error. My problem is that I'm not able to have a verbose error :(

Running it in WSL in RStudio server I have this

[aborruso]

You could try running the ipynb generated from the Qmd in jupyter by using quarto convert to convert from .qmd to .ipynb.
Test here should be about the bash engine. We need more information about the error or I need to reproduce;

The converted notebook seems to work

[cderv]

Running it in WSL in RStudio server I have this

It does not look the same as my Quarto output:

I don't think yours is correct.

When you click Render you should have the Job pane opening which similar content

Starting Jupyter kernel...Done

Executing 'test.ipynb'
  Cell 1/1...Done

pandoc 
  to: html
  output-file: test.html
  standalone: true
  section-divs: true
  html-math-method: mathjax
  wrap: none
  default-image-extension: png
  filters:
    - crossref
  
metadata
  document-css: false
  link-citations: true
  lang: en
  title: bash demo
  jupyter: bash
  
Output created: test.html

Watching files for changes

Be sure to have RStudio IDE updated also. Using last daily is best with Quarto.

There is something not right currently with your environment that prevent us to reproduce.

[jjallaire]

For the use case of running bash commands within Python cells (w/ magics, etc.) we now have a fix that will automatically turn off daemon mode if bash commands are detected: 537cf3c

[bvancil]

I have a related question that surfaces on Windows (not under WSL) with Quarto 0.9.611. Here's the code. Note the lowercase {sas}.

---
title: "sas_kernel test"
format:
  html:
    code-fold: true
jupyter: sas
execute:
  enabled: true
---

```{sas}
data BPressure;
    length PatientID $2;
    input PatientID $ Systolic Diastolic @@;
    datalines;
CK 120 50  SS 96  60 FR 100 70
CP 120 75  BL 140 90 ES 120 70
CP 165 110 JI 110 40 MC 119 66
FC 125 76  RW 133 60 KD 108 54
DS 110 50  JW 130 80 BH 120 65
JW 134 80  SB 118 76 NS 122 78
GS 122 70  AB 122 78 EC 112 62
HH 122 82
;

title 'Systolic and Diastolic Blood Pressure';
ods select BasicMeasures Quantiles;
proc univariate data=BPressure;
    var Systolic Diastolic;
run;
REMOVEME```

Then quarto render sas_kernel.qmd --execute-debug yields:

ERROR: SyntaxError: Invalid regular expression: /^*\s*\| ?/: Nothing to repeat

If I change the lowercase {sas} to {SAS}, running the same command yields:

- Connecting to kernel

Starting sas kernel...2022-06-22 10:35:25,222 - root - Level 25 - starting notebook server subprocess
- Kernel connection successful
- Sending execute command to kernel
- Execute command sent, reading response
Done

Executing 'sas_kernel.ipynb'

- Server request complete


ERROR: SyntaxError: Invalid regular expression: /^*\s*\| ?/: Nothing to repeat

Here's the output of quarto capabilities:

{
  "formats": [
    "html",
    "pdf",
    "docx",
    "odt",
    "pptx",
    "beamer",
    "revealjs",
    "gfm",
    "hugo",
    "epub",
    "asciidoc",
    "asciidoctor",
    "commonmark",
    "commonmark_x",
    "context",
    "docbook",
    "docbook4",
    "docbook5",
    "dokuwiki",
    "dzslides",
    "epub2",
    "epub3",
    "fb2",
    "haddock",
    "html4",
    "html5",
    "icml",
    "ipynb",
    "jats",
    "jats_archiving",
    "jats_articleauthoring",
    "jats_publishing",
    "jira",
    "json",
    "latex",
    "man",
    "markdown",
    "markdown_github",
    "markdown_mmd",
    "markdown_phpextra",
    "markdown_strict",
    "markua",
    "mediawiki",
    "ms",
    "muse",
    "native",
    "opendocument",
    "opml",
    "org",
    "plain",
    "rst",
    "rtf",
    "s5",
    "slideous",
    "slidy",
    "tei",
    "texinfo",
    "textile",
    "xwiki",
    "zimwiki"
  ],
  "themes": [
    "default",
    "cerulean",
    "cosmo",
    "cyborg",
    "darkly",
    "flatly",
    "journal",
    "litera",
    "lumen",
    "lux",
    "materia",
    "minty",
    "morph",
    "pulse",
    "quartz",
    "sandstone",
    "simplex",
    "sketchy",
    "slate",
    "solar",
    "spacelab",
    "superhero",
    "united",
    "vapor",
    "yeti",
    "zephyr"
  ],
  "python": {
    "versionMajor": 3,
    "versionMinor": 9,
    "versionPatch": 9,
    "versionStr": "3.9.9 (tags/v3.9.9:ccb0e6a, Nov 15 2021, 18:08:50) [MSC v.1929 64 bit (AMD64)]",
    "conda": false,
    "execPrefix": "C:/Users/bvancil/.virtualenvs/sas_kernel-xT3AzAu4-python",
    "executable": "C:/Users/bvancil/.virtualenvs/sas_kernel-xT3AzAu4-python/Scripts/python.exe",
    "jupyter_core": "4.10.0",
    "nbformat": "5.4.0",
    "nbclient": "0.6.4",
    "ipykernel": "6.15.0",
    "pyLauncher": true,
    "kernels": [
      {
        "name": "julia-1.7",
        "language": "julia",
        "display_name": "Julia 1.7.3"
      },
      {
        "name": "sas",
        "language": "sas",
        "display_name": "SAS"
      },
      {
        "name": "python3",
        "language": "python",
        "display_name": "Python 3 (ipykernel)"
      }
    ],
    "venv": true
  }
}

I realize this is hard to replicate without a working SAS installation, and I'm pretty sure the regexp error is unrelated to the Bash problem outlined above, but I noticed the similarity with behavior between language and display_name here. Is there another not officially supported but open source Jupyter kernel with a similar case difference that someone might recommend? I'd be happy to follow up by trying an open-source kernel, I'm just forced to use SAS by my employer.

I also noticed a difference in the converted ipynb files. I'm attaching my original and the one converted by quarto.
sas_kernel.zip

[jjallaire]

I think the problem is not related to case but rather to how we are handling SAS option comments (specifically we are not escaping the * when we build the regex to look for comments). Fix for this is here: ebfcb48

Fix is available in this build: https://github.com/quarto-dev/quarto-cli/releases/tag/v0.9.613

Note that option comments need to be valid SAS comment lines including the trailing ;, e.g.

```{sas}
*| echo: false ;

```

[bvancil]

You are right on both counts. Using uppercase {SAS} just included the code verbatim, whereas {sas} executed it. It only looked like the uppercase version got further in the process. With version 0.9.613, everything works great. Thank you so much, and I apologize for SAS's funky comments!