Testing if the notebook runs fine with pyodide-kernel

[kp992]

I am trying to see if I can automate the testing of .ipynb notebooks after they are served using jupyter-lite. The problem is given a python notebook, the script should output whether the execution completed successfully or not. When I installed using

pip install jupyterlite-pyodide-kernel

I can't see the kernel in the kernel list:

% jupyter kernelspec list
Available kernels:
  python3                     /Users/kpl/mambaforge/envs/qe/share/jupyter/kernels/python3
  xpython-raw                 /Users/kpl/mambaforge/envs/qe/share/jupyter/kernels/xpython-raw

How do I test the notebooks or how do I fix the installation to see this in kernel list.

[agriyakhetarpal]

A workaround, for now, could be in this form (if you use the Pyodide kernel):

# convert the notebook to a Python file
python -m pip install jupytext
python -m jupytext --to py your_notebook.ipynb # or py:percent if you like

# install Pyodide and activate
python -m pip install pyodide-build
pyodide xbuildenv install 0.27.0 # or your preferred Pyodide version
pyodide venv .venv-pyodide
source .venv-pyodide/bin/activate

# run the notebook as a Python file
python my_notebook.py

As a more robust solution, maybe it can be possible to run the notebook as one usually does – it would require making the Pyodide/Xeus kernels visible in the kernel list and making it possible for notebook testing tools like papermill or nbmake to connect to them.

[kp992]

Thank you. That did work till some extent. I not sure how to install packages like numpy? I am getting this error:

% python  
Python 3.11.3 (main, Mar 31 2024, 11:27:51) [Clang 18.0.0 (https://github.com/llvm/llvm-project 75501f53624de92aafce2f1da698 on emscripten
Type "help", "copyright", "credits" or "license" for more information.
>>> import micropip
>>> await micropip.install('regex') 
  File "<stdin>", line 1
SyntaxError: 'await' outside function
>>> micropip.install('regex') 
<coroutine object install at 0x15c06c0>
>>> import regex
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: The module 'regex' is included in the Pyodide distribution, but it is not installed.
You can install it by calling:
  await micropip.install("regex") in Python, or
  await pyodide.loadPackage("regex") in JavaScript
See https://pyodide.org/en/stable/usage/loading-packages.html for more details.

[agriyakhetarpal]

Ah – micropip implements a command-line interface that's highly similar to pip, so, in an activated Pyodide venv, you should be able to run pip install numpy to install NumPy. The await micropip.install(<...>) syntax is more relevant for running in the context of a browser.

[kp992]

Thanks @agriyakhetarpal for the response. Executing pip install on CLI does work, but is numba supported by pyodide?

I don't think we can install numba because of missing pure python wheels? Please correct me if I am wrong but I was able to install numba.

[agriyakhetarpal]

I don't think we can install numba because of missing pure python wheels?

Yes, @kp992, numba isn't supported yet in Pyodide. xref: pyodide/pyodide-recipes#192

Please correct me if I am wrong but I was able to install numba.

I think you mistyped or missed a word here – "(not) able to install numba"?

[kp992]

No, I was able to install.

[agriyakhetarpal]

Well, we certainly haven't compiled or added numba :)

Error trace

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[1], line 1
----> 1 await __import__("piplite").install(**{'requirements': ['numba']})

File /lib/python3.12/site-packages/piplite/piplite.py:117, in _install(requirements, keep_going, deps, credentials, pre, index_urls, verbose)
    115 """Invoke micropip.install with a patch to get data from local indexes"""
    116 with patch("micropip.package_index.query_package", _query_package):
--> 117     return await micropip.install(
    118         requirements=requirements,
    119         keep_going=keep_going,
    120         deps=deps,
    121         credentials=credentials,
    122         pre=pre,
    123         index_urls=index_urls,
    124         verbose=verbose,
    125     )

File /lib/python3.12/site-packages/micropip/_commands/install.py:142, in install(requirements, keep_going, deps, credentials, pre, index_urls, verbose)
    130     index_urls = package_index.INDEX_URLS[:]
    132 transaction = Transaction(
    133     ctx=ctx,
    134     ctx_extras=[],
   (...)
    140     index_urls=index_urls,
    141 )
--> 142 await transaction.gather_requirements(requirements)
    144 if transaction.failed:
    145     failed_requirements = ", ".join([f"'{req}'" for req in transaction.failed])

File /lib/python3.12/site-packages/micropip/transaction.py:55, in Transaction.gather_requirements(self, requirements)
     52 for requirement in requirements:
     53     requirement_promises.append(self.add_requirement(requirement))
---> 55 await asyncio.gather(*requirement_promises)

File /lib/python3.12/site-packages/micropip/transaction.py:62, in Transaction.add_requirement(self, req)
     59     return await self.add_requirement_inner(req)
     61 if not urlparse(req).path.endswith(".whl"):
---> 62     return await self.add_requirement_inner(Requirement(req))
     64 # custom download location
     65 wheel = WheelInfo.from_url(req)

File /lib/python3.12/site-packages/micropip/transaction.py:151, in Transaction.add_requirement_inner(self, req)
    148     if self._add_requirement_from_pyodide_lock(req):
    149         return
--> 151     await self._add_requirement_from_package_index(req)
    152 else:
    153     try:

File /lib/python3.12/site-packages/micropip/transaction.py:190, in Transaction._add_requirement_from_package_index(self, req)
    182 """
    183 Find requirement from package index. If the requirement is found,
    184 add it to the package list and return True. Otherwise, return False.
    185 """
    186 metadata = await package_index.query_package(
    187     req.name, self.fetch_kwargs, index_urls=self.index_urls
    188 )
--> 190 wheel = find_wheel(metadata, req)
    192 # Maybe while we were downloading pypi_json some other branch
    193 # installed the wheel?
    194 satisfied, ver = self.check_version_satisfied(req)

File /lib/python3.12/site-packages/micropip/transaction.py:281, in find_wheel(metadata, req)
    278     if best_wheel is not None:
    279         return wheel
--> 281 raise ValueError(
    282     f"Can't find a pure Python 3 wheel for '{req}'.\n"
    283     f"See: {FAQ_URLS['cant_find_wheel']}\n"
    284     "You can use `await micropip.install(..., keep_going=True)` "
    285     "to get a list of all packages with missing wheels."
    286 )

ValueError: Can't find a pure Python 3 wheel for 'numba'.
See: https://pyodide.org/en/stable/usage/faq.html#why-can-t-micropip-find-a-pure-python-wheel-for-a-package
You can use `await micropip.install(..., keep_going=True)` to get a list of all packages with missing wheels.

[kp992]

Hmm, I got this:

(.venv-pyodide) (qe) kpl@kpl-MacBook-Air book % python -m pip install numba    
Looking in indexes: https://pypi.org/simple, file:///Users/kpl/repos/intro-wasm-demo/.pyodide-xbuildenv/xbuildenv/pyodide-root/pypa_index
WARNING: Location 'file:/Users/kpl/repos/intro-wasm-demo/.pyodide-xbuildenv/xbuildenv/pyodide-root/pypa_index/numba/' is ignored: it is neither a file nor a directory.
Collecting numba
  Using cached numba-0.60.0-cp311-cp311-macosx_11_0_arm64.whl.metadata (2.7 kB)
Requirement already satisfied: llvmlite<0.44,>=0.43.0dev0 in /Users/kpl/repos/intro-wasm-demo/.venv-pyodide/lib/python3.11/site-packages (from numba) (0.43.0)
Requirement already satisfied: numpy<2.1,>=1.22 in /Users/kpl/repos/intro-wasm-demo/.venv-pyodide/lib/python3.11/site-packages (from numba) (1.26.4)
Using cached numba-0.60.0-cp311-cp311-macosx_11_0_arm64.whl (2.6 MB)
Installing collected packages: numba
Successfully installed numba-0.60.0

[agriyakhetarpal]

Right, that means that the virtual environment was probably not activated correctly, or there's a bug on our side (slightly less likely), as it downloaded the macOS wheel in your case:

Using cached numba-0.60.0-cp311-cp311-macosx_11_0_arm64.whl (2.6 MB)
Installing collected packages: numba

A wheel for Pyodide/WASM would have had the filename as numba-0.60.0-cp312-cp312-pyodide_2024_0_wasm32.whl

Could you please recreate the environment and try again, @kp992? I am unable to reproduce:

agriyakhetarpal@Agriyas-MacBook-Pro ~ % pyodide venv .venv-pyodide
Downloading Pyodide cross-build environment from
https://github.com/pyodide/pyodide/releases/download/0.27.0/xbuildenv-0.27.0.tar.bz2                                    
Installing Pyodide cross-build environment                                                                              
Using Pyodide cross-build environment version: 0.27.0                                                                   
Creating Pyodide virtualenv at .venv-pyodide                                                                            
... Configuring virtualenv                                                                                              
... Installing standard library                                                                                         
Successfully created Pyodide virtual environment!                                                                       
(venv) (base) agriyakhetarpal@Agriyas-MacBook-Pro ~ % source .venv-pyodide/bin/activate
(.venv-pyodide) (base) agriyakhetarpal@Agriyas-MacBook-Pro ~ % which python
/Users/agriyakhetarpal/.venv-pyodide/bin/python
(.venv-pyodide) (base) agriyakhetarpal@Agriyas-MacBook-Pro ~ % which pip 
/Users/agriyakhetarpal/.venv-pyodide/bin/pip
(.venv-pyodide) (base) agriyakhetarpal@Agriyas-MacBook-Pro ~ % python -m pip --version
pip 24.3.1 from /Users/agriyakhetarpal/.venv-pyodide/lib/python3.12/site-packages/pip (python 3.12)
(.venv-pyodide) (base) agriyakhetarpal@Agriyas-MacBook-Pro ~ % pip install numba
Looking in indexes: https://pypi.org/simple, file:///Users/agriyakhetarpal/.pyodide-xbuildenv-0.29.2/0.27.0/xbuildenv/pyodide-root/package_index
WARNING: Location 'file:/Users/agriyakhetarpal/.pyodide-xbuildenv-0.29.2/0.27.0/xbuildenv/pyodide-root/package_index/numba/' is ignored: it is neither a file nor a directory.
ERROR: Could not find a version that satisfies the requirement numba (from versions: none)
WARNING: Location 'file:/Users/agriyakhetarpal/.pyodide-xbuildenv-0.29.2/0.27.0/xbuildenv/pyodide-root/package_index/pip/' is ignored: it is neither a file nor a directory.
ERROR: No matching distribution found for numba
(.venv-pyodide) (base) agriyakhetarpal@Agriyas-MacBook-Pro ~ % python -m pip install numba
Looking in indexes: https://pypi.org/simple, file:///Users/agriyakhetarpal/.pyodide-xbuildenv-0.29.2/0.27.0/xbuildenv/pyodide-root/package_index
WARNING: Location 'file:/Users/agriyakhetarpal/.pyodide-xbuildenv-0.29.2/0.27.0/xbuildenv/pyodide-root/package_index/numba/' is ignored: it is neither a file nor a directory.
ERROR: Could not find a version that satisfies the requirement numba (from versions: none)
ERROR: No matching distribution found for numba

[kp992]

Thanks @agriyakhetarpal for looking into it. Here is the trace from my terminal

(qe) kpl@kpl-MacBook-Air intro-wasm-demo % pyodide venv .venv-pyodide
xbuild environment already exists, skipping download                            
Installing xbuild environment                                                   
Creating Pyodide virtualenv at .venv-pyodide                                    
... Configuring virtualenv                                                      
... Installing standard library                                                 
Successfully created Pyodide virtual environment!                               
(qe) kpl@kpl-MacBook-Air intro-wasm-demo % source .venv-pyodide/bin/activate
(.venv-pyodide) (qe) kpl@kpl-MacBook-Air intro-wasm-demo % which python
/Users/kpl/repos/intro-wasm-demo/.venv-pyodide/bin/python
(.venv-pyodide) (qe) kpl@kpl-MacBook-Air intro-wasm-demo % which pip
/Users/kpl/repos/intro-wasm-demo/.venv-pyodide/bin/pip
(.venv-pyodide) (qe) kpl@kpl-MacBook-Air intro-wasm-demo % python -m pip --version
pip 24.3.1 from /Users/kpl/repos/intro-wasm-demo/.venv-pyodide/lib/python3.11/site-packages/pip (python 3.11)
(.venv-pyodide) (qe) kpl@kpl-MacBook-Air intro-wasm-demo % pip install numba
Looking in indexes: https://pypi.org/simple, file:///Users/kpl/repos/intro-wasm-demo/.pyodide-xbuildenv/xbuildenv/pyodide-root/pypa_index
WARNING: Location 'file:/Users/kpl/repos/intro-wasm-demo/.pyodide-xbuildenv/xbuildenv/pyodide-root/pypa_index/numba/' is ignored: it is neither a file nor a directory.
Collecting numba
  Using cached numba-0.60.0-cp311-cp311-macosx_11_0_arm64.whl.metadata (2.7 kB)
WARNING: Location 'file:/Users/kpl/repos/intro-wasm-demo/.pyodide-xbuildenv/xbuildenv/pyodide-root/pypa_index/llvmlite/' is ignored: it is neither a file nor a directory.
Collecting llvmlite<0.44,>=0.43.0dev0 (from numba)
  Using cached llvmlite-0.43.0-cp311-cp311-macosx_11_0_arm64.whl.metadata (4.8 kB)
Collecting numpy<2.1,>=1.22 (from numba)
  Downloading numpy-2.0.2-cp311-cp311-macosx_11_0_arm64.whl.metadata (60 kB)
Using cached numba-0.60.0-cp311-cp311-macosx_11_0_arm64.whl (2.6 MB)
Using cached llvmlite-0.43.0-cp311-cp311-macosx_11_0_arm64.whl (28.8 MB)
Downloading numpy-2.0.2-cp311-cp311-macosx_11_0_arm64.whl (13.7 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 13.7/13.7 MB 53.0 MB/s eta 0:00:00
Installing collected packages: numpy, llvmlite, numba
Successfully installed llvmlite-0.43.0 numba-0.60.0 numpy-2.0.2

[agriyakhetarpal]

Thanks for that, @kp992 – I think there's some issue with it here:

pip 24.3.1 from /Users/kpl/repos/intro-wasm-demo/.venv-pyodide/lib/python3.11/site-packages/pip (python 3.11)

We use Python 3.12 for Pyodide 0.26.0 and 0.27.0, and

pip install numba

is likely pointing to some other pip that the virtual environment fails to rule over, perhaps, and therefore installing the macOS arm64 wheels for numba. However, you used python -m pip in #145 (comment), so it's baffling to me. I tried again, and I'm unable to reproduce with either python -m pip or pip on my machine. Could you please open a bug report at https://github.com/pyodide/pyodide-build if the error persists after deleting both the Pyodide virtual environment and the xbuildenv folder?