Can't access many notebook types

[tdudgeon]

Only the minimal notebook seems to work at the moment.
I'm sure other types have worked in the past, but now when you try to access the Jupyter Lab app you get a 502 Bad Gateway from nginx.

The problem might be that the Jupyter Hub server version has changed. The one used by the jupyter/minimal-notebook image claims to be 1.17.1 whilst that run by jupyter/tensorflow-notebook is 1.18.1. In the pod logs there are message like this:

[W 2022-07-19 15:24:09.076 LabApp] 'token' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2022-07-19 15:24:09.076 LabApp] 'base_url' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.

and there are more of these from the 1.18.1 server, but these are messages not errors, so not sure whether the server version is really the problem (though the way the container is launched and the Ingress created are the same for ll notebook images).

A core part of the problem is that we are not using tagged images so latest is evolving over time.

[tdudgeon]

Further to that description, the behaviour when you try to curl the pod directly from within the cluster is different. The jupyter/minimal-notebook instances will respond to http:://<pod-id>:8888/<dm-instance-id>/lab?<token> whereas the jupyter/tensorflow-notebook does not respond.

I haven't been able to work out what is wrong as nothing useful seems to be logged.

[tdudgeon]

Seems that the configuration of the available notebooks is defined in the DM API here.
Looks like the value that is defined there (e.g. tensorflow) is transformed into a image name (e.g. jupyter/tensorflow-notebook), presumably by prepending jupyter/ and appending -notebook.

This does not allow an image tag to be specified and only allows to use notebooks conforming to that pattern.

We need to be able to support multiple versions of each type of notebook, and probably to allow his to be defined by the system administrator (e.g. as a ConfigMap).

For instance, for the tensorflow notebooks we should allow the use to choose between different versions e.g. https://hub.docker.com/r/jupyter/tensorflow-notebook/tags?page=1&name=tensorflow-

Probably we need to map a human readable name (Tensorflow 2.8.1) to a image name (jupyter/tensorflow-notebook:tensorflow-2.8.1).

This won't fix the current problem, but would allow to test whether it is caused by an incompatible update to the container image.

It might also be useful to somehow allow an admin to somehow specify a complete container image name (with tag) to allow different notebook images to be tested prior to adding them to the selectable list.

[tdudgeon]

This seems to be the result of a misconfiguration of the newer notebook images.
See jupyter/docker-stacks#1760

[tdudgeon]

In fact it's probably this problem: jupyter/docker-stacks#1761