PlanktoScope OS (Sunray) v2024.0.0-beta.0

[ethanjli]

The v2024.0.0 software release, which is part of the "Sunray" series of software releases started with v2023.9.0, includes major fixes for various problems and improvements to existing functionality with the PlanktoScope software.

About this pre-release

Starting with this beta pre-release, we are calling the software we provide in our SD card images the "PlanktoScope OS" (because it is an operating system for the PlanktoScope).

PlanktoScope OS v2024.0.0-beta.0 is a pre-release version for everyone to test out. It may have undiscovered software bugs, breakage, and other problems. If you decide to try out this pre-release, please file bug reports on GitHub Issues and/or let us know about any bugs you discover in the #6-dev-software channel on the PlanktoScope Slack workspace!

Contributors

The PlanktoScope OS is a community effort, and the v2024.0.0 release would not have been possible without the PlanktoScope community:

• Thanks to Wassim Chakroun (@W7CH) for code contributions in the new picamera2-based image-acquisition module!
• Thanks to (in alphabetical order) Ana Fernandez Carrera, Adam Larson (@babo989), Fabien Lombard (@fabienlombard), Kevin Du Clos (@kevin-duclos), Satoshi Kitajima (@kitazy52), Pierre Kostyrka (@pkostyrka), Tanguy Cebron, and Thibaut Pollina (@tpollina), for contributing feature requests, bug reports, and software testing efforts for this release!

Changelog since v2023.9.0

This changelog only lists items we think users are likely to notice or to need to know about; for a full list of changes, please refer to software/CHANGELOG.md.

Added:

• The PlanktoScope now shares internet access over its Wi-Fi hotspot and Ethernet port (and a USB-to-Ethernet adapter, if one is attached) from a phone connected to it in USB tethering mode.
• On the Node-RED dashboard, the System Monitoring page will now prompt you to reset the PlanktoScope's system clock to match the time from your web browser, if the two times are very different. This enables you to temporarily fix an inaccurate system clock on your PlanktoScope, at least until the next time you unplug the PlanktoScope from power.

Changed:

• The pscopehat version of the SD card image has been renamed to planktoscopehat for more consistent naming.
• On the planktoscopehat version of the SD card image, the default hardware configuration has been changed from the v2.5 PlanktoScope hardware to the v2.6 hardware. If you're upgrading the software on a PlanktoScope with v2.5 hardware, you will need to change the hardware configuration back to v2.5 on the "Hardware Settings" page of the Node-RED dashboard.
• The hardware controller's camera driver has been fully rewritten to fix various problems we had with raspimjpeg (which has now been replaced with a different camera control mechanism). This change may require you to use different ISO and white balance gains than before; accordingly, the Node-RED dashboard's ISO selector has been changed to a slider to enable finer adjustment of the ISO setting.
• Previously, the segmenter's default behavior was to subtract consecutive masks to try to mitigate image-processing issues with objects which get stuck to the flowcell during imaging. However, when different objects occupied the same space in consecutive frames, the subtraction behavior would subtract one object's mask from the mask of the other object in the following frame, which would produce clearly incorrect masks. This behavior is no longer enabled by default.
• The PlanktoScope OS is now built as a 64-bit OS, instead of a 32-bit OS. This change speeds up the segmenter, potentially by a factor of 2.
• On the Node-RED dashboard, the System Monitoring page’s monitoring panel has been replaced with an embedded Grafana dashboard, with additional graphs available in an expandable subpanel.

Deprecated:

• The current Node-RED dashboard (both the version for the Adafruit HAT and the version for the PlanktoScope HAT) is transitioning to maintenance mode: no new features will be added, and any bugs will be only be fixed if someone volunteers to fix them. The current Node-RED dashboard will be completely replaced by a fully-rewritten Node-RED dashboard, though there is no timeline for completion of that new dashboard. Currently, our plan for deprecating and eventually removing the current Node-RED dashboard is as follows: maintenance mode (no new features, only some bugfixes), then deprecation (no maintenance; not enabled by default, but still installed), then removal (not installed by default, but anyone is free to install it and see if it still works); deprecation will not occur before the rewritten Node-RED dashboard is stable for general-purpose usage. If you have concerns, please share your feedback on GitHub or on the PlanktoScope Slack.
• The "USB backup" functionality of the Node-RED dashboard will be removed in v2024.1.0 (the next release after v2024.0.0). Instead, you should use the datasets file browser (which you can access by a link on the PlanktoScope's landing page) for backing up and deleting dataset files on your PlanktoScope.
• The raspimjpeg-based imaging module in the Python hardware controller has not yet been deleted, so that you can change the Python hardware controller code to switch back from the new picamera2-based imaging module if picamera2 ends up causing big problems for you. However, we are deprecating the raspimjpeg-based imaging module, and we will fully delete it in a future release after we get sufficient information about the picamera2-based imaging module's reliability for PlanktoScope users.
• Portainer will no longer be installed/provided by default after v2024.0.0, because it takes up lots of disk space and it is difficult to set up and its functionality overlaps with the much simpler Dozzle. After we remove Portainer from the default SD card image, you can use Forklift to install Portainer.
• Now that the PlanktoScope project is building 64-bit versions of the PlanktoScope OS, 32-bit versions of the PlanktoScope OS are no longer built or officially supported by the project. However, for v2024.0.0 you can still use the installer scripts (with the PlanktoScope OS's non-standard installation process) to set up the PlanktoScope OS on 32-bit base images of the Raspberry Pi OS. After v2024.0.0 we may stop testing whether the installer scripts work on 32-bit Raspberry Pi OS images.

Fixed:

• The Node-RED dashboard’s “Sample” page’s Filtered Volume field is now exported as the sample_total_volume metadata field for all sample types; previously it was only exported for horizontal plankton tow samples.
• Startup is now faster.

Changelog since v2024.0.0-alpha.2

Changed

• The PlanktoScope OS is now built as a 64-bit OS. This change speeds up the segmenter, potentially by a factor of 2.

• The Node-RED dashboard's ISO selector has been changed to a slider to enable finer adjustment of the ISO setting.

Deprecated

• Now that the PlanktoScope project is building 64-bit versions of the PlanktoScope OS, 32-bit versions of the PlanktoScope OS are no longer built or officially supported by the project. However, for v2024.0.0 you can still use the installer scripts (with the PlanktoScope OS's non-standard installation process) to set up the PlanktoScope OS on 32-bit base images of the Raspberry Pi OS.

Fixed

• Because v2024.0.0-alpha.2 had fixed an incorrect formula for converting the ISO setting to an image gain for the camera hardware, in that pre-release it was possible (in the planktoscopehat build of the PlanktoScope OS) to use the Node-RED dashboard to attempt to set an ISO value (anything above 650) which would be rejected and ignored by the Python hardware controller because it resulted in an image gain value above the maximum allowed value. Now the planktoscopehat version of the Node-RED dashboard limits the ISO setting to 650.

Known Problems

• (Known as of v2024.0.0-beta.0) In some cases, changing the ISO setting may cause the camera preview to disappear. You can reload the camera preview by refreshing your web browser tab. We will fix this problem before the v2024.0.0 stable release.
• (Known as of v2023.9.0-beta.1) You may encounter weird behavior with the system time if you try to use Cockpit to manually adjust the system clock. If you experience this problem, please report it and your testing circumstances (e.g. your GPS situation, and the time you tried to change the system clock to), to help us determine why this problem is happening for you, so that we can fix it!
• (Known as of v2023.9.0-beta.0) In certain cases, some computers connected to both a Wi-Fi network providing internet access and to a PlanktoScope may be unable to access anything on the internet. If you experience this problem, please report it and your testing circumstances (e.g. your OS, your network configuration for internet access, etc.), to help us determine why this problem is happening for you, so that we can fix it!
• (Known as of v2023.9.0-beta.0) If you restart the Node-RED dashboard (e.g. via Cockpit), you will also need to manually restart the hardware controller, or else the Node-RED dashboard won’t know the type of camera installed in the PlanktoScope. We are working on a solution to this problem, for a future release after v2024.0.0.

Planning, management, and maintenance of the v2024.0.0 release of PlanktoScope OS is led by Ethan Li (@ethanjli), with input and oversight from attendees of the weekly PlanktoScope software development meetings. If this release causes new problems for how you are using PlanktoScope, Ethan will take responsibility for addressing any new issues caused by this release - just tag them in a new Github Discussions post or in the #6-dev-software channel on the PlanktoScope Slack workspace!

What's Changed

• build(deps): bump poethepoet from 0.25.0 to 0.26.1 in /documentation by @dependabot in build(deps): bump poethepoet from 0.25.0 to 0.26.1 in /documentation #406
• build(deps): bump mkdocs-material from 9.5.17 to 9.5.18 in /documentation by @dependabot in build(deps): bump mkdocs-material from 9.5.17 to 9.5.18 in /documentation #403
• build(deps): bump pymdown-extensions from 10.7.1 to 10.8.1 in /documentation by @dependabot in build(deps): bump pymdown-extensions from 10.7.1 to 10.8.1 in /documentation #405
• build(deps): bump mkdocs-git-revision-date-localized-plugin from 1.2.4 to 1.2.5 in /documentation by @dependabot in build(deps): bump mkdocs-git-revision-date-localized-plugin from 1.2.4 to 1.2.5 in /documentation #404
• build(deps): bump jinja2 from 3.1.3 to 3.1.4 in /documentation by @dependabot in build(deps): bump jinja2 from 3.1.3 to 3.1.4 in /documentation #407
• Remove FAQ item about PlanktoScope cost by @ethanjli in Remove FAQ item about PlanktoScope cost #409
• Bump mkdocs-material to fix CI build by @ethanjli in Bump mkdocs-material to fix CI build #410
• Deploy machine-name via Forklift with pallet-standard rather than the OS setup scripts by @ethanjli in Deploy machine-name via Forklift with pallet-standard rather than the OS setup scripts #411
• Try to fix docker/build-push-action in PRs from external forks by @ethanjli in Try to fix docker/build-push-action in PRs from external forks #413
• build(deps): bump requests from 2.31.0 to 2.32.0 in /documentation by @dependabot in build(deps): bump requests from 2.31.0 to 2.32.0 in /documentation #414
• Bump mkdocs-git-revision-date-localized-plugin from 1.2.5 to 1.2.6 in /documentation by @dependabot in Bump mkdocs-git-revision-date-localized-plugin from 1.2.5 to 1.2.6 in /documentation #417
• Bump caddy from 2.7.6 to 2.8.1 in /documentation by @dependabot in Bump caddy from 2.7.6 to 2.8.1 in /documentation #420
• Bump pygments from 2.17.2 to 2.18.0 in /documentation by @dependabot in Bump pygments from 2.17.2 to 2.18.0 in /documentation #419
• Bump mkdocs-glightbox from 0.3.7 to 0.4.0 in /documentation by @dependabot in Bump mkdocs-glightbox from 0.3.7 to 0.4.0 in /documentation #418
• Start building 64-bit OS images in GitHub Actions by @ethanjli in Start building 64-bit OS images in GitHub Actions #416
• Try to fix installer versioning in CI OS builds outside PRs by @ethanjli in Try to fix installer versioning in CI OS builds outside PRs #421
• Bump version numbers for v2024.0.0-beta.0 by @ethanjli in Bump version numbers for v2024.0.0-beta.0 #423

Full Changelog: software/v2024.0.0-alpha.2...software/v2024.0.0-beta.0

---

This discussion was created from the release Software: Sunray v2024.0.0-beta.0.