Feature augmented slide overview by beniroquai · Pull Request #234 · openUC2/ImSwitch

beniroquai · 2026-02-27T20:19:05Z

This pull request introduces a new "Overview Camera Registration" feature to the frontend, allowing users to register overview camera images and overlay them onto the well selector canvas. It adds a Redux slice to manage the wizard and overlay state, backend API integration for registration and overlay data, and new UI controls for overlay visibility and opacity.

Overview Camera Registration & Overlay Feature:

Added a new Redux slice OverviewRegistrationSlice.js to manage wizard state, overlay visibility, opacity, and per-slide registration data.
Integrated backend APIs for overview registration and overlay management, including endpoints for fetching overlay data, registration config/status, registering slides, refreshing overlay images, and snapping overview images. [1] [2] [3] [4] [5] [6]

UI Enhancements:

Updated WellSelectorComponent.js to include controls for opening the registration wizard, toggling overlay visibility, and adjusting overlay opacity. Overlay data is loaded on demand. [1] [2]
Added OverviewRegistrationWizard component and related imports for the registration workflow.

Canvas Rendering:

Modified WellSelectorCanvas.js to draw overview camera overlay images on the canvas, using cached HTML Image objects and the overlay state from Redux. Overlay respects opacity and visibility settings. [1] [2] [3] [4]

State Management Integration:

Registered the new overviewRegistrationState in the Redux store for global access and persistence. [1] [2]

Backend Model Import:

Imported OverviewRegistrationService and related types in the backend controller, preparing for registration service integration.

Introduce an Overview Overlay registration feature: adds a new React wizard (OverviewRegistrationWizard) for snapping overview camera images, picking slide corner points, and saving per-slide registrations. Adds Redux state slice (OverviewRegistrationSlice) and connects it in the store; provides frontend API wrappers for snapshot/registration/overlay endpoints. Integrates overlay rendering and controls into WellSelector (canvas drawing, opacity toggle, wizard entry) and preloads overlay images for display. Extends backend ExperimentController to initialize an OverviewRegistrationService, expose endpoints (config, snap, register, refresh, overlay data/status), and save/serve snapshot and registration data. Enables manual registration workflow so overlays can be generated and shown on the WellSelector canvas.

Make layout/slot serialization and overlay generation more robust: handle missing layouts with a clear error, prefer Pydantic v2 APIs (model_dump / model_dump_json) falling back to dict/json for compatibility, and use model_dump for slots when available. Replace fixed px_per_um scaling with aspect-preserving sizing capped by MAX_OVERLAY_PX (keeps overlays reasonably small: 1024 in one path, 2048 in another) and enforce a minimum size, since the frontend stretches overlays to fit. Also keep the existing fallback Heidstar layout behavior.

Copilot

Pull request overview

Adds an “Overview Camera Registration” workflow that lets users snap overview images, manually pick slide corners to compute per-slot homographies, and render registered overview overlays on the WellSelector canvas.

Changes:

Backend: introduced an OverviewRegistrationService with persistence, registration, overlay refresh, and overlay-data endpoints exposed via ExperimentController.
Frontend: added a Redux slice plus API helpers, a multi-step registration wizard, and WellSelector controls to toggle overlay visibility/opacity.
Canvas: renders per-slot overlay images with simple caching and opacity support.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
imswitch/imcontrol/model/overview_registration.py	New backend service + models for homography registration, persistence, overlay warping, and overlay-data aggregation
imswitch/imcontrol/controller/controllers/ExperimentController.py	Wires service into controller and exposes registration/snapshot/overlay endpoints
frontend/src/state/store.js	Registers `overviewRegistrationState` reducer in the Redux store
frontend/src/state/slices/OverviewRegistrationSlice.js	New Redux slice for wizard state + overlay visibility/opacity/data
frontend/src/backendapi/apiSnapOverviewImage.js	API helper for snapping overview images
frontend/src/backendapi/apiRegisterOverviewSlide.js	API helper for slide registration submission
frontend/src/backendapi/apiRefreshOverviewSlideImage.js	API helper for refreshing a slot overlay using existing registration
frontend/src/backendapi/apiGetOverviewRegistrationStatus.js	API helper for fetching registration status
frontend/src/backendapi/apiGetOverviewRegistrationConfig.js	API helper for fetching wizard config + slot definitions
frontend/src/backendapi/apiGetOverviewOverlayData.js	API helper for fetching overlay data/images
frontend/src/axon/WellSelectorComponent.js	Adds overlay controls and mounts the registration wizard
frontend/src/axon/WellSelectorCanvas.js	Draws overlay images onto the canvas with opacity and caching
frontend/src/axon/OverviewRegistrationWizard.js	New multi-step UI for selecting slots, snapping, corner picking, and enabling overlay

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-02-27T20:24:20Z

frontend/src/axon/WellSelectorCanvas.js

+      const bounds = slideData.stageBounds;
+      const imgKey = slideData.slotId + "_" + (slideData.updatedAt || "");
+
+      // Check if image is already cached
+      let cachedImg = overlayImagesRef.current[imgKey];
+      if (!cachedImg) {
+        // Load image and cache it
+        const img = new Image();
+        img.src = `data:${slideData.imageMimeType || "image/png"};base64,${slideData.imageBase64}`;
+        img.onload = () => {
+          overlayImagesRef.current[imgKey] = img;
+          // Trigger re-render by requesting animation frame
+          requestAnimationFrame(() => renderCanvas());


The overlay image cache key includes updatedAt (slotId + "_" + updatedAt), but old cache entries are never removed. Over time (especially with repeated refreshes) this will grow overlayImagesRef.current unbounded. Consider caching per slotId and replacing the entry when updatedAt changes, or pruning keys that are no longer present in overlayData.slides.

Copilot · 2026-02-27T20:24:21Z

frontend/src/axon/OverviewRegistrationWizard.js

+  IconButton,
+  Chip,
+  LinearProgress,
+  Slider,


Slider is imported but never used. With the project’s CRA/ESLint setup, this will raise a no-unused-vars lint error and can fail the frontend build. Remove the import (or use it if intended).

Suggested change

Slider,

Copilot · 2026-02-27T20:24:21Z

frontend/src/axon/OverviewRegistrationWizard.js

+const OverviewRegistrationWizard = () => {
+  const dispatch = useDispatch();
+  const canvasRef = useRef(null);
+  const imgRef = useRef(null);


imgRef is declared but never used. This triggers no-unused-vars under the current ESLint config and may break the frontend build. Remove the ref (or wire it up where needed).

Suggested change

const imgRef = useRef(null);

Copilot · 2026-02-27T20:24:21Z

imswitch/imcontrol/model/overview_registration.py

+    warped = cv2.warpPerspective(
+        image, H, (output_width, output_height),
+        flags=cv2.INTER_LINEAR,
+        borderMode=cv2.BORDER_CONSTANT,
+        borderValue=(0, 0, 0, 0),
+    )


warp_slide_image() claims it supports BGR/grayscale input, but it calls cv2.warpPerspective with a 4-channel borderValue=(0,0,0,0). OpenCV expects the borderValue tuple length to match the image channel count, so this can break if the function is used with 1- or 3-channel images. Either ensure the input is converted to BGRA inside this helper (and update the docstring), or set borderValue conditionally based on the number of channels.

Copilot · 2026-02-27T20:24:21Z

imswitch/imcontrol/model/overview_registration.py

+        # Compute output size: cap at MAX_OVERLAY_PX, frontend stretches to fill slot area
+        MAX_OVERLAY_PX = 2048
+        tl = slot_stage_corners[0]
+        br = slot_stage_corners[2]
+        stage_w = abs(br.x - tl.x)
+        stage_h = abs(br.y - tl.y)
+        aspect = stage_w / stage_h if stage_h > 0 else 1.0
+        if stage_w >= stage_h:
+            out_w = MAX_OVERLAY_PX
+            out_h = max(100, int(MAX_OVERLAY_PX / aspect))
+        else:
+            out_h = MAX_OVERLAY_PX
+            out_w = max(100, int(MAX_OVERLAY_PX * aspect))


register_slide() caps overlay images at 1024 px (MAX_OVERLAY_PX = 1024), but refresh_overlay_image() uses 2048 px. This means a refreshed overlay will have a different resolution than the initially registered one, which can cause inconsistent quality and larger payloads. Consider using a single shared constant (or a parameter) for both paths.

Copilot · 2026-02-27T20:24:22Z

imswitch/imcontrol/model/overview_registration.py

+    def get_overlay_data(self, camera_name: str, layout_name: str) -> dict:
+        """
+        Return all overlay data needed by the frontend WellSelector canvas.
+        Each completed slide includes base64-encoded PNG overlay + slot bounds.
+        """
+        key = self._store_key(camera_name, layout_name)
+        store = self._stores.get(key)
+        result = {
+            "cameraName": camera_name,
+            "layoutName": layout_name,
+            "slides": {},
+        }
+        if store is None:
+            return result
+
+        sdir = self._store_dir(camera_name, layout_name)
+
+        for sid, reg in store.slides.items():
+            if not reg.overlayImageRef:
+                continue
+            overlay_path = os.path.join(sdir, reg.overlayImageRef)
+            if not os.path.isfile(overlay_path):
+                continue
+
+            # Read and base64-encode the overlay image
+            with open(overlay_path, "rb") as f:
+                img_bytes = f.read()
+            b64 = base64.b64encode(img_bytes).decode("ascii")


get_overlay_data() base64-encodes full PNGs into the JSON response. Even with only a few slides this can produce multi-megabyte payloads, increase memory pressure on both backend and frontend, and make overlay refreshes slow. Consider returning lightweight metadata + a URL/endpoint to fetch each overlay image (with HTTP caching) instead of embedding the image bytes in JSON.

Copilot · 2026-02-27T20:24:22Z

frontend/src/axon/WellSelectorCanvas.js

+      const imgKey = slideData.slotId + "_" + (slideData.updatedAt || "");
+
+      // Check if image is already cached
+      let cachedImg = overlayImagesRef.current[imgKey];
+      if (!cachedImg) {
+        // Load image and cache it
+        const img = new Image();
+        img.src = `data:${slideData.imageMimeType || "image/png"};base64,${slideData.imageBase64}`;
+        img.onload = () => {
+          overlayImagesRef.current[imgKey] = img;
+          // Trigger re-render by requesting animation frame
+          requestAnimationFrame(() => renderCanvas());
+        };
+        return; // Skip this slide until image is loaded


drawOverviewOverlay() creates a new Image() on every render until onload fires because the image isn’t placed into the cache until after load. If renderCanvas() is triggered frequently, this can start multiple duplicate loads for the same slide. Cache the Image object immediately (or track a "loading" sentinel) so subsequent renders reuse the in-flight request.

beniroquai added 3 commits February 26, 2026 22:53

Copilot AI review requested due to automatic review settings February 27, 2026 20:19

Copilot started reviewing on behalf of beniroquai February 27, 2026 20:19 View session

Copilot AI reviewed Feb 27, 2026

View reviewed changes

beniroquai merged commit ba2f0ae into master Feb 27, 2026
17 checks passed

beniroquai deleted the feature-augmented-slideoverview branch February 27, 2026 20:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature augmented slide overview#234

Feature augmented slide overview#234
beniroquai merged 3 commits intomasterfrom
feature-augmented-slideoverview

beniroquai commented Feb 27, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Feb 27, 2026

Uh oh!

Copilot AI Feb 27, 2026

Uh oh!

Copilot AI Feb 27, 2026

Uh oh!

Copilot AI Feb 27, 2026

Uh oh!

Copilot AI Feb 27, 2026

Uh oh!

Copilot AI Feb 27, 2026

Uh oh!

Copilot AI Feb 27, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

beniroquai commented Feb 27, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants