Feature/voxel annotation

[briossant]

This Draft Pull Request introduces an interactive voxel annotation feature, allowing users to perform manual segmentation by painting directly onto volumetric layers. This implementation is based on the proposal in Issue #851 and incorporates the feedback from @jbms.

Key Changes & Architectural Overview

Following the discussion, this implementation has been significantly revised from the initial prototype:

1. Integration with Existing Layers: Instead of a new vox layer type, the voxel editing functionality is now integrated directly into ImageUserLayer and SegmentationUserLayer via a UserLayerWithVoxelEditingMixin. This mixin adds a new "Draw" tab in the UI.

2. New Tool System: The Brush and Flood Fill tools are implemented as toggleable LayerTools, while the Picker tool is a one-shot tool. All integrate with Neuroglancer's new tool system. The drawing action is bound to Ctrl + Left Click.

3. Optimistic Preview for Compressed Chunks: To provide immediate visual feedback and solve the performance problem with compressed chunks, edits are now rendered through an optimistic preview layer.

• When a user paints, edits are first applied to an InMemoryVolumeChunkSource.
• This preview source is rendered by a second instance of the layer's primary RenderLayer (e.g., ImageRenderLayer or SegmentationRenderLayer). This ensures the preview perfectly matches the user's existing shader and display settings.
• The base data chunk is not modified on the frontend, avoiding the need to decompress/recompress it.

Data-flow

sequenceDiagram
participant User
participant Tool as VoxelBrushTool
participant ControllerFE as VoxelEditController (FE)
participant EditSourceFE as OverlayChunkSource (FE)
participant BaseSourceFE as VolumeChunkSource (FE)
participant ControllerBE as VoxelEditController (BE)
participant BaseSourceBE as VolumeChunkSource (BE)

    User->>Tool: Mouse Down/Drag
    Tool->>ControllerFE: paintBrushWithShape(mouse, ...)
    ControllerFE->>ControllerFE: Calculates affected voxels and chunks

    ControllerFE->>EditSourceFE: applyLocalEdits(chunkKeys, ...)
    activate EditSourceFE
    EditSourceFE->>EditSourceFE: Modifies its own in-memory chunk data
    note over EditSourceFE: This chunk's texture is re-uploaded to the GPU
    deactivate EditSourceFE

    ControllerFE->>ControllerBE: commitEdits(edits, ...) [RPC]

    activate ControllerBE
    ControllerBE->>ControllerBE: Debounces and batches edits
    ControllerBE->>BaseSourceBE: applyEdits(chunkKeys, ...)
    activate BaseSourceBE
    BaseSourceBE-->>ControllerBE: Returns VoxelChange (for undo stack)
    deactivate BaseSourceBE
    ControllerBE->>ControllerFE: callChunkReload(chunkKeys) [RPC]
    activate ControllerFE
    ControllerFE->>BaseSourceFE: invalidateChunks(chunkKeys)
    note over BaseSourceFE: BaseSourceFE re-fetches chunk with the now-permanent edit.
    ControllerFE->>EditSourceFE: clearOptimisticChunk(chunkKeys)
    deactivate ControllerFE

    ControllerBE->>ControllerBE: Pushes change to Undo Stack & enqueues for downsampling
    deactivate ControllerBE

    loop Downsampling & Reload Cascade
        ControllerBE->>ControllerBE: downsampleStep(chunkKeys)
        ControllerBE->>ControllerFE: callChunkReload(chunkKeys) [RPC]
        activate ControllerFE
        ControllerFE->>BaseSourceFE: invalidateChunks(chunkKeys)
        note over BaseSourceFE: BaseSourceFE re-fetches chunk with the now-permanent edit.
        ControllerFE->>EditSourceFE: clearOptimisticChunk(chunkKeys)
        deactivate ControllerFE
    end

Loading

4. Writable source selection Aside every activated volume sub-sources of a writable datasource, an additional checkbox lets the user mark the sub-source as writable, then neuroglancer will try to write in it.

Data sources & Kvstores

Currently, there is a very limited set of supported data sources and kvstores, which are:

• datasources:
  • zarr with no compression
• kvstores:
  • s3+http: mostly used for local development, paired with minio
  • opfs: in-browser storage, also used for local development at some point, the relevancy can be discussed.
  • ssa+https: a kvstore linked to an in development project, which is a stateless (thanks to OAuth 2.0) worker providing signed urls to read/write in s3 stores

This lack of support is the first limitation of the current implementation.

Open Questions & Future Work

This PR focuses on establishing the core architecture. Several larger topics from the original discussion are noted here as future work:

• Efficient Low-Resolution Drawing: As discussed, efficient, multi-resolution drawing with upsampling is a complex challenge that requires a new data format.
• 3D Drawing Tools: As suggested by @fcollman, 3D-specific tools like interpolation between slices are out of scope for this initial PR but could be a valuable direction for future work.

---

Checklist

• Completed the todo list found in src/voxel_annotations/TODOs.md
• Added support to more (every?) datasources and kvstores
• Signed the CLA.

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[jbms]

Can you complete the CLA?

[jbms]

You can use the hidden property to ensure this is only visible if there is a writable source.

[jbms]

If this is intended to show the color for segmentation layers, it shouldn't be editable, and also it doesn't always seem to stay in sync.

[briossant]

yeah, this is pretty much useless, I will remove it

[jbms]

This should be limited to the valid range for the data type.

[jbms]

The brush hover outline (circle where the mouse pointer is) seems to go away in some cases when changing the zoom level.

[briossant]

I need to rewrite history, my commits are signed with the wrong email, I will open a new PR I made a mistake with this force push nevermind its fixed

[chrisj]

I would avoid casting as any, it makes it hard to follow. kvStore in zarr is explictly a ReadableKvStore so I think you want to change that to a normal KvStore.

[briossant]

You are right, I will look into that. I originally did this as part of my non-destructive workflow (e.g. I wanted to modify as little as possible of preexisting neuroglancer code while prototyping), but this makes no more sense now.