Decoupling trame-slicer core logic from Vue-based UI for React/wslink integration

[hieund4438]

I am developing a medical imaging application using React (likely OHIF viewer) for the frontend and a Python (VTK) backend. Currently, I am using wslink library (https://github.com/Kitware/wslink) to establish a WebSocket connection and vtkRemoteView (from vtk.js) to render the server-side frames.

I am very interested in the slicing capabilities of trame-slicer, but our project architecture does not use Vue.js. I want to leverage the robust slicing and volume rendering logic of trame-slicer as a "headless" service or a core library that I can communicate with via custom wslink protocols.

Current Setup
Frontend: React, vtk.js, wslink.

Backend: Python VTK

Reference: Following the vtk.js RemoteView example.

The Problem
The current trame-slicer implementation seems tightly coupled with Trame's UI components (which are built on Vue). This makes it difficult to extract just the VTK pipeline logic (the "Core") to use in a custom protocol setup where the UI state is managed by React.

Questions
Is there an existing way to initialize the trame-slicer engine without triggering the UI generation?

Are there plans to modularize trame-slicer so the VTK pipeline can be used independently of the Trame web components?

[Thibault-Pelletier]

Hi @hieund4438,

trame-slicer is actually very decoupled from trame.
The only components which are coupled are :

• The remote controlled views (uses trame-rca)
• The view layout management system (uses trame / vuetify templates)
• The examples which show how to connect the framework to trame

The core components / logic / view factory are wrapped into the SlicerApp class which is independent from trame.

You can have a look at the tests if you want to see how things are instantiated from scratch. For instance, the segmentation tests are independent from trame while using the Slicer core features with interactive VTK render windows through the --render-interactive 1 test flag.

Note that it may be easier to integrate trame-slicer in react using an iframe with the trame-react library.

[hieund4438]

@Thibault-Pelletier thank you — I’m now able to run trame-slicer independently.

I’m currently trying to create a custom interactor style for ThreeDView render window. However, I’m observing an unexpected behavior:

The volume starts rotating as soon as I move the mouse, even without any mouse button being pressed. My expectation is that the rotation should only happen while dragging (e.g. left mouse button down), similar to the default Trackball interaction.

Below is a minimal reproducible example. In this test, I explicitly set the interactor style to vtkInteractorStyleTrackballCamera, but the volume still reacts to simple mouse movement.

import os
from trame_slicer.core import SlicerApp
from trame_slicer.views import ThreeDView
import vtk

def test_3d_minimal(dicom_path):
    app = SlicerApp()

    volume_node = app.volumes_logic.AddArchetypeVolume(
        os.path.abspath(dicom_path),
        "MainVolume",
        0,
    )
    if not volume_node:
        print("Load failed")
        return

    view = ThreeDView(
        app.scene,
        app.app_logic,
        "View1",
    )
    app.volume_rendering._logic.SetDefaultRenderingMethod("vtkMRMLCPURayCastVolumeRenderingDisplayNode")
    vr_display_node = app.volume_rendering.create_display_node(
        volume_node,
        "CT-AAA",
    )

    if vr_display_node:
        vr_display_node.AddViewNodeID(view.get_view_node().GetID())

    view.fit_view_to_content()
    view.render()
    view.render_window().SetOffScreenRendering(False)
    view.render_window().SetSize(600, 600)

    interactor = view.render_window().GetInteractor()
    track_ball_style = vtk.vtkInteractorStyleTrackballCamera()
    interactor.SetInteractorStyle(track_ball_style)

    view.start_interactor()


if __name__ == "__main__":
    path = "CT 0.625MM KHONG THUOC/CT000001.dcm"
    if os.path.exists(path):
        test_3d_minimal(path)

[Thibault-Pelletier]

HI @hieund4438 ,

The VTK trackball camera is like that, it rotates when you left click.

If you plan on keeping the builtin 3D slicer interactions (for instance for placing / moving landmarks and more), you shouldn't replace the interactor style.

The interactor style dispatches calls to 3D Slicer's displayable manager

Changing the interaction in 3D views should be done using its associated displayable manager

[hieund4438]

Thank you @Thibault-Pelletier
I have a follow-up question regarding volume rendering customization.

I’m currently working on a tool that allows modifying how the volume is displayed. As a simple proof of concept, I created a small demo that hides half of a patient’s 3D volume (for example, clipping or masking one side of the volume).

Below is a simplified demo illustrating the idea:

import os
import vtk
import vtkmodules.util.numpy_support as vtk_np
from trame_slicer.core import SlicerApp
from trame_slicer.views import ThreeDView

# --- MASK LOGIC ---
def apply_half_mask(volume_node):
    image_data = volume_node.GetImageData()
    if not image_data:
        return
    
    dims = image_data.GetDimensions()
    shape = (dims[2], dims[1], dims[0])
    scalars = image_data.GetPointData().GetScalars()
    np_array = vtk_np.vtk_to_numpy(scalars).reshape(shape)

    # Mask upper half on Z axis
    z_half = dims[2] // 2
    np_array[z_half:, :, :] = -3024 

    scalars.Modified()
    image_data.Modified()
    volume_node.Modified()
    print(f"[DEBUG] Mask applied: Upper {dims[2]-z_half} slices filled.")

class CustomBindingStyle(vtk.vtkInteractorStyle):
    def __init__(self, volume_node):
        super().__init__()
        self.volume_node = volume_node
        self.AddObserver(vtk.vtkCommand.RightButtonPressEvent, self.__on_right_click)

    def __on_right_click(self, obj, event):
        apply_half_mask(self.volume_node)
        pass

# --- MAIN APP ---
def test_binding_flow(dicom_path):
    app = SlicerApp()
    
    volume_node = app.volumes_logic.AddArchetypeVolume(os.path.abspath(dicom_path), "MainVolume", 0)
    if not volume_node:
        return

    view = ThreeDView(app.scene, app.app_logic, "View1")
    
    # Volume Rendering
    app.volume_rendering._logic.SetDefaultRenderingMethod("vtkMRMLCPURayCastVolumeRenderingDisplayNode")
    vr_display_node = app.volume_rendering.create_display_node(volume_node, "CT-AAA")
    if vr_display_node:
        vr_display_node.AddViewNodeID(view.get_view_node().GetID())

    # --- DISPLAYABLE MANAGER CONFIG ---
    manager_group = view._displayable_manager_group
    camera_dm = manager_group.GetDisplayableManagerByClassName("vtkMRMLCameraDisplayableManager")

    if camera_dm:
        camera_widget = camera_dm.GetCameraWidget()
        
        # 1. Map Left-Click to Rotate (Standard Slicer)
        camera_widget.SetEventTranslationClickAndDrag(
            camera_widget.WidgetStateIdle, vtk.vtkCommand.LeftButtonPressEvent, vtk.vtkEvent.NoModifier,
            camera_widget.WidgetStateRotate, camera_widget.WidgetEventRotateStart, camera_widget.WidgetEventRotateEnd
        )

        # 2. Disable Right-Click Zoom in the widget (Block default translation)
        camera_widget.SetEventTranslationClickAndDrag(
            camera_widget.WidgetStateIdle, vtk.vtkCommand.RightButtonPressEvent, vtk.vtkEvent.NoModifier,
            camera_widget.WidgetStateRotate, vtk.vtkWidgetEvent.NoEvent, vtk.vtkWidgetEvent.NoEvent
        )

    # --- SET CUSTOM STYLE ---
    # This style handles the Right-Click logic while Left-Click is handled by the CameraWidget above
    custom_style = CustomBindingStyle(volume_node)
    view.interactor().SetInteractorStyle(custom_style)

    view.fit_view_to_content()
    view.render_window().SetSize(800, 800)

    view.render_window().SetOffScreenRendering(False)
    view.start_interactor()

if __name__ == "__main__":
    path = "CT 0.625MM KHONG THUOC/CT000001.dcm"
    if os.path.exists(path):
        test_binding_flow(path)
    else:
        print(f"Path not found: {path}")

Is it recommended to directly access and modify the underlying image_data from the volume node, as done in some examples?
If not, what would be the preferred or “correct” way to implement this?

Thank you for your enthusiasm.

[Thibault-Pelletier]

For volume cropping, you can use the ROI cropping

It provides interactive ROI cropping features.

For other types of masking, for instance by segmentation areas, then modifying the image data is fine.

[hieund4438]

Hi @Thibault-Pelletier
I’m trying to customize the 2D slice view functionality, Specifically, I want to use SliceView() in a more standalone way. At the moment, I’m able to render the slice and scroll through it with the mouse wheel, but I can’t get the intersection crosslines to appear (the horizontal and vertical lines shown when using MPR).

Below is a minimal example of the code I’m currently using:

import os
import vtk
from trame_slicer.core import SlicerApp
from trame_slicer.views import SliceView, DirectRendering
from slicer import vtkMRMLApplicationLogic, vtkMRMLSliceDisplayNode

def test_2d_views_with_intersections(dicom_path):
    # --- STEP 1: INIT APP ---
    app = SlicerApp()
    scene = app.scene
    app_logic = app.app_logic
    
    # --- STEP 2: LOAD VOLUME ---
    volume_node = app.volumes_logic.AddArchetypeVolume(os.path.abspath(dicom_path), "MainVolume", 0)
    if not volume_node:
        print("[-] Load Volume FAILED")
        return

    # --- STEP 3: SETUP 3 VIEWS (Axial, Sagittal, Coronal) ---
    views = []
    view_configs = [
        ("Red", "Axial", (1, 0, 0), (0, 0)),
        ("Yellow", "Sagittal", (1, 1, 0), (800, 0)),
        ("Green", "Coronal", (0, 1, 0), (0, 800))
    ]

    for name, ori, color, pos in view_configs:
        view = SliceView(scene, app_logic, name)
        s_node = view.get_view_node()
        
        s_node.SetOrientation(ori)
        s_node.SetLayoutColor(color)
        view.set_background_volume_id(volume_node.GetID())
        
        # Cấu hình cửa sổ
        rw = view.render_window()
        rw.SetSize(300, 300)
        rw.SetPosition(*pos)
        rw.SetWindowName(f"View {name}: {ori}")

        #  
        display_node = vtkMRMLSliceDisplayNode()
        scene.AddNode(display_node)
        # Liên kết DisplayNode với SliceNode
        display_node.SetNodeReferenceID("slice", s_node.GetID())
        display_node.SetIntersectingSlicesVisibility(True)
        
        views.append(view)

    # --- STEP 4: GLOBAL INTERACTION SETUP ---
    # Đồng bộ volume lên các view
    app_logic.GetSelectionNode().SetReferenceActiveVolumeID(volume_node.GetID())
    app_logic.PropagateVolumeSelection()

    # Explain why this doesn't work pls :((  I comment this in mpr_interaction_button_logic.py then the intersection crossline disappeared
    app_logic.SetIntersectingSlicesEnabled(vtkMRMLApplicationLogic.IntersectingSlicesVisibility, True)
    app_logic.SetIntersectingSlicesEnabled(vtkMRMLApplicationLogic.IntersectingSlicesInteractive, True)
    app_logic.SetIntersectingSlicesEnabled(vtkMRMLApplicationLogic.IntersectingSlicesRotation, True)
    
    app_logic.SetIntersectingSlicesIntersectionMode(1) 
    app_logic.SetIntersectingSlicesLineThicknessMode(2)

    # --- STEP 5: RENDER & START EVENT LOOP ---
    for view in views:
        view.fit_view_to_content()
        view.get_view_node().Modified()
        view.render_window().Render()
        view.render_window().SetOffScreenRendering(False)

    print("[+] MPR Views Ready. Intersection lines should be visible.")
    print("[+] Interaction: Use Mouse to drag/rotate lines.")


    views[0].interactor().Start()

if __name__ == "__main__":
    path = "CT 0.625MM KHONG THUOC/CT000001.dcm"
    if os.path.exists(path):
        test_2d_views_with_intersections(path)

With this setup, scrolling works correctly, but the intersection crosslines are not displayed. I tried to run like this, it still not work
Turn on slice intersection

[Thibault-Pelletier]

Hi @hieund4438,

Your problem is likely linked to the missing view.set_mapped_in_layout flag which is normally set by the layout manager when switching view layouts and ultimately defines if an intersection should be visible or not.

See usage here.

[hieund4438]

@Thibault-Pelletier Thanks a lot — I followed your suggestion and I’m now able to display the 2D view correctly.
However, the rendering seems a bit choppy / low frame rate, especially during interaction. I’m not sure whether this is expected or if I might be missing some performance-related configuration.
Additionally, I have a question about interaction customization:

1. Is it possible to hide the intersection lines while keeping the existing interactions (scrolling, slice movement, linking, etc.)?
2. Or would it be better to fully redefine/customize the interaction logic myself (e.g. use sphere handles instead of intersection
   line)?

[Thibault-Pelletier]

Hi @hieund4438,

The performances lag is probably coming from the render rate which is unbound in your code.
In 3D Slicer an FPS rate limit is applied to the views to avoid triggering the update too often with a value around 30FPS.

Note that the 2D views are rendered on the CPU in 3D Slicer so if the changes are too numerous or too many views are displayed and refreshed at the same time, then it can lag behind.

In trame-slicer, we use the trame-rca RCA render scheduler for this.

We also have a few other performances tweaks:

• Interaction throttling (also done through trame-rca)
• Lower refresh rate for views without focus
• No refresh for not displayed views

Regarding your other question, no, the SliceIntersectionWidget doesn't support keeping the interaction while hiding the lines.

If you want a different interaction widget, then it's best to implement it and deactivate the intersection widget.

For its implementation, you can either use builtin 3D Slicer primitives (for instance markups points) or you can dive lower an implement a custom displayable pipeline using the LayerDM library

Of course, you can also implement that in pure VTK / VTK widgets, but this takes usually more code.

[hieund4438]

Thanks @Thibault-Pelletier
After two days, I figured out how to use ScheduledRenderStrategy. The lag because of using DirectRender() which always render while user dragging.

It appears that ScheduledRenderStrategy is responsible for managing the rendering of non-active viewports, while the currently active window is fully driven by direct user interactions.

Applying an FPS limit at this level works well and produces good results in practice.

[Thibault-Pelletier]

Hi @hieund4438,

Glad you're making progress.
Your analysis is slightly mistaken. The schedule render also affects user interactions.

The displayable manager group trigger an UpdateEvent when an interaction was handled by a displayable manager. This UpdateEvent is connected to the abstract view schedule_render method which delegates to the schedule render strategy.

[hieund4438]

Hi @Thibault-Pelletier
I’m noticing that the Trame Slicer web window is able to change the mouse icon when the user hovers over the intersection center, rotation handles, or translation regions, and this behavior matches exactly what happens in the native desktop window.

Is this achieved via Trame RCA, or some other mechanism?

Is there a way to send mouse icon state from the server to the web client?

I’m trying to implement a similar mechanism in vtkRemoteView. Currently, vtkRemoteView is using the mouse cursor by default.

[Thibault-Pelletier]

Hi @hieund4438,

The cursor is set from the DisplayableManagers as VTK mouse cursor int values and is forwarded to the vtkRenderWindow.
In trame-slicer, these values are then passed to trame using a state variable (in trame-rca) with HTML format.

The cursor is then set in the DIV

[hieund4438]

Thanks @Thibault-Pelletier
Basically, I’m already able to retrieve mouse information via an rpcCommand, but the mouse position does not seem to be accurate relative to the current cursor position on the offscreen window.

I noticed that Trame Slicer continuously sends data over WebSocket even when I only move the mouse, which looks like a synchronization mechanism.

In contrast, my application is built on vtkRemoteView, where by default an rpcCommand is only sent on mouse click events.
To continuously get mouse information, I had to manually add a mouse-move event listener on the

that contains the canvas, and then call something like:

self._vtk_render_window.GetCurrentCursor()

However, this seems to return the cursor position from the last click, not the current mouse position, which suggests the state is not being synchronized properly.

So my question is: how do Trame RCA / trame-slicer RCA handle this internally?
Is there a continuous mouse-state synchronization mechanism between client and server that vtkRemoteView does not enable by default?

[Thibault-Pelletier]

Hi @hieund4438,

What you are accessing when using self._vtk_render_window.GetCurrentCursor() is actually the int representing the mouse cursor icon which is what I thought you meant by accessing the cursor.

In the VTK world, the mouse position, click events and keyboards go through the vtk render window interactor. In Slicer, the interactions are then forwarded to the displayable manager instances which handle the interaction with the nodes in the scene.

For instance, the markups node have dedicated displayable manager which will then delegate to VTK widgets.

If you want to customize interactions in 3D Slicer and trame slicer, then you have two options:

• Use existing node primitives to handle interaction (for instance markups / models / rois / etc.) and manage VTK actors in the renderer
• Implement a new displayable manager to handle the custom interaction. This can be done using the layer displayable manager extension which is integrated in trame Slicer and used by the segmentation effects.