Possibility to render mesh objects (or their SDFs) onto slice render image?

[trainedmoose]

Hello :)

First of all, thank you for this tool! I've been trying to set it up to display the slice renderer image in a virtual MRI intervention scene which works really great!

My problem: I'm trying to add 3D meshes to the whole process and I've been wondering whether I'm trying to achieve a functionality that already exists. In short, here's what I try to do:

1. Load a DICOM set.
2. Add a slicing plane.
3. Copy the slicing plane material to a "display" (e.g. a screen in a virtual MRI intervention room).
4. Place objects inside volume rendered object (e.g. a needle or a tumour)
5. Make the additional objects show up on the slicing plane. (not in a physically correct way, just their mesh surface)

Steps 1 to 4 are working fine but where it gets tricky (at least for me), is the 5th step. I cannot seem to get the volume data of the additional meshes to be displayed in the slice renderer image.

Here's the process I've tried so far:

1. Create an SDF from a (primitive) mesh (e.g. a cube or capsule) using Unity's SDF Bake Tool .
2. Couple the SDFs to their respective 3D objects in the scene via a script that fetches their worldToLocal matrices and updates them in the SliceRenderingShader.
3. Sample the SDFs using the vertex world position.
4. Override the slice renderer material with a certain colour if the sampled value is below 0 (i.e. the vertex is inside the SDF).

The result is quite promising but I cannot actually achieve to sample the SDFs correctly. At the moment, I only correctly sync the worldToLocal matrices to the shader. The problem is that I'm new to shaders so I don't fully understand every detail necessary to achieve this. Having different scale settings when using multiple meshes or their SDFs adds to the problem of making this flexible for non-primitive objects.

The box shape you see on the "display" to the left is actually just the limited coordinate range in the shader for that SDF. If I wouldn't limit that, everything would be tinted white. So I think I'm not even close to an answer here. And that's why I would like to know if you know a better way towards achieving this. I'll keep trying and keep you updated. I can also submit the code I already have so far. I just didn't want to overload this post too much.

Anyway, thanks for reading if you made it this far :)

[mlavik1]

Hi!
Thanks for reaching out - sounds like an interesting use case!

First of all, I think I mostly understand what you want to achieve, but I have two questions:

1. Do you need to support arbitrary meshes, or only certain geometrical shapes? (cube, capsule, sphere
2. How do you expect the mesh/shape to show up in the slice view? Do you have any reference images you could share with me?

The reason I'm asking is:

• With SDF I believe you would at best only be able to show the outline of the mesh/shape - not what's inside. This is because an SDF gives you distance field containing distance to surface.
• If you don't need arbitrary meshes, but instead can achieve what you want with simpler geometric shapes, then you could probably do this mathematically in the shader.

I have some suggestions, depending on exactly what you want to achieve:

• If you only need to support geometric shapes (capsule, etc.): Edit the shader to mathematically check if you're inside the shape, and then add a colour.
• If you only want to see the outline of the mesh/shape: SDF might be an alternative
• If you want to show the shape as if you view it from above: Create an orthographic camera placed in front of the slice, and render the mesh/shape to a texture - then look up this texture in the slice shader. Alternatively you could do the same, but with depth only.

The mathematical solution (that only works with geometric shapes) is by far the easiest. The orthographic camera solution should also not be too hard - but it may of course take some time to get it right.

[trainedmoose]

Thanks for your reply! I will probably update you on my progress tomorrow. I have quite good results already using the SDF approach. Just need to smooth a few things out.

[mlavik1]

Ok, that's great to hear!
Looking forward to the update :)

[trainedmoose]

@mlavik1

Alright, thanks for waiting. I hope it was worth it. At least, I learned a lot :)

First: What can you do with the tool I wrote (partially with help of AI):

• Convert any mesh object to an SDF (using Unity's SDF Bake Tool) and display it in the slice renderer.
• Use as many SDFs as you want. No need to adapt the shader.
• Define a tint for each SDF so you can differentiate between certain objects (if desired).

What you cannot do with this tool:

• Use different SDF sizes (bounding box) or resolutions (3d texture). Any differences will "break" the shader.
• Mix large objects and small objects well. Since the bounding boxes need to be the same, smaller objects will lose their details easily upon SDF baking (and become a meaningless blob) compared to the larger objects because the distance to the bounding box is so large (unless you use a high resolution for all objects). You can somewhat circumvent this by approximating an object (if possible) by a downscaled larger object. For example, I used a larger cube, created an SDF from it and downscaled it so it would fit the needle position. I did this because the SDF of the needle mesh did have some issues at the thinner parts.

Why I still prefer this approach:

• It avoids (some) code duplication because the shader doesn't have any hard coded SDF references.
  • This might depend on the use case. If you only ever use a few objects and it makes sense to have different SDF resolutions and bounding boxes, my approach might be the wrong way.
• I will highly likely only use one tool (a needle) anyway. Maybe a second one. But they are roughly the same size.

Some of the limitations might be able to be circumvented but I didn't have the time to adapt the shader or experiment further.

---

SDFBinder script (above) and the in game view (below).

---

Regarding the attached files:

DynamicSDFBinder.cs allows you to select your tracked game objects and connect them with their respective SDF. It basically creates and fills the shader's SDF atlas which stores all SDFs stacked along its z-axis. The transforms of the SDFs are updated using their game objects' world-to-local matrices.

SliceRenderingShader.shader is based on the shader you had before your recent restructure commits. It still uses CG instead of HLSL (I didn't figure out a quick way to adapt the code to HLSL yet). It continues to look up the medical volume data first and then iterates through the SDFs. Using the vertex world position and the SDFs' world-to-local matrices it samples the SDF in their local space and checks whether the sampled values are below 0 (= inside the mesh / SDF) or above 0 (= outside of the mesh / SDF). It then overwrites the medical slice image with the defined SDF tint where the slice intersects with an SDF.

I hope the files are commented good enough to answer any remaining questions. Nevertheless, feel free to ask if there is anything that's left unclear.

Cheers

Attachment (updated)

EDIT: I updated the script and the shader to not block the DICOM slice rendering if the SDFBinder isn't used. Basically the script now enables or disables a preprocessor directive in the shader to include or exclude the SDF sampling part.

[mlavik1]

Thanks for sharing!
This looks great - definitely a useful feature, and for a use case like this I think SDFs might be a great solution!
And I agree, it's definitely best to avoid any hard coded SDF references in the slice rendering shader.

And regarding this:

It still uses CG instead of HLSL

You may or may not have seen this, but I just merged a PR for updating the plugin to support URP and HDRP. I haven't made a release yet though. I need to first decide on how to deal with some of the Unity Asset Store requirements, haha.

Anyway, great work on this!

[trainedmoose]

Thanks for the feedback. I appreciate it a lot!

I actually tried using the package through the package manager but I always get an error. I wasn't sure if that's because you haven't released the update yet but I thought I'd share it here. I could also open an issue.

This error appears in multiple (if not all) HDRP shaders:

Shader error in 'VolumeRendering/HDRP/SliceRenderingShader': unrecognized identifier 'GLOBAL_CBUFFER_START' at /projectname/Library/PackageCache/com.unity.render-pipelines.high-definition@7f3ce09990d6/Runtime/ShaderLibrary/ShaderVariablesGlobal.cs.hlsl(14) (on d3d11)

[mlavik1]

Oh, thank you! I'll have a look at it.
I suppose you're using HDRP when testing, right?
Thanks for the feedback!

[trainedmoose]

Yep, I'm using HDRP.

[mlavik1]

I've fixed the issues, HDRP should now work fine :)
There's one issue with the histogram rendering though - I'll test some more and make sure everything works soon. Hoping to prepare a new release soon!

Don't hesitate to let me know if you run into any issues.