Comprehensive Review of UltraShape

[realstevewarner]

First and foremost, thank you SO much for this amazing tool. I spent time getting it installed on an Ubuntu build running in WSL on my Windows 11 machine with a 4090 GPU. Then I ran numerous tests to evaluate quality and processing time. I detailed these in a post on X. You can find that here:

https://x.com/SteveWarnerFL/status/2008012464231976995

In short, UltraShape is incredible. It can take starting meshes from just about any 3D mesh generator (e.g. Hunyuan, Trellis, etc.) and kick out a vastly better mesh that more closely adheres to the reference image.

I'll be doing an extensive evaluation against the Refiner in Trellis2, but in my initial tests, I found that UltraShape did a far superior job.

What's most interesting is how UltraShape handles memory. In Trellis2, I was only able to refine at 512 resolution. At 1024 resolution, I hit Out Of Memory (OOM) errors. However UltraShape never had that issue. At 512 Octree Resolution, I was generating ~2 Million poly meshes and never hitting an OOM error. Even better, UltraShape does not produce interior geometry where Trellis2 does.

Here you can see Trellis2 on the left and UltraShape on the right. The red indicates interior facing polys. The fact that Trellis2 only has interior faces around the perimeter means that all that grey on the inside (which is red in the UltraShape mesh on the right) is actual geometry. Let me state that again. Trellis2 is using 50% or more of its polygons on the inside of your mesh where you'll never see them. Because of this, the 2 million polygons in the UltraShape mesh are all going towards the refined shape. Whereas the 2 million polygons in the Trellis2 mesh are being split between interior and exterior, resulting in a mesh with notably less detail.

Anyway, I wanted to say thank you for the amazing work on this tool. It's currently the best open source tool out there for refining 3D meshes. Congratulations to you and your team!

[StarRealMan]

Thanks a lot for the thorough review, I think maybe we can keep this issue open?

Thank you, that was very informative.

You mentioned "...It can take starting meshes from just about any 3D mesh generator (e.g., Hunyuan, Trellis, etc.)..." : doesn't that mean we can take any existing 3D model (for example, ripped from an old game or done manually in Blender as a rough sketch), take a screenshot of her, use some AI to add details and texture to screenshot, and feed it to UltraShape 1.0, essentially using it as an "enhancer"?

After all, if it accepts input models in OBJ format, then what difference does it make whether it was created by AI or not from the start?

[realstevewarner]

You mentioned "...It can take starting meshes from just about any 3D mesh generator (e.g., Hunyuan, Trellis, etc.)..." : doesn't that mean we can take any existing 3D model (for example, ripped from an old game or done manually in Blender as a rough sketch), take a screenshot of her, use some AI to add details and texture to screenshot, and feed it to UltraShape 1.0, essentially using it as an "enhancer"?

I haven't explicitly tested this, but my guess is yes. You should be able to take a rough model (e.g. an old game model) and then an enhanced render from AI and use UltraShape to create an enhanced mesh. The key to getting this to work would be making sure your AI render was derived from a screenshot of the low poly 3D model. For example, import the game model into Blender, grab a screenshot (make sure it's square), then run that through SDXL or Flux with ControlNet using Open Pose, Canny, etc. to ensure the generation closely matches your screen grab. Then pass the mesh and image to UltraShape for processing.

I just posted a breakdown of how UltraShape handles different source meshes. You can find that here:

https://x.com/SteveWarnerFL/status/2008259639583101284

The big takeaway is that input mesh can be almost anything. The more divergent it is from the reference image, the more wonky the results will be. But there's potentially some fun to be had in exploring this as well.

[mrchalk]

Regarding the "interior faces of TRELLIS.2" section, it should be noted that TRELLIS.2 uses PaMO as a remesh post-processing step, which converts TRELLIS.2's single-layer mesh output into a double-layer mesh. Setting TRELLIS.2 to output twice as many faces would likely be fairer. These unnecessary faces could possibly be removed by detecting interior faces through visibility checks or a flood-fill method and deleting them.

[realstevewarner]

Regarding the "interior faces of TRELLIS.2" section, it should be noted that TRELLIS.2 uses PaMO as a remesh post-processing step, which converts TRELLIS.2's single-layer mesh output into a double-layer mesh. Setting TRELLIS.2 to output twice as many faces would likely be fairer. These unnecessary faces could possibly be removed by detecting interior faces through visibility checks or a flood-fill method and deleting them.

I've heard it's a "feature not a bug" because it's designed to handle transparent surfaces better, but TBH, if that's the case there needs to be a clear way to disable it in the workflows. Thus far, I've not seen that. Could be there, but I don't see an option in the workflows I'm using. And every mesh I've generated with Trellis.2 has had connecting polygons between the outer shell and inner shell. That makes selecting the interior geometry really time consuming in post. This isn't a slam against Trellis.2. It's a remarkable tool, but the fact that every mesh comes with interior faces is a problem for both detail refinement as well as PBR texturing, as the interior faces factor into the overall Atlas UV map.