Animating mesh in cloth_style3d example

[mdeth]

Hi team, I am trying to change the vertex positions of the currently static mesh representing a human during the simulation in the example cloth_style3d.
To test this, I currently simply scale it like this at the beginning of simulate():

self.model.shape_source[0].mesh.points *= scale
self.model.shape_source[0].mesh..refit()

The cloth actually seems to be colliding with something becoming increasingly different from the human, but the static human mesh remains static. How can I adjust the static mesh to show the scaled vertices?
Also, how could I copy custom vertex positions for each frame to the human mesh instead of just scaling?

[preist-nvidia]

@WenchaoHuang can you please comment on this?

[WenchaoHuang]

@mdeth Thanks for your feedback!

As far as I know, the current ViewerGL implementation doesn't expose any public method to update the rendered buffer from shape_source. The rendering vertex data of the body is only created once when calling viewer.set_model(), and there's no public method to update it afterward. Therefore, directly modifying shape_source in the model won't trigger updates to the rendered buffers.
​​
A quick workaround is to manually call viewer.log_mesh() in the example.render() function. However, you'll need to manually compute the world-space vertex positions based on shape_source and state.body_q, and then pass them to log_mesh(). Here is an approach I used previously:

    for i in range(self.model.body_count):
        shape_indices = self.model.body_shapes[i]
        for shape_idx in shape_indices:
            if isinstance(self.model.shape_source[shape_idx], newton.Mesh):
                if self.shape_flags[shape_idx] & 1 == 0:
                    continue
      
                @wp.kernel
                def transform_vertices_kernel(
                    shape_index: wp.int32,
                    scale: float,
                    shape_body: wp.array(dtype=int),
                    vertices_in: wp.array(dtype=wp.vec3),
                    scaling3d: wp.array(dtype=wp.vec3),
                    transforms0: wp.array(dtype=wp.transform),
                    transforms1: wp.array(dtype=wp.transform),
                    vertices_out: wp.array(dtype=wp.vec3),
                ):
                    tid = wp.tid()
                    scaling = scaling3d[shape_index] * scale
                    new_pos = wp.transform_point(transforms0[shape_index], vertices_in[tid])
                    new_pos = wp.transform_point(transforms1[shape_body[shape_index]], new_pos)
                    new_pos[0] *= scaling[0]
                    new_pos[1] *= scaling[1]
                    new_pos[2] *= scaling[2]
                    vertices_out[tid] = new_pos
      
                shape_vertices = wp.array(self.model.shape_source[shape_idx].vertices, dtype=wp.vec3)
                shape_indices = wp.array(self.model.shape_source[shape_idx].indices.flatten(), dtype=int)
      
                wp.launch(
                    transform_vertices_kernel,
                    dim=len(shape_vertices),
                    inputs=[
                        shape_idx,
                        1.0,
                        self.model.shape_body,
                        shape_vertices,
                        self.model.shape_scale,
                        self.model.shape_transform,
                        self.state.body_q,
                    ],
                    outputs=[shape_vertices],
                )
                self.viewer.log_mesh(
                    f"/model/triangles_{self.model.shape_key[shape_idx]}", shape_vertices, shape_indices
                )

This approach is a bit complicated and not recommended, but it works.

Getting back to your requirement. Based on my experience, directly updating the collider's position every frame won't produce good collision results — unless the timestep is extremely small or the motion is very slow. This is because the velocity information between frames is lost in this approach. A more robust solution would be to properly develop new functionality that supports collision respond for such externally-driven meshes.

So, your question suggests a new feature: collision support for externally-driven meshes. I think this is quite a common and feasible requirement. Noted — I’ll record this request and let you know when there’s any development progress. : )

[mdeth]

@WenchaoHuang thanks a lot for the suggestion and excuse my late reply. I did try what you proposed, and did indeed see the changes rendered. However, the mesh also kept rotating around the x or y axis by steps of 90 degrees. I see a new example in master using a humanoid robot. Could I change it to have my animated mesh instead of the robot colliding with the cloth?

Also, can you elaborate why the velocity information is lost? Is it reset somewhere or is it due the solver not knowing about the mesh animation? How could handle this properly in the Style3d solver?

[WenchaoHuang]

However, the mesh also kept rotating around the x or y axis by steps of 90 degrees

This is likely because your model's 'up direction' differs from the viewer's setting (Style3D Assets export uses Y-up by default, while ViewerGL defaults to Z-up).

Could I change it to have my animated mesh instead of the robot colliding with the cloth?

Newton does not officially support externally-driven animated meshes currently, as the Model data structure does not provide such information. While a workaround exists to forcibly modify a rigid body mesh, this approach will lead to the poor collision behavior I mentioned earlier.

why the velocity information is lost?

Currently, in Model.collide(...), the velocity of a rigid body mesh is calculated based on the rules of rigid body motion. This calculation operates under the assumption of purely rigid deformation. More specifically, the system only has access to the initial mesh geometry at one time step, along with its transforms from the previous and current frames. If your mesh is also deforming, calculating its velocity requires, at a minimum, access to the vertex positions from both the current and previous frames. If the rigid body velocity calculation method is applied to a deforming mesh, the result will be inaccurate. Any inaccuracies only become negligible when using extremely small time steps (~1ms).

How could handle this properly in the Style3d solver?

You can test the current implementation with a small time step (e.g., at 60 FPS and ~10 substeps) to evaluate the results and see if it meets your requirements.

[mdeth]

@WenchaoHuang thanks again. I am considering adding support for collisions with externally-driven animated meshes and would appreciate any help or hint you could give me for that, since I have no experience in extending newton so far.

[WenchaoHuang]

You can reference examples like example_cloth_h1.py, which demonstrate robot-cloth interaction.

Here's what you'll need to implement:

1. Perform mesh-cloth face-vertex collision detection and add valid pairs to the contact system.
2. Implement a non-rigidbody collision response by adapting the approach form evaluate_body_particle_contact function.
3. Apply interpolation to the mesh between substeps.