+
+
+
+ 
- We present the first method capable of photorealistically reconstructing a non-rigidly - deforming scene using photos/videos captured casually from mobile phones. + Rigging and skinning are essential steps to create realistic 3D animations, + often requiring significant expertise and manual effort. Traditional attempts + at automating these processes rely heavily on geometric heuristics and often + struggle with objects of complex geometry. Recent data-driven approaches + show potential for better generality, but are often constrained by limited + training data.
- Our approach augments neural radiance fields - (NeRF) by optimizing an - additional continuous volumetric deformation field that warps each observed point into a - canonical 5D NeRF. - We observe that these NeRF-like deformation fields are prone to local minima, and - propose a coarse-to-fine optimization method for coordinate-based models that allows for - more robust optimization. - By adapting principles from geometry processing and physical simulation to NeRF-like - models, we propose an elastic regularization of the deformation field that further - improves robustness. -
-- We show that Nerfies can turn casually captured selfie - photos/videos into deformable NeRF - models that allow for photorealistic renderings of the subject from arbitrary - viewpoints, which we dub "nerfies". We evaluate our method by collecting data - using a - rig with two mobile phones that take time-synchronized photos, yielding train/validation - images of the same pose at different viewpoints. We show that our method faithfully - reconstructs non-rigidly deforming scenes and reproduces unseen views with high - fidelity. + We present the Anymate Dataset, a large-scale dataset of + 230K 3D assets paired with expert-crafted rigging and skinning information— + 70 times larger than existing datasets. Using this dataset, we propose a + learning-based auto-rigging framework with three sequential modules for + joint, connectivity, and skinning weight prediction. We systematically de + sign and experiment with various architectures as baselines for each module + and conduct comprehensive evaluations on our dataset to compare their + performance. Our models significantly outperform existing methods, providing + a foundation for comparing future methods in automated rigging and + skinning.
- Using nerfies you can create fun visual effects. This Dolly zoom effect - would be impossible without nerfies since it would require going through a wall. + (Sample 1)
- ++ (Sample 5) +
+ ++ (Sample 9) +
+- As a byproduct of our method, we can also solve the matting problem by ignoring - samples that fall outside of a bounding box during rendering. -
- -+ (Sample 2) +
+ ++ (Sample 6) +
+ ++ (Sample 10) +
+- We can also animate the scene by interpolating the deformation latent codes of two input - frames. Use the slider here to linearly interpolate between the left frame and the right - frame. + (Sample 3)
+ ++ (Sample 7) +
+ ++ (Sample 11) +
+
- Start Frame
-
- End Frame
-- Using Nerfies, you can re-render a video from a novel - viewpoint such as a stabilized camera by playing back the training deformations. + (Sample 4)
-+ (Sample 8) +
+ ++ (Sample 12) +
+- Progressive Encoding for Neural Optimization introduces an idea similar to our windowed position encoding for coarse-to-fine optimization. + RigAnything, MagicArticulate and One Model to Rig Them All + employ autoregressive model for skeleton prediction.
- D-NeRF and NR-NeRF - both use deformation fields to model non-rigid scenes. -
+ MagicArticulate introduces the Articulation-XL Dataset + derived from Objaverse-XL and comprising over 33,000 models annotated with rigging details.- Some works model videos with a NeRF by directly modulating the density, such as Video-NeRF, NSFF, and DyNeRF + One Model to Rig Them All contributes two datasets: Rig-XL Dataset a curated subset of about 14,000 rigged models from Objaverse-XL and the + VRoid Dataset, a collection of 2,000 stylized characters from VRoidHub
- There are probably many more by the time you are reading this. Check out Frank Dellart's survey on recent NeRF papers, and Yen-Chen Lin's curated list of NeRF papers. -
+ Together, these works highlight the growing emphasis on scalable, data-driven rigging solutions. + +