From 3ec97f94e2dd41053a42d2765997fef403649bd7 Mon Sep 17 00:00:00 2001
From: animate3d 1CASIA NeurIPS 2024Animate3D: Animating Any 3D Model with Multi-view Video Diffusion
+
+
+ 2DAMO Academy, Alibaba GroupAbstract
Specifically, for MV-VDM, we design a new spatiotemporal attention module to enhance spatial and temporal consistency by integrating 3D and video diffusion models.
Additionally, we leverage the static 3D model's multi-view renderings as conditions to preserve its identity.
For animating 3D models, an effective two-stage pipeline is proposed: we first reconstruct motions directly from generated multi-view videos, followed by the introduced 4D-SDS to refine both appearance and motion.
+ Benefiting from accurate motion learning, we could achieve straightforward mesh animation.
Qualitative and quantitative experiments demonstrate that Animate3D significantly outperforms previous approaches.
Data, code, and models will be open-released.
The video is best viewed in 4K mode.
+
+ Our training dataset, MV-Video, comprises 115K animations that are available under a public license, consisting of about 53K animated 3D objects at all,
+ which are rendered into over 1.8M multi-view videos.
+ Notably, our training data is manually selected and with high-quality. It includes the highest quality part of Objaverse (around 7K animated 3D objects), while the rest (around 46K animated 3D objects) are collected by ourselves.
+
+ [1] SC4D: Sparse-Controlled Video-to-4D Generation and Motion Transfer (ECCV 2024)
+ [2] STAG4D: Spatial-Temporal Anchored Generative 4D Gaussians (ECCV 2024)
+ [3] Consistent4D: Consistent 360° Dynamic Object Generation from Monocular Video (ICLR 2024)
+
Some 3D assets for animation are downloaded from sketchfab, under CC Attribution and CC Attribution-NonCommercial. - We would like to thank the creatorsfor sharing great 3D assets. + We would like to thank the creatorsfor sharing great 3D assets.