🕺🕺🕺 Lodge 💃💃💃
A Coarse to Fine Diffusion Network for Long Dance Generation Guided by the Characteristic Dance Primitives (CVPR 2024)

Ronghui Li, Yuxiang Zhang, Yachao Zhang, Hongwen Zhang, Jie Guo, Yan Zhang, Yebin Liu and Xiu Li

💃💃💃 Abstract

TL;DR: We propose a two-stage diffusion modle that can generate extremely long dance from given music in a parallel manner.

CLICK for full abstract

We propose Lodge, a network capable of generating extremely long dance sequences conditioned on given music. We design Lodge as a two-stage coarse to fine diffusion architecture, and propose the characteristic dance primitives that possess significant expressiveness as intermediate representations between two diffusion models. The first stage is global diffusion, which focuses on comprehending the coarse-level music-dance correlation and production characteristic dance primitives. In contrast, the second-stage is the local diffusion, which parallelly generates detailed motion sequences under the guidance of the dance primitives and choreographic rules. In addition, we propose a Foot Refine Block to optimize the contact between the feet and the ground, enhancing the physical realism of the motion. Our approach can parallelly generate dance sequences of extremely long length, striking a balance between global choreographic patterns and local motion quality and expressiveness. Extensive experiments validate the efficacy of our method.

🎤🎤🎤 Todo

• Release the code and config for teaser
• [] Release the checkpoints
• Release more applications

🍻🍻🍻 Setup Environment

Our method is trained using cuda11, pytorch-lightning 1.9.5 on Nvidia A100.

conda env create -f lodge.yml

Our environment is similar to EDGE (official. You may check them for more details.

💃💃💃 Training

python train.py --cfg configs/lodge/finedance_fea139.yaml --cfg_assets configs/data/assets.yaml 

🕺🕺🕺 Inference

Once the training is done, run inference:

python infer_lodge.py --cfg configs/lodge/finedance_fea139.yaml --cfg_assets configs/data/assets.yaml 

🎼🎼🎼 Citation

If you think this project is helpful, please feel free to leave a star⭐️⭐️⭐️ and cite our paper:

👯👯👯 Acknowledgements

This repository borrows heavily from EDGE and Bailando. Thanks the authors for sharing their code and models.