Diffusion Shape Prior for Wrinkle-Accurate Cloth Registration

1University of Minnesota 2Meta Reality Labs 3Carnegie Mellon University
3DV 2024

We learn a strong shape prior from pre-captured 4D data using a diffusion model, and apply it to texture-less registration of the clothing with highly complex deformations.


Registering clothes from 4D scans with vertex-accurate correspondence is challenging, yet important for dynamic appearance modeling and physics parameter estimation from real-world data. However, previous methods either rely on texture information, which is not always reliable, or achieve only coarse-level alignment. In this work, we present a novel approach to enabling accurate surface registration of texture-less clothes with large deformation. Our key idea is to effectively leverage a shape prior learned from pre-captured clothing using diffusion models. We also propose a multi-stage guidance scheme based on learned functional maps, which stabilizes registration for large-scale deformation even when they vary significantly from training data. Using high-fidelity real captured clothes, our experiments show that the proposed approach based on diffusion models generalizes better than surface registration with VAE or PCA-based priors, outperforming both optimization-based and learning-based non-rigid registration methods for both interpolation and extrapolation tests.

Diffusion-based shape prior

Diffusion model for cloth deformation. In the forward process, we gradually add noise to the UV displacement map to acquire an isotropic Gaussian distribution. To sample from the learned data distribution, we recover the clean UV displacement map by gradually denoising the corrupted UV displacement map.

Non-rigid registration via manifold-guidance

The manifold guidance maximizes the log-likelihood of every diffusion state given the observation. We use a multi-stage posterior sampling process, where the early stage of the denoising process is guided by a learning-based coarse registration approach, and the later stage is refined with spatial proximity guidance.

Results on T-shirts

Results on Skirts


  title={Diffusion Shape Prior for Wrinkle-Accurate Cloth Registration}, 
  author={Guo, Jingfan and Prada, Fabian and Xiang, Donglai and Romero, Javier and 
          Wu, Chenglei and Park, Hyun Soo and Shiratori, Takaaki and Saito, Shunsuke},
  journal={arXiv preprint arXiv:2311.05828}