GAD: Geometry Attention Distillation for Consistent Video Generation

GAD distills geometry teacher attention into video attention during training. Compared with Wan, it preserves scene geometry across viewpoint changes and object structure during motion, without added inference cost. — Compared with the Wan baseline, GAD better preserves scene geometry along long-range camera trajectories and maintains object structure during motion, reducing geometric drift across changing viewpoints as well as structural degradation and object disappearance over time.

Abstract

Video diffusion models generate visually compelling short-term sequences but often struggle to preserve scene geometry and object structure over time. In this paper, we first demonstrate that current video attention cannot identify reliable correspondences across frames, with responses dispersed across multiple regions even after recognizable scene structure has emerged. We argue that reliable cross-frame associations are essential for information exchange as scene structure forms in video generation. Thus, we introduce Geometry Attention Distillation (GAD), which transfers correspondences from a frozen geometry model's attention to video attention. Extracted from clean videos, these correspondences link the same static surface or moving object across frames. Since structure formation begins in deeper layers at higher noise and shallower layers at lower noise, we apply attention distillation at a student layer selected according to the noise timestep. After training, video attention focuses more consistently on corresponding locations. Experiments across all three video backbones demonstrate improved long-range scene consistency and dynamic temporal consistency, with geometric gains across generation lengths. On Wan2.2 5B, GAD improves cross-frame matching accuracy by at least 30% over the Wan baseline on both long-range benchmarks. Notably, GAD achieves these gains without additional inference modules, offering a promising approach to efficient and consistent video-based world simulation.

Motivation

Video attention often responds to multiple unrelated regions, even when recognizable scene structure has already emerged. Rather than focusing on the same scene element across frames, it can spread across regions with similar appearance, making cross-frame associations ambiguous and weakening scene and object consistency. In contrast, geometry model attention can more reliably follow the same static surface or moving object across frames. Building on this observation, GAD uses geometry model attention as a source of supervision during training. By distilling its more accurate attention patterns into video attention, GAD encourages the model to associate corresponding scene elements across frames, helping preserve scene geometry and object structure over time.

Method

GAD overview: a frozen geometry teacher extracts correspondence targets from clean videos, which supervise video attention at a noise-selected layer during training.

GAD guides video attention with geometric correspondence supervision through three key components:

Extract reliable correspondences. Use a frozen VGGT-Ω teacher to extract cross-frame attention from clean videos, align it with the student's token grid, and weight correspondence targets by their reliability.
Guide attention where structure forms. Select the student layer based on the noise timestep and apply geometry-teacher supervision as scene structure and cross-frame associations emerge.
Learn spatial and temporal consistency. Combine pairwise and trackwise losses with the generation objective to improve correspondence accuracy between frame pairs and maintain consistent associations across frames.

The geometry teacher's attention and additional supervision modules are needed only during training; inference uses the video model alone, with no additional modules or computational cost.

Results

Long-Range Video Generation

Compared with competing methods, GAD better preserves scene layout and geometric structure throughout long-range camera trajectories, reducing geometric drift as the viewpoint changes.

Dynamic Video Generation

GAD also enables moving characters and foreground objects to maintain more consistent structure, while keeping background layouts more stable across changing viewpoints.

Video Attention Comparison

GAD focuses attention more consistently on corresponding regions across frames, improving scene and object consistency. Red dots indicate query points, and heatmaps visualize attention in subsequent views.

Static scene · 1 / 2

Static scene: GT, Wan and GAD views and attention maps, with red query points marked on the building in the input views. — Static scene: cross-view associations become more localized with GAD.

Citation

If you find our work helpful, please cite GAD:

@article{ni2026gad,
  title={GAD: Geometry Attention Distillation for Consistent Video Generation},
  author={Ni, Junfeng and Zhu, Ziyu and Jiang, Fan and Liu, Yu and Wang, Chiyu and Zhang, Jiawei and Sun, Zhaoxu and Xu, Mu and Huang, Siyuan and Chen, Yixin},
  journal={arXiv preprint arXiv:2610.18779},
  year={2026}
}