TriFold encodes variable-cardinality 3D Gaussian scenes into fixed-layout triplane latents that jointly represent geometry and appearance, and decodes content-adaptive Gaussian sets with variable local densities.
Recent progress in 3D generation has motivated growing interest in compact generative representations of complex 3D scenes. For native 3D Gaussian scenes, however, learning such representations remains challenging due to widely varying primitive counts, nonuniform spatial density, and differences in geometry and appearance even among spatially neighboring primitives. In this work, we introduce TriFold, a native 3D Gaussian VAE that encodes Gaussian scenes into a fixed-layout triplane latent and jointly reconstructs their geometry and appearance. The key insight is to impose structural regularity on the latent space while preserving the spatial adaptivity of the underlying Gaussians. During encoding, we use cell-wise learned queries to aggregate projected Gaussians into multiple slots per cell, reducing the information loss caused by single-feature pooling. During decoding, occupancy prediction identifies active regions, where query-based set prediction and candidate selection reconstruct local Gaussian sets with spatially varying primitive counts. Building on this representation, we train a latent flow-matching model for conditional scene-level 3DGS generation, enabling point-cloud-conditioned synthesis and single-image-conditioned amodal generation. Experiments demonstrate that TriFold achieves high-fidelity reconstruction and generation from a single image that provides only a partial observation of the scene, validating its effectiveness for native 3D Gaussian representation learning.
The triplane encoder maps a variable-cardinality Gaussian set to a fixed-layout variational latent, and the triplane decoder reconstructs spatially adaptive local Gaussian sets. The learned latent further supports image- and point-cloud-conditioned scene generation with flow matching.
TriFold is a variational autoencoder that operates directly on native 3D Gaussian primitives. The triplane encoder extracts context-aware primitive features with a point transformer (PTv3) and projects them onto three orthogonal planes. Instead of pooling all primitives that collide in a cell into a single vector, Q learned queries per cell aggregate them into multiple feature slots, and a sparse cross-plane attention module exchanges information between corresponding cells before a variational bottleneck produces the triplane latent.
The triplane decoder adopts a set-based decoding paradigm without rigid 3D discretization. An occupancy predictor first localizes active regions; for each occupied voxel, a DETR-style set decoder predicts Gaussian candidates, and entropy-regularized optimal transport matches them to the local target set. The predefined query count only bounds local capacity — both the spatial allocation and the per-voxel primitive count adapt to scene content. Gaussian centers are supervised through optimal-transport matching, while the remaining geometric and appearance attributes are constrained by a full-scene rendering loss.
On the frozen latent space, a conditional latent flow-matching model supports point-cloud-conditioned synthesis from complete spatial observations and single-image-conditioned amodal generation, inferring plausible geometry and appearance for unobserved regions.
We compare TriFold against Can3Tok, DiffGS, and GaussianGPT under the same scene-level training and evaluation protocol. TriFold achieves the best results across all metrics, improving PSNR over GaussianGPT by 5.15 dB while reducing LPIPS and Chamfer Distance.
Compare the ground-truth scene-level 3DGS against the TriFold VAE reconstruction interactively. Pick a scene on the left, toggle the variant on the right. The viewer loads on demand (compressed PLY, 11–36 MB per model). Drag to orbit, scroll to zoom, right-drag to pan.
The TriFold VAE reconstruction preserves layout, appearance, and fine-grained structures of the ground-truth scene. Camera starts at the same viewpoint used in the comparison videos below.
| Method | PSNR ↑ | SSIM ↑ | LPIPS ↓ | CD (×103) ↓ |
|---|---|---|---|---|
| Can3Tok | 4.53 | 0.012 | 0.920 | 10.96 |
| DiffGS | 12.43 | 0.719 | 0.374 | 3.21 |
| GaussianGPT | 21.40 | 0.856 | 0.157 | 2.62 |
| TriFold (Ours) | 26.55 | 0.913 | 0.088 | 2.23 |
Quantitative comparison with existing 3D Gaussian representation methods. Best results are in bold, second-best are underlined.
Qualitative VAE reconstruction comparison. Compared with GaussianGPT, TriFold better captures scene geometry and appearance.
VAE reconstruction demo (GT | TriFold (Ours) | GaussianGPT).
VAE reconstruction demo (GT | TriFold (Ours) | GaussianGPT).
Adaptive 3DGS allocation. The decoder retains a content-dependent number of Gaussians per occupied voxel (heatmap, right), instead of a fixed per-cell budget.
Additional VAE reconstructions. Under the identical evaluation setup, TriFold preserves more visible layout, appearance, and fine-grained local structures.
Representative reconstruction failures of the adapted Can3Tok and DiffGS baselines. Under the same scene-level training split and optimization budget, distinct input scenes can be reconstructed as similar low-detail structures.
Built on the TriFold latent space, a latent flow-matching model supports two conditioning settings: point-cloud-conditioned synthesis from complete spatial observations, and single-image-conditioned amodal generation, where a single image provides only a partial observation of the scene.
Explore 3DGS scenes generated by TriFold from a single input image directly in your browser. Click a thumbnail below to load the corresponding scene (compressed PLY, 28–52 MB; first load takes a few seconds). Drag to orbit, scroll to zoom, right-drag to pan. The thumbnails show the conditioning input images.
Scene-level 3DGS generated by TriFold (465K–849K Gaussians each, full spherical harmonics), rendered in-browser with GaussianSplats3D. Thumbnails are the single conditioning images.
Point-cloud-conditioned generation. Given a full point cloud, the latent flow model generates native 3DGS with coherent geometry and appearance across novel views.
Additional point-cloud-conditioned generations. Each panel shows a Gaussian scene sampled from a different point-cloud condition, illustrating the diversity of generated indoor layouts and scene content.
Single-image-conditioned scene generation demos (Input Image | TriFold (Ours)).
