Gaussian Splatting Decoder for 3D‑aware Generative Adversarial Networks

Accepted at CVPRW 2024

Abstract

NeRF-based 3D-aware Generative Adversarial Networks like EG3D or GIRAFFE have shown very high rendering quality under large representational variety. However, rendering with Neural Radiance Fields poses several challenges for most 3D applications: First, the significant computational demands of NeRF rendering preclude its use on low-power devices, such as mobiles and VR/AR headsets. Second, implicit representations based on neural networks are difficult to incorporate into explicit 3D scenes, such as VR environments or video games. 3D Gaussian Splatting (3DGS) overcomes these limitations by providing an explicit 3D representation that can be rendered efficiently at high frame rates. In this work, we present a novel approach that combines the high rendering quality of NeRF-based 3D-aware Generative Adversarial Networks with the flexibility and computational advantages of 3DGS. By training a decoder that maps implicit NeRF representations to explicit 3D Gaussian Splatting attributes, we can integrate the representational diversity and quality of 3D GANs into the ecosystem of 3D Gaussian Splatting for the first time. Additionally, our approach allows for a high resolution GAN inversion and real-time GAN editing with 3D Gaussian Splatting scenes.

Method

We propose a novel 3D Gaussian Splatting decoder that converts high quality results from pre-trained 3D-aware GANs into Gaussian Splatting scenes in real-time for efficient and high resolution rendering.

Above is a visualization of our method (orange parts are optimized). We initially clone the backbone of the 3D-aware GAN. Afterwards, we iteratively optimize the Gaussian Splatting decoder by comparing the output of the pre-trained GAN, after super-resolution, with the output of the decoder. The xyz coordinates at which the tri-plane is sampled originates from the density information of the NeRF renderer.

Latent Interpolation

Here we convert the EG3D scenes, trained on FFHQ, directly into 3D Gaussian Splatting scenes in real-time. We use this python based 3D Gaussian splatting viewer: Gaussian-Viewer

Citation

@INPROCEEDINGS{10677856,
  author={Barthel, Florian and Beckmann, Arian and Morgenstern, Wieland and Hilsmann, Anna and Eisert, Peter},
  booktitle={2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)},
  title={Gaussian Splatting Decoder for 3D-aware Generative Adversarial Networks},
  year={2024},
  volume={},
  number={},
  pages={7963-7972},
  keywords={Training;Visualization;Video games;Three-dimensional displays;Image resolution;Generative adversarial networks;Rendering (computer graphics);3D Gaussian Splatting;Gaussian Splatting Decoder;3D-aware GANs;NeRF2Gaussian},
  doi={10.1109/CVPRW63382.2024.00794}}

Check out my other publications here.

License

The structure of this page is taken and modified from nvlabs.github.io/eg3d which was published under the Creative Commons CC BY-NC 4.0 license .

Acknowledgments

This research has partly been funded by the German Research Foundation (3DIL, grant no. 502864329), the European Union's Horizon Europe research and innovation programme (Luminous, grant no. 101135724), and the German Ministry of Education and Research (MoDL, grant no. 01IS20044).

Gaussian web viewer: github.com/quadjr/aframe-gaussian-splatting