Computergrafik

SHINOBI: Shape and Illumination using Neural Object Decomposition via BRDF Optimization In-the-wild

Andreas Engelhardt1, Amit Raj2Mark Boss3, Yunthi Zhang4, Abhishek Kar2, Yuanzhen Li2, Deqing Sun2, Rcardo Martin Brualla2, Jonathan T. Barron2Hendrik P. A. Lensch1 and Varun Jampani3
University of Tübingen1 , Google2, Stability AI3, Stanford University4

CVPR 2024

Abstract

We present SHINOBI, an end-to-end framework for the reconstruction of shape, material, and illumination from object images captured with varying lighting, pose, and background. Inverse rendering of an object based on unconstrained image collections is a long-standing challenge in computer vision and graphics and requires a joint optimization over shape, radiance, and pose. We show that an implicit shape representation based on a multi-resolution hash encoding enables faster and robust shape reconstruction with joint camera alignment optimization that outperforms prior work. Further, to enable the editing of illumination and object reflectance (i.e. material) we jointly optimize BRDF and illumination together with the object’s shape. Our method is class-agnostic and works on in-the-wild image collections of objects to produce relightable 3D assets for several use cases such as AR/VR, movies, games, etc.

 

Links

Bibtex

@inproceedings{engelhardt2024-shinobi, author ={Engelhardt, Andreas and Raj, Amit and Boss, Mark and Zhang, Yunzhi and Kar, Abhishek and Li, Yuanzhen and Sun, Deqing and Barron, Jonathan T. and Lensch, Hendrik P.A. and Jampani, Varun},title ={{SHINOBI}: {Sh}ape and {I}llumination using {N}eural {O}bject Decomposition via {B}RDF Optimization {I}n-the-wild},booktitle ={CVPR},year ={2024}}