We present a photograph relighting method that enables explicit control over light sources akin to CG pipelines. Users can insert different types of light sources, such as spot lights, point lights, or environmental illumination into the scene. We achieve this in a pipeline involving mid-level computer vision, physically-based rendering, and neural rendering. We introduce a self-supervised training methodology using differentiable rendering to train our neural renderer with real-world photograph collections for in-the-wild generalization.
Abstract
We present a self-supervised approach to in-the-wild image relighting that enables fully controllable, physically based illumination editing.
We achieve this by combining the physical accuracy of traditional rendering with the photorealistic appearance made possible by neural rendering.
Our pipeline works by inferring a colored mesh representation of a given scene using monocular estimates of geometry and intrinsic components.
This representation allows users to define their desired illumination configuration in 3D. The scene under the new lighting can then be rendered using a path-tracing engine.
We send this approximate rendering of the scene through a feed-forward neural renderer to predict the final photorealistic relighting result.
We develop…