Relighting Neural Radiance Fields with Shadow and Highlight Hints

Chong Zeng · Guojun Chen · Yue Dong · Pieter Peers · Hongzhi Wu · Xin Tong

SIGGRAPH 2023 Conference Proceedings

Project Page | Paper | arXiv | Data

Setup

Environment

The code is developed and tested on Linux servers with NVIDIA GPU(s). We support Python 3.8+ and PyTorch 1.11+. After getting a required Python environment, you can setup the rest of requirements by running:

bash git clone https://github.com/iamNCJ/NRHints.git cd NRHints pip install -r requirements.txt

Data

Our data is compatible with NeRF Blender Dataset, except that we have extra fields in each frame for point light position.

You can download our data here.

Usage

Configuration System

We use tyro for configuration management. Description to all configurations can be found by running python main.py -h.

Training

bash python3 main.py config:nr-hints --config.data.path /path/to/data/ --config.scene-name XXX

Refer to train_synthetic.sh and train_real.sh for training on synthetic and real data, respectively.

Note: 1. Our code automatically detects the number of GPUs and uses all of them for training. If you want to use a subset of GPUs, you can set the CUDA_VISIBLE_DEVICES environment variable. 2. For training on real captured scenes, we recommend turning on camera optimization by using config:nr-hints-cam-opt, which can significantly reduce the blurry effects. Since this is an improvement after the paper submission, details are described in the author's version.

Testing

bash python3 main.py config:nr-hints --config.data.path /path/to/data/ --config.scene-name XXX --config.evaluation-only True

Refer to eval_synthetic.sh and eval_real.sh for testing on synthetic and real data, respectively.

Our pretrained models can be downloaded here.

Data and Models

Real Captured Scenes

| Object | Data | Pre-trained model | | ----------- | :----------: | :-----------: | | Cat | Link | Link | | Cluttered Scene | Link | Link | | Pixiu Statuette | Link | Link | | Ornamental Fish | Link | Link | | Cat on Decor | Link | Link | | Cup and Fabric | Link | Link | | Pikachu Statuette | Part1 Part2 | Link |

Synthetic Rendered Scenes

Note: Our synthetic data rendering scripts are released at here.

| Object | Data | Pre-trained model | | ----------- | :----------: | :-----------: | | Diffuse | Link | Link | | Metallic | Link | Link | | Glossy-Metal | Link | Link | | Rough-Metal | Link | Link | | Anisotropic-Metal | Link | Link | | Plastic | Link | Link | | Glossy-Plastic | Link | Link | | Rough-Plastic | Link | Link | | Short-Fur | Link | Link | | Long-Fur | Link | Link | | Translucent | Link | Link | | Fur-Ball | Link | Link | | Basket | Link | Link | | Layered Woven Ball | Link | Link | | Drums | Link | Link | | Hotdog | Link | Link | | Lego | Link | Link |

You can use the script download_data.sh to download all data.

Citation

Cite as below if you find this repository is helpful to your project:

@inproceedings {zeng2023nrhints, title = {Relighting Neural Radiance Fields with Shadow and Highlight Hints}, author = {Chong Zeng and Guojun Chen and Yue Dong and Pieter Peers and Hongzhi Wu and Xin Tong}, booktitle = {ACM SIGGRAPH 2023 Conference Proceedings}, year = {2023} }

Acknowledgement

Some code snippets are borrowed from NeuS and Nerfstudio. Thanks for these great projects.