https://github.com/ansj11/learnneuralrecon

https://github.com/ansj11/learnneuralrecon

Repository

https://github.com/ansj11/LearnNeuralRecon/blob/main/

# NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video
### [Project Page](https://zju3dv.github.io/neuralrecon) | [Paper](https://arxiv.org/pdf/2104.00681.pdf)



> NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video  
> [Jiaming Sun](https://jiamingsun.ml)\*, [Yiming Xie](https://ymingxie.github.io)\*, [Linghao Chen](https://github.com/f-sky), [Xiaowei Zhou](http://www.cad.zju.edu.cn/home/xzhou/), [Hujun Bao](http://www.cad.zju.edu.cn/bao/)  
> CVPR 2021 (Oral Presentation)


![real-time video](assets/neucon-demo.gif)




## TODO List and ETA
- [x] Code (with detailed comments) for training and inference, and the data preparation scripts (2021-5-2).
- [x] Pretrained models on ScanNet (2021-5-2).
- [x] Real-time reconstruction demo on custom ARKit data with instructions (2021-5-7).
- [ ] Evaluation code and metrics (expected 2021-6-10).

## How to Use

### Installation
```shell
# Ubuntu 18.04 and above is recommended.
sudo apt install libsparsehash-dev  # you can try to install sparsehash with conda if you don't have sudo privileges.
conda env create -f environment.yaml
conda activate neucon
```


### Pretrained Model on ScanNet
Download the [pretrained weights](https://drive.google.com/file/d/1zKuWqm9weHSm98SZKld1PbEddgLOQkQV/view?usp=sharing) and put it under 
`PROJECT_PATH/checkpoints/`.
You can also use [gdown](https://github.com/wkentaro/gdown) to download it in command line:
```bash
mkdir checkpoints && cd checkpoints
gdown --id 1zKuWqm9weHSm98SZKld1PbEddgLOQkQV
```

### Real-time Demo on Custom Data with Camera Poses from ARKit.
We provide a real-time demo of NeuralRecon running with self-captured ARKit data.
Please refer to [DEMO.md](DEMO.md) for details.

### Data Preperation for ScanNet
Download and extract ScanNet by following the instructions provided at http://www.scan-net.org/.

  
[Expected directory structure of ScanNet (click to expand)]

```
DATAROOT
scannet
   scans
   |   scene0000_00
   |       color
   |             0.jpg
   |             1.jpg
   |             ...
   |          ...
   scans_test
   |   scene0707_00
   |       color
   |             0.jpg
   |             1.jpg
   |             ...
   |          ...
|   scannetv2_test.txt
|   scannetv2_train.txt
|   scannetv2_val.txt
```


Next run the data preparation script which parses the raw data format into the processed pickle format.
This script also generates the ground truth TSDFs using TSDF Fusion.  

  
[Data preparation script]

```bash
# Change PATH_TO_SCANNET and OUTPUT_PATH accordingly.
# For the training/val split:
python tools/tsdf_fusion/generate_gt.py --data_path PATH_TO_SCANNET --save_name all_tsdf_9 --window_size 9
# For the test split
python tools/tsdf_fusion/generate_gt.py --test --data_path PATH_TO_SCANNET --save_name all_tsdf_9 --window_size 9
```



### Inference on ScanNet test-set
```bash
python main.py --cfg ./config/test.yaml
```

The reconstructed meshes will be saved to `LOGDIR`, which is `./checkpoints/` by default.

### Training on ScanNet

Start training by running `./train.sh`.
More info about training (e.g. GPU requirements, convergence time, etc.) to be added soon.

  
[train.sh]

```bash
#!/usr/bin/env bash
export CUDA_VISIBLE_DEVICES=0,1
python -m torch.distributed.launch --nproc_per_node=2 main.py --cfg ./config/train.yaml
```


The training is seperated to two phases and the switching between phases is controlled manually for now:

-  Phase 1 (the first 0-20 epoch), training single fragments.
`MODEL.FUSION.FUSION_ON=False, MODEL.FUSION.FULL=False`


- Phase 2 (the remaining 21-50 epoch), with `GRUFusion`.
`MODEL.FUSION.FUSION_ON=True, MODEL.FUSION.FULL=True`

## Citation

If you find this code useful for your research, please use the following BibTeX entry.

```bibtex
@article{sun2021neucon,
  title={{NeuralRecon}: Real-Time Coherent {3D} Reconstruction from Monocular Video},
  author={Sun, Jiaming and Xie, Yiming and Chen, Linghao and Zhou, Xiaowei and Bao, Hujun},
  journal={CVPR},
  year={2021}
}
```

## Acknowledgment
We would like to specially thank Reviewer 3 for the insightful and constructive comments. We would like to thank Sida Peng , Siyu Zhang and Qi Fang for the proof-reading.
Some of the code in this repo is borrowed from [MVSNet_pytorch](https://github.com/xy-guo/MVSNet_pytorch), thanks Xiaoyang!

## Copyright
This work is affiliated with ZJU-SenseTime Joint Lab of 3D Vision, and its intellectual property belongs to SenseTime Group Ltd.

```
Copyright SenseTime. All Rights Reserved.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
```

Owner

GitHub Events