📢 News

• 2025.1.24: 🎉🎉🎉 GenPercept has been accepted by ICLR 2025. 🎉🎉🎉
• 2024.10.25: Update GenPercept Huggingface App demo.
• 2024.10.24: Release latest training and inference code, which is armed with the accelerate library and based on Marigold.
• 2024.10.24: Release arXiv v3 paper. We reorganize the structure of the paper and offer more detailed analysis.
• 2024.4.30: Release checkpoint weights of surface normal and dichotomous image segmentation.
• 2024.4.7: Add HuggingFace App demo.
• 2024.4.6: Release inference code and depth checkpoint weight of GenPercept in the GitHub repo.
• 2024.3.15: Release arXiv v2 paper, with supplementary material.
• 2024.3.10: Release arXiv v1 paper.

📚 Download Resource Summary

• Space-Huggingface demo: https://huggingface.co/spaces/guangkaixu/GenPercept.
• Models-all (including ablation study): https://huggingface.co/guangkaixu/genpercept-exps.
• Models-main-paper: https://huggingface.co/guangkaixu/genpercept-models.
• Models-depth: https://huggingface.co/guangkaixu/genpercept-depth.
• Models-normal: https://huggingface.co/guangkaixu/genpercept-normal.
• Models-dis: https://huggingface.co/guangkaixu/genpercept-dis.
• Models-matting: https://huggingface.co/guangkaixu/genpercept-matting.
• Models-seg: https://huggingface.co/guangkaixu/genpercept-seg.
• Models-disparity: https://huggingface.co/guangkaixu/genpercept-disparity.
• Models-disparity-dpt-head: https://huggingface.co/guangkaixu/genpercept-disparity-dpt-head.
• Datasets-input demo: https://huggingface.co/datasets/guangkaixu/genpercept-input-demo.
• Datasets-evaluation data: https://huggingface.co/datasets/guangkaixu/genperceptdatasetseval.
• Datasets-evaluation results: https://huggingface.co/datasets/guangkaixu/genpercept-exps-eval.

🖥️ Dependencies

bash conda create -n genpercept python=3.10 conda activate genpercept pip install -r requirements.txt pip install -e .

🚀 Inference

Using Command-line Scripts

Download the stable-diffusion-2-1 and our trained models from HuggingFace and put the checkpoints under ./pretrained_weights/ and ./weights/, respectively. You can download them with the script script/download_sd21.sh and script/download_weights.sh, or download the weights of depth, normal, Dichotomous Image Segmentation, matting, segmentation, disparity, disparitydpthead seperately.

Then, place images in the ./input/ dictionary. We offer demo images in Huggingface, and you can also download with the script script/download_sample_data.sh. Then, run inference with scripts as below.

```bash

Depth

source script/infer/mainpaper/inferencegenpercept_depth.sh

Normal

source script/infer/mainpaper/inferencegenpercept_normal.sh

Dis

source script/infer/mainpaper/inferencegenpercept_dis.sh

Matting

source script/infer/mainpaper/inferencegenpercept_matting.sh

Seg

source script/infer/mainpaper/inferencegenpercept_seg.sh

Disparity

source script/infer/mainpaper/inferencegenpercept_disparity.sh

Disparitydpthead

source script/infer/mainpaper/inferencegenperceptdisparitydpt_head.sh ```

If you would like to change the input folder path, unet path, and output path, input these parameters like: ```bash

Assign a values

inputrgbdir=... unet=... output_dir=...

Take depth as example

source script/infer/mainpaper/inferencegenperceptdepth.sh $inputrgbdir $unet $outputdir For a general inference script, please seescript/infer/inference_general.sh``` in detail.

Thanks to our one-step perception paradigm, the inference process runs much faster. (Around 0.4s for each image on an A800 GPU card.)

Using torch.hub

TODO

🔥 Train

NOTE: We implement the training with the accelerate library, but find a worse training accuracy with multi gpus compared to one gpu, with the same training effective_batch_size and max_iter. Your assistance in resolving this issue would be greatly appreciated. Thank you very much!

Preparation

Datasets: TODO

Place training datasets unser datasets/

Download the stable-diffusion-2-1 from HuggingFace and put the checkpoints under ./pretrained_weights/. You can also download with the script script/download_sd21.sh.

Start Training

The reproduction training scripts in arxiv v3 paper is released in script/, whose configs are stored in config/. Models with max_train_batch_size > 2 are trained on an H100 and max_train_batch_size <= 2 on an RTX 4090. Run the train script:

```bash

Take depth training of main paper as an example

source script/trainsd21mainpaper/sd21trainaccelerategenpercept1cardensuredepthbs8peraccupixelmsessigrad_loss.sh ```

🎖️ Eval

Preparation

1. Download evaluation datasets and place them in datasets_eval.
2. Download our trained models of main paper and ablation study in Section 3 of arxiv v3 paper, and place them in weights/genpercept-exps.

Start Evaluation

The evaluation scripts are stored in script/eval_sd21.

```bash

Take "ensemble1 + step1" as an example

source script/evalsd21/evalensemble1step1/0inferevalall.sh ```

📖 Recommanded Works

• Marigold: Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation. arXiv, GitHub.
• GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Image. arXiv, GitHub.
• FrozenRecon: Pose-free 3D Scene Reconstruction with Frozen Depth Models. arXiv, GitHub.

👍 Results in Paper

Depth and Surface Normal

Dichotomous Image Segmentation

Image Matting

Image Segmentation

🎫 License

For non-commercial academic use, this project is licensed under the 2-clause BSD License. For commercial use, please contact Chunhua Shen.

🎓 Citation

@article{xu2024diffusion, title={What Matters When Repurposing Diffusion Models for General Dense Perception Tasks?}, author={Xu, Guangkai and Ge, Yongtao and Liu, Mingyu and Fan, Chengxiang and Xie, Kangyang and Zhao, Zhiyue and Chen, Hao and Shen, Chunhua}, journal={arXiv preprint arXiv:2403.06090}, year={2024} }