If you find DiffuScene useful for your work please cite: ``` @inproceedings{ tang2023diffuscene, title={DiffuScene: Scene Graph Denoising Diffusion Probabilistic Model for Generative Indoor Scene Synthesis}, author={Tang, Jiapeng and Nie Yinyu and Markhasin Lev and Dai Angela and Thies Justus and Nie{\ss}ner, Matthias}, booktitle={arxiv}, year={2023}, } ``` Contact [Jiapeng Tang](mailto:tangjiapengtjp@gmail.com) for questions, comments and reporting bugs. ## Installation & Dependencies You can create a conda environment called ```diffuscene``` using ``` conda env create -f environment.yaml conda activate diffuscene ``` Next compile the extension modules. You can do this via ``` python setup.py build_ext --inplace pip install -e . ``` Install ChamferDistancePytorch ``` cd ChamferDistancePytorch/chamfer3D python setup.py install ``` ## Dataset The training and evaluation are based on the [3D-FRONT](https://tianchi.aliyun.com/specials/promotion/alibaba-3d-scene-dataset) and the [3D-FUTURE](https://www.google.com/search?q=3d-future&oq=3d-fut&aqs=chrome.1.69i57j0j0i30l8.3909j0j7&sourceid=chrome&ie=UTF-8) dataset. To download both datasets, please refer to the instructions provided in the dataset's [webpage](https://tianchi.aliyun.com/specials/promotion/alibaba-3d-scene-dataset). ### Pickle the 3D-FUTURE dataset To accelerate the preprocessing speed, we can sepcify the `PATH_TO_SCENES` environment variable for all scripts. This filepath contains the parsed `ThreedFutureDataset` after being pickled. To pickle it, you can simply run this script as follows: ``` python pickle_threed_future_dataset.py path_to_output_dir path_to_3d_front_dataset_dir path_to_3d_future_dataset_dir path_to_3d_future_model_info --dataset_filtering room_type ``` Based on the pickled ThreedFutureDataset, we also provide a script to pickle the sampled point clouds of object CAD models, which are used to shape autoencoder training and latent shape code extraction. ``` python pickle_threed_future_pointcloud.py path_to_output_dir path_to_3d_front_dataset_dir path_to_3d_future_dataset_dir path_to_3d_future_model_info --dataset_filtering room_type ``` For example, ``` python pickle_threed_future_dataset.py /cluster/jtang/3d_front_processed/ /cluster/jtang/3D-FRONT/ /cluster/jtang/3D-FUTURE-model /cluster/balrog/jtang/3D-FUTURE-model/model_info.json --dataset_filtering threed_front_livingroom --annotation_file ../config/livingroom_threed_front_splits.csv PATH_TO_SCENES="/cluster/jtang/3d_front_processed/threed_front.pkl" python pickle_threed_fucture_pointcloud.py /cluster/jtang/3d_front_processed/ /cluster/jtang/3D-FRONT/ /cluster/jtang/3D-FUTURE-model /cluster/jtang/3D-FUTURE-model/model_info.json --dataset_filtering threed_front_livingroom --annotation_file ../config/livingroom_threed_front_splits.csv ``` Note that these two scripts should be separately executed for different room types containing different objects. For the case of 3D-FRONT this is for the bedrooms and the living/dining rooms, thus you have to run this script twice with different `--dataset_filtering` and `--annotation_file`options. Please check the help menu for additional details. ### Train shape autoencoder Then you can train the shape autoencoder using all models from bedrooms/diningrooms/livingrooms. ``` cd ./scripts PATH_TO_SCENES="/cluster/jtang/3d_front_processed/threed_front.pkl" python train_objautoencoder.py ../config/obj_autoencoder/bed_living_diningrooms_lat32.yaml your_objae_output_directory --experiment_tag "bed_living_diningrooms_lat32" --with_wandb_logger ``` ### Pickle Latent Shape Code Next, you can use the pre-train checkpoint of shape autoencoder to extract latent shape codes for each room type. Take the bedrooms for example: ``` PATH_TO_SCENES="/cluster/jtang/3d_front_processed/threed_front.pkl" python generate_objautoencoder.py ../config/objautoencoder/bedrooms.yaml your_objae_output_directory --experiment_tag "bed_living_diningrooms_lat32" ``` ### Preprocess 3D-Front dataset with latent shape codes Finally, you can run `preprocessing_data.py` to read and pickle object properties (class label, location, orientation, size, and latent shape features) of each scene. ``` PATH_TO_SCENES="/cluster/jtang/3d_front_processed/threed_front.pkl" python preprocess_data.py /cluster/jtang/3d_front_processed/livingrooms_objfeats_32_64 /cluster/jtang/3D-FRONT/ /cluster/jtang/3D-FUTURE-model /cluster/jtang/3D-FUTURE-model/model_info.json --dataset_filtering threed_front_livingroom --annotation_file ../config/livingroom_threed_front_splits.csv --add_objfeats ``` The proprossed datasets can also be downloaded from [here](https://drive.google.com/file/d/1UNSFN0kULyOzUErDPVvkKYbmfzA-4MsG/view?usp=sharing). ## Training & Evaluate Diffuscene To train diffuscene on 3D Front-bedrooms, you can run ``` ./run/train.sh ``` To generate the scene of unconditional diffusion model, you can run ``` ./run/generate.sh ``` ## Relevant Research Please also check out the following papers that explore similar ideas: - Fast and Flexible Indoor Scene Synthesis via Deep Convolutional Generative Models [pdf](https://arxiv.org/pdf/1811.12463.pdf) - Sceneformer: Indoor Scene Generation with Transformers [pdf](https://arxiv.org/pdf/2012.09793.pdf) - ATISS: Autoregressive Transformers for Indoor Scene Synthesis [pdf](https://arxiv.org/pdf/2110.03675.pdf) - Indoor Scene Generation from a Collection of Semantic-Segmented Depth Images [pdf](https://arxiv.org/abs/2108.09022) - Scene Synthesis via Uncertainty-Driven Attribute Synchronization [pdf](https://arxiv.org/abs/2108.13499) - LEGO-Net: Learning Regular Rearrangements of Objects in Rooms[pdf](https://arxiv.org/abs/2301.09629) ## Acknowledgements Most of the code is borrowed from [ATISS](https://github.com/nv-tlabs/ATISS). We thank for Despoina Paschalidou her great works and repos.