Neural Scene Designer: Self-Styled Semantic Image Manipulation

Detailed inference and training procedures coming soon.

Getting Started

Environment Requirement 🌍

NSD has been implemented and tested on Pytorch 1.12.1 with Python 3.9.

Clone the repo and setup environment

bash git clone https://github.com/jianmanlincjx/NSD.git conda create -n diffusers python=3.9 -y conda activate diffusers python -m pip install --upgrade pip pip install torch==1.12.1+cu116 torchvision==0.13.1+cu116 torchaudio==0.12.1 --extra-index-url https:// pip install -e . cd examples/NSD/ pip install -r requirements.txt

Data Download

NSD uses BrushData and BrushBench for training and testing. You can download the dataset through this link. At the same time, NSD proposes an indoor dataset for specialized self-styled editing of indoor scenes. This dataset is still being organized. Once the dataset is ready, you can organize it in JSON format within the "data" folder for training.

Running Scripts

Training 🤖

You can train with segmentation mask using the script:

```bash

sd v1.5

# sd v1.5 follow IP-adapter

accelerate launch --numprocesses 3 examples/NSD/trainNSD.py \ --pretrainedmodelnameorpath /data1/JM/code/NSD/pretrainmodel/stable-diffusion-v1-5 \ --outputdir runs/logs/brushnetsegmentationmask \ --resolution 512 \ --learningrate 1e-5 \
--trainbatchsize 2 \ --trackerprojectname NSD \ --reportto tensorboard \ --resumefromcheckpoint latest \ --checkpointingsteps 100000 \ --jsonfile /data1/JM/code/NSD/data/trainsmall.json \ --brushnetmodelnameorpath /data1/JM/code/NSD/pretrainmodel/segmentationmaskbrushnetckpt \ --mixedprecision 'fp16' \ --validationimage /data1/JM/code/NSD/data/datatrainsmall/image/000075.png \ --validationmask /data1/JM/code/NSD/data/datatrainsmall/mask/chair/000075.png \ --validationprompt 'A delicate sofa in the room. ' \ --validationsteps 1000 \ --imageencoderpath /data1/JM/code/NSD/pretrainmodel/image_encoder ```

Inference 🎨

bash python examples/NSD/test_NSD.py

🚨 Important Notes

• Ensure all pre-trained models are correctly downloaded and placed in the specified locations
• Training can be performed on a GPU with 24GB VRAM (e.g., RTX 3090, RTX 4090)
• For inference, a GPU with 12GB VRAM is sufficient
• CUDA 11.7 or higher is recommended