We offer a novel approach to image composition, which integrates multiple input images into a single, coherent image. Rather than concentrating on specific use cases such as appearance editing (image harmonization) or semantic editing (semantic image composition), we showcase the potential of utilizing the powerful generative prior inherent in large-scale pre-trained diffusion models to accomplish generic image composition applicable to both scenarios. We observe that the pre-trained diffusion models automatically identify simple copy-paste boundary areas as low-density regions during denoising. Building on this insight, we propose to optimize the composed image towards high-density regions guided by the diffusion prior. In addition, we introduce a novel mask-guided loss to further enable flexible semantic image composition. Extensive experiments validate the superiority of our approach in achieving generic zero-shot image composition. Additionally, our approach shows promising potential in various tasks, such as object removal and multi-concept customization. ## Method

FreeCompose overview. Our FreeCompose pipeline consists of three phases: object removal, image harmonization, and semantic image composition. In each phase, the pipeline takes an input image and two text prompts to calculate the loss. In the object removal phase, an additional mask is required to select K, V values. In the semantic image composition phase, text prompts can be replaced by other formats, and an additional K, V replacement is implemented for identity consistency. ## Demos ### Object Removal

### Imgae Harmonization

### Semantic Image Composition

### SDXL Plug-and-Play

## Code Structure - `assets/`: contains the data for cases. - `eval/`: contains the evaluation code, including `fid`, `lpips` and so on. - `utils/`: contains the utility functions for the project. - `main.ipynb`: the main notebook for the project. ## Environment Setup ```bash conda create -n freecompose python=3.9.11 conda activate freecompose pip install -r requirements.txt ``` Note: Because of proxy reason, we use *[hf-mirror]()* as the mirror for huggingface. However, maybe you need to use the original mirror. Then, comment or delete the following line in some files: ```python os.environ['HF_ENDPOINT'] = "https://hf-mirror.com" ``` ## Usage We provide a notebook for the project. You can run the notebook to see the results of the project. ## Acknowledgement We would like to thank the authors of the following repositories for their great work: - [Prompt-2-prompt](https://github.com/google/prompt-to-prompt) - [Controlcom](https://github.com/bcmi/ControlCom-Image-Composition) - [Paint-by-Example](https://github.com/Fantasy-Studio/Paint-by-Example) ## License For non-commercial academic use, this project is licensed under [the 2-clause BSD License](https://opensource.org/license/bsd-2-clause). For commercial use, please contact [Chunhua Shen](mailto:chhshen@gmail.com). ## Citing If you find our work useful, please consider citing: ``` @article{FreeCompose, title={FreeCompose: Generic Zero-Shot Image Composition with Diffusion Prior}, author={Chen, Zhekai and Wang, Wen and Yang, Zhen and Yuan, Zeqing and Chen, Hao and Shen, Chunhua}, journal={arXiv preprint arXiv:2407.04947}, year={2024} } ```