https://github.com/airen3339/fatezero
Science Score: 23.0%
This score indicates how likely this project is to be science-related based on various indicators:
-
○CITATION.cff file
-
✓codemeta.json file
Found codemeta.json file -
○.zenodo.json file
-
○DOI references
-
✓Academic publication links
Links to: arxiv.org, scholar.google -
○Academic email domains
-
○Institutional organization owner
-
○JOSS paper metadata
-
○Scientific vocabulary similarity
Low similarity (11.0%) to scientific vocabulary
Last synced: 9 months ago
·
JSON representation
Repository
Basic Info
- Host: GitHub
- Owner: airen3339
- License: mit
- Language: Jupyter Notebook
- Default Branch: main
- Size: 190 MB
Statistics
- Stars: 0
- Watchers: 1
- Forks: 0
- Open Issues: 0
- Releases: 0
Created about 3 years ago
· Last pushed about 3 years ago
Metadata Files
Readme
License
README.md
FateZero : Fusing Attentions for Zero-shot Text-based Video Editing
Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen
 |
 |
 |
| "silver jeep ➜ posche car" | "+ Van Gogh style" | "squirrel,Carrot ➜ rabbit,eggplant" |
CLICK for the full abstract
> The diffusion-based generative models have achieved remarkable success in text-based image generation. However, since it contains enormous randomness in generation progress, it is still challenging to apply such models for real-world visual content editing, especially in videos. In this paper, we propose FateZero, a zero-shot text-based editing method on real-world videos without per-prompt training or use-specific mask. To edit videos consistently, we propose several techniques based on the pre-trained models. Firstly, in contrast to the straightforward DDIM inversion technique, our approach captures intermediate attention maps during inversion, which effectively retain both structural and motion information. These maps are directly fused in the editing process rather than generated during denoising. To further minimize semantic leakage of the source video, we then fuse self-attentions with a blending mask obtained by cross-attention features from the source prompt. Furthermore, we have implemented a reform of the self-attention mechanism in denoising UNet by introducing spatial-temporal attention to ensure frame consistency. Yet succinct, our method is the first one to show the ability of zero-shot text-driven video style and local attribute editing from the trained text-to-image model. We also have a better zero-shot shape-aware editing ability based on the text-to-video model. Extensive experiments demonstrate our superior temporal consistency and editing capability than previous works.Click for xformers installation
We find its installation not stable. You may try the following wheel: ```bash wget https://github.com/ShivamShrirao/xformers-wheels/releases/download/4c06c79/xformers-0.0.15.dev0+4c06c79.d20221201-cp38-cp38-linux_x86_64.whl pip install xformers-0.0.15.dev0+4c06c79.d20221201-cp38-cp38-linux_x86_64.whl ```Click for the bash command:
``` mkdir ./ckpt cd ./ckpt # download from huggingface face, takes 20G space git lfs install git clone https://huggingface.co/CompVis/stable-diffusion-v1-4 ```The result is saved at `./result` . (Click for directory structure)
``` result ├── teaser │ ├── jeep_posche │ ├── jeep_watercolor │ ├── cross-attention # visualization of cross-attention during inversion │ ├── sample # result │ ├── train_samples # the input video ```Click for the bash command:
``` mkdir ./ckpt cd ./ckpt # download from huggingface face, takes 10G space git lfs install git clone https://huggingface.co/chenyangqi/jeep_tuned_200 ```The directory structure should be like this: (Click for directory structure)
``` ckpt ├── stable-diffusion-v1-4 ├── jeep_tuned_200 ... data ├── car-turn │ ├── 00000000.png │ ├── 00000001.png │ ├── ... video_diffusion ```Click for wget bash command:
``` wget https://github.com/ChenyangQiQi/FateZero/releases/download/v0.0.1/attribute.zip wget https://github.com/ChenyangQiQi/FateZero/releases/download/v0.0.1/style.zip wget https://github.com/ChenyangQiQi/FateZero/releases/download/v0.0.1/shape.zip ```Click for the bash command:
``` mkdir ./ckpt cd ./ckpt # download from huggingface face, takes 10G space git lfs install git clone https://huggingface.co/chenyangqi/man_skate_250 git clone https://huggingface.co/chenyangqi/swan_150 ```Click for the bash command example:
``` cd ./data wget https://github.com/ChenyangQiQi/FateZero/releases/download/v0.0.1/negative_reg.zip unzip negative_reg cd .. accelerate launch train_tune_a_video.py --config config/tune/jeep.yaml ``` |
 |
 |
| "+ Ukiyo-e style" | "+ watercolor painting" | "+ Monet style" |
 |
 |
 |
| "+ Pokémon cartoon style" | "+ Makoto Shinkai style" | "+ cartoon style" |
Attribute Editing Results with Stable Diffusion
 |
 |
 |
| "rabbit, strawberry ➜ white rabbit, flower" | "rabbit, strawberry ➜ squirrel, carrot" | "rabbit, strawberry ➜ white rabbit, leaves" |
 |
|
 |
| "squirrel ➜ robot squirrel" | "squirrel, Carrot ➜ rabbit, eggplant" | "squirrel, Carrot ➜ robot mouse, screwdriver" |
 |
 |
 |
| "bear ➜ a red tiger" | "bear ➜ a yellow leopard" | "bear ➜ a brown lion" |
 |
 |
 |
| "cat ➜ black cat, grass..." | "cat ➜ red tiger" | "cat ➜ Shiba-Inu" |
 |
 |
 |
| "bus ➜ GPU" | "gray dog ➜ yellow corgi" | "gray dog ➜ robotic dog" |
 |
 |
 |
| "white duck ➜ yellow rubber duck" | "grass ➜ snow" | "white fox ➜ grey wolf" |
Shape and large motion editing with Tune-A-Video
 |
 |
 |
| "silver jeep ➜ posche car" | "Swan ➜ White Duck" | "Swan ➜ Pink flamingo" |
 |
 |
 |
| "A man ➜ A Batman" | "A man ➜ A Wonder Woman, With cowboy hat" | "A man ➜ A Spider-Man" |
🕹 Online Demo
Thanks to AK and the team from Hugging Face for providing computing resources to support our Hugging-face Demo, which supports up to 30 steps DDIM steps.
You may use the UI for testing FateZero built with gradio locally. ``` git clone https://huggingface.co/spaces/chenyangqi/FateZero python app_fatezero.py
we will merge the FateZero on hugging face with that in github repo latter
```
We also provide a Colab demo, which supports 10 DDIM steps.
You may launch the colab as a jupyter notebook on your local machine.
We will refine and optimize the above demos in the following days.
📀 Demo Video
https://user-images.githubusercontent.com/45789244/225698509-79c14793-3153-4bba-9d6e-ede7d811d7f8.mp4
The video here is compressed due to the size limit of GitHub. The original full-resolution video is here.
📍 Citation
@article{qi2023fatezero,
title={FateZero: Fusing Attentions for Zero-shot Text-based Video Editing},
author={Chenyang Qi and Xiaodong Cun and Yong Zhang and Chenyang Lei and Xintao Wang and Ying Shan and Qifeng Chen},
year={2023},
journal={arXiv:2303.09535},
}
💗 Acknowledgements
This repository borrows heavily from Tune-A-Video and prompt-to-prompt. Thanks to the authors for sharing their code and models.
🧿 Maintenance
This is the codebase for our research work. We are still working hard to update this repo, and more details are coming in days. If you have any questions or ideas to discuss, feel free to contact Chenyang Qi or Xiaodong Cun.
Owner
- Login: airen3339
- Kind: user
- Repositories: 187
- Profile: https://github.com/airen3339
GitHub Events
Total
Last Year
Dependencies
requirements.txt
pypi
- accelerate ==0.15.0
- bitsandbytes ==0.35.4
- click *
- diffusers ==0.11.1
- einops *
- ftfy *
- imageio *
- modelcards *
- omegaconf *
- opencv-python *
- tensorboard *
- transformers ==4.25.1
- triton *