Data

Tools

Indexes

Applications

Experiments

Open Source Science

https://github.com/aim-uofa/styledrop-pytorch

This is an unofficial PyTorch implementation of StyleDrop: Text-to-Image Generation in Any Style.

https://github.com/aim-uofa/styledrop-pytorch

Science Score: 36.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ○
    DOI references
  • ✓
    Academic publication links
    Links to: arxiv.org
  • ○
    Academic email domains
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (10.2%) to scientific vocabulary
Last synced: 10 months ago · JSON representation

Repository

This is an unofficial PyTorch implementation of StyleDrop: Text-to-Image Generation in Any Style.

Basic Info
  • Host: GitHub
  • Owner: aim-uofa
  • License: mit
  • Language: Python
  • Default Branch: main
  • Homepage:
  • Size: 43.1 MB
Statistics
  • Stars: 221
  • Watchers: 10
  • Forks: 15
  • Open Issues: 7
  • Releases: 0
Created almost 3 years ago · Last pushed almost 3 years ago
Metadata Files
Readme License

README.md

StyleDrop

Huggingface

This is an unofficial PyTorch implementation of StyleDrop: Text-to-Image Generation in Any Style.

Unlike the parameters in the paper in (Round 1), we set $\lambdaA=2.0$, $\lambdaB=5.0$ and d_prj=32, is_shared=False, which we found work better, these hyperparameters can be seen in configs/custom.py.

we release them to facilitate community research.

result1

result2

result3

result4

result5

News

  • [07/06/2023] Online Gradio Demo is available here

Todo List

  • [x] Release the code.
  • [x] Add gradio inference demo (runs in local).
  • [ ] Add iterative training (Round 2).

Data & Weights Preparation

First, download VQGAN from this link (from MAGE, thanks!), and put the downloaded VQGAN in assets/vqgan_jax_strongaug.ckpt.

Then, download the pre-trained checkpoints from this link to assets/ckpts for evaluation or to continue training for more iterations.

finally, prepare emptyfeature by runnig command `python extractempty_feature.py`

And the final directory structure is as follows: ``` . ├── assets │ ├── ckpts │ │ ├── cc3m-285000.ckpt │ │ │ ├── lrscheduler.pth │ │ │ ├── nnetema.pth │ │ │ ├── nnet.pth │ │ │ ├── optimizer.pth │ │ │ └── step.pth │ │ └── imagenet256-450000.ckpt │ │ ├── lrscheduler.pth │ │ ├── nnetema.pth │ │ ├── nnet.pth │ │ ├── optimizer.pth │ │ └── step.pth │ ├── fidstats │ │ ├── fidstatscc3mval.npz │ │ └── fidstatsimagenet256guideddiffusion.npz │ ├── pipeline.png | ├── contexts │ │ └── emptycontext.npy └── └── vqganjax_strongaug.ckpt

```

Dependencies

Same as MUSE-PyTorch. conda install pytorch torchvision torchaudio cudatoolkit=11.3 pip install accelerate==0.12.0 absl-py ml_collections einops wandb ftfy==6.1.1 transformers==4.23.1 loguru webdataset==0.2.5 gradio

Train

All style data in the paper are placed in the data directory

  1. Modify data/one_style.json (It should be noted that one_style.json and style data must be in the same directory), The format is file_name:[object,style]

json {"image_03_05.jpg":["A bear","in kid crayon drawing style"]} 2. Training script as follows. ```shell

!/bin/bash

unset EVALCKPT unset ADAPTER export OUTPUTDIR="outputdir/for/this/experiment" accelerate launch --numprocesses 8 --mixedprecision fp16 traint2icustomv2.py --config=configs/custom.py ```

Inference

The pretrained style_adapter weights can be downloaded from 🤗 Hugging Face. ```shell

!/bin/bash

export EVALCKPT="assets/ckpts/cc3m-285000.ckpt" export ADAPTER="path/to/your/styleadapter"

export OUTPUT_DIR="output/for/this/experiment"

accelerate launch --numprocesses 8 --mixedprecision fp16 traint2icustom_v2.py --config=configs/custom.py ```

Gradio Demo

Put the style_adapter weights in ./style_adapter folder and run the following command will launch the demo:

shell python gradio_demo.py

The demo is also hosted on HuggingFace.

Citation

bibtex @article{sohn2023styledrop, title={StyleDrop: Text-to-Image Generation in Any Style}, author={Sohn, Kihyuk and Ruiz, Nataniel and Lee, Kimin and Chin, Daniel Castro and Blok, Irina and Chang, Huiwen and Barber, Jarred and Jiang, Lu and Entis, Glenn and Li, Yuanzhen and others}, journal={arXiv preprint arXiv:2306.00983}, year={2023} }

Acknowlegment

Star History

Owner

  • Name: Advanced Intelligent Machines (AIM)
  • Login: aim-uofa
  • Kind: organization
  • Location: China

A research team at Zhejiang University, focusing on Computer Vision and broad AI research ...

GitHub Events

Total
  • Issues event: 1
  • Watch event: 22
  • Fork event: 3
Last Year
  • Issues event: 1
  • Watch event: 22
  • Fork event: 3

Issues and Pull Requests

Last synced: 10 months ago

All Time
  • Total issues: 10
  • Total pull requests: 0
  • Average time to close issues: 18 days
  • Average time to close pull requests: N/A
  • Total issue authors: 10
  • Total pull request authors: 0
  • Average comments per issue: 0.6
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 1
  • Pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Issue authors: 1
  • Pull request authors: 0
  • Average comments per issue: 0.0
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
  • carldaws (1)
  • maxin-cn (1)
  • fypbiqi (1)
  • vicc (1)
  • yatoubusha (1)
  • yxxshin (1)
  • AmanKishore (1)
  • Laidawang (1)
  • EnigmaHong (1)
Pull Request Authors
Top Labels
Issue Labels
Pull Request Labels