VectorFusion: Text-to-SVG by Abstracting Pixel-Based Diffusion Models

In this work, the authors show that a text-conditioned diffusion model trained on pixel representations of images can be used to generate SVG-exportable vector graphics.

official website: https://vectorfusion.github.io/

Updates

• [01/2024] 🔥 We released the SVGDreamer. SVGDreamer is a novel text-guided vector graphics synthesis method. This method considers both the editing of vector graphics and the quality of the synthesis.
• [12/2023] 🔥 We released the PyTorch-SVGRender. Pytorch-SVGRender is the go-to library for state-of-the-art differentiable rendering methods for image vectorization.
• [10/2023] 🔥 We released the DiffSketcher code. A method of synthesizing vector sketches by text prompts.
• [10/2023] 🔥 We reproduce the VectorFusion code.

Installation Guide

To quickly get started with DiffSketcher, follow the steps below.
These instructions will help you run quick inference locally.

🚀 Option 1: Standard Installation

Run the following command in the top-level directory:

shell chmod +x script/install.sh bash script/install.sh

🐳 Option 2: Using Docker

shell chmod +x script/run_docker.sh sudo bash script/run_docker.sh

Quickstart

Case: Sydney Opera House

Prompt: the Sydney Opera House.
Style: iconography
Preview:

| | | | |-----------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------| | (a) Sample raster image with Stable Diffusion | (b) Convert raster image to a vector via LIVE | (c) VectorFusion: Fine tune by LSDS |

LIVE Rendering Process:

| iter 0 | iter 500 | iter 1000 | iter 1500 | iter 2500 | iter 3500 | |:-----------------------------------------------------:|:-------------------------------------------------------:|:--------------------------------------------------------:|:--------------------------------------------------------:|----------------------------------------------------------|----------------------------------------------------------| | | | | | | |

VectorFusion Rendering Process:

| iter 0 | iter 100 | iter 300 | iter 400 | iter 700 | iter 1000 | |:----------------------------------------------------:|:------------------------------------------------------:|:------------------------------------------------------:|:------------------------------------------------------:|--------------------------------------------------------|------------------------------------------------------| | | | | | | |

Script:

shell python run_painterly_render.py \ -c vectorfusion.yaml \ -pt "the Sydney Opera House. minimal flat 2d vector icon. lineal color. on a white background. trending on artstation" \ -save_step 50 \ -update "K=6" \ -respath workdir/SydneyOperaHouse \ -d 15486 \ --download

• -c a.k.a --config: configuration file.
• -save_step: the step size used to save the result (too frequent calls will result in longer times).
• -update: a tool for editing the hyper-params of the configuration file, so you don't need to create a new yaml.
• -pt a.k.a --prompt: text prompt.
• -respath a.k.a --results_path: the folder to save results.
• -d a.k.a --seed: random seed.
• --download: download models from huggingface automatically when you first run them.

optional:

• -npt, a.k.a --negative_prompt: negative text prompt.
• -mv, a.k.a --make_video: make a video of the rendering process (it will take much longer).
• -frame_freq, a.k.a --video_frame_freq: the interval of the number of steps to save the image.
• -framerate, a.k.a --video_frame_rate: control the playback speed of the output video.

Case: Ming Dynasty Vase

Prompt: A photo of a Ming Dynasty vase on a leather topped table.
Style: iconography
Preview:

| | | | |---------------------------------------------------------------------------------|-------------------------------------------------------------------------------|-----------------------------------------------------------------------------------| | (a) Sample raster image with Stable Diffusion | (b) Convert raster image to a vector via LIVE | (c) VectorFusion: Fine tune by LSDS |

Script:

shell python run_painterly_render.py -c vectorfusion.yaml -pt "A photo of a Ming Dynasty vase on a leather topped table. minimal flat 2d vector icon. lineal color. on a white background. trending on artstation" -save_step 50 -respath ./workdir/vase -d 683692

Case: Astronaut

Prompt: An astronaut figure.
Style: iconography
Preview:

| | | | |----------------------------------------------------------------------------------|--------------------------------------------------------------------------------|------------------------------------------------------------------------------------| | (a) Sample raster image with Stable Diffusion | (b) Convert raster image to a vector via LIVE | (c) VectorFusion: Fine tune by LSDS |

Script:

shell python run_painterly_render.py -c vectorfusion.yaml -pt "An astronaut figure. minimal flat 2d vector icon. lineal color. on a white background. trending on artstation" -save_step 50 -respath ./workdir/astronaut -d 522178

Case: Guitar

Prompt: Electric guitar.
Style: Pixel-Art
Preview:

| | | | |--------------------------------------------------------------------------------|------------------------------------------------------------------------------|----------------------------------------------------------------------------------| | (a) Sample raster image with Stable Diffusion | (b) Convert raster image to a vector via LIVE | (c) VectorFusion: Fine tune by LSDS |

Script:

shell python run_painterly_render.py -c vectorfusion.yaml -pt "Electric guitar. pixel art. trending on artstation" -save_step 50 -respath ./workdir/guitar -update "style=pixelart" -d 445997

Case: Dragon

Prompt: watercolor painting of a firebreathing dragon.
Style: Sketch
Preview:

| | | | |-----------------------------------------------|-------------------------------------------------|----------------------------------------------| | SVG initialization | VectorFusion fine-tune 500 step | VectorFusion fine-tune 1500 step |

Script:

shell python run_painterly_render.py -c vectorfusion.yaml -pt "watercolor painting of a firebreathing dragon. minimal 2d line drawing. trending on artstation" -save_step 50 -respath ./workdir/dragon-sketch -update "style=sketch num_segments=5 radius=0.5 sds.num_iter=1500" -d 106764

Other Cases

```shell

Sketch style

CUDAVISIBLEDEVICES=0 python runpainterlyrender.py -c vectorfusion.yaml -pt "watercolor painting of a firebreathing dragon. minimal 2d line drawing. trending on artstation" -savestep 50 -respath ./workdir/dragon-sketch -update "style=sketch skiplive=True numpaths=32 numsegments=5 radius=0.5 sds.numiter=1500" -rdbz CUDAVISIBLEDEVICES=0 python runpainterlyrender.py -c vectorfusion.yaml -pt "A cat. minimal 2d line drawing. trending on artstation" -savestep 50 -respath ./workdir/cat-sketch -update "style=sketch skiplive=True numpaths=32 numsegments=5 radius=0.5 sds.numiter=1500" -rdbz ```

More Examples:

• check the Examples.md for more cases.

More Scripts:

• check the Run.md for more scripts.

Acknowledgement

The project is built based on the following repository:

• BachiLi/diffvg
• huggingface/diffusers
• ximinng/DiffSketcher

We gratefully thank the authors for their wonderful works.

Citation

If you use this code for your research, please cite the following work:

@inproceedings{jain2023vectorfusion, title={Vectorfusion: Text-to-svg by abstracting pixel-based diffusion models}, author={Jain, Ajay and Xie, Amber and Abbeel, Pieter}, booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages={1911--1920}, year={2023} } @inproceedings{xing2023diffsketcher, title={DiffSketcher: Text Guided Vector Sketch Synthesis through Latent Diffusion Models}, author={XiMing Xing and Chuang Wang and Haitao Zhou and Jing Zhang and Qian Yu and Dong Xu}, booktitle={Thirty-seventh Conference on Neural Information Processing Systems}, year={2023}, url={https://openreview.net/forum?id=CY1xatvEQj} }

Licence

This repo is licensed under a MIT License.