Evolutionary-Diffusion

Combining Evolutionary Computing with Diffusion Models

Images

• 🎨 Aesthetics Maximization/Minimization using LAION Aesthetics Predictor V2
• 📊 Multi-Objective Optimization with CLIP-IQA metrics
• 🛡️ Evading AI-Image Detection by optimizing against a fine-tuned SDXL AI-Image-Detector
• 🧭 Navigating the CLIP-Score Landscape for Prompt-Matching

Audio

• 🔊 Optimize Audiobox Aesthetics from Meta

Try it out in Google Colab

| Notebook | Link | |--------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Genetic Algorithm | | | Island Genetic Algorithm | | | NSGA | |

Image results will be saved in your Google Drive in the folder evolutionary. Each generation creates a new folder where the images will be saved in. You can change the folders in the notebook.

Sometimes Google Collab causes dependency problems which break the notebook. If you have any issues executing this in a Collab environment, please do not hesitate to create a new issue.

Running locally

Optionally but recommended to use a venv. Clone the repo or download the .zip, then install the dependencies via:

pip install -e ".[all]" 

Now you are ready to go with the notebooks or custom code. CUDA and MPS are supported.

Example - Creating the most Aesthetic Image

Optimizing for Aesthetics using the Aesthetics Predictor V2 from LAION with a GA and SDXL-Turbo

Optimizing the aesthetics predictor as a maximization problem, the algorithm came to a max Aesthetics score of 8.67. This score is higher than the examples from the real LAION English Subset dataset have, with the red line showing the limit. A wide variety of prompts (inspired by parti prompts) was used for the initial population.

https://github.com/malthee/evolutionary-diffusion-results/assets/18032233/4841d671-639a-4ac4-b7a8-ee5a66fab28d

Parameters: ```python populationsize = 100 numgenerations = 200 batch_size = 1 elitism = 1

creator = SDXLPromptEmbeddingImageCreator(pipelinefactory=setuppipeline, batchsize=batchsize, inferencesteps=3) evaluator = AestheticsImageEvaluator()
crossover = PooledArithmeticCrossover(0.5, 0.5) mutationarguments = UniformGaussianMutatorArguments(mutationrate=0.1, mutationstrength=2, clamprange=(-900, 900)) mutationargumentspooled = UniformGaussianMutatorArguments(mutationrate=0.1, mutationstrength=0.3, clamprange=(-8, 8)) mutator = PooledUniformGaussianMutator(mutationarguments, mutationargumentspooled) selector = TournamentSelector(tournamentsize=3) ```

Example - Island GA with Artists on each Island

Performing an Island GA by creating random embeddings and mixing them with artist embeddings to get mixtures of styles and new ideas.

| | | | | | --- | --- | --- | --- | | | | | |
| | | |

More images

Example - Improving Audiobox Aesthetics Score

Starting from noisy random samples, evolving to better sounds. Using the sum of all fitness criteria Audiobox Aesthetics offers.

https://github.com/user-attachments/assets/25be32eb-188b-483a-a787-9cea4d7f5eeb

https://github.com/user-attachments/assets/17f4bef7-1f5a-47a4-8f29-e4f42519a4fe

Detailed Results and Notebooks

More detailed results can be found in a separate repository dedicated to the results of the experiments: https://github.com/malthee/evolutionary-diffusion-results

Evaluators

• AestheticsImageEvaluator: Uses the LAION Aesthetics Predictor V2. Blog: https://laion.ai/blog/laion-aesthetics/
• AestheticsPredictorV25ImageEvaluator: Uses the Aesthetic Predictor V2.5 from discus0434
• CLIPScoreEvaluator: Using the torchmetrics implementation for CLIP-Score
• (Single/Multi)CLIPIQAEvaluator: Using the torchmetrics implementation for CLIP Image Quality Assessment.
• AIDetectionImageEvaluator: Using the original Version from HuggingFace, or the fine-tuned one for SDXL generated images
• AudioboxAestheticsEvaluator: Using Audiobox Aesthetics from Meta

Image Creators

Current supported creators working in the prompt embedding space: * SDXLPromptEmbeddingImageCreator: Supports the SDXL pipeline, creates both prompt- and pooled-prompt-embeddings. * SDPromptEmbeddingImageCreator: Only has prompt-embeddings, is faster but produces less quality results than SDXL.

Audio Creators

Supporting only AudioLDM because it works simply on the CLAP embedding space (suitable for this kind of operation). Other embeddings have shown to not work well with evolutionary operations (like the T5 encoder for example) * AudioLDMSoundCreator: Works with any AudioLDMPipeline, default is audioldm-l-full

Package Structure and Base Classes

(Pre-Testing) Evaluating Models for Evolutionary use

There are multiple notebooks exploring the speed and quality of models for generation and fitness-evaluation. These notebooks also allow for simple inference so that any model can be tried out easily.

• diffusionmodelcomparison: tries out different diffusion models with varying arguments (inference steps, batch size) to find out the optimal model for image generation in an evolutionary context (generation speed & quality)
• clip_evaluators: uses torch metrics with CLIPScore and CLIP IQA. CLIPScore could define the fitness for "prompt fulfillment" or "image alignment" while CLIP IQA has many possible metrics like "quality, brightness, happiness..."
• aidetectionevaluator: uses a pre-trained model for AI image detection. This could be a fitness criteria to minimize "AI-likeness" in images.
• aesthetics_evaluator: uses a pre-trained model from the maintainers of the LAION image dataset, which scores an image 0-10 depending on how "aesthetic" it is. Could be used as a maximization criteria for the fitness of images.
• clamp_range: testing the usual prompt-embedding min and max values for different models, so that a CLAMP range can be set in the mutator for example. Using the parti prompts.
• crossovermutationexperiments: testing different crossover and mutation strategies to see how they work in the prompt embedding space
• embedding_relations: experimenting with TensorBoard and integrating it into our embedding model