Data

Tools

Indexes

Applications

Experiments

Open Source Science

https://github.com/compvis/loradapter

https://github.com/compvis/loradapter

Science Score: 23.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ○
    CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ○
    .zenodo.json file
  • ○
    DOI references
  • ✓
    Academic publication links
    Links to: arxiv.org
  • ○
    Committers with academic emails
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (15.1%) to scientific vocabulary
Last synced: 11 months ago · JSON representation

Repository

Basic Info
  • Host: GitHub
  • Owner: CompVis
  • Language: Python
  • Default Branch: main
  • Size: 20.7 MB
Statistics
  • Stars: 116
  • Watchers: 16
  • Forks: 5
  • Open Issues: 2
  • Releases: 0
Created about 2 years ago · Last pushed almost 2 years ago
Metadata Files
Readme

README.md

Conditional LoRAdapter for Efficient 0-Shot Control & Altering of T2I Models

Nick Stracke1 · Stefan A. Baumann1 · Josh Susskind2 · Miguel A. Bautista2 · Björn Ommer1

1 CompVis Group @ LMU Munich
2 Apple

ECCV 2024

Project Page Paper

This repository contains an implementation of the paper "CTRLorALTer: Conditional LoRAdapter for Efficient 0-Shot Control & Altering of T2I Models".

We present LoRAdapter, an approach that unifies both style and structure conditioning under the same formulation using a novel conditional LoRA block that enables zero-shot control. LoRAdapter is an efficient, powerful, and architecture-agnostic approach to condition text-to-image diffusion models, which enables fine-grained control conditioning during generation and outperforms recent state-of-the-art approaches.

teaser

🔥 Updates

  • Implemented B-LoRA implicit content and style disentangle using LoRAdapter
  • Released Code and Weights for inference

💪 TODO

  • [x] Add training Code
  • [ ] Add more structure conditioning checkpoints (including SDXL)
  • [ ] Experiment with SD3

Setup

Create the conda environment

conda env create -f environment.yaml

Activate the conda environment

conda activate loradapter

Weights

All weights are available on HuggingFace.

For ease of you, you can also use the provided bash script download_weights.sh to automatically download all available weights and place them in the the right directory.

Usage

Sampling works according to the following schema: python sample.py experiment=<check ./config/experiments> All currently available experiments are listed in /config/experiments. Feel free to adjust the configs according to you own needs.

B-LoRA

Sampling using the B-LoRA LoRAdapter is possible using the config sample_b-lora_sdxl.yaml. By default this will condition on both content and style of the image. For conditioning on only content or only style, change the adaption_mode to either b-lora_content or b-lora_style. Also set ignore_check to true as we are only loading the checkpoint partially.

For best results provide information about the missing modality via the text prompt or using another LoRAdapter.

🎓 Citation

If you use this codebase or otherwise found our work valuable, please cite our paper:

bibtex @misc{stracke2024loradapter, title={CTRLorALTer: Conditional LoRAdapter for Efficient 0-Shot Control & Altering of T2I Models}, author={Nick Stracke and Stefan Andreas Baumann and Joshua Susskind and Miguel Angel Bautista and Björn Ommer}, year={2024}, eprint={2405.07913}, archivePrefix={arXiv}, primaryClass={cs.CV} }

Owner

  • Name: CompVis - Computer Vision and Learning LMU Munich
  • Login: CompVis
  • Kind: organization
  • Email: assist.mvl@lrz.uni-muenchen.de
  • Location: Germany

Computer Vision and Learning research group at Ludwig Maximilian University of Munich (formerly Computer Vision Group at Heidelberg University)

GitHub Events

Total
  • Issues event: 2
  • Watch event: 26
  • Issue comment event: 7
  • Fork event: 4
Last Year
  • Issues event: 2
  • Watch event: 26
  • Issue comment event: 7
  • Fork event: 4

Committers

Last synced: about 1 year ago

All Time
  • Total Commits: 9
  • Total Committers: 1
  • Avg Commits per committer: 9.0
  • Development Distribution Score (DDS): 0.0
Past Year
  • Commits: 5
  • Committers: 1
  • Avg Commits per committer: 5.0
  • Development Distribution Score (DDS): 0.0
Top Committers
Name Email Commits
Nick Stracke m****l@n****v 9
Committer Domains (Top 20 + Academic)
nickstracke.dev: 1

Issues and Pull Requests

Last synced: about 1 year ago

All Time
  • Total issues: 3
  • Total pull requests: 0
  • Average time to close issues: 4 months
  • Average time to close pull requests: N/A
  • Total issue authors: 3
  • Total pull request authors: 0
  • Average comments per issue: 4.33
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
Past Year
  • Issues: 2
  • Pull requests: 0
  • Average time to close issues: N/A
  • Average time to close pull requests: N/A
  • Issue authors: 2
  • Pull request authors: 0
  • Average comments per issue: 4.5
  • Average comments per pull request: 0
  • Merged pull requests: 0
  • Bot issues: 0
  • Bot pull requests: 0
View more stats
Top Authors
Issue Authors
  • xavier-airilab (1)
  • bghira (1)
  • atonderski (1)
Pull Request Authors
Top Labels
Issue Labels
Pull Request Labels

Dependencies

environment.yaml pypi
  • Pillow *
  • accelerate *
  • basicsr *
  • diffusers ==0.25.0
  • einops *
  • hydra-core *
  • jaxtyping *
  • numpy *
  • open-clip-torch *
  • tensorboard *
  • torch-fidelity *
  • torchvision *
  • tqdm *
  • transformers *