Data

Tools

Indexes

Applications

Experiments

Open Source Science

controlnet

Self-implemented Control-Net for final proj of DATA303 2025 spring

https://github.com/kyne0127/controlnet

Science Score: 54.0%

This score indicates how likely this project is to be science-related based on various indicators:

  • ✓
    CITATION.cff file
    Found CITATION.cff file
  • ✓
    codemeta.json file
    Found codemeta.json file
  • ✓
    .zenodo.json file
    Found .zenodo.json file
  • ○
    DOI references
  • ✓
    Academic publication links
    Links to: arxiv.org
  • ○
    Academic email domains
  • ○
    Institutional organization owner
  • ○
    JOSS paper metadata
  • ○
    Scientific vocabulary similarity
    Low similarity (11.1%) to scientific vocabulary
Last synced: 10 months ago · JSON representation ·

Repository

Self-implemented Control-Net for final proj of DATA303 2025 spring

Basic Info
  • Host: GitHub
  • Owner: kyne0127
  • License: apache-2.0
  • Language: Python
  • Default Branch: main
  • Size: 56.7 MB
Statistics
  • Stars: 0
  • Watchers: 1
  • Forks: 0
  • Open Issues: 0
  • Releases: 0
Created about 1 year ago · Last pushed 11 months ago
Metadata Files
Readme Contributing License Code of conduct Citation

README.md

ControlNet Customed with 🤗 Diffusers

This repository provides examples and instructions for fine-tuning ControlNet on the circle-filling dataset using 🤗 Diffusers. You can train with PyTorch or Flax/JAX, leverage Accelerate for distributed/multi-GPU training, and apply various memory optimizations for GPUs of different sizes.

📦 Installation

  1. Clone and install Diffusers

bash pip install -e . 2. Install example requirements

bash cd examples/controlnet pip install -r requirements.txt 3. Initialize Accelerate

  • Interactive setup:

    bash accelerate config

  • Default configuration:

    bash accelerate config default

  • Non-interactive (e.g., notebook):

    python from accelerate.utils import write_basic_config write_basic_config()

📂 Dataset

We use the circle-filling dataset (re-hosted for 🤗 Datasets compatibility) under fusing/fill50k. Dataloading is handled inside the training script.

🖼️ Download Conditioning Images

bash wget https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/controlnet_training/conditioning_image_1.png wget https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/controlnet_training/conditioning_image_2.png

🚀 Training (PyTorch)

```bash export MODELDIR="stable-diffusion-v1-5/stable-diffusion-v1-5" export OUTPUTDIR="output"

accelerate launch mytraincontrolnet.py \ --pretrainedmodelnameorpath=$MODELDIR \ --outputdir=$OUTPUTDIR \ --datasetname=fusing/fill50k \ --resolution=512 \ --learningrate=1e-5 \ --validationimage "./conditioningimage1.png" "./conditioningimage2.png" \ --validationprompt "red circle with blue background" "cyan circle with brown floral background" \ --trainbatchsize=4 --reportto=wandb ```

📖 References

Owner

  • Login: kyne0127
  • Kind: user

Citation (CITATION.cff) 

cff-version: 1.2.0
title: 'Diffusers: State-of-the-art diffusion models'
message: >-
  If you use this software, please cite it using the
  metadata from this file.
type: software
authors:
  - given-names: Patrick
    family-names: von Platen
  - given-names: Suraj
    family-names: Patil
  - given-names: Anton
    family-names: Lozhkov
  - given-names: Pedro
    family-names: Cuenca
  - given-names: Nathan
    family-names: Lambert
  - given-names: Kashif
    family-names: Rasul
  - given-names: Mishig
    family-names: Davaadorj
  - given-names: Dhruv
    family-names: Nair
  - given-names: Sayak
    family-names: Paul
  - given-names: Steven
    family-names: Liu
  - given-names: William
    family-names: Berman
  - given-names: Yiyi
    family-names: Xu
  - given-names: Thomas
    family-names: Wolf
repository-code: 'https://github.com/huggingface/diffusers'
abstract: >-
  Diffusers provides pretrained diffusion models across
  multiple modalities, such as vision and audio, and serves
  as a modular toolbox for inference and training of
  diffusion models.
keywords:
  - deep-learning
  - pytorch
  - image-generation
  - hacktoberfest
  - diffusion
  - text2image
  - image2image
  - score-based-generative-modeling
  - stable-diffusion
  - stable-diffusion-diffusers
license: Apache-2.0
version: 0.12.1

GitHub Events

Total
  • Member event: 2
  • Push event: 10
  • Create event: 1
Last Year
  • Member event: 2
  • Push event: 10
  • Create event: 1

Dependencies

.github/actions/setup-miniconda/action.yml actions
  • actions/cache v2 composite
.github/workflows/benchmark.yml actions
  • actions/checkout v3 composite
  • actions/upload-artifact v4 composite
.github/workflows/build_docker_images.yml actions
  • actions/checkout v3 composite
  • docker/build-push-action v3 composite
  • docker/login-action v2 composite
  • docker/setup-buildx-action v1 composite
  • huggingface/hf-workflows/.github/actions/post-slack main composite
  • jitterbit/get-changed-files v1 composite
.github/workflows/build_documentation.yml actions
.github/workflows/build_pr_documentation.yml actions
.github/workflows/mirror_community_pipeline.yml actions
  • actions/checkout v3 composite
  • actions/setup-python v4 composite
.github/workflows/nightly_tests.yml actions
  • actions/checkout v3 composite
  • actions/download-artifact v4 composite
  • actions/upload-artifact v4 composite
.github/workflows/notify_slack_about_release.yml actions
  • actions/checkout v3 composite
  • actions/setup-python v4 composite
.github/workflows/pr_dependency_test.yml actions
  • actions/checkout v3 composite
  • actions/setup-python v4 composite
.github/workflows/pr_flax_dependency_test.yml actions
  • actions/checkout v3 composite
  • actions/setup-python v4 composite
.github/workflows/pr_style_bot.yml actions
.github/workflows/pr_test_fetcher.yml actions
  • actions/checkout v3 composite
  • actions/upload-artifact v3 composite
  • actions/upload-artifact v4 composite
.github/workflows/pr_tests.yml actions
  • actions/checkout v3 composite
  • actions/setup-python v4 composite
  • actions/upload-artifact v4 composite
.github/workflows/pr_tests_gpu.yml actions
  • actions/checkout v3 composite
  • actions/setup-python v4 composite
  • actions/upload-artifact v4 composite
.github/workflows/pr_torch_dependency_test.yml actions
  • actions/checkout v3 composite
  • actions/setup-python v4 composite
.github/workflows/push_tests.yml actions
  • actions/checkout v3 composite
  • actions/upload-artifact v4 composite
.github/workflows/push_tests_fast.yml actions
  • actions/checkout v3 composite
  • actions/upload-artifact v4 composite
.github/workflows/push_tests_mps.yml actions
  • ./.github/actions/setup-miniconda * composite
  • actions/checkout v3 composite
  • actions/upload-artifact v4 composite
.github/workflows/pypi_publish.yaml actions
  • actions/checkout v3 composite
  • actions/setup-python v4 composite
.github/workflows/release_tests_fast.yml actions
  • actions/checkout v3 composite
  • actions/upload-artifact v4 composite
.github/workflows/run_tests_from_a_pr.yml actions
  • actions/checkout v4 composite
.github/workflows/ssh-pr-runner.yml actions
  • actions/checkout v3 composite
  • huggingface/tailscale-action main composite
.github/workflows/ssh-runner.yml actions
  • actions/checkout v3 composite
  • huggingface/tailscale-action main composite
.github/workflows/stale.yml actions
  • actions/checkout v2 composite
  • actions/setup-python v1 composite
.github/workflows/trufflehog.yml actions
  • actions/checkout v4 composite
  • trufflesecurity/trufflehog main composite
.github/workflows/typos.yml actions
  • actions/checkout v3 composite
  • crate-ci/typos v1.12.4 composite
.github/workflows/update_metadata.yml actions
  • actions/checkout v3 composite
.github/workflows/upload_pr_documentation.yml actions
docker/diffusers-doc-builder/Dockerfile docker
  • ubuntu 20.04 build
docker/diffusers-flax-cpu/Dockerfile docker
  • ubuntu 20.04 build
docker/diffusers-flax-tpu/Dockerfile docker
  • ubuntu 20.04 build
docker/diffusers-onnxruntime-cpu/Dockerfile docker
  • ubuntu 20.04 build
docker/diffusers-onnxruntime-cuda/Dockerfile docker
  • nvidia/cuda 12.1.0-runtime-ubuntu20.04 build
docker/diffusers-pytorch-cpu/Dockerfile docker
  • ubuntu 20.04 build
docker/diffusers-pytorch-cuda/Dockerfile docker
  • nvidia/cuda 12.1.0-runtime-ubuntu20.04 build
docker/diffusers-pytorch-minimum-cuda/Dockerfile docker
  • nvidia/cuda 12.1.0-runtime-ubuntu20.04 build
docker/diffusers-pytorch-xformers-cuda/Dockerfile docker
  • nvidia/cuda 12.1.0-runtime-ubuntu20.04 build
examples/advanced_diffusion_training/requirements.txt pypi
  • Jinja2 *
  • accelerate >=0.31.0
  • ftfy *
  • peft >=0.11.1
  • sentencepiece *
  • tensorboard *
  • torchvision *
  • transformers >=4.41.2
examples/advanced_diffusion_training/requirements_flux.txt pypi
  • Jinja2 *
  • accelerate >=0.31.0
  • ftfy *
  • peft >=0.11.1
  • sentencepiece *
  • tensorboard *
  • torchvision *
  • transformers >=4.41.2
examples/cogvideo/requirements.txt pypi
  • Jinja2 *
  • accelerate >=0.31.0
  • decord >=0.6.0
  • ftfy *
  • imageio-ffmpeg *
  • peft >=0.11.1
  • sentencepiece *
  • tensorboard *
  • torchvision *
  • transformers >=4.41.2
examples/cogview4-control/requirements.txt pypi
  • accelerate ==1.2.0
  • peft >=0.14.0
  • torch *
  • torchvision *
  • transformers ==4.47.0
  • wandb *
examples/consistency_distillation/requirements.txt pypi
  • Jinja2 *
  • accelerate >=0.16.0
  • ftfy *
  • tensorboard *
  • torchvision *
  • transformers >=4.25.1
  • webdataset *
examples/controlnet/requirements.txt pypi
  • accelerate >=0.16.0
  • datasets *
  • ftfy *
  • tensorboard *
  • torchvision *
  • transformers >=4.25.1
examples/controlnet/requirements_flax.txt pypi
  • Jinja2 *
  • datasets *
  • flax *
  • ftfy *
  • optax *
  • tensorboard *
  • torch *
  • torchvision *
  • transformers >=4.25.1
examples/controlnet/requirements_flux.txt pypi
  • Jinja2 *
  • SentencePiece *
  • accelerate >=0.16.0
  • datasets *
  • ftfy *
  • tensorboard *
  • torchvision *
  • transformers >=4.25.1
  • wandb *
examples/controlnet/requirements_sd3.txt pypi
  • Jinja2 *
  • accelerate >=0.16.0
  • datasets *
  • ftfy *
  • tensorboard *
  • torchvision *
  • transformers >=4.25.1
  • wandb *
examples/controlnet/requirements_sdxl.txt pypi
  • Jinja2 *
  • accelerate >=0.16.0
  • datasets *
  • ftfy *
  • tensorboard *
  • torchvision *
  • transformers >=4.25.1
  • wandb *
examples/custom_diffusion/requirements.txt pypi
  • Jinja2 *
  • accelerate *
  • ftfy *
  • tensorboard *
  • torchvision *
  • transformers >=4.25.1
examples/dreambooth/requirements.txt pypi
  • Jinja2 *
  • accelerate >=0.16.0
  • ftfy *
  • peft ==0.7.0
  • tensorboard *
  • torchvision *
  • transformers >=4.25.1
examples/dreambooth/requirements_flax.txt pypi
  • Jinja2 *
  • flax *
  • ftfy *
  • optax *
  • tensorboard *
  • torch *
  • torchvision *
  • transformers >=4.25.1
examples/dreambooth/requirements_flux.txt pypi
  • Jinja2 *
  • accelerate >=0.31.0
  • ftfy *
  • peft >=0.11.1
  • sentencepiece *
  • tensorboard *
  • torchvision *
  • transformers >=4.41.2
examples/dreambooth/requirements_hidream.txt pypi
  • Jinja2 *
  • accelerate >=1.4.0
  • ftfy *
  • peft >=0.14.0
  • sentencepiece *
  • tensorboard *
  • torchvision *
  • transformers >=4.50.0
examples/dreambooth/requirements_sana.txt pypi
  • Jinja2 *
  • accelerate >=1.0.0
  • ftfy *
  • peft >=0.14.0
  • sentencepiece *
  • tensorboard *
  • torchvision *
  • transformers >=4.47.0
examples/dreambooth/requirements_sd3.txt pypi
  • Jinja2 *
  • accelerate >=0.31.0
  • ftfy *
  • peft ==0.11.1
  • sentencepiece *
  • tensorboard *
  • torchvision *
  • transformers >=4.41.2
examples/dreambooth/requirements_sdxl.txt pypi
  • Jinja2 *
  • accelerate >=0.16.0
  • ftfy *
  • peft ==0.7.0
  • tensorboard *
  • torchvision *
  • transformers >=4.25.1
examples/flux-control/requirements.txt pypi
  • accelerate ==1.2.0
  • peft >=0.14.0
  • torch *
  • torchvision *
  • transformers ==4.47.0
  • wandb *
examples/instruct_pix2pix/requirements.txt pypi
  • accelerate >=0.16.0
  • datasets *
  • ftfy *
  • tensorboard *
  • torchvision *
  • transformers >=4.25.1
examples/kandinsky2_2/text_to_image/requirements.txt pypi
  • Jinja2 *
  • accelerate >=0.16.0
  • datasets *
  • ftfy *
  • tensorboard *
  • torchvision *
  • transformers >=4.25.1
examples/model_search/requirements.txt pypi
  • huggingface-hub >=0.26.2
examples/research_projects/autoencoderkl/requirements.txt pypi
  • Pillow *
  • accelerate >=0.16.0
  • bitsandbytes *
  • datasets *
  • huggingface_hub *
  • lpips *
  • numpy *
  • packaging *
  • taming_transformers *
  • torch *
  • torchvision *
  • tqdm *
  • transformers *
  • wandb *
  • xformers *
examples/research_projects/colossalai/requirement.txt pypi
  • Jinja2 *
  • diffusers *
  • ftfy *
  • tensorboard *
  • torch *
  • torchvision *
  • transformers *
examples/research_projects/consistency_training/requirements.txt pypi
  • Jinja2 *
  • accelerate >=0.16.0
  • ftfy *
  • tensorboard *
  • torchvision *
  • transformers >=4.25.1
examples/research_projects/diffusion_dpo/requirements.txt pypi
  • Jinja2 *
  • accelerate >=0.16.0
  • ftfy *
  • peft *
  • tensorboard *
  • torchvision *
  • transformers >=4.25.1
  • wandb *
examples/research_projects/diffusion_orpo/requirements.txt pypi
  • accelerate *
  • datasets *
  • peft *
  • torchvision *
  • transformers *
  • wandb *
  • webdataset *
examples/research_projects/dreambooth_inpaint/requirements.txt pypi
  • Jinja2 *
  • accelerate >=0.16.0
  • diffusers ==0.9.0
  • ftfy *
  • tensorboard *
  • torchvision *
  • transformers >=4.21.0
examples/research_projects/gligen/requirements.txt pypi
  • Jinja2 *
  • accelerate >=0.16.0
  • diffusers *
  • fairscale *
  • ftfy *
  • scipy *
  • tensorboard *
  • timm *
  • torchvision *
  • transformers >=4.25.1
  • wandb *
examples/research_projects/intel_opts/textual_inversion/requirements.txt pypi
  • Jinja2 *
  • accelerate >=0.16.0
  • ftfy *
  • intel_extension_for_pytorch >=1.13
  • tensorboard *
  • torchvision *
  • transformers >=4.21.0
examples/research_projects/intel_opts/textual_inversion_dfq/requirements.txt pypi
  • accelerate *
  • ftfy *
  • modelcards *
  • neural-compressor *
  • tensorboard *
  • torchvision *
  • transformers >=4.25.0
examples/research_projects/ip_adapter/requirements.txt pypi
  • accelerate *
  • ip_adapter *
  • torchvision *
  • transformers >=4.25.1
examples/research_projects/lora/requirements.txt pypi
  • Jinja2 *
  • accelerate >=0.16.0
  • datasets *
  • ftfy *
  • tensorboard *
  • torchvision *
  • transformers >=4.25.1
examples/research_projects/multi_subject_dreambooth/requirements.txt pypi
  • Jinja2 *
  • accelerate >=0.16.0
  • ftfy *
  • tensorboard *
  • torchvision *
  • transformers >=4.25.1
examples/research_projects/multi_subject_dreambooth_inpainting/requirements.txt pypi
  • Jinja2 *
  • accelerate >=0.16.0
  • datasets >=2.16.0
  • ftfy *
  • tensorboard *
  • torchvision *
  • transformers >=4.25.1
  • wandb >=0.16.1
examples/research_projects/multi_token_textual_inversion/requirements.txt pypi
  • Jinja2 *
  • accelerate >=0.16.0
  • ftfy *
  • tensorboard *
  • torchvision *
  • transformers >=4.25.1
examples/research_projects/multi_token_textual_inversion/requirements_flax.txt pypi
  • Jinja2 *
  • flax *
  • ftfy *
  • optax *
  • tensorboard *
  • torch *
  • torchvision *
  • transformers >=4.25.1
examples/research_projects/onnxruntime/text_to_image/requirements.txt pypi
  • accelerate >=0.16.0
  • datasets *
  • ftfy *
  • modelcards *
  • tensorboard *
  • torchvision *
  • transformers >=4.25.1
examples/research_projects/onnxruntime/textual_inversion/requirements.txt pypi
  • accelerate >=0.16.0
  • ftfy *
  • modelcards *
  • tensorboard *
  • torchvision *
  • transformers >=4.25.1
examples/research_projects/onnxruntime/unconditional_image_generation/requirements.txt pypi
  • accelerate >=0.16.0
  • datasets *
  • tensorboard *
  • torchvision *
examples/research_projects/pixart/requirements.txt pypi
  • SentencePiece *
  • controlnet-aux *
  • datasets *
  • torchvision *
  • transformers *
examples/research_projects/pytorch_xla/training/text_to_image/requirements.txt pypi
  • Jinja2 *
  • accelerate >=0.16.0
  • datasets >=2.19.1
  • ftfy *
  • peft ==0.7.0
  • tensorboard *
  • torchvision *
  • transformers >=4.25.1
examples/research_projects/realfill/requirements.txt pypi
  • Jinja2 ==3.1.6
  • accelerate ==0.23.0
  • diffusers ==0.20.1
  • ftfy ==6.1.1
  • peft ==0.5.0
  • tensorboard ==2.14.0
  • torch ==2.2.0
  • torchvision >=0.16
  • transformers ==4.38.0
examples/research_projects/wuerstchen/text_to_image/requirements.txt pypi
  • accelerate >=0.16.0
  • bitsandbytes *
  • deepspeed *
  • peft >=0.6.0
  • torchvision *
  • transformers >=4.25.1
  • wandb *
examples/server/requirements.in pypi
  • aiohttp *
  • fastapi *
  • prometheus-fastapi-instrumentator >=7.0.0
  • prometheus_client >=0.18.0
  • py-consul *
  • sentencepiece *
  • torch *
  • transformers ==4.46.1
  • uvicorn *
examples/server/requirements.txt pypi
  • aiohappyeyeballs ==2.4.3
  • aiohttp ==3.10.10
  • aiosignal ==1.3.1
  • annotated-types ==0.7.0
  • anyio ==4.6.2.post1
  • attrs ==24.2.0
  • certifi ==2024.8.30
  • charset-normalizer ==3.4.0
  • click ==8.1.7
  • fastapi ==0.115.3
  • filelock ==3.16.1
  • frozenlist ==1.5.0
  • fsspec ==2024.10.0
  • h11 ==0.14.0
  • huggingface-hub ==0.26.1
  • idna ==3.10
  • jinja2 ==3.1.4
  • markupsafe ==3.0.2
  • mpmath ==1.3.0
  • multidict ==6.1.0
  • networkx ==3.4.2
  • numpy ==2.1.2
  • packaging ==24.1
  • prometheus-client ==0.21.0
  • prometheus-fastapi-instrumentator ==7.0.0
  • propcache ==0.2.0
  • py-consul ==1.5.3
  • pydantic ==2.9.2
  • pydantic-core ==2.23.4
  • pyyaml ==6.0.2
  • regex ==2024.9.11
  • requests ==2.32.3
  • safetensors ==0.4.5
  • sentencepiece ==0.2.0
  • sniffio ==1.3.1
  • starlette ==0.41.0
  • sympy ==1.13.3
  • tokenizers ==0.20.1
  • torch ==2.4.1
  • tqdm ==4.66.5
  • transformers ==4.46.1
  • typing-extensions ==4.12.2
  • urllib3 ==2.2.3
  • uvicorn ==0.32.0
  • yarl ==1.16.0
examples/t2i_adapter/requirements.txt pypi
  • accelerate >=0.16.0
  • datasets *
  • ftfy *
  • safetensors *
  • tensorboard *
  • torchvision *
  • transformers >=4.25.1
  • wandb *
examples/text_to_image/requirements.txt pypi
  • Jinja2 *
  • accelerate >=0.16.0
  • datasets >=2.19.1
  • ftfy *
  • peft ==0.7.0
  • tensorboard *
  • torchvision *
  • transformers >=4.25.1
examples/text_to_image/requirements_flax.txt pypi
  • Jinja2 *
  • datasets *
  • flax *
  • ftfy *
  • optax *
  • tensorboard *
  • torch *
  • torchvision *
  • transformers >=4.25.1
examples/text_to_image/requirements_sdxl.txt pypi
  • Jinja2 *
  • accelerate >=0.22.0
  • datasets *
  • ftfy *
  • peft ==0.7.0
  • tensorboard *
  • torchvision *
  • transformers >=4.25.1
examples/textual_inversion/requirements.txt pypi
  • Jinja2 *
  • accelerate >=0.16.0
  • ftfy *
  • tensorboard *
  • torchvision *
  • transformers >=4.25.1
examples/textual_inversion/requirements_flax.txt pypi
  • Jinja2 *
  • flax *
  • ftfy *
  • optax *
  • tensorboard *
  • torch *
  • torchvision *
  • transformers >=4.25.1
examples/unconditional_image_generation/requirements.txt pypi
  • accelerate >=0.16.0
  • datasets *
  • torchvision *
examples/vqgan/requirements.txt pypi
  • accelerate >=0.16.0
  • datasets *
  • numpy *
  • tensorboard *
  • timm *
  • torchvision *
  • tqdm *
  • transformers >=4.25.1
pyproject.toml pypi
setup.py pypi
  • deps *