mirror of
https://github.com/kohya-ss/sd-scripts.git
synced 2026-04-09 06:45:09 +00:00
103 lines
5.0 KiB
Markdown
103 lines
5.0 KiB
Markdown
# About ControlNet-LLLite
|
|
|
|
__This is an extremely experimental implementation and may change significantly in the future.__
|
|
|
|
日本語版は[こちら](./train_lllite_README-ja.md)
|
|
|
|
## Overview
|
|
|
|
ControlNet-LLLite is a lightweight version of [ControlNet](https://github.com/lllyasviel/ControlNet). It is a "LoRA Like Lite" that is inspired by LoRA and has a lightweight structure. Currently, only SDXL is supported.
|
|
|
|
## Sample weight file and inference
|
|
|
|
Sample weight file is available here: https://huggingface.co/kohya-ss/controlnet-lllite
|
|
|
|
A custom node for ComfyUI is available: https://github.com/kohya-ss/ControlNet-LLLite-ComfyUI
|
|
|
|
Sample images are at the end of this page.
|
|
|
|
## Model structure
|
|
|
|
A single LLLite module consists of a conditioning image embedding that maps a conditioning image to a latent space and a small network with a structure similar to LoRA. The LLLite module is added to U-Net's Linear and Conv in the same way as LoRA. Please refer to the source code for details.
|
|
|
|
Due to the limitations of the inference environment, only CrossAttention (attn1 q/k/v, attn2 q) is currently added.
|
|
|
|
## Model training
|
|
|
|
### Preparing the dataset
|
|
|
|
In addition to the normal dataset, please store the conditioning image in the directory specified by `conditioning_data_dir`. The conditioning image must have the same basename as the training image. The conditioning image will be automatically resized to the same size as the training image.
|
|
|
|
```toml
|
|
[[datasets.subsets]]
|
|
image_dir = "path/to/image/dir"
|
|
caption_extension = ".txt"
|
|
conditioning_data_dir = "path/to/conditioning/image/dir"
|
|
```
|
|
|
|
At the moment, random_crop cannot be used.
|
|
|
|
As a training data, it seems to be better to use the images generated by the original model as training images and the images processed from them as conditioning images. If you use images with a different style from the original model as training images, the model will have to learn not only the control but also the style. ControlNet-LLLite is not suitable for style learning because of its small capacity.
|
|
|
|
If you use images other than the generated images as training images, please increase the dimension as described below.
|
|
|
|
### Training
|
|
|
|
Run `sdxl_train_control_net_lllite.py`. You can specify the dimension of the conditioning image embedding with `--cond_emb_dim`. You can specify the rank of the LoRA-like module with `--network_dim`. Other options are the same as `sdxl_train_network.py`, but `--network_module` is not required.
|
|
|
|
Since a large amount of memory is used during training, please enable memory-saving options such as cache and gradient checkpointing. It is also effective to use BFloat16 with the `--full_bf16` option (requires RTX 30 series or later GPU). It has been confirmed to work with 24GB VRAM.
|
|
|
|
For the sample Canny, the dimension of the conditioning image embedding is 32. The rank of the LoRA-like module is also 64. Adjust according to the features of the conditioning image you are targeting.
|
|
|
|
(The sample Canny is probably quite difficult. It may be better to reduce it to about half for depth, etc.)
|
|
|
|
The following is an example of a .toml configuration.
|
|
|
|
```toml
|
|
pretrained_model_name_or_path = "/path/to/model_trained_on.safetensors"
|
|
max_train_epochs = 12
|
|
max_data_loader_n_workers = 4
|
|
persistent_data_loader_workers = true
|
|
seed = 42
|
|
gradient_checkpointing = true
|
|
mixed_precision = "bf16"
|
|
save_precision = "bf16"
|
|
full_bf16 = true
|
|
optimizer_type = "adamw8bit"
|
|
learning_rate = 2e-4
|
|
xformers = true
|
|
output_dir = "/path/to/output/dir"
|
|
output_name = "output_name"
|
|
save_every_n_epochs = 1
|
|
save_model_as = "safetensors"
|
|
vae_batch_size = 4
|
|
cache_latents = true
|
|
cache_latents_to_disk = true
|
|
cache_text_encoder_outputs = true
|
|
cache_text_encoder_outputs_to_disk = true
|
|
network_dim = 64
|
|
cond_emb_dim = 32
|
|
dataset_config = "/path/to/dataset.toml"
|
|
```
|
|
|
|
### Inference
|
|
|
|
If you want to generate images with a script, run `sdxl_gen_img.py`. You can specify the LLLite model file with `--control_net_lllite_models`. The dimension is automatically obtained from the model file.
|
|
|
|
Specify the conditioning image to be used for inference with `--guide_image_path`. Since preprocess is not performed, if it is Canny, specify an image processed with Canny (white line on black background). `--control_net_preps`, `--control_net_weights`, and `--control_net_ratios` are not supported.
|
|
|
|
## Credit
|
|
|
|
I would like to thank lllyasviel, the author of ControlNet, furusu, who provided me with advice on implementation and helped me solve problems, and ddPn08, who implemented the ControlNet dataset.
|
|
|
|
## Sample
|
|
|
|
Canny
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|