* feat: Add LoHa/LoKr network support for SDXL and Anima
- networks/network_base.py: shared AdditionalNetwork base class with architecture auto-detection (SDXL/Anima) and generic module injection
- networks/loha.py: LoHa (Low-rank Hadamard Product) module with HadaWeight custom autograd, training/inference classes, and factory functions
- networks/lokr.py: LoKr (Low-rank Kronecker Product) module with factorization, training/inference classes, and factory functions
- library/lora_utils.py: extend weight merge hook to detect and merge LoHa/LoKr weights alongside standard LoRA
Linear and Conv2d 1x1 layers only; Conv2d 3x3 (Tucker decomposition) support will be added separately.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: Enhance LoHa and LoKr modules with Tucker decomposition support
- Added Tucker decomposition functionality to LoHa and LoKr modules.
- Implemented new methods for weight rebuilding using Tucker decomposition.
- Updated initialization and weight handling for Conv2d 3x3+ layers.
- Modified get_diff_weight methods to accommodate Tucker and non-Tucker modes.
- Enhanced network base to include unet_conv_target_modules for architecture detection.
* fix: rank dropout handling in LoRAModule for Conv2d and Linear layers, see #2272 for details
* doc: add dtype comment for load_safetensors_with_lora_and_fp8 function
* fix: enhance architecture detection to support InferSdxlUNet2DConditionModel for gen_img.py
* doc: update model support structure to include Lumina Image 2.0, HunyuanImage-2.1, and Anima-Preview
* doc: add documentation for LoHa and LoKr fine-tuning methods
* Update networks/network_base.py
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update docs/loha_lokr.md
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* fix: refactor LoHa and LoKr imports for weight merging in load_safetensors_with_lora_and_fp8 function
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Support network_reg_alphas and fix bug when setting rank_dropout in training lora for anima model
* Update anima_train_network.md
* Update anima_train_network.md
* Remove network_reg_alphas
* Update document
* fix: update extend-exclude list in _typos.toml to include configs
* fix: exclude anima tests from pytest
* feat: add entry for 'temperal' in extend-words section of _typos.toml for Qwen-Image VAE
* fix: update default value for --discrete_flow_shift in anima training guide
* feat: add Qwen-Image VAE
* feat: simplify encode_tokens
* feat: use unified attention module, add wrapper for state dict compatibility
* feat: loading with dynamic fp8 optimization and LoRA support
* feat: add anima minimal inference script (WIP)
* format: format
* feat: simplify target module selection by regular expression patterns
* feat: kept caption dropout rate in cache and handle in training script
* feat: update train_llm_adapter and verbose default values to string type
* fix: use strategy instead of using tokenizers directly
* feat: add dtype property and all-zero mask handling in cross-attention in LLMAdapterTransformerBlock
* feat: support 5d tensor in get_noisy_model_input_and_timesteps
* feat: update loss calculation to support 5d tensor
* fix: update argument names in anima_train_utils to align with other archtectures
* feat: simplify Anima training script and update empty caption handling
* feat: support LoRA format without `net.` prefix
* fix: update to work fp8_scaled option
* feat: add regex-based learning rates and dimensions handling in create_network
* fix: improve regex matching for module selection and learning rates in LoRANetwork
* fix: update logging message for regex match in LoRANetwork
* fix: keep latents 4D except DiT call
* feat: enhance block swap functionality for inference and training in Anima model
* feat: refactor Anima training script
* feat: optimize VAE processing by adjusting tensor dimensions and data types
* fix: wait all block trasfer before siwtching offloader mode
* feat: update Anima training guide with new argument specifications and regex-based module selection. Thank you Claude!
* feat: support LORA for Qwen3
* feat: update Anima SAI model spec metadata handling
* fix: remove unused code
* feat: split CFG processing in do_sample function to reduce memory usage
* feat: add VAE chunking and caching options to reduce memory usage
* feat: optimize RMSNorm forward method and remove unused torch_attention_op
* Update library/strategy_anima.py
Use torch.all instead of all.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update library/safetensors_utils.py
Fix duplicated new_key for concat_hook.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update anima_minimal_inference.py
Remove unused code.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update anima_train.py
Remove unused import.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update library/anima_train_utils.py
Remove unused import.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* fix: review with Copilot
* feat: add script to convert LoRA format to ComfyUI compatible format (WIP, not tested yet)
* feat: add process_escape function to handle escape sequences in prompts
* feat: enhance LoRA weight handling in model loading and add text encoder loading function
* feat: improve ComfyUI conversion script with prefix constants and module name adjustments
* feat: update caption dropout documentation to clarify cache regeneration requirement
* feat: add clarification on learning rate adjustments
* feat: add note on PyTorch version requirement to prevent NaN loss
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
