DKnight54 1567ce1e17 Enable distributed sample image generation on multi-GPU enviroment (#1061) ...

* Update train_util.py

Modifying to attempt enable multi GPU inference

* Update train_util.py

additional VRAM checking, refactor check_vram_usage to return string for use with accelerator.print

* Update train_network.py

* Update train_util.py

* Update train_util.py

* Update train_util.py

* Update train_util.py

* Update train_util.py

* Update train_util.py

* Update train_util.py

* Update train_util.py

* Update train_util.py

* Update train_util.py

* Update train_util.py

* Update train_util.py

* Update train_util.py

* Update train_util.py

* Update train_util.py

* Update train_util.py

remove sample image debug outputs

* Update train_util.py

* Update train_util.py

* Update train_network.py

* Update train_util.py

* Update train_util.py

* Update train_util.py

* Update train_util.py

* Update train_util.py

* Update train_util.py

* Update train_util.py

* Update train_util.py

* Update train_util.py

* Update train_network.py

* Update train_util.py

* Update train_network.py

* Update train_network.py

* Update train_network.py

* Cleanup of debugging outputs

* adopt more elegant coding

Co-authored-by: Aarni Koskela <akx@iki.fi>

* Update train_util.py

Fix leftover debugging code
attempt to refactor inference into separate function

* refactor in function generate_per_device_prompt_list() generation of distributed prompt list

* Clean up missing variables

* fix syntax error

* Update train_util.py

* Update train_util.py

* Update train_util.py

* Update train_util.py

* Update train_util.py

* Update train_util.py

* Update train_util.py

* Update train_util.py

* true random sample image generation

update code to reinitialize random seed to true random if seed was set

* true random sample image generation

* simplify per process prompt

* Update train_util.py

* Update train_util.py

* Update train_util.py

* Update train_util.py

* Update train_util.py

* Update train_util.py

* Update train_util.py

* Update train_util.py

* Update train_network.py

* Update train_network.py

* Update train_network.py

---------

Co-authored-by: Aarni Koskela <akx@iki.fi>

2024-02-03 21:46:31 +09:00

..

ipex

IPEX torch.tensor FP64 workaround

2024-01-30 01:52:32 +03:00

__init__.py

Proposed file structure rework and required file changes

2022-12-18 15:24:46 -05:00

attention_processors.py

update diffusers to 1.16 | train_network

2023-06-01 20:39:26 +09:00

config_util.py

Add network_multiplier for dataset and train LoRA

2024-01-20 16:24:43 +09:00

custom_train_functions.py

Fix min-snr-gamma for v-prediction and ZSNR.

2023-11-07 23:02:25 +01:00

huggingface_util.py

fix error in huggingface_path_in_repo=None

2023-08-06 22:08:30 +09:00

hypernetwork.py

update diffusers to 1.16 | train_network

2023-06-01 20:39:26 +09:00

ipex_interop.py

Deduplicate ipex initialization code

2024-01-19 18:07:36 +02:00

lpw_stable_diffusion.py

Update dependencies ref #1024

2024-01-04 19:53:25 +09:00

model_util.py

Deduplicate ipex initialization code

2024-01-19 18:07:36 +02:00

original_unet.py

Fix typo --spda (it's --sdpa)

2024-01-16 14:32:48 +02:00

sai_model_spec.py

fix crashing in saving lora with clipskip

2023-08-08 21:03:16 +09:00

sdxl_lpw_stable_diffusion.py

Fix dtype issue in PyTorch 2.0 for generating samples in training sdxl network

2024-01-22 12:43:37 +09:00

sdxl_model_util.py

Update dependencies ref #1024

2024-01-04 19:53:25 +09:00

sdxl_original_unet.py

make separate U-Net for inference

2023-11-26 18:11:30 +09:00

sdxl_train_util.py

Fix gradients synchronization for multi-GPUs training (#989)

2023-12-07 22:01:42 +09:00

slicing_vae.py

make slicing vae compatible with latest diffusers

2023-11-26 18:12:04 +09:00

train_util.py

Enable distributed sample image generation on multi-GPU enviroment (#1061)

2024-02-03 21:46:31 +09:00

utils.py

small fix

2023-04-02 23:21:17 +09:00