Kohya S
11aced3500
simplify multi-GPU sample generation
2024-02-03 22:25:29 +09:00
DKnight54
1567ce1e17
Enable distributed sample image generation on multi-GPU enviroment ( #1061 )
...
* Update train_util.py
Modifying to attempt enable multi GPU inference
* Update train_util.py
additional VRAM checking, refactor check_vram_usage to return string for use with accelerator.print
* Update train_network.py
* Update train_util.py
* Update train_util.py
* Update train_util.py
* Update train_util.py
* Update train_util.py
* Update train_util.py
* Update train_util.py
* Update train_util.py
* Update train_util.py
* Update train_util.py
* Update train_util.py
* Update train_util.py
* Update train_util.py
* Update train_util.py
* Update train_util.py
* Update train_util.py
remove sample image debug outputs
* Update train_util.py
* Update train_util.py
* Update train_network.py
* Update train_util.py
* Update train_util.py
* Update train_util.py
* Update train_util.py
* Update train_util.py
* Update train_util.py
* Update train_util.py
* Update train_util.py
* Update train_util.py
* Update train_network.py
* Update train_util.py
* Update train_network.py
* Update train_network.py
* Update train_network.py
* Cleanup of debugging outputs
* adopt more elegant coding
Co-authored-by: Aarni Koskela <akx@iki.fi >
* Update train_util.py
Fix leftover debugging code
attempt to refactor inference into separate function
* refactor in function generate_per_device_prompt_list() generation of distributed prompt list
* Clean up missing variables
* fix syntax error
* Update train_util.py
* Update train_util.py
* Update train_util.py
* Update train_util.py
* Update train_util.py
* Update train_util.py
* Update train_util.py
* Update train_util.py
* true random sample image generation
update code to reinitialize random seed to true random if seed was set
* true random sample image generation
* simplify per process prompt
* Update train_util.py
* Update train_util.py
* Update train_util.py
* Update train_util.py
* Update train_util.py
* Update train_util.py
* Update train_util.py
* Update train_util.py
* Update train_network.py
* Update train_network.py
* Update train_network.py
---------
Co-authored-by: Aarni Koskela <akx@iki.fi >
2024-02-03 21:46:31 +09:00
Kohya S
c576f80639
Fix ControlNetLLLite training issue #1069
2024-01-25 18:43:07 +09:00
Kohya S
fef172966f
Add network_multiplier for dataset and train LoRA
2024-01-20 16:24:43 +09:00
Kohaku-Blueleaf
9cfa68c92f
[Experimental Feature] FP8 weight dtype for base model when running train_network (or sdxl_train_network) ( #1057 )
...
* Add fp8 support
* remove some debug prints
* Better implementation for te
* Fix some misunderstanding
* as same as unet, add explicit convert
* better impl for convert TE to fp8
* fp8 for not only unet
* Better cache TE and TE lr
* match arg name
* Fix with list
* Add timeout settings
* Fix arg style
* Add custom seperator
* Fix typo
* Fix typo again
* Fix dtype error
* Fix gradient problem
* Fix req grad
* fix merge
* Fix merge
* Resolve merge
* arrangement and document
* Resolve merge error
* Add assert for mixed precision
2024-01-20 09:46:53 +09:00
Kohya S
09ef3ffa8b
Merge branch 'main' into dev
2024-01-14 21:49:25 +09:00
Nir Weingarten
ab716302e4
Added cli argument for wandb session name
2024-01-03 11:52:38 +02:00
Plat
62e7516537
feat: support torch.compile
2023-12-27 02:17:24 +09:00
Kohya S
3efd90b2ad
fix sampling in training with mutiple gpus ref #989
2023-12-15 22:35:54 +09:00
Kohya S
d309a27a51
change option names, add ddp kwargs if needed ref #1000
2023-12-13 21:02:26 +09:00
Kohya S
471d274803
Merge pull request #1000 from Isotr0py/dev
...
Fix multi-gpu SDXL training
2023-12-13 20:52:11 +09:00
Kohya S
35f4c9b5c7
fix an error when keep_tokens_separator is not set ref #975
2023-12-12 21:43:21 +09:00
Kohya S
034a49c69d
Merge pull request #975 from Linaqruf/dev
...
Add keep_tokens_separator as alternative for keep_tokens
2023-12-12 21:28:32 +09:00
Isotr0py
bb5ae389f7
fix DDP SDXL training
2023-12-12 19:58:44 +08:00
Kohya S
4a2cef887c
fix lllite training not working ref #913
2023-12-10 09:23:37 +09:00
Kohya S
42750f7846
fix error on pool_workaround in sdxl TE training ref #994
2023-12-10 09:18:33 +09:00
Isotr0py
db84530074
Fix gradients synchronization for multi-GPUs training ( #989 )
...
* delete DDP wrapper
* fix train_db vae and train_network
* fix train_db vae and train_network unwrap
* network grad sync
---------
Co-authored-by: Kohya S <52813779+kohya-ss@users.noreply.github.com >
2023-12-07 22:01:42 +09:00
Kohya S
81a440c8e8
Merge pull request #955 from xzuyn/paged_adamw
...
Add PagedAdamW
2023-12-03 21:22:38 +09:00
Kohya S
f24a3b5282
show seed in generating samples
2023-12-03 21:15:30 +09:00
Kohya S
383b4a2c3e
Merge pull request #907 from shirayu/add_option_sample_at_first
...
Add option --sample_at_first
2023-12-03 21:00:32 +09:00
Kohya S
df59822a27
Merge pull request #906 from shirayu/accept_scheduler_designation_in_training
...
Accept sampler designation in sampling of training
2023-12-03 20:46:16 +09:00
Furqanil Taqwa
1bdd83a85f
remove unnecessary debug print
2023-11-28 17:26:27 +07:00
Furqanil Taqwa
1624c239c2
added keep_tokens_separator to dynamically keep token for being shuffled
2023-11-28 17:23:55 +07:00
Kohya S
f312522cef
Merge pull request #913 from KohakuBlueleaf/custom-seperator
...
Add custom seperator for shuffle caption
2023-11-19 21:32:01 +09:00
xzuyn
da5a144589
Add PagedAdamW
2023-11-18 07:47:27 -05:00
Kohya S
6231aa91e2
common lr logging, set default None to ddp_timeout
2023-11-05 19:09:17 +09:00
Kohaku-Blueleaf
583e2b2d01
Fix typo
2023-10-30 20:02:04 +08:00
Kohaku-Blueleaf
5dc2a0d3fd
Add custom seperator
2023-10-30 19:55:30 +08:00
Yuta Hayashibe
fea810b437
Added --sample_at_first to generate sample images before training
2023-10-29 21:44:57 +09:00
Yuta Hayashibe
40d917b0fe
Removed incorrect comments
2023-10-29 21:02:44 +09:00
Yuta Hayashibe
cf876fcdb4
Accept --ss to set sample_sampler dynamically
2023-10-29 20:15:04 +09:00
Yuta Hayashibe
291c29caaf
Added a function line_to_prompt_dict() and removed duplicated initializations
2023-10-29 19:57:25 +09:00
Yuta Hayashibe
01e00ac1b0
Make a function get_my_scheduler()
2023-10-29 19:46:02 +09:00
Kohya S
a9ed4ed8a8
Merge pull request #900 from xzuyn/paged_adamw_32bit
...
Add PagedAdamW32bit
2023-10-29 15:01:55 +09:00
Kohya S
9d6a5a0c79
Merge pull request #899 from shirayu/use_moving_average
...
Show moving average loss in the progress bar
2023-10-29 14:37:58 +09:00
Kohya S
fb97a7aab1
Merge pull request #898 from shirayu/update_repare_buckets_latents
...
Fix a typo and add assertions in making buckets
2023-10-29 14:29:53 +09:00
Kohaku-Blueleaf
1cefb2a753
Better implementation for te autocast ( #895 )
...
* Better implementation for te
* Fix some misunderstanding
* as same as unet, add explicit convert
* Better cache TE and TE lr
* Fix with list
* Add timeout settings
* Fix arg style
2023-10-28 15:49:59 +09:00
xzuyn
d8f68674fb
Update train_util.py
2023-10-27 07:05:53 -04:00
Yuta Hayashibe
0d21925bdf
Use @property
2023-10-27 18:14:27 +09:00
Yuta Hayashibe
3d2bb1a8f1
Add LossRecorder and use moving average in all places
2023-10-27 17:49:49 +09:00
Yuta Hayashibe
837a4dddb8
Added assertions
2023-10-26 13:34:36 +09:00
Isotr0py
592014923f
Support JPEG-XL on windows
2023-10-04 21:48:25 +08:00
Yuta Hayashibe
27f9b6ffeb
updated typos to v1.16.15 and fix typos
2023-10-01 21:51:24 +09:00
Kohya S
360af27749
fix ControlNetDataset not working
2023-09-03 12:27:58 +09:00
Kohya S
0ee75fd75d
fix typos, add comments etc.
2023-09-03 12:24:15 +09:00
Kohya S
2eae9b66d0
Merge pull request #798 from vvern999/vvern999-patch-1
...
add input perturbation noise
2023-09-03 10:51:23 +09:00
Kohya S
948cf17499
add caption_prefix/suffix to dataset
2023-09-02 16:17:12 +09:00
Kohya S
497051c14b
Merge pull request #786 from Isotr0py/jxl
...
Support JPEG XL
2023-09-02 15:30:07 +09:00
vvern999
e0beb6a999
add input perturbation noise
...
from https://arxiv.org/abs/2301.11706
2023-09-02 07:33:27 +03:00
Kohya S
7e850f3b7e
Merge branch 'main' into sdxl
2023-09-01 07:59:26 +09:00
