Kohya S
8d1b1acd33
Merge pull request #1266 from Zovjsra/feature/disable-mmap
...
Add "--disable_mmap_load_safetensors" parameter
2024-05-12 17:43:44 +09:00
Kohya S
02298e3c4a
Merge pull request #1331 from kohya-ss/lora-plus
...
Lora plus
2024-05-12 17:04:58 +09:00
Kohya S
44190416c6
update docs etc.
2024-05-12 17:01:20 +09:00
Kohya S
3c8193f642
revert lora+ for lora_fa
2024-05-12 17:00:51 +09:00
Kohya S
c6a437054a
Merge branch 'dev' into lora-plus
2024-05-12 16:18:57 +09:00
Kohya S
1ffc0b330a
fix typo
2024-05-12 16:18:43 +09:00
Kohya S
e01e148705
Merge branch 'dev' into lora-plus
2024-05-12 16:17:52 +09:00
Kohya S
e9f3a622f4
Merge branch 'dev' into lora-plus
2024-05-12 16:17:27 +09:00
Kohya S
7983d3db5f
Merge pull request #1319 from kohya-ss/fused-backward-pass
...
Fused backward pass
2024-05-12 15:09:39 +09:00
Kohya S
bee8cee7e8
update README for fused optimizer
2024-05-12 15:08:52 +09:00
Kohya S
f3d2cf22ff
update README for fused optimizer
2024-05-12 15:03:02 +09:00
Kohya S
6dbc23cf63
Merge branch 'dev' into fused-backward-pass
2024-05-12 14:21:56 +09:00
Kohya S
c1ba0b4356
update readme
2024-05-12 14:21:10 +09:00
Kohya S
607e041f3d
chore: Refactor optimizer group
2024-05-12 14:16:41 +09:00
Kohya S
b56d5f7801
add experimental option to fuse params to optimizer groups
2024-05-06 21:35:39 +09:00
Kohya S
017b82ebe3
update help message for fused_backward_pass
2024-05-06 15:05:42 +09:00
Kohya S
2a359e0a41
Merge pull request #1259 from 2kpr/fused_backward_pass
...
Adafactor fused backward pass and optimizer step, lowers SDXL (@ 1024 resolution) VRAM usage to BF16(10GB)/FP32(16.4GB)
2024-05-06 15:01:56 +09:00
Kohya S
3fd8cdc55d
fix dylora loraplus
2024-05-06 14:03:19 +09:00
Kohya S
7fe81502d0
update loraplus on dylora/lofa_fa
2024-05-06 11:09:32 +09:00
Kohya S
52e64c69cf
add debug log
2024-05-04 18:43:52 +09:00
Kohya S
58c2d856ae
support block dim/lr for sdxl
2024-05-03 22:18:20 +09:00
Kohya S
969f82ab47
move loraplus args from args to network_args, simplify log lr desc
2024-04-29 20:04:25 +09:00
Kohya S
834445a1d6
Merge pull request #1233 from rockerBOO/lora-plus
...
Add LoRA+ support
2024-04-29 18:05:12 +09:00
Kohya S
0540c33aca
pop weights if available #1247
2024-04-21 17:45:29 +09:00
Kohya S
52652cba1a
disable main process check for deepspeed #1247
2024-04-21 17:41:32 +09:00
Zovjsra
64916a35b2
add disable_mmap to args
2024-04-16 16:40:08 +08:00
2kpr
4f203ce40d
Fused backward pass
2024-04-14 09:56:58 -05:00
rockerBOO
68467bdf4d
Fix unset or invalid LR from making a param_group
2024-04-11 17:33:19 -04:00
rockerBOO
75833e84a1
Fix default LR, Add overall LoRA+ ratio, Add log
...
`--loraplus_ratio` added for both TE and UNet
Add log for lora+
2024-04-08 19:23:02 -04:00
Kohya S
71e2c91330
Merge pull request #1230 from kohya-ss/dependabot/github_actions/crate-ci/typos-1.19.0
...
Bump crate-ci/typos from 1.17.2 to 1.19.0
2024-04-07 21:14:18 +09:00
Kohya S
bfb352bc43
change huber_schedule from exponential to snr
2024-04-07 21:07:52 +09:00
Kohya S
c973b29da4
update readme
2024-04-07 20:51:52 +09:00
Kohya S
683f3d6ab3
Merge pull request #1212 from kohya-ss/dev
...
Version 0.8.6
2024-04-07 20:42:41 +09:00
Kohya S
dfa30790a9
update readme
2024-04-07 20:34:26 +09:00
Kohya S
d30ebb205c
update readme, add metadata for network module
2024-04-07 14:58:17 +09:00
kabachuha
90b18795fc
Add option to use Scheduled Huber Loss in all training pipelines to improve resilience to data corruption ( #1228 )
...
* add huber loss and huber_c compute to train_util
* add reduction modes
* add huber_c retrieval from timestep getter
* move get timesteps and huber to own function
* add conditional loss to all training scripts
* add cond loss to train network
* add (scheduled) huber_loss to args
* fixup twice timesteps getting
* PHL-schedule should depend on noise scheduler's num timesteps
* *2 multiplier to huber loss cause of 1/2 a^2 conv.
The Taylor expansion of sqrt near zero gives 1/2 a^2, which differs from a^2 of the standard MSE loss. This change scales them better against one another
* add option for smooth l1 (huber / delta)
* unify huber scheduling
* add snr huber scheduler
---------
Co-authored-by: Kohya S <52813779+kohya-ss@users.noreply.github.com >
2024-04-07 13:54:21 +09:00
Kohya S
089727b5ee
update readme
2024-04-07 12:42:49 +09:00
Kohya S
921036dd91
Merge pull request #1240 from kohya-ss/verify-command-line-args
...
verify command line args if wandb is enabled
2024-04-07 12:27:03 +09:00
ykume
cd587ce62c
verify command line args if wandb is enabled
2024-04-05 08:23:03 +09:00
rockerBOO
1933ab4b48
Fix default_lr being applied
2024-04-03 12:46:34 -04:00
Kohya S
b748b48dbb
fix attention couple+deep shink cause error in some reso
2024-04-03 12:43:08 +09:00
rockerBOO
c7691607ea
Add LoRA-FA for LoRA+
2024-04-01 15:43:04 -04:00
rockerBOO
f99fe281cb
Add LoRA+ support
2024-04-01 15:38:26 -04:00
dependabot[bot]
80e9f72234
Bump crate-ci/typos from 1.17.2 to 1.19.0
...
Bumps [crate-ci/typos](https://github.com/crate-ci/typos ) from 1.17.2 to 1.19.0.
- [Release notes](https://github.com/crate-ci/typos/releases )
- [Changelog](https://github.com/crate-ci/typos/blob/master/CHANGELOG.md )
- [Commits](https://github.com/crate-ci/typos/compare/v1.17.2...v1.19.0 )
---
updated-dependencies:
- dependency-name: crate-ci/typos
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com >
2024-04-01 01:50:22 +00:00
Kohya S
2258a1b753
add save/load hook to remove U-Net/TEs from state
2024-03-31 15:50:35 +09:00
Kohya S
059ee047f3
fix typo
2024-03-30 23:02:24 +09:00
Kohya S
2c2ca9d726
update tagger doc
2024-03-30 22:55:56 +09:00
Kohya S
f5323e3c4b
update tagger doc
2024-03-30 22:10:37 +09:00
Kohya S
cae5aa0a56
update wd14 tagger and doc
2024-03-30 21:48:22 +09:00
Kohya S
6ba84288d9
Merge pull request #1216 from Disty0/dev
...
Rating support for WD Tagger
2024-03-30 18:50:49 +09:00
