sdbds
5e45df722d
update gemma2 train attention layer
2025-03-04 08:07:33 +08:00
青龍聖者@bdsqlsz
09c4710d1e
Merge pull request #22 from rockerBOO/sage_attn
...
Add Sage Attention for Lumina
2025-03-03 10:26:02 +08:00
青龍聖者@bdsqlsz
dfe1ab6c50
Merge pull request #21 from rockerBOO/lumina-torch-dynamo-gemma2
...
fix torch compile/dynamo for Gemma2
2025-03-02 18:31:13 +08:00
青龍聖者@bdsqlsz
b6e4194ea5
Merge pull request #20 from rockerBOO/lumina-system-prompt-special-token
...
Lumina system prompt special token
2025-03-02 18:30:49 +08:00
青龍聖者@bdsqlsz
b5d1f1caea
Merge pull request #19 from rockerBOO/lumina-block-swap
...
Lumina block swap
2025-03-02 18:30:37 +08:00
青龍聖者@bdsqlsz
d6c3e6346e
Merge pull request #18 from rockerBOO/fix-sample-batch-norms
...
Fix sample batch norms
2025-03-02 18:30:24 +08:00
青龍聖者@bdsqlsz
800d068e37
Merge pull request #17 from rockerBOO/lumina-cache-text-encoder-outputs
...
Lumina cache text encoder outputs
2025-03-02 18:30:08 +08:00
青龍聖者@bdsqlsz
3817b65b45
Merge pull request #16 from rockerBOO/lumina
...
Merge SD3 into Lumina
2025-03-02 18:29:44 +08:00
rockerBOO
a69884a209
Add Sage Attention for Lumina
2025-03-01 20:37:45 -05:00
rockerBOO
cad182d29a
fix torch compile/dynamo for Gemma2
2025-02-28 18:35:19 -05:00
rockerBOO
a2daa87007
Add block swap for uncond (neg) for sample images
2025-02-28 14:22:47 -05:00
rockerBOO
1bba7acd9a
Add block swap in sample image timestep loop
2025-02-28 14:12:13 -05:00
rockerBOO
d6f7e2e20c
Fix block swap for sample images
2025-02-28 14:08:27 -05:00
rockerBOO
9647f1e324
Fix validation block swap. Add custom offloading tests
2025-02-27 20:36:36 -05:00
rockerBOO
42fe22f5a2
Enable block swap for Lumina
2025-02-27 03:21:24 -05:00
rockerBOO
ce2610d29b
Change system prompt to inject Prompt Start special token
2025-02-27 02:47:04 -05:00
rockerBOO
0886d976f1
Add block swap
2025-02-27 02:31:50 -05:00
rockerBOO
542f980443
Fix sample norms in batches
2025-02-27 00:00:20 -05:00
rockerBOO
70403f6977
fix cache text encoder outputs if not using disk. small cleanup/alignment
2025-02-26 23:33:50 -05:00
rockerBOO
7b83d50dc0
Merge branch 'sd3' into lumina
2025-02-26 22:13:56 -05:00
Kohya S
3d79239be4
docs: update README to include recent improvements in validation loss calculation
2025-02-26 21:21:04 +09:00
Kohya S
ec350c83eb
Merge branch 'dev' into sd3
2025-02-26 21:17:29 +09:00
Kohya S.
49651892ce
Merge pull request #1903 from kohya-ss/val-loss-improvement
...
Val loss improvement
2025-02-26 21:15:14 +09:00
Kohya S
1fcac98280
Merge branch 'sd3' into val-loss-improvement
2025-02-26 21:09:10 +09:00
Kohya S.
b286304e5f
Merge pull request #1953 from Disty0/dev
...
Update IPEX libs
2025-02-26 21:03:09 +09:00
Kohya S
ae409e83c9
fix: FLUX/SD3 network training not working without caching latents closes #1954
2025-02-26 20:56:32 +09:00
sdbds
a1a5627b13
fix shift
2025-02-26 11:35:38 +08:00
sdbds
ce37c08b9a
clean code and add finetune code
2025-02-26 11:20:03 +08:00
Disty0
f68702f71c
Update IPEX libs
2025-02-25 21:27:41 +03:00
sdbds
5f9047c8cf
add truncation when > max_length
2025-02-26 01:00:35 +08:00
Kohya S.
6e90c0f86c
Merge pull request #1909 from rockerBOO/progress_bar
...
Move progress bar to account for sampling image first
2025-02-24 18:57:44 +09:00
Kohya S
67fde015f7
Merge branch 'dev' into sd3
2025-02-24 18:56:15 +09:00
Kohya S.
386b7332c6
Merge pull request #1918 from tsukimiya/fix_vperd_warning
...
Remove v-pred warning.
2025-02-24 18:55:25 +09:00
Kohya S
905f081798
Merge branch 'dev' into sd3
2025-02-24 18:54:28 +09:00
Kohya S.
59ae9ea20c
Merge pull request #1945 from yidiq7/dev
...
Remove position_ids for V2
2025-02-24 18:53:46 +09:00
sdbds
fc772affbe
1、Implement cfg_trunc calculation directly using timesteps, without intermediate steps.
...
2、Deprecate and remove the guidance_scale parameter because it used in inference not train
3、Add inference command-line arguments --ct for cfg_trunc_ratio and --rc for renorm_cfg to control CFG truncation and renormalization during inference.
2025-02-24 14:10:24 +08:00
青龍聖者@bdsqlsz
653621de57
Merge pull request #15 from rockerBOO/samples-training
...
Fix samples, LoRA training. Add system prompt, use_flash_attn
2025-02-24 11:24:53 +08:00
rockerBOO
2c94d17f05
Fix typo
2025-02-23 20:21:06 -05:00
rockerBOO
48e7da2d4a
Add sample batch size for Lumina
2025-02-23 20:19:24 -05:00
rockerBOO
ba725a84e9
Set default discrete_flow_shift to 6.0. Remove default system prompt.
2025-02-23 18:01:09 -05:00
rockerBOO
42a801514c
Fix system prompt in datasets
2025-02-23 13:48:37 -05:00
rockerBOO
6d7bec8a37
Remove non-used code
2025-02-23 01:46:47 -05:00
rockerBOO
025cca699b
Fix samples, LoRA training. Add system prompt, use_flash_attn
2025-02-23 01:29:18 -05:00
Kohya S
efb2a128cd
fix wandb val logging
2025-02-21 22:07:35 +09:00
Yidi
13df47516d
Remove position_ids for V2
...
The postions_ids cause errors for the newer version of transformer.
This has already been fixed in convert_ldm_clip_checkpoint_v1() but
not in v2.
The new code applies the same fix to convert_ldm_clip_checkpoint_v2().
2025-02-20 04:49:51 -05:00
青龍聖者@bdsqlsz
6597631b90
Merge pull request #14 from rockerBOO/samples-attention
...
Samples attention
2025-02-19 13:08:00 +08:00
Kohya S
4a36996134
modify log step calculation
2025-02-18 22:05:08 +09:00
Kohya S
dc7d5fb459
Merge branch 'sd3' into val-loss-improvement
2025-02-18 21:34:30 +09:00
Kohya S.
894037f2c6
Merge pull request #1943 from rockerBOO/validation-split-test
...
Fix validation split and add test
2025-02-18 21:30:32 +09:00
rockerBOO
bd16bd13ae
Remove unused attention, fix typo
2025-02-18 01:21:18 -05:00
