4 Commits

Author	SHA1	Message	Date
araleza	f6f3d6e34e	Now exp_avg[_sq] are stored on cpu in 24 bit format. Also changed some final Flux stages to f32.	2025-09-28 15:30:02 +01:00
araleza	657813346b	Changed u16 pow() factor from 16.0 to 8.0, seems to better distribute the exp_avg[_sq] values	2025-08-31 18:51:50 +01:00
araleza	5ea1ada2e2	Changed cpu storage of exp_avg[_sq] from bf16 to powed/scaled u16	2025-08-31 12:46:12 +01:00
araleza	225ea36285	Support for fused (N)AdamW + Kahan + momentum offloading FFT on a 5090.	2025-08-24 16:00:38 +01:00