* doc: move sample prompt file documentation, and remove history for branch
* doc: remove outdated FLUX.1 and SD3 training information from README
* doc: update README and training documentation for clarity and structure
* feat: add pyramid noise and noise offset options to generation script
* fix: fix to work with SD1.5 models
* doc: update to match with latest gen_img.py
* doc: update README to clarify script capabilities and remove deprecated sections
* fix: support dataset with metadata
* feat: support another tagger model
* fix: improve handling of image size and caption/tag processing in FineTuningDataset
* fix: enhance metadata loading to support JSONL format in FineTuningDataset
* feat: enhance image loading and processing in ImageLoadingPrepDataset with batch support and output options
* fix: improve image path handling and memory management in dataset classes
* Update finetune/tag_images_by_wd14_tagger.py
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* fix: add return type annotation for process_tag_replacement function and ensure tags are returned
* feat: add artist category threshold for tagging
* doc: add comment for clarification
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
- Merged redundant test files
- Removed 'comprehensive' from file and docstring names
- Improved test organization and clarity
- Ensured all tests continue to pass
- Simplified test documentation
- Add explicit warning and tracking for multiple unique latent shapes
- Simplify test imports by removing unused modules
- Minor formatting improvements in print statements
- Ensure log messages provide clear context about dimension mismatches
- Make FAISS import optional with try/except
- CDCPreprocessor raises helpful ImportError if FAISS unavailable
- train_util.py catches ImportError and returns None
- train_network.py checks for None and warns user
- Training continues without CDC-FM if FAISS not installed
- Remove benchmark file (not needed in repo)
This allows users to run training without FAISS dependency.
CDC-FM will be automatically disabled with a warning if FAISS is missing.
- Add --cdc_adaptive_k flag to enable adaptive k based on bucket size
- Add --cdc_min_bucket_size to set minimum bucket threshold (default: 16)
- Fixed mode (default): Skip buckets with < k_neighbors samples
- Adaptive mode: Use k=min(k_neighbors, bucket_size-1) for buckets >= min_bucket_size
- Update CDCPreprocessor to support adaptive k per bucket
- Add metadata tracking for adaptive_k and min_bucket_size
- Add comprehensive pytest tests for adaptive k behavior
This allows CDC-FM to work effectively with multi-resolution bucketing where
bucket sizes may vary widely. Users can choose between strict paper methodology
(fixed k) or pragmatic approach (adaptive k).
- Add ss_use_cdc_fm, ss_cdc_k_neighbors, ss_cdc_k_bandwidth, ss_cdc_d_cdc, ss_cdc_gamma
- Ensures CDC-FM training parameters are tracked in model metadata
- Enables reproducibility and model provenance tracking
- Add --cdc_debug flag to enable verbose bucket-by-bucket output
- When debug=False (default): Show tqdm progress bar, concise logging
- When debug=True: Show detailed bucket information, no progress bar
- Improves user experience during CDC cache generation
Fixes shape mismatch bug in multi-subset training where CDC preprocessing
and training used different index calculations, causing wrong CDC data to
be loaded for samples.
Changes:
- CDC cache now stores/loads data using image_key strings instead of integer indices
- Training passes image_key list instead of computed integer indices
- All CDC lookups use stable image_key identifiers
- Improved device compatibility check (handles "cuda" vs "cuda:0")
- Updated all 30 CDC tests to use image_key-based access
Root cause: Preprocessing used cumulative dataset indices while training
used sorted keys, resulting in mismatched lookups during shuffled multi-subset
training.
- Treat cuda and cuda:0 as compatible devices
- Only warn on actual device mismatches (cuda vs cpu)
- Eliminates warning spam during multi-subset training
- Check that noise and CDC matrices are on same device
- Automatically transfer noise if device mismatch detected
- Warn user when device transfer occurs
- Add tests to verify device handling
- Track warned samples in global set to prevent log spam
- Each sample only warned once per training session
- Prevents thousands of duplicate warnings during training
- Add tests to verify throttling behavior
- Remove @torch.no_grad() decorator from compute_sigma_t_x()
- Gradients now properly flow through CDC transformation during training
- Add comprehensive gradient flow tests for fast/slow paths and fallback
- All 25 CDC tests passing
- Create apply_cdc_noise_transformation() for better modularity
- Implement fast path for batch processing when all shapes match
- Implement slow path for per-sample processing on shape mismatch
- Clone noise tensors in fallback path for gradient consistency
- Cache all shapes during GammaBDataset initialization
- Eliminates file I/O on every training step (9.5M accesses/sec)
- Reduces get_shape() from file operation to dict lookup
- Memory overhead: ~126 bytes/sample (~12.6 MB per 100k images)
