[Experimental] Add cache mechanism for dataset groups to avoid long waiting time for initilization (#1178)

* support meta cached dataset

* add cache meta scripts

* random ip_noise_gamma strength

* random noise_offset strength

* use correct settings for parser

* cache path/caption/size only

* revert mess up commit

* revert mess up commit

* Update requirements.txt

* Add arguments for meta cache.

* remove pickle implementation

* Return sizes when enable cache

---------

Co-authored-by: Kohya S <52813779+kohya-ss@users.noreply.github.com>
This commit is contained in:
Kohaku-Blueleaf
2024-03-24 14:40:18 +08:00
committed by GitHub
parent 381c44955e
commit ae97c8bfd1
5 changed files with 173 additions and 22 deletions

View File

@@ -6,6 +6,7 @@ import sys
import random
import time
import json
import pickle
from multiprocessing import Value
import toml
@@ -23,7 +24,7 @@ from library import model_util
import library.train_util as train_util
from library.train_util import (
DreamBoothDataset,
DreamBoothDataset, DatasetGroup
)
import library.config_util as config_util
from library.config_util import (