New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

it seems that hash of traindata is lost, so it's impossible to continue finetune after stop #483

Open

drazdra opened this issue Mar 16, 2024 · 0 comments

drazdra commented Mar 16, 2024

upon every restart of finetune i see:
"train data seems to have changed. restarting shuffled epoch."

i looked up where it happens, added debugging line and it turned out that
train->shuffle_samples_hash returns zero.

so, it's either not being stored or not being read. so, finetuning always starts from scratch and can not be continued.

at least that's how it is for me :).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment