-
Notifications
You must be signed in to change notification settings - Fork 126
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Missing batch statistics #946
Labels
Comments
Datahttps://drive.google.com/file/d/1hr-RzBz-5zCMhwbi4ogzPulVB1ZLCDUI/view?usp=sharing Marian version
Command
Logfilehttps://gist.github.com/alvations/9da72d5458c409e8971ee3c65d550a85 Hardware
|
This is interesting,
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Bug description
At a certain point of the data, the batch statistics somehow "disappears" or went missing. I recall there was an issue that outputs the batch to a temp file, but I've been using
--shuffle-in-ram
so not sure where to find the bad batch. Maybe it's #480 ?Will try to re-run without
--shuffle-in-ram
and see if goes pass that same batch. Meanwhile, is it possible to skip batch and move to the next one if one batch's statistics is missing?How to reproduce
See next comment.
Context
v1.11.0 f00d0621 2022-02-08 08:39:24 -0800
The text was updated successfully, but these errors were encountered: