New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

The averaged learning rate in log.txt #254

Open

haohang96 opened this issue Sep 22, 2023 · 0 comments

haohang96 commented Sep 22, 2023

I find that in https://dl.fbaipublicfiles.com/dino/dino_deitsmall16_pretrain/dino_deitsmall16_pretrain_log.txt, the average learning rate of each epoch is not the same as the log in my reproduce results.

According to the deit-small args(https://dl.fbaipublicfiles.com/dino/dino_deitsmall16_pretrain/args.txt) and the cos_schedule code( warmup_schedule = np.linspace(start_warmup_value, base_value, warmup_iters)), the averaged learning rate in epoch0 is: np.linspace(0,0.002,12510)[:1251].mean() = 9.992805180270206e-05. But the average lr in https://dl.fbaipublicfiles.com/dino/dino_deitsmall16_pretrain/dino_deitsmall16_pretrain_log.txt is 9.999999999999726e-06.

I wonder what causes such a difference.

Thanks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment