Sudden drop in accuracy

[wing212]

Hello, I want to ask why the accuracy has suddenly dropped, and the accuracy of my reproduced article is much lower than that of the original text. I use a single 3090ti graphics card for training.

[voldemortX]

@wing212 Hi! Did you run code from the latest master branch? And could I ask what is your exact setup? e.g. PyTorch & TorchVision version, CUDA version, whether mixed precision is used, etc.

Btw, I don't think there is 3090ti in the market yet, did you mean 3090?

[voldemortX]

Did you accidentally closed the issue?

[wing212]

@wing212 Hi! Did you run code from the latest master branch? And could I ask what is your exact setup? e.g. PyTorch & TorchVision version, CUDA version, whether mixed precision is used, etc.

Btw, I don't think there is 3090ti in the market yet, did you mean 3090?
It is 3090. I have used ape


x.

[wing212]

@wing212 Hi! Did you run code from the latest master branch? And could I ask what is your exact setup? e.g. PyTorch & TorchVision version, CUDA version, whether mixed precision is used, etc.

Btw, I don't think there is 3090ti in the market yet, did you mean 3090?

should I adjust my learning rate?

[voldemortX]

The performance should be reproduceable from my provided shells. Although if you use different PyTorch versions and NVIDIA's apex, it could lead to problems like gradient explosions. Could you give me more details about your shell commands and code version? Instead of using a different learning rate, I recommend full precision training first to isolate the problem.

EDIT: And your baseline (p0) performance also seems off.

[wing212]

Yes, there are problems with the baseline, I don't know why. I am using the code of the single card training version


.

[voldemortX]

@wing212 Firstly, if you are using PyTorch >= 1.6, You best use the master branch. Also a clear environment without the standalone apex package. I suspect the problem is from there. Maybe refer to #2 and check your loss in tensorboard.

Secondly, for 1/30 cityscapes baselines, the number of epochs should be longer. See Appendix B in the paper. Although yours seems good enough there, we still should align the settings.

[important] Your lr for cityscapes seem wrong? It should be 0.004 from Table 1.

[wing212]

@wing212首先，如果您使用的是 PyTorch >= 1.6，则最好使用主分支。也是一个没有独立顶点包的清晰环境。我怀疑问题出在哪里。也许参考#2并检查你在张量板中的损失。

其次，对于1/30的城市景观基线，纪元的数量应该更长。见文件附录B。虽然你的看起来足够好，但我们仍然应该对齐设置。

[重要]你对城市景观的了解似乎不对吗？它应为表 1 中的 0.004。
I originally wanted to try training with other learning rates to see what the results are. I forgot to change the learning rate to 0.001. I will change the epoch to 329 without using apex.

[voldemortX]

@wing212 Great! Let me know if you still can't reproduce results. FYI, the std for Cityscapes 1/30 experiments is 2.56 in my records.

[wing212]

@wing212伟大！如果您仍然无法重现结果，请告诉我。仅供参考，在我的记录中，Cityscapes 1/30实验的标准是2.56。

Thank you, I will keep trying. I would like to ask what is 2.56?

[voldemortX]

@wing212伟大！如果您仍然无法重现结果，请告诉我。仅供参考，在我的记录中，Cityscapes 1/30实验的标准是2.56。

Thank you, I will keep trying. I would like to ask what is 2.56?

STD (标准差)

[wing212]

@wing212 great! If you still can't reproduce the results, please let me know. For informational purposes only, the standard for the Cityscapes 1/30 experiment is 2.56 in my records.

Thank you, I will keep trying. I would like to ask what is 2.56?

STD (STANDARD DIFFERENCE)

ok

[wing212]

Sorry to interrupt you again, I did not use mixed-precision this time, but the result is still wrong. The 1/50 challenge of VOC data set is very low accuracy, and the 1/30 challenge of cityscapes data set is abnormally high

.

[wing212]

[voldemortX]

@wing212 Actually, your results look fine. I did get a 57.24 for city-30-1, that is why we need to take averages in these experiments (the STD is quite large). My voc-50-1 was 63.50 (it was weirdly low but I kept it to be fair), try 50-0 or 50-2 the results should be higher and give you an average around 67.

[wing212]

@wing212 Actually, your results look fine. I did get a 57.24 for city-30-1, that is why we need to take averages in these experiments (the STD is quite large). My voc-50-1 was 63.50 (it was weirdly low but I kept it to be fair), try 50-0 or 50-2 the results should be higher and give you an average around 67.

I will continue， thank you.

[voldemortX]

But it looks like you have sid=2 for the voc experiment, do check the dataset and data splits if you can't get similar avg from 3 runs.

[wing212]

但看起来 voc 实验的 sid=2，如果无法从 3 次运行中获得类似的平均值，请检查数据集和数据拆分。

Ok， I will do it.