ET evaluation problem after the data/dataloader update

[cuteyyt]

Dear Organizers,

After updating the code for fixing up the data/data_loader bug, we rerun the code with both our custom functions and the notebook's default settings directly and found that the ET dice score is very low while the WT and TC seem OK. The figure below shows the last round's evaluation results using our custom functions training 40 rounds,

which seems abnormal.

Could you kindly help us figure out this issue?

Thanks!

[HongzhengYang]

I have the same question. @psfoley @sbakas @sarthakpati @msheller

[GorkemP]

We did not encounter such issue for the Task-1 (when running the baseline code).

[cuteyyt]

Thanks! We had run more rounds yesterday and the ET scores were up to 60. As we haven't seen this issue before (when the data/data_loader bug hasn't been fixed, we found that our algorithm‘s convergence speed will obviously slow down after 40-60 rounds), so it confused us.

[GorkemP]

We face with the high RAM consumption; therefore, get out-of-memory error after certain rounds. Have you experienced such issue when you train with more rounds?

[cuteyyt]

We did not encounter the RAM problem. However, we met #106 (program randomly killed) from time to time.