Processes get killed at random

[Linardos]

This has been the most major roadblock in this challenge for me: My processes are arbitrarily getting killed even though I am using GPUs with 24GB for each experiment. It always happens if I try to load a checkpoint from another experiment to continue from that point with other parameters. But it also just happens at random, and if it happens during checkpoint storing, the pickle file gets corrupted and I lose all my progress. The only feedback I get from the code when this happens is "Killed"

Did anyone else have this issue?

[sarthakpati]

Hi @Linardos,

I am unfortunately unable to replicate this issue. Tagging @alexey-gruzdev @psfoley for more clarification.

[Linardos]

Could you perhaps tell me what other resources are being used other than GPU and to what extent? Right now it seems to need 11GB from a GPU that is 24GB for one process but there is a lot of memory left in the GPU when the "killed" happens. And somehow this issue happens when there are multiple experiments that have progressed further in the training. Some days ago I could train 5 at the same time when they were just starting, when they progressed though some started getting killed and if I try to run a killed process again one of the others gets killed. Dunno if that helps for a hint

[sarthakpati]

How much RAM do you have? And what type of CPU?

[Linardos]

RAM:

CPU:

[sarthakpati]

Hmm, interesting. There shouldn't be any issues with the kind of resources you have. The DataLoader uses the CPU and RAM for performing the necessary augmentations, but it should not be using anything that would cause the process to get killed. Do you have any ideas, @psfoley @alexey-gruzdev ?

[psfoley]

@Linardos I unfortunately don't have a great explanation for this either as I haven't run into this myself. Is there anything reported in the system logs? The only thing I could think of that would help would be a retry mechanism for the checkpoint serialization so you don't lose your progress. The other thing you may want to do if you see this happening repeatedly is manually backup those checkpoints into a different folder between rounds.