Task-1: ValueError: A model output tensor was found to have nan values.

[GorkemP]

Dear organizers,

We sometimes get the following error just after a new round starts:

It seems like an exploding gradient problem. In our aggregation method, we check returned aggregated values and there is no nan value but we know that it can be still possible during the forward pass. What we would like to ask you is that is there anything we can do in our side to prevent this? Normally, this issue can be prevented with gradient/value clipping, we applied those but did not work.

We always get this nan value error in the output tensor; therefore, we also suspect that there might be some function in the final output layer that may cause this behaviour.

Thank you.

[sarthakpati]

Hi @GorkemP,

This does seem like an exploding gradient issue. We will investigate to see if there is anything we can put in for the short term to address this. Tagging @alexey-gruzdev and @psfoley

Cheers,
Sarthak

[sarthakpati]

Not yet.

[GorkemP]

Hi, we have solved this issue on our side. We were trying to implement SGD with momentum and during calculations, we were using all tensors (whatever the tensor_name is in the aggregator function). Later, we have realized that there are parameters related to batchnorm (running_mean, running_var , num_batches_tracked). We excluded them in the momentum part (we have only used tensors that ends with weight and bias in momentum calculations) and nan problem disappeared. Hope it helps.

[sarthakpati]

Thanks @GorkemP!

[dskhanirfan]

@GorkemP Thanks. Is your code available to review?

[eceisik]

@dskhanirfan, you can check here

https://github.com/eceisik/FeTS_Challenge_METU_FL_Team/blob/db0db894b971dc4aba7f20fdcb3585e2f4e0ece1/FeTS_METU_FL_ALL.py#L531

[dskhanirfan]

I ran some experiments, I observed if there are 4 participating colabs in a federation round and the local_tensors values are0like : [array([118.], dtype=float32), array([118.], dtype=float32), array([118.], dtype=float32), array([118.], dtype=float32)] then weights returned are nan like : [nan, nan, nan, nan]. Can you verify? what can be a potential reason for getting local_tensors values as zero and how can we resolve this issue? @GorkemP @psfoley @sbakas @sarthakpati