RuntimeError: 'AddmmBackward0' Returned NaN Values

[yxwang8775]

Hello,
While running your code, I encountered the following error:
RuntimeError: Function 'AddmmBackward0' returned nan values in its 1th output.
I noticed that in your configuration, you've explicitly set detect_anomaly=True, which is set to False by default. I wonder if you have also encountered this issue during development. And could you please provide guidance on how to resolve it?
Thank you in advance for your help!

Best regards

[yxwang8775]

The command is
python run.py profile=re_twostage model.model.model_path="cnut1648/biolinkbert-large-mnli-snli"
model.model.margin=0.7 model.model.temperature=0.01
model.save_epoch=1
trainer.detect_anomaly=True
datamodule.batch_size=4 trainer.accumulate_grad_batches=16
datamodule=DDI model.num_negs=4 model.model.hypo_version=v1