User jobs failing with 'Maximum of reschedulings reached'

[ernstpijper]

Hi,

Hope you can shed some light on this problem that might help me find the solution.

DIRAC py3 server 7.3.17, py3 pilot jobs and py3 diracos2 client

Many user jobs are failing with the following MinorStatus:
Maximum of reschedulings reached

The job logging shows the following Application Status:
On Hold: after rescheduling 3

In the WorkloadManagement/Optimizers/log/current I see the following error message (that may be related)
2024-01-29 15:01:39 UTC WorkloadManagement/Optimizers/WorkloadManagement/SandboxMetadataDB ERROR: _update (INSERT INTO sb_EntityMapping ( entityId, entitySetup, Type, SBId ) VALUES ( "Job:348732", "DIRAC-Production", "Input", 324286 )): Execution failed. 1062: Duplicate entry '324286-DIRAC-Production-Job:348732-Input' for key 'sb_EntityMapping.Mapping'

Job #348732 that is being referred to is one of those failed jobs.

Is it maybe immediately clear to you what may cause this? I understand you might need more information.

Thanks,

Ernst
SURF

[ernstpijper]

Just noticed there's another discussion on the same topic. Sorry.

[ernstpijper]

This one: #5821

[fstagni]

I am not sure the 2 are related.

Do you errors similar to

2024-01-29 15:01:39 UTC WorkloadManagement/Optimizers/WorkloadManagement/SandboxMetadataDB ERROR: _update (INSERT INTO sb_EntityMapping ( entityId, entitySetup, Type, SBId ) VALUES ( "Job:348732", "DIRAC-Production", "Input", 324286 )): Execution failed. 1062: Duplicate entry '324286-DIRAC-Production-Job:348732-Input' for key 'sb_EntityMapping.Mapping'

for other jobs?

[ernstpijper]

Yes there lots and lots of similar error messages in the logs for other jobs.

I also checked the proxies and they seem just fine so indeed doesn't seem related to the other discussion.

[fstagni]

The error is pretty clear: it is trying to insert the "same" job in the DB more than once. I can't really know how it happened without looking deeply at your logs, which of course I can't!

BTW, you are using an old version of DIRAC, we really suggest you update.