[infer] Fix inference assert and add extra check

[infer] Fix inference assert and add extra check #6155