Submit critical service check whenever connection fails

[iliakur]

What does this PR do?

Before

We submit CRITICAL service check only if our first connection attempt to mongodb fails. If the connection breaks after the check has run a few times successfully, we continue sending OK service check even if the connection dies.

After

We submit the service check at the very end of the integration check run, at which point we can catch connection errors and submit a CRITICAL service check every time the connection fails.

Motivation

Several support cases. Folks are saying their service check remains OK even though the metric submissions are missing and the logs show connection errors.

Additional Notes

Review checklist (to be filled by reviewers)

• Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
• PR title must be written as a CHANGELOG entry (see why)
• Files changes must correspond to the primary purpose of the PR as described in the title (small unrelated changes should have their own PR)
• PR must have changelog/ and integration/ labels attached
• If the PR doesn't need to be tested during QA, please add a qa/skip-qa label.

[github-actions]

Test Results

  16 files    16 suites   3m 17s ⏱️
  92 tests   92 ✔️     0 💤 0 ❌
768 runs  580 ✔️ 188 💤 0 ❌

Results for commit fbbb024.

♻️ This comment has been updated with latest results.

[codecov]

Codecov Report

Merging #15208 (fbbb024) into master (ca31f3c) will decrease coverage by 0.01%.
Report is 40 commits behind head on master.
The diff coverage is 88.23%.

Flag	Coverage Δ
mongo	95.99% <88.23%> (-0.56%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

[FlorentClarret]

Left some questions but looks good to me 👍

[alopezz]

👍

[vivek-datadog]

LGTM if test failures are resolved 👍

[vivek-datadog]

👍