Investigate flaky tests on CI

[masih]

Set up a temporary workflow to run all flaky tests repeatedly on CI for further investigation.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 64.82%. Comparing base (23ce86a) to head (486b9a8).
Report is 4 commits behind head on main.

❗ There is a different number of reports uploaded between BASE (23ce86a) and HEAD (486b9a8). Click for more details.

HEAD has 2 uploads less than BASE

Flag	BASE (23ce86a)	HEAD (486b9a8)
	4	2

Additional details and impacted files

@@             Coverage Diff             @@
##             main     #700       +/-   ##
===========================================
- Coverage   76.32%   64.82%   -11.50%     
===========================================
  Files          69       69               
  Lines        5596     5584       -12     
===========================================
- Hits         4271     3620      -651     
- Misses        911     1630      +719     
+ Partials      414      334       -80     

see 26 files with indirect coverage changes

[Stebalien]

Well, this looks like it helped a bit.

[Stebalien]

Hm. Well, I'm going to try increasing the timeout.

[masih]

@Stebalien Tests seem to only fail when run in conjunction with other tests. To me so far this seems like some kind of congestion issue deep in libp2p/pubsub stack.

[Stebalien]

Tests seem to only fail when run in conjunction with other tests. To me so far this seems like some kind of congestion issue deep in libp2p/pubsub stack.

Or just a timing issue? We're using a mock network, so at least it's not some kind of port issue.

But yeah, it sounds like the solution is to simply not run them in parallel.

[masih]

Findings:

• Repeated individual tests do not fail (repeated up to 100 times each).
• They do fail when run in conjunction with other tests.
  • More specifically in conjunction with other tests in f3_test.go
• These tests are time-based, and use an artificial clock. Having reviewed the artificial clock code nothing immediately stands out.
• The cause of failure is waiting for a condition (e.g. instance number) that is never satisfied regardless of how long we wait (tested up to 30m timeout).
• Tests pass consistently after removing t.Parallel() (repeated 50 times).