Coverage

[MetRonnie]

I was trying to look up coverage and a look at https://codecov.io/gh/cylc/cylc-flow reveals it hasn't updated in 2 months. I'm guessing this was the move from Travis to GH Actions?

Pull requests welcome!

[kinow]

I think we disabled it to ease the migration to GH actions, but there were problems with the reported values.

IIRC, codecov didn't/doesn't merge the reports well, giving a different coverage value; see codecov/codecov-python#244

For this, we could either use line-coverage only, but I think branch-coverage is more helpful to guide someone writing tests; or use the number reported in codecov as a guideline/indicator, but still run the report locally for new features... or look at SonarCloud/coveralls/codacy/etc and see if their reporters log the correct value.

Also, it wasn't reporting subcommand coverage (there's an issue for that somewhere); see #3488

For this, there's a solution in the issue above that could work. An alternative is to monkey-patch cylc-flow classes to trigger a command telling coverage module that the tests is done. I remember looking into that issue, and with debug/trace enabled, it's possible to confirm that the coverage session was started successfully.

But because of the way we handle subcommands, the python process running the coverage session gets terminated without flushing the session data (hence empty report). If you set a breakpoint before the session is terminated, and flush it, the coverage is successfully reported. atexit could work, but would have to be only for tests (thus monkey-patching).

The solution proposed in the issue is probably the clean/simplest.

[MetRonnie]

I now see there's already #3598 for this - it didn't show up when I searched yesterday for some reason

[hjoliver]

Closing as duplicate...