feat: add E2E tests for cases that peers going offline

[BruceAko]

Description

Verifies that Scheduler can clear offline peer metadata when the host exits normally/abnormally by following steps:

1. Get the number of all hosts in the scheduler.
2. After a normal exit of a host, verify that the number of hosts in the scheduler decreases by 1.
3. After an abnormal exit of a host, verify that the number of hosts in the scheduler decreases by 1 after twice the announce host interval.

Related Issue

close #3523

Motivation and Context

Earlier we have added functionality so that hosts send LeaveHost() requests to the Scheduler on normal exit and hosts on abnormal exit are also sensed by the Scheduler GC after a set amount of time and the metadata is cleared. Now, we are going to add E2E tests to verify that this functionality is correct.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation Update (if none of the other choices apply)

Checklist

• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING document.
• I have added tests to cover my changes.

[codecov]

Codecov Report

Attention: Patch coverage is 66.38655% with 40 lines in your changes missing coverage. Please review.

Project coverage is 36.04%. Comparing base (2793851) to head (75b8d03).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
pkg/rpc/scheduler/client/mocks/client_v2_mock.go	0.00%	14 Missing ⚠️
pkg/rpc/scheduler/client/client_v2.go	0.00%	9 Missing ⚠️
scheduler/rpcserver/scheduler_server_v2.go	0.00%	9 Missing ⚠️
manager/job/sync_peers.go	0.00%	5 Missing ⚠️
scheduler/resource/standard/host.go	0.00%	2 Missing ⚠️
client/daemon/daemon.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3524      +/-   ##
==========================================
+ Coverage   35.93%   36.04%   +0.11%     
==========================================
  Files         335      335              
  Lines       36848    36961     +113     
==========================================
+ Hits        13240    13323      +83     
- Misses      22769    22801      +32     
+ Partials      839      837       -2     

Flag	Coverage Δ
unittests	36.04% <66.38%> (+0.11%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
pkg/idgen/host_id.go	100.00% <100.00%> (ø)
pkg/rpc/scheduler/client/client_v1.go	0.00% <ø> (ø)
scheduler/metrics/metrics.go	10.89% <ø> (ø)
scheduler/resource/standard/seed_peer_client.go	76.92% <100.00%> (ø)
scheduler/service/service_v2.go	80.23% <100.00%> (+1.42%)	⬆️
client/daemon/daemon.go	0.00% <0.00%> (ø)
scheduler/resource/standard/host.go	88.23% <0.00%> (-1.51%)	⬇️
manager/job/sync_peers.go	0.00% <0.00%> (ø)
pkg/rpc/scheduler/client/client_v2.go	0.00% <0.00%> (ø)
scheduler/rpcserver/scheduler_server_v2.go	1.26% <0.00%> (-0.07%)	⬇️
... and 1 more

... and 3 files with indirect coverage changes

[BruceAko]

All problems resolved, please take a look, thanks. @gaius-qi

[gaius-qi]

LGTM