test(integration/manual): support GKE Container-Optimized OS

[c3y1huang]

Which issue(s) this PR fixes:

Issue longhorn/longhorn#6165

What this PR does / why we need it:

• Integration: The replica_auto_balance_zone test cases failed when Kubernetes is running on GKE with COS_CONTAINERD. This is because the node zone label is periodically updated with the actual GCP zone. To address this, implement a workaround by refreshing the zone label with each retry iteration to maintain the expected zone label.
• Manual: test dependencies configure/setup.

Alternative

Skip zone-related tests when the cluster is running on COS.

Special notes for your reviewer:

None

Additional documentation or context

None

[c3y1huang]

Test result for test_replica_auto_balance:

platform linux -- Python 3.11.8, pytest-6.2.4, py-1.11.0, pluggy-0.13.1 -- /usr/bin/python3.11
cachedir: .pytest_cache
rootdir: /integration, configfile: pytest.ini
plugins: order-1.0.1, repeat-0.9.1
collecting ... collected 1125 items / 1086 deselected / 39 selected

test_scheduling.py::test_replica_auto_balance_node_least_effort[1-3] PASSED [  2%]
test_scheduling.py::test_replica_auto_balance_node_least_effort[2-3] PASSED [  5%]
test_scheduling.py::test_replica_auto_balance_node_least_effort[3-3] PASSED [  7%]
test_scheduling.py::test_replica_auto_balance_node_best_effort[1-3] PASSED [ 10%]
test_scheduling.py::test_replica_auto_balance_node_best_effort[2-3] PASSED [ 12%]
test_scheduling.py::test_replica_auto_balance_node_best_effort[3-3] PASSED [ 15%]
test_scheduling.py::test_replica_auto_balance_with_data_locality[1-3] SKIPPED [ 17%]
test_scheduling.py::test_replica_auto_balance_with_data_locality[2-3] SKIPPED [ 20%]
test_scheduling.py::test_replica_auto_balance_with_data_locality[3-3] SKIPPED [ 23%]
test_scheduling.py::test_replica_auto_balance_disabled_volume_spec_enabled[1-3] PASSED [ 25%]
test_scheduling.py::test_replica_auto_balance_disabled_volume_spec_enabled[2-3] PASSED [ 28%]
test_scheduling.py::test_replica_auto_balance_disabled_volume_spec_enabled[3-3] PASSED [ 30%]
test_zone.py::test_replica_auto_balance_zone_least_effort[1-3] PASSED    [ 33%]
test_zone.py::test_replica_auto_balance_zone_least_effort[2-3] PASSED    [ 35%]
test_zone.py::test_replica_auto_balance_zone_least_effort[3-3] PASSED    [ 38%]
test_zone.py::test_replica_auto_balance_zone_best_effort[1-3] PASSED     [ 41%]
test_zone.py::test_replica_auto_balance_zone_best_effort[2-3] PASSED     [ 43%]
test_zone.py::test_replica_auto_balance_zone_best_effort[3-3] PASSED     [ 46%]
test_zone.py::test_replica_auto_balance_when_disabled_disk_scheduling_in_zone[1-3] PASSED [ 48%]
test_zone.py::test_replica_auto_balance_when_disabled_disk_scheduling_in_zone[2-3] PASSED [ 51%]
test_zone.py::test_replica_auto_balance_when_disabled_disk_scheduling_in_zone[3-3] PASSED [ 53%]
test_zone.py::test_replica_auto_balance_when_no_storage_available_in_zone[1-3] PASSED [ 56%]
test_zone.py::test_replica_auto_balance_when_no_storage_available_in_zone[2-3] PASSED [ 58%]
test_zone.py::test_replica_auto_balance_when_no_storage_available_in_zone[3-3] PASSED [ 61%]
test_zone.py::test_replica_auto_balance_when_replica_on_unschedulable_node[1-3] PASSED [ 64%]
test_zone.py::test_replica_auto_balance_when_replica_on_unschedulable_node[2-3] PASSED [ 66%]
test_zone.py::test_replica_auto_balance_when_replica_on_unschedulable_node[3-3] PASSED [ 69%]
test_zone.py::test_replica_auto_balance_zone_best_effort_with_data_locality[1-3] PASSED [ 71%]
test_zone.py::test_replica_auto_balance_zone_best_effort_with_data_locality[2-3] PASSED [ 74%]
test_zone.py::test_replica_auto_balance_zone_best_effort_with_data_locality[3-3] PASSED [ 76%]
test_zone.py::test_replica_auto_balance_node_duplicates_in_multiple_zones[1-3] PASSED [ 79%]
test_zone.py::test_replica_auto_balance_node_duplicates_in_multiple_zones[2-3] PASSED [ 82%]
test_zone.py::test_replica_auto_balance_node_duplicates_in_multiple_zones[3-3] PASSED [ 84%]
test_zone.py::test_replica_auto_balance_zone_best_effort_with_uneven_node_in_zones[1-3] SKIPPED [ 87%]
test_zone.py::test_replica_auto_balance_zone_best_effort_with_uneven_node_in_zones[2-3] SKIPPED [ 89%]
test_zone.py::test_replica_auto_balance_zone_best_effort_with_uneven_node_in_zones[3-3] SKIPPED [ 92%]
test_zone.py::test_replica_auto_balance_should_respect_node_selector[1-3] PASSED [ 94%]
test_zone.py::test_replica_auto_balance_should_respect_node_selector[2-3] PASSED [ 97%]
test_zone.py::test_replica_auto_balance_should_respect_node_selector[3-3] PASSED [100%]

=== 33 passed, 6 skipped, 1086 deselected, 12 warnings in 4702.26s (1:18:22) ===

[khushboo-rancher]

@yangchiu Could you please help with the review?

[yangchiu]

LGTM