[sonic-mgmt][dualtor-aa] Fix fdb/test_fdb_mac_learning.py failures

[vkjammala-arista]

Description of PR

Summary: [dualtor-aa] Fix "fdb/test_fdb_mac_learning.py" failures
Fixes # https://github.com/aristanetworks/sonic-qual.msft/issues/329

Type of change

• Bug fix
• Testbed and Framework(new/improvement)
• Test case(new/improvement)

Back port request

• 202012
• 202205
• 202305
• 202311
• 202405

Approach

What is the motivation for this PR?

Test is currently failing on dualtor-aa topologies due to

1. Packet sometimes going to unselected dut (due to active-active topology) and thus lead to mac learning failure.

2. After bringing up interfaces (from shutdown state), there is time.sleep of 30 seconds which seem to be not enough for muxcable status on duthost to become consistent with mux server_status (see SERVER_STATUS shown as unknown below). We need to wait for SERVER_STATUS to match with STATUS field for mac learning to happen.

PORT       STATUS    SERVER_STATUS    HEALTH     HWSTATUS      LAST_SWITCHOVER_TIME
---------  --------  ---------------  ---------  ------------  ----------------------
Ethernet0  active    unknown          unhealthy  inconsistent

3. As test is bringing down all the interfaces (including portchannels), ERR swss#tunnel_packet_handler.py: All portchannels failed to come up within 3 minutes, exiting. is coming during the test and causing test faiure (as log_analyzer is complaining)

How did you do it?

1. Add fixture to setup topo in active-standby mode. This is needed to make sure packets goto selected dut (for mac
   learning to happen correctly).
2. Introduce logic to wait for mux status to become consistent before sending traffic (instead of relying on time.sleep delay).
3. Ignore "All port channels failed to come up ..." syslog, which seems to be expected as test is bringing down all the
   portchannels.

How did you verify/test it?

Stressed the test on Arista-7260CX3-D108C8 platform with dualtor-aa[-56] deployed and test is passing.

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

[mssonicbld]

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed
fix end of files.........................................................Passed
check yaml...........................................(no files to check)Skipped
check for added large files..............................................Passed
check python ast.........................................................Passed
flake8...................................................................Failed
- hook id: flake8
- exit code: 1

tests/fdb/test_fdb_mac_learning.py:17:1: E302 expected 2 blank lines, found 1
tests/fdb/test_fdb_mac_learning.py:29:1: E302 expected 2 blank lines, found 1
tests/fdb/test_fdb_mac_learning.py:195:43: E225 missing whitespace around operator
tests/fdb/test_fdb_mac_learning.py:235:121: E501 line too long (128 > 120 characters)

flake8...............................................(no files to check)Skipped
check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

1. Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
   the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
   docker container.
2. Ensure that the pre-commit package is installed:

sudo pip install pre-commit

3. Go to repository root folder
4. Install the pre-commit hooks:

pre-commit install

5. Use pre-commit to check staged file:

pre-commit

6. Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

[lolyu]

do we need to check if this is dualtor testbed first? What if this is a t0?

[vkjammala-arista]

Thanks @lolyu for catching this, yeah for t0 mux status is irrelevant (as muxcable is specific to dualtor), will update check_mux_status_consistency method to handle this case.