[Bug][202405]: test/test_acl.py Fail with AssertionError: Rule counters should be ready!

[arista-nwolfe]

Issue Description

test/test_acl.py fails on all tests with:

    def set_up_acl_rules_single_dut(self, acl_table,
                                    conn_graph_facts, dut_to_analyzer_map, duthost, # noqa F811
                                    ip_version, localhost,
                                    populate_vlan_arp_entries, tbinfo):
        logger.info("{}: ACL rule application started".format(duthost.hostname))
        if duthost.is_supervisor_node():
            return
        loganalyzer = LogAnalyzer(ansible_host=duthost, marker_prefix="acl_rules")
        loganalyzer.load_common_config()
        dut_to_analyzer_map[duthost] = loganalyzer
        try:
            loganalyzer.expect_regex = [LOG_EXPECT_ACL_RULE_CREATE_RE]
            # Ignore any other errors to reduce noise
            loganalyzer.ignore_regex = [r".*"]
            with loganalyzer:
                self.setup_rules(duthost, acl_table, ip_version)
                # Give the dut some time for the ACL rules to be applied and LOG message generated
                wait_until(300, 20, 0, check_msg_in_syslog,
                           duthost, LOG_EXPECT_ACL_RULE_CREATE_RE)

            self.post_setup_hook(duthost, localhost, populate_vlan_arp_entries, tbinfo, conn_graph_facts)

>           assert self.check_rule_counters(duthost), "Rule counters should be ready!"
E           AssertionError: Rule counters should be ready!

This appears to be due to the counters showing N/A for the ACL rules:

root@nfc405-7:~# aclshow -a
RULE NAME     TABLE NAME                PRIO  PACKETS COUNT    BYTES COUNT
------------  ----------------------  ------  ---------------  -------------
RULE_1        DATA_INGRESS_IPV4_TEST    9999  N/A              N/A
RULE_2        DATA_INGRESS_IPV4_TEST    9998  N/A              N/A
RULE_3        DATA_INGRESS_IPV4_TEST    9997  N/A              N/A
RULE_4        DATA_INGRESS_IPV4_TEST    9996  N/A              N/A
RULE_5        DATA_INGRESS_IPV4_TEST    9995  N/A              N/A
RULE_6        DATA_INGRESS_IPV4_TEST    9994  N/A              N/A
RULE_7        DATA_INGRESS_IPV4_TEST    9993  N/A              N/A
RULE_8        DATA_INGRESS_IPV4_TEST    9992  N/A              N/A
RULE_9        DATA_INGRESS_IPV4_TEST    9991  N/A              N/A
RULE_10       DATA_INGRESS_IPV4_TEST    9990  N/A              N/A
RULE_11       DATA_INGRESS_IPV4_TEST    9989  N/A              N/A
RULE_12       DATA_INGRESS_IPV4_TEST    9988  N/A              N/A
RULE_13       DATA_INGRESS_IPV4_TEST    9987  N/A              N/A
RULE_14       DATA_INGRESS_IPV4_TEST    9986  N/A              N/A
RULE_15       DATA_INGRESS_IPV4_TEST    9985  N/A              N/A
RULE_16       DATA_INGRESS_IPV4_TEST    9984  N/A              N/A
RULE_17       DATA_INGRESS_IPV4_TEST    9983  N/A              N/A
RULE_18       DATA_INGRESS_IPV4_TEST    9982  N/A              N/A
RULE_19       DATA_INGRESS_IPV4_TEST    9981  N/A              N/A
RULE_20       DATA_INGRESS_IPV4_TEST    9980  N/A              N/A
RULE_21       DATA_INGRESS_IPV4_TEST    9979  N/A              N/A
RULE_22       DATA_INGRESS_IPV4_TEST    9978  N/A              N/A
RULE_23       DATA_INGRESS_IPV4_TEST    9977  N/A              N/A
RULE_24       DATA_INGRESS_IPV4_TEST    9976  N/A              N/A
RULE_25       DATA_INGRESS_IPV4_TEST    9975  N/A              N/A
RULE_26       DATA_INGRESS_IPV4_TEST    9974  N/A              N/A
RULE_27       DATA_INGRESS_IPV4_TEST    9973  N/A              N/A
RULE_28       DATA_INGRESS_IPV4_TEST    9972  N/A              N/A
RULE_29       DATA_INGRESS_IPV4_TEST    9971  N/A              N/A
RULE_30       DATA_INGRESS_IPV4_TEST    9970  N/A              N/A
RULE_31       DATA_INGRESS_IPV4_TEST    9969  N/A              N/A
RULE_32       DATA_INGRESS_IPV4_TEST    9968  N/A              N/A
RULE_33       DATA_INGRESS_IPV4_TEST    9967  N/A              N/A
DEFAULT_RULE  DATA_INGRESS_IPV4_TEST       1  N/A              N/A

We confirmed the issue appears to have been introduced by #20555
After reverting this change we no longer see test/test_acl.py failing:

acl/test_acl.py::TestBasicAcl::test_ingress_unmatched_blocked[ipv4-ingress-downlink->uplink-default-no_vlan] PASSED                        [  0%]
acl/test_acl.py::TestBasicAcl::test_egress_unmatched_forwarded[ipv4-ingress-downlink->uplink-default-no_vlan] SKIPPED (Only run for eg...) [  1%]
acl/test_acl.py::TestBasicAcl::test_source_ip_match_forwarded[ipv4-ingress-downlink->uplink-default-no_vlan] PASSED                        [  1%]
acl/test_acl.py::TestBasicAcl::test_rules_priority_forwarded[ipv4-ingress-downlink->uplink-default-no_vlan] PASSED                         [  2%]
acl/test_acl.py::TestBasicAcl::test_rules_priority_dropped[ipv4-ingress-downlink->uplink-default-no_vlan] PASSED                           [  2%]
acl/test_acl.py::TestBasicAcl::test_dest_ip_match_forwarded[ipv4-ingress-downlink->uplink-default-no_vlan] PASSED                          [  3%]
...
acl/test_acl.py::TestBasicAcl::test_tcp_flags_match_dropped[ipv6-ingress-uplink->downlink-default-no_vlan] PASSED                          [ 99%]
acl/test_acl.py::TestBasicAcl::test_icmp_match_forwarded[ipv6-ingress-uplink->downlink-default-no_vlan] PASSED                             [100%]

Results you see

test/test_acl.py is seeing aclshow -a display N/A for the rule counters

Results you expected to see

When test/test_acl.py creates ACL rules aclshow -a show display an integer for the rule counter

Is it platform specific

generic

Relevant log output

No response

Output of show version

No response

Attach files (if any)

No response

[Javier-Tan]

Investigated and found the same thing yesterday

For more reference, this is exactly where aclshow -a fails to pull the counters and delivers N/A

admin@sonic:~$ sonic-db-cli -n asic0 COUNTERS_DB HGETALL "ACL_COUNTER_RULE_MAP"
{'DATA_INGRESS_IPV4_TEST:DEFAULT_RULE': 'oid:0x9000000000947', 'DATA_INGRESS_IPV4_TEST:RULE_1': 'oid:0x9000000000949', 'DATA_INGRESS_IPV4_TEST:RULE_10': 'oid:0x900000000094b', 'DATA_INGRESS_IPV4_TEST:RULE_11': 'oid:0x900000000094e', 'DATA_INGRESS_IPV4_TEST:RULE_12': 'oid:0x9000000000951', 'DATA_INGRESS_IPV4_TEST:RULE_13': 'oid:0x9000000000953', 'DATA_INGRESS_IPV4_TEST:RULE_14': 'oid:0x9000000000955', ...

admin@sonic:~$ sonic-db-cli -n asic0 COUNTERS_DB keys COUNTERS:oid:90*
(nothing returned -> `aclshow -a` returns N/A for each counter it can't find)

[rlhui]

@stepanblyschak could you please help check this? Thanks.

[bingwang-ms]

Hi @Javier-Tan Is there any other counter issue with this change #20555?

[stepanblyschak]

@bingwang-ms This change should come together with #20554 and sonic-net/sonic-swss#3326