Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[subnet_decap] Unbreak orchagent on ASICs that don't support IPINIP tunnels #3377

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

wdoekes
Copy link

@wdoekes wdoekes commented Nov 19, 2024

The Tomahawk 3 chipset on an Accton AS9716-32D does not do SAI_TUNNEL_TERM_TABLE_ENTRY_TYPE_P2MP tunnels and returns SAI_STATUS_NOT_SUPPORTED.

Since #3117, such a tunnel is created unconditionally whenever an IP address is added to Loopback0. The SAI_STATUS_NOT_SUPPORTED is handled as an unrecoverable error. That takes the orchagent down and syncd along with it.

root@spine2:0:~# dmidecode 2>/dev/null | grep -A1 'Manufacturer:' | head -n2
Manufacturer: Accton
Product Name: AS9716-32D

root@spine2:0:~# printf '%s\n' 'bsv' 'show unit' 'ver' |
    xargs -d\\n -n1 bcmcmd -t 1 | grep '^[A-Z]'
BRCM SAI ver: [11.2.13.1], OCP SAI ver: [1.14.0], SDK ver: [sdk-6.5.30-SP4]
Unit 0 chip BCM56980_B0 (current)
Broadcom Command Monitor: Copyright (c) 1998-2024 Broadcom
Release: sdk-6.5.30-SP4 built 20241016 (Wed Oct 16 13:33:02 2024)
From root@7ec56acb7b44:/__w/1/s/output/x86-xgsall-deb/xgs-sdk-src/hsdk-all-6.5.30
Platform: X86
OS: Unix (Posix)

Logs output:

NOTICE swss#orchagent: :- addDecapTunnel: Create overlay loopback router interface oid:60000000005b2
NOTICE swss#orchagent: :- doDecapTunnelTask: Tunnel IPINIP_TUNNEL added to ASIC_DB.
ERR syncd#syncd: :- sendApiResponse: api SAI_COMMON_API_CREATE failed in syncd mode: SAI_STATUS_NOT_SUPPORTED
ERR syncd#syncd: :- processQuadEvent: attr: SAI_TUNNEL_TERM_TABLE_ENTRY_ATTR_VR_ID: oid:0x3000000000021a
...
ERR swss#orchagent: :- create: create status: SAI_STATUS_NOT_SUPPORTED
ERR swss#orchagent: :- addDecapTunnelTermEntry: Failed to create tunnel decap term entry 10.1.0.1/32.
ERR swss#orchagent: :- handleSaiCreateStatus: Encountered failure in create operation, exiting orchagent, SAI API: SAI_API_TUNNEL, status: SAI_STATUS_NOT_SUPPORTED
NOTICE swss#orchagent: :- notifySyncd: sending syncd: SYNCD_INVOKE_DUMP
NOTICE syncd#syncd: :- processNotifySyncd: Invoking SAI failure dump

This changeset adds a check for SAI_STATUS_NOT_SUPPORTED, turning the error into a warning and going back to behaviour before subnet_decap was added:

ERR swss#orchagent: :- create: create status: SAI_STATUS_NOT_SUPPORTED
WARNING swss#orchagent: :- addDecapTunnelTermEntry: Creating SAI_API_TUNNEL returned SAI_STATUS_NOT_SUPPORTED for tunnel IPINIP_TUNNEL IP 10.1.0.1/32.
ERR swss#orchagent: :- doDecapTunnelTermTask: IPINIP_TUNNEL:10.1.0.1: failed to add tunnel decap term to ASIC_DB.

Resolves: sonic-net/sonic-buildimage#20837

What I did

Replace error and failure handling with a warning and false return.

Why I did it

Because the Tomahawk 3 is unusable without this fix.

How I verified it

Adding the code resolved the issue. All IP addresses (local and remote) still responded to ping.

Details if related

Also needs cherry-pick to 202405.

…unnels

The Tomahawk 3 chipset on an Accton AS9716-32D does not do
SAI_TUNNEL_TERM_TABLE_ENTRY_TYPE_P2MP tunnels and returns
SAI_STATUS_NOT_SUPPORTED.

Since sonic-net#3117, such a tunnel is created unconditionally whenever an IP
address is added to Loopback0. The SAI_STATUS_NOT_SUPPORTED is handled
as an unrecoverable error. That takes the orchagent down and syncd along
with it.

    root@spine2:0:~# dmidecode 2>/dev/null | grep -A1 'Manufacturer:' | head -n2
    Manufacturer: Accton
    Product Name: AS9716-32D

    root@spine2:0:~# printf '%s\n' 'bsv' 'show unit' 'ver' |
        xargs -d\\n -n1 bcmcmd -t 1 | grep '^[A-Z]'
    BRCM SAI ver: [11.2.13.1], OCP SAI ver: [1.14.0], SDK ver: [sdk-6.5.30-SP4]
    Unit 0 chip BCM56980_B0 (current)
    Broadcom Command Monitor: Copyright (c) 1998-2024 Broadcom
    Release: sdk-6.5.30-SP4 built 20241016 (Wed Oct 16 13:33:02 2024)
    From root@7ec56acb7b44:/__w/1/s/output/x86-xgsall-deb/xgs-sdk-src/hsdk-all-6.5.30
    Platform: X86
    OS: Unix (Posix)

Logs output:

    NOTICE swss#orchagent: :- addDecapTunnel: Create overlay loopback router
      interface oid:60000000005b2
    NOTICE swss#orchagent: :- doDecapTunnelTask: Tunnel IPINIP_TUNNEL added
      to ASIC_DB.
    ERR syncd#syncd: :- sendApiResponse: api SAI_COMMON_API_CREATE failed in
      syncd mode: SAI_STATUS_NOT_SUPPORTED
    ERR syncd#syncd: :- processQuadEvent: attr:
      SAI_TUNNEL_TERM_TABLE_ENTRY_ATTR_VR_ID: oid:0x3000000000021a
    ...
    ERR swss#orchagent: :- create: create status: SAI_STATUS_NOT_SUPPORTED
    ERR swss#orchagent: :- addDecapTunnelTermEntry: Failed to create tunnel
      decap term entry 10.1.0.1/32.
    ERR swss#orchagent: :- handleSaiCreateStatus: Encountered failure in
      create operation, exiting orchagent, SAI API: SAI_API_TUNNEL, status:
      SAI_STATUS_NOT_SUPPORTED
    NOTICE swss#orchagent: :- notifySyncd: sending syncd: SYNCD_INVOKE_DUMP
    NOTICE syncd#syncd: :- processNotifySyncd: Invoking SAI failure dump

This changeset adds a check for SAI_STATUS_NOT_SUPPORTED, turning the
error into a warning and going back to behaviour before subnet_decap was
added:

    ERR swss#orchagent: :- create: create status: SAI_STATUS_NOT_SUPPORTED
    WARNING swss#orchagent: :- addDecapTunnelTermEntry: Creating SAI_API_TUNNEL
      returned SAI_STATUS_NOT_SUPPORTED for tunnel IPINIP_TUNNEL IP 10.1.0.1/32.
    ERR swss#orchagent: :- doDecapTunnelTermTask: IPINIP_TUNNEL:10.1.0.1: failed
      to add tunnel decap term to ASIC_DB.

Resolves: sonic-net/sonic-buildimage#20837
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[orchagent/syncd] Adding Loopback0 IP causes orchagent/syncd crash/dump on Accton-AS9716-32D Tomahawk 3
1 participant