Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

broadcom SAI bug for Trident3-X3 #21247

Open
bradh352 opened this issue Dec 20, 2024 · 7 comments
Open

broadcom SAI bug for Trident3-X3 #21247

bradh352 opened this issue Dec 20, 2024 · 7 comments
Assignees
@LaveenBrcm
Labels
BRCM Triaged this issue has been triaged

Comments

@bradh352
Copy link
Contributor

bradh352 commented Dec 20, 2024 •
edited
Loading

Description

On a Dell N3248TE switch running 202411 and master (as of 20241222) I'm getting a syncd termination due to timeout of removal of an ipv6 link-local neighbor address.

2024 Dec 21 04:42:05.260261 swmgmt NOTICE syncd#syncd: [none] SAI_API_NEXT_HOP:brcm_sai_remove_next_hop:441 Removing nhid 44 if_id 100095
2024 Dec 21 04:42:05.261548 swmgmt NOTICE swss#orchagent: :- removeNeighbor: Removed next hop fe80::1c52:a783:9a48:4d0d on Vlan2
2024 Dec 21 04:42:05.262580 swmgmt NOTICE syncd#syncd: [none] SAI_API_DASH_DIRECTION_LOOKUP:_brcm_sai_l2_ecmp_nbr_mac_delete:267 FDB : MAC:B2-80-80-F7-0C-1E vfi:0x2, is_fdb_del 0, dir 3
2024 Dec 21 04:42:35.655137 swmgmt ERR syncd#syncd: :- threadFunction: time span WD exceeded 30392 ms for remove:SAI_OBJECT_TYPE_NEIGHBOR_ENTRY:{"ip":"fe80::1c52:a783:9a48:4d0d","rif":"oid:0x6000000000768","switch_id":"oid:0x21000000000000"}
2024 Dec 21 04:42:35.655194 swmgmt ERR syncd#syncd: :- logEventData: op: remove, key: SAI_OBJECT_TYPE_NEIGHBOR_ENTRY:{"ip":"fe80::1c52:a783:9a48:4d0d","rif":"oid:0x6000000000768","switch_id":"oid:0x21000000000000"}
2024 Dec 21 04:43:04.592860 swmgmt WARNING swss#supervisor-proc-exit-listener: Process 'orchagent' is stuck in namespace 'host' (1.0 minutes).
2024 Dec 21 04:43:05.322286 swmgmt ERR swss#orchagent: :- wait: SELECT operation result: TIMEOUT on getresponse
2024 Dec 21 04:43:05.322601 swmgmt ERR swss#orchagent: :- wait: failed to get response for getresponse
2024 Dec 21 04:43:05.322783 swmgmt ERR swss#orchagent: :- remove: remove status: SAI_STATUS_FAILURE
2024 Dec 21 04:43:05.322956 swmgmt ERR swss#orchagent: :- removeNeighbor: Failed to remove neighbor b2:80:80:f7:0c:1e on Vlan2, rv:-1
2024 Dec 21 04:43:05.323127 swmgmt ERR swss#orchagent: :- handleSaiRemoveStatus: Encountered failure in remove operation, exiting orchagent, SAI API: SAI_API_NEIGHBOR, status: SAI_STATUS_FAILURE
2024 Dec 21 04:43:05.323301 swmgmt NOTICE swss#orchagent: :- notifySyncd: sending syncd: SYNCD_INVOKE_DUMP

In the SAI replay log I can see the neighbor was added 5 minutes prior:

2024-12-21.04:38:20.373960|c|SAI_OBJECT_TYPE_NEIGHBOR_ENTRY:{"ip":"fe80::1c52:a783:9a48:4d0d","rif":"oid:0x6000000000768","switch_id":"oid:0x21000000000000"}|SAI_NEIGHBOR_ENTRY_ATTR_DST_MAC_ADDRESS=B2:80:80:F7:0C:1E
2024-12-21.04:38:20.376885|c|SAI_OBJECT_TYPE_NEXT_HOP:oid:0x40000000007a7|SAI_NEXT_HOP_ATTR_TYPE=SAI_NEXT_HOP_TYPE_IP|SAI_NEXT_HOP_ATTR_IP=fe80::1c52:a783:9a48:4d0d|SAI_NEXT_HOP_ATTR_ROUTER_INTERFACE_ID=oid:0x6000000000768
...
2024-12-21.04:42:05.259290|r|SAI_OBJECT_TYPE_NEXT_HOP:oid:0x40000000007a7
2024-12-21.04:42:05.261649|r|SAI_OBJECT_TYPE_NEIGHBOR_ENTRY:{"ip":"fe80::1c52:a783:9a48:4d0d","rif":"oid:0x6000000000768","switch_id":"oid:0x21000000000000"}
2024-12-21.04:43:05.321719|E|SAI_STATUS_FAILURE
2024-12-21.04:43:05.321862|a|SYNCD_INVOKE_DUMP
2024-12-21.04:44:05.381753|A|SAI_STATUS_FAILURE

Steps to reproduce the issue:

  1. Configure similar to:
{
      "PORT": {
        "Ethernet0": {
            "admin_status": "up",
            "alias": "oneGigE1/1",
            "autoneg": "on",
            "fec": "none",
            "index": "1",
            "lanes": "1",
            "mtu": "9100",
            "speed": "1000"
        }
    },
    "VLAN": {
        "Vlan2": {
            "mtu": "1500",
            "vlanid": "2"
        }
    },
    "VLAN_INTERFACE": {
	"Vlan2": {
           "ipv6_use_link_local_only": "enable",
         },
    	 "Vlan2|192.168.1.55/24": {}
    },
    "VLAN_MEMBER": {
        "Vlan2|Ethernet0": {
            "tagging_mode": "untagged"
        }
    }
}
  1. Make sure there are other ipv6 hosts on the network with link-local addresses
  2. Wait for neighbor timeout and removal and see crash

Describe the results you received:

All switch ports go down due to error.

Describe the results you expected:

switch ports stay up

Output of show version:

SONiC Software Version: SONiC.202411.2-bradh352
SONiC OS Version: 12
Distribution: Debian 12.8
Kernel: 6.1.0-22-2-amd64
Build commit: f7a56440d
Build date: Thu Dec 19 14:13:14 UTC 2024
Built by: brad@github-runner-ubuntu-2004

Platform: x86_64-dellemc_n3248te_c3338-r0
HwSKU: DellEMC-N3248TE
ASIC: broadcom
ASIC Count: 1
Serial Number: FSB8PK2
Model Number: 0HXY4C
Hardware Revision: 
Uptime: 07:04:50 up  1:34,  1 user,  load average: 1.22, 1.31, 1.14
Date: Sat 21 Dec 2024 07:04:50

Docker images:
REPOSITORY                    TAG                 IMAGE ID       SIZE
docker-orchagent              202411.2-bradh352   fdf05d43b3f5   354MB
docker-orchagent              latest              fdf05d43b3f5   354MB
docker-fpm-frr                202411.2-bradh352   d4979ae35a83   375MB
docker-fpm-frr                latest              d4979ae35a83   375MB
docker-nat                    202411.2-bradh352   062fe24aeb03   344MB
docker-nat                    latest              062fe24aeb03   344MB
docker-macsec                 latest              425734c782d0   344MB
docker-sflow                  202411.2-bradh352   5a8d07a4304a   342MB
docker-sflow                  latest              5a8d07a4304a   342MB
docker-teamd                  202411.2-bradh352   d16caeb5ea04   341MB
docker-teamd                  latest              d16caeb5ea04   341MB
docker-dhcp-relay             latest              979953fe7f4d   321MB
docker-platform-monitor       202411.2-bradh352   fe0edaeb6113   431MB
docker-platform-monitor       latest              fe0edaeb6113   431MB
docker-snmp                   202411.2-bradh352   1dc7700ca6c8   356MB
docker-snmp                   latest              1dc7700ca6c8   356MB
docker-sonic-mgmt-framework   202411.2-bradh352   2d5eadfe5a72   399MB
docker-sonic-mgmt-framework   latest              2d5eadfe5a72   399MB
docker-syncd-brcm             202411.2-bradh352   9881e0e377a1   753MB
docker-syncd-brcm             latest              9881e0e377a1   753MB
docker-sonic-bmp              202411.2-bradh352   1514a68a170a   313MB
docker-sonic-bmp              latest              1514a68a170a   313MB
docker-router-advertiser      202411.2-bradh352   6284b6c7fed5   312MB
docker-router-advertiser      latest              6284b6c7fed5   312MB
docker-mux                    202411.2-bradh352   475e47e7903a   363MB
docker-mux                    latest              475e47e7903a   363MB
docker-lldp                   202411.2-bradh352   96c59972cb35   357MB
docker-lldp                   latest              96c59972cb35   357MB
docker-sonic-gnmi             202411.2-bradh352   075f9d9da560   401MB
docker-sonic-gnmi             latest              075f9d9da560   401MB
docker-database               202411.2-bradh352   f6385c5a1998   320MB
docker-database               latest              f6385c5a1998   320MB
docker-eventd                 202411.2-bradh352   ae3cbb2b95a5   312MB
docker-eventd                 latest              ae3cbb2b95a5   312MB
docker-gbsyncd-broncos        202411.2-bradh352   4545d6f00888   351MB
docker-gbsyncd-broncos        latest              4545d6f00888   351MB
docker-gbsyncd-credo          202411.2-bradh352   02408068f2c2   324MB
docker-gbsyncd-credo          latest              02408068f2c2   324MB```

#### Output of `show techsupport`:



#### Additional information you deem important (e.g. issue happens only occasionally):
@bradh352
Copy link
Contributor Author

Ping broadcom contributors on this issue: @Praveen-Brcm @geans-pin @Gfrom2016

@tomvil
Copy link

tomvil commented Dec 23, 2024

This issue looks very similar to #20587 ?

@bradh352
Copy link
Contributor Author

This issue looks very familiar to #20587 ?

@tomvil Interesting. I have Dell S5248F switches as well and haven't observed that there, but the configs are significantly different. But yes, the issue in the logs is indeed the same.

@bradh352 bradh352 mentioned this issue Dec 23, 2024
Open
@wumiaont
Copy link
Contributor

wumiaont commented Jan 3, 2025

We here using Nokia chassis also see similar errors from the log sometimes. The err logs could cause some OC tests to fail by log analyzer.

2024 Oct 25 16:08:56.461374 ixre-egl-board40 ERR syncd0#syncd: [06:00.0] SAI_API_NEIGHBOR:brcm_sai_remove_neighbor_entry:457 Neighbor vr_id 1 NH 1 references found.
2024 Oct 25 16:08:56.461374 ixre-egl-board40 ERR syncd0#syncd: :- asic_process_event: failed to execute api: remove, key: SAI_OBJECT_TYPE_NEIGHBOR_ENTRY:{"ip":"10.0.0.1","rif":"oid:0x60000000008be","switch_id":"oid:0x21000000000000"}, status: SAI_STATUS_OBJECT_IN_USE
2024 Oct 25 16:08:56.463038 ixre-egl-board40 NOTICE syncd0#syncd: :- executeOperationsOnAsic: asic apply took 0.189187 sec
2024 Oct 25 16:08:56.463038 ixre-egl-board40 ERR syncd0#syncd: :- executeOperationsOnAsic: Error while executing asic operations, ASIC is in inconsistent state: :- asic_process_event: failed to execute api: remove, key: SAI_OBJECT_TYPE_NEIGHBOR_ENTRY:{"ip":"10.0.0.1","rif":"oid:0x60000000008be","switch_id":"oid:0x21000000000000"}, status: SAI_STATUS_OBJECT_IN_USE
2024 Oct 25 16:08:56.471668 ixre-egl-board40 ERR syncd0#syncd: :- run: Runtime error: :- asic_process_event: failed to execute api: remove, key: SAI_OBJECT_TYPE_NEIGHBOR_ENTRY:{"ip":"10.0.0.1","rif":"oid:0x60000000008be","switch_id":"oid:0x21000000000000"}, status: SAI_STATUS_OBJECT_IN_USE
2024 Oct 25 16:08:56.471668 ixre-egl-board40 NOTICE syncd0#syncd: :- sendShutdownRequest: sending switch_shutdown_request notification to OA for switch: oid:0x21000000000000
2024 Oct 25 16:08:56.471833 ixre-egl-board40 ERR swss0#orchagent: :- syncd_apply_view: Failed to notify syncd APPLY_VIEW -1
2024 Oct 25 16:09:48.952927 ixre-egl-board40 WARNING swss0#supervisor-proc-exit-listener: Process 'orchagent' is stuck in namespace 'asic0' (1.0 minutes).
2024 Oct 25 16:09:51.578735 ixre-egl-board40 INFO systemd[1]: run-docker-runtime\x2drunc-moby-f6fd37976410169649ab03e4e2fb7e05c5f0539802c39ab8eb9d3bfa309f2413-runc.4InEUH.mount: Deactivated successfully.
2024 Oct 25 16:09:56.531496 ixre-egl-board40 ERR swss0#orchagent: :- wait: SELECT operation result: TIMEOUT on notify
2024 Oct 25 16:09:56.531623 ixre-egl-board40 ERR swss0#orchagent: :- wait: failed to get response for notify
2024 Oct 25 16:09:56.531623 ixre-egl-board40 ERR swss0#orchagent: :- handleSaiFailure: Failed to take sai failure dump -1

@bradh352 bradh352 mentioned this issue Jan 8, 2025
Open
@zjswhhh zjswhhh added Triaged this issue has been triaged BRCM labels Jan 15, 2025
@adyeung
Copy link
Collaborator

adyeung commented Jan 15, 2025

@LaveenBrcm is taking a look

@adyeung adyeung assigned LaveenBrcm and unassigned adyeung Jan 15, 2025
@audmas
Copy link

audmas commented Feb 7, 2025

hi @LaveenBrcm maybe you found some time to look at this issue?

@fountzou
Copy link
Contributor

fountzou commented Feb 13, 2025 •
edited
Loading

Experiencing the same issue on the Trident4 (BCM56780) chipset.

2025 Feb  8 13:40:59.525548 ixr-7220-d4-nysa NOTICE swss#orchagent: :- removeNeighbor: Removed next hop 192.168.0.252 on Vlan1000
2025 Feb  8 13:40:59.525931 ixr-7220-d4-nysa NOTICE syncd#syncd: [none] (null):_brcm_sai_l2_ecmp_nbr_mac_delete:316 FDB : MAC:72-06-00-01-00-01 vfi:0x3e8, is_fdb_del 0, dir 3
2025 Feb  8 13:41:30.036214 ixr-7220-d4-nysa NOTICE syncd#syncd: :- threadFunction: time span 30510 ms for 'remove:SAI_OBJECT_TYPE_NEIGHBOR_ENTRY:{"ip":"192.168.0.252","rif":"oid:0x60000000005d6","switch_id":"oid:0x21000000000000"}'
2025 Feb  8 13:41:30.036214 ixr-7220-d4-nysa ERR syncd#syncd: :- threadFunction: time span WD exceeded 30510 ms for remove:SAI_OBJECT_TYPE_NEIGHBOR_ENTRY:{"ip":"192.168.0.252","rif":"oid:0x60000000005d6","switch_id":"oid:0x21000000000000"}
2025 Feb  8 13:41:30.036214 ixr-7220-d4-nysa ERR syncd#syncd: :- logEventData: op: remove, key: SAI_OBJECT_TYPE_NEIGHBOR_ENTRY:{"ip":"192.168.0.252","rif":"oid:0x60000000005d6","switch_id":"oid:0x21000000000000"}
2025 Feb  8 13:41:59.549458 ixr-7220-d4-nysa ERR swss#orchagent: :- wait: SELECT operation result: TIMEOUT on getresponse
2025 Feb  8 13:41:59.549518 ixr-7220-d4-nysa ERR swss#orchagent: :- wait: failed to get response for getresponse
2025 Feb  8 13:41:59.549518 ixr-7220-d4-nysa ERR swss#orchagent: :- remove: remove status: SAI_STATUS_FAILURE
2025 Feb  8 13:41:59.549538 ixr-7220-d4-nysa ERR swss#orchagent: :- removeNeighbor: Failed to remove neighbor 72:06:00:01:00:01 on Vlan1000, rv:-1
2025 Feb  8 13:41:59.549556 ixr-7220-d4-nysa ERR swss#orchagent: :- handleSaiRemoveStatus: Encountered failure in remove operation, exiting orchagent, SAI API: SAI_API_NEIGHBOR, status: SAI_STATUS_FAILURE
2025 Feb  8 13:41:59.549556 ixr-7220-d4-nysa NOTICE swss#orchagent: :- notifySyncd: sending syncd: SYNCD_INVOKE_DUMP
2025 Feb  8 13:41:59.600389 ixr-7220-d4-nysa WARNING swss#supervisor-proc-exit-listener: Process 'orchagent' is stuck in namespace 'host' (1.0 minutes).

From SAI perspective

2025-02-08.13:38:01.494635|c|SAI_OBJECT_TYPE_NEIGHBOR_ENTRY:{"ip":"192.168.0.252","rif":"oid:0x60000000005d6","switch_id":"oid:0x21000000000000"}|SAI_NEIGHBOR_ENTRY_ATTR_DST_MAC_ADDRESS=72:06:00:01:00:01
2025-02-08.13:38:01.496401|c|SAI_OBJECT_TYPE_NEXT_HOP:oid:0x40000000006a1|SAI_NEXT_HOP_ATTR_TYPE=SAI_NEXT_HOP_TYPE_IP|SAI_NEXT_HOP_ATTR_IP=192.168.0.252|SAI_NEXT_HOP_ATTR_ROUTER_INTERFACE_ID=oid:0x60000000005d6
2025-02-08.13:39:19.421173|c|SAI_OBJECT_TYPE_ACL_ENTRY:oid:0x80000000006be|SAI_ACL_ENTRY_ATTR_TABLE_ID=oid:0x7000000000602|SAI_ACL_ENTRY_ATTR_PRIORITY=9998|SAI_ACL_ENTRY_ATTR_ADMIN_STATE=true|SAI_ACL_ENTRY_ATTR_ACTION_COUNTER=oid:0x90000000006bd|SAI_ACL_ENTRY_ATTR_FIELD_DST_IP=192.168.0.252&mask:255.255.255.255|SAI_ACL_ENTRY_ATTR_FIELD_ETHER_TYPE=2048&mask:0xffff|SAI_ACL_ENTRY_ATTR_ACTION_PACKET_ACTION=SAI_PACKET_ACTION_FORWARD
2025-02-08.13:40:59.524672|r|SAI_OBJECT_TYPE_NEXT_HOP:oid:0x40000000006a1
2025-02-08.13:40:59.525460|r|SAI_OBJECT_TYPE_NEIGHBOR_ENTRY:{"ip":"192.168.0.252","rif":"oid:0x60000000005d6","switch_id":"oid:0x21000000000000"}
2025-02-08.13:41:59.549245|E|SAI_STATUS_FAILURE     
2025-02-08.13:41:59.549419|a|SYNCD_INVOKE_DUMP

Has any CSP raised to track this one?

# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees

@LaveenBrcm LaveenBrcm

Labels
BRCM Triaged this issue has been triaged
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@wumiaont @tomvil @bradh352 @zjswhhh @adyeung @audmas @LaveenBrcm @fountzou