Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

NAT DVS tests fail consistently #1199

Closed
daall opened this issue Feb 20, 2020 · 0 comments · Fixed by #1205
Closed

NAT DVS tests fail consistently #1199

daall opened this issue Feb 20, 2020 · 0 comments · Fixed by #1205
Assignees

Comments

@daall
Copy link
Contributor

daall commented Feb 20, 2020

It looks like some dependencies might be missing from the docker virtual switch:

Feb 20 22:16:00.273533 f2be27c28a0c INFO #supervisord: natmgrd sh: 1: /sbin/iptables: not found
Feb 20 22:18:56.293810 f2be27c28a0c INFO #supervisord: natmgrd sh: 1: conntrack: not found

Even after installing these dependencies, the tests are still failing and I'm seeing this error throughout:

Feb 20 22:28:02.795680 f2be27c28a0c NOTICE #natmgrd: :- sigterm_handler: Got SIGTERM
Feb 20 22:28:02.800283 f2be27c28a0c NOTICE #natmgrd: :- sigterm_handler: Sending notification to orchagent to cleanup NAT entries in REDIS/ASIC
Feb 20 22:25:23.602449 f2be27c28a0c INFO #supervisord: intfmgrd RTNETLINK answers: File exists
Feb 20 22:28:02.800302 f2be27c28a0c INFO #supervisord: natmgrd conntrack v1.4.4 (conntrack-tools): connection tracking table has been emptied.
Feb 20 22:28:02.800707 f2be27c28a0c NOTICE #orchagent: :- doTask: Received RedisDB and ASIC  cleanup notification on NAT docker stop
Feb 20 22:28:10.472606 f2be27c28a0c INFO #supervisord 2020-02-20 22:28:02,793 WARN received SIGTERM indicating exit request
Feb 20 22:28:10.472858 f2be27c28a0c INFO #supervisord 2020-02-20 22:28:02,794 INFO waiting for natmgrd, neighsyncd, vrfmgrd, natsyncd, zebra, teamsyncd, rsyslogd, teammgrd, portsyncd, orchagent, vxlanmgrd, intfmgrd, syncd, buffermgrd, portmgrd, staticd, vlanmgrd, redis-server, fpmsyncd, sflowmgrd, nbrmgrd to die
Feb 20 22:28:10.472866 f2be27c28a0c INFO #supervisord 2020-02-20 22:28:02,795 INFO stopped: natsyncd (terminated by SIGTERM)
Feb 20 22:28:10.473872 f2be27c28a0c INFO #supervisord 2020-02-20 22:28:05,807 INFO waiting for natmgrd, neighsyncd, vrfmgrd, zebra, teamsyncd, rsyslogd, teammgrd, portsyncd, orchagent, vxlanmgrd, intfmgrd, syncd, buffermgrd, portmgrd, staticd, vlanmgrd, redis-server, fpmsyncd, sflowmgrd, nbrmgrd to die
Feb 20 22:28:10.473895 f2be27c28a0c INFO #supervisord 2020-02-20 22:28:08,813 INFO waiting for natmgrd, neighsyncd, vrfmgrd, zebra, teamsyncd, rsyslogd, teammgrd, portsyncd, orchagent, vxlanmgrd, intfmgrd, syncd, buffermgrd, portmgrd, staticd, vlanmgrd, redis-server, fpmsyncd, sflowmgrd, nbrmgrd to die
Feb 20 22:28:17.420409 f2be27c28a0c INFO #liblogging-stdlog:  [origin software="rsyslogd" swVersion="8.24.0" x-pid="141" x-info="http://www.rsyslog.com"] start
Feb 20 22:28:17.420616 f2be27c28a0c ERR #liblogging-stdlog: Could not open output pipe '/dev/xconsole':: No such file or directory [v8.24.0 try http://www.rsyslog.com/e/2039 ]
Feb 20 22:28:17.420885 f2be27c28a0c INFO #supervisord: rsyslogd rsyslogd: Could not open output pipe '/dev/xconsole':: No such file or directory [v8.24.0 try http://www.rsyslog.com/e/2039 ]
Feb 20 22:28:17.420995 f2be27c28a0c WARNING #liblogging-stdlog: action 'action 11' suspended, next retry is Thu Feb 20 22:28:47 2020 [v8.24.0 try http://www.rsyslog.com/e/2007 ]
Feb 20 22:28:17.420456 f2be27c28a0c INFO #supervisord 2020-02-20 22:28:11,821 INFO waiting for natmgrd, neighsyncd, vrfmgrd, zebra, teamsyncd, rsyslogd, teammgrd, portsyncd, orchagent, vxlanmgrd, intfmgrd, syncd, buffermgrd, portmgrd, staticd, vlanmgrd, redis-server, fpmsyncd, sflowmgrd, nbrmgrd to die
Feb 20 22:28:17.420462 f2be27c28a0c INFO #supervisord 2020-02-20 22:28:12,823 WARN killing 'natmgrd' (676) with SIGKILL
Feb 20 22:28:17.420577 f2be27c28a0c INFO #supervisord 2020-02-20 22:28:12,825 INFO stopped: natmgrd (terminated by SIGKILL)

Tests were based off latest master build: https://sonic-jenkins.westus2.cloudapp.azure.com/job/vs/job/buildimage-vs-all/218/

@daall daall linked a pull request Feb 25, 2020 that will close this issue
EdenGri pushed a commit to EdenGri/sonic-swss that referenced this issue Feb 28, 2022
- What I did
To remove the list of hardcoded order-dependent lists of services to stop/restart/reset-failed.

- How I did it
Used sonic.target to stop/restart/reset-failed.

- How to verify it
Execute config reload and observe the services do restart.

Signed-off-by: Stepan Blyshchak <stepanb@nvidia.com>
oleksandrivantsiv pushed a commit to oleksandrivantsiv/sonic-swss that referenced this issue Mar 1, 2023
What I did:

Moved the SAI header to v1.8.1.
   7cd3a7ed84db3fc9cec13496a5339b6fe1888bb7 (HEAD, tag: v1.8.1, origin/v1.8) Update SAI version to V1.8.1 (sonic-net#1218)
   5913e4cdd0c9c7ae859baa2e18086327b39a94da Fix error when compiling Broadcom SAI with v1.8.0 (sonic-net#1216)
   5a98bc3c7e86c01f3cf702054f9af7c7c5ca6daf (HEAD, tag: v1.8.0, origin/master, origin/HEAD, master) Update version to 1.8.0 (sonic-net#1207)
   b3244ceceb45184ffe37da55bb9a98ef126050ce saineighbor.h: Updated SAI_NEIGHBOR_ENTRY_ATTR_ENCAP_INDEX and deprecated SAI_NEIGHBOR_ENTRY_ATTR_ENCAP_IMPOSE_INDEX (sonic-net#1202)
   8731ca6e09c7ba99b0b009e5821d80598e216756 Add source/dest/double NAPT entry available attributes (sonic-net#1194)
   f053d899feb9517f2db43ee462589a30572b5ed1 Add switch attributes for hash offset configuration. (sonic-net#1195)
   13e5cd6940f9a0da1878d00f08e5941e09f16e7f PRBS RX State Data Type (sonic-net#1179)
   9755845a06525a3c17f03e7b936a70783e8ef068 Packet header based VRF classification (sonic-net#1185)
   2369ecb59fff1a5cae948d41eea06bf8b71330b2 SAI versioning (sonic-net#1183)
   744279839c176e68b19734657975e3f5ec6f1a32 Replaced SAI_SWITCH_ATTR_MACSEC_OBJECT_ID with SAI_SWITCH_ATTR_MACSEC_OBJECT_LIST (sonic-net#1199)
   584c724864fe565357e82d097ddcc7363bddefac [CI] Set up CI&PR with Azure Pipelines (sonic-net#1200)
   08192237963174cc60edae9b4812a39c43b291fd Add attribute to query available packet DMA pool size (sonic-net#1198)
   f092ef1e3ce695fc3f9552721025695312b961a2 Add IPv6 flow label hash attribute. (sonic-net#1192)
   cbc9562bb7a8f2c3a79702b99be55f3b3afa6957 Override VRF (sonic-net#1186)
   1eb35afdb2146baf40e6c2b8f2f8bfe99075eaee Add SAI_SWITCH_ATTR_SWITCH_HARDWARE_INFO format for GB MDIO sysfs access   (sonic-net#1171)
   b2d4c9a57c7f00b2632c35ca5eb3dd6480f7916a Switch scoped tunnel attributes (sonic-net#1173)
   96adc95bf8316e1905143d9ecd21f32a43e80d7f Enhancements for MPLS support (sonic-net#1181)
   3dcf1f2028da4060b345ad78e8a0c87d225bf5d0 Support for ACL extensions in metadata (sonic-net#1178)
   24076be95b871e8f82ecaeb908cad951dc68896c [meta] Add support for allow empty list tag (sonic-net#1190)
   a2b3344cdde0bf5a4f8e98e1c676a658c0c615b0 spell check fixes (sonic-net#1189)
   bf4271bab6e8884bd96050bcba44e3134adaaec3 Do not call sai_metadata_sai get APIs before checking if they are allocated (sonic-net#1182)
   5d5784dc3dbfc713c92ae7d2c472680b837bb002 [macsec]: Separate XPN configuration attribute from read-only attribute (sonic-net#1169)
   6d5a9bf5ad17cb82621cabbe2449524320930606 [macsec]: add SAI_MACSEC_ATTR_SUPPORTED_CIPHER_SUITE_LIST (sonic-net#1172)
   e72c8f3a0cc543cb228554be82c97a63db917740 [meta] Print each tool version in Makefile (sonic-net#1177)
   8f19677da88c7494d563ef7c5acb0529ecbd0b6e [meta] Add check for START, END and RANBE_BASE enums (sonic-net#1175)
   24ad7906f145930b2e25682b6248909289d39e72 [meta] Create sai_switch_pointers_t struct (sonic-net#1174)
   4f5f84df3fcd0e146707df41d3e2837c48f7c760 Tunnel loopback packet action as resource (sonic-net#1163)
   8a0e82c57aa0e22e696158735516904e7dc14052 [meta] Add create only oid attribute check on switch object (sonic-net#1170)
   14cf50772e478551920963ecf11f4fd019a0c106 Remove obsolete stub folder (sonic-net#1168)
   f14f406340e4f5f1b1d674f6fdd5fd861a54c877 [meta] Use safer calloc for integer overflow check (sonic-net#1166)

Also this PR include changes of this sonic-net#815

SAI commit b2d4c9a57c7f00b2632c35ca5eb3dd6480f7916a Switch scoped tunnel attributes (sonic-net#1173) needed change in sai_redis_switch.cpp and sai_vs_switch.cpp for compilation.

How I verify:

Verify Build is fine of libsairedis*.deb, syncd*.deb, swss*.deb

Co-authored-by: Ann Pokora <apokora@juniper.net>
# for free to join this conversation on GitHub. Already have an account? # to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants