SNMP service started and triggered SWSS start in the middle of warm-reboot #2750
Assignees
Labels
Comments
|
@stepanblyschak great finding! Should we proactively stop snmp before warm reboot? |
|
It is hard to solve this issue unless we have an indication of system boot up is done. This timer is a one time timer since boot up. snmp services wait for 3.5 minutes before starting service. Can we fix snmp service to not depending on the delay? This delay is a result of delaying creating counters. why did we delay creating counters? @pavel-shirshov can you fill in the information? Per discussion in warm reboot meeting, we would like to see if we could remove this 3.5 minutes delay.
@stepanblyschak will drive the action items. |
This was referenced Apr 12, 2019
dgsudharsan
added a commit
to dgsudharsan/sonic-buildimage
that referenced
this issue
Jul 11, 2023
Update sonic-utilities submodule pointer to include the following: * ff380e04 [hash]: Implement GH frontend ([sonic-net#2580](sonic-net/sonic-utilities#2580)) * 61bad064 [db_migrator] Set correct CURRENT_VERSION, extend UT ([sonic-net#2895](sonic-net/sonic-utilities#2895)) * 6b8ee47c [CLI][Show][BGP] Show BGP Change for no neighbor scenario ([sonic-net#2885](sonic-net/sonic-utilities#2885)) * 73d8d633 [doc] Update Command-Reference.md, change show bgp peer command to show bfd peer ([sonic-net#2750](sonic-net/sonic-utilities#2750)) * 7bc08c28 [db_migrator] Remove hardcoded config and migrate config from minigraph ([sonic-net#2887](sonic-net/sonic-utilities#2887)) * b1aa9426 [generate_dump]: Enhance show techsupport for Marvell platform ([sonic-net#2676](sonic-net/sonic-utilities#2676)) * 316b14c0 Add support for secure upgrade ([sonic-net#2698](sonic-net/sonic-utilities#2698)) * dc2945bc [dns] Implement config and show commands for static DNS. ([sonic-net#2737](sonic-net/sonic-utilities#2737)) * 8414a709 [chassis][multi asic] change acl_loader to use tcp socket for db communication ([sonic-net#2525](sonic-net/sonic-utilities#2525)) * 0b629ba1 Revert [chassis][voq] Clear fabric counters queue/port (2789) ([sonic-net#2882](sonic-net/sonic-utilities#2882)) * 3ba8241a [db_migtrator] Add migration of FLEX_COUNTER_DELAY_STATUS during 1911->master upgrade + fast-reboot. Add UT. ([sonic-net#2839](sonic-net/sonic-utilities#2839)) * fceef2ed [chassis][voq] Clear fabric counters queue/port ([sonic-net#2789](sonic-net/sonic-utilities#2789)) Signed-off-by: dgsudharsan <sudharsand@nvidia.com>
liat-grozovik
pushed a commit
that referenced
this issue
Jul 11, 2023
Update sonic-utilities submodule pointer to include the following: * ff380e04 [hash]: Implement GH frontend ([#2580](sonic-net/sonic-utilities#2580)) * 61bad064 [db_migrator] Set correct CURRENT_VERSION, extend UT ([#2895](sonic-net/sonic-utilities#2895)) * 6b8ee47c [CLI][Show][BGP] Show BGP Change for no neighbor scenario ([#2885](sonic-net/sonic-utilities#2885)) * 73d8d633 [doc] Update Command-Reference.md, change show bgp peer command to show bfd peer ([#2750](sonic-net/sonic-utilities#2750)) * 7bc08c28 [db_migrator] Remove hardcoded config and migrate config from minigraph ([#2887](sonic-net/sonic-utilities#2887)) * b1aa9426 [generate_dump]: Enhance show techsupport for Marvell platform ([#2676](sonic-net/sonic-utilities#2676)) * 316b14c0 Add support for secure upgrade ([#2698](sonic-net/sonic-utilities#2698)) * dc2945bc [dns] Implement config and show commands for static DNS. ([#2737](sonic-net/sonic-utilities#2737)) * 8414a709 [chassis][multi asic] change acl_loader to use tcp socket for db communication ([#2525](sonic-net/sonic-utilities#2525)) * 0b629ba1 Revert [chassis][voq] Clear fabric counters queue/port (2789) ([#2882](sonic-net/sonic-utilities#2882)) * 3ba8241a [db_migtrator] Add migration of FLEX_COUNTER_DELAY_STATUS during 1911->master upgrade + fast-reboot. Add UT. ([#2839](sonic-net/sonic-utilities#2839)) * fceef2ed [chassis][voq] Clear fabric counters queue/port ([#2789](sonic-net/sonic-utilities#2789)) Signed-off-by: dgsudharsan <sudharsand@nvidia.com>
mssonicbld
added a commit
that referenced
this issue
Jul 11, 2023
…atically (#15456) #### Why I did it src/sonic-utilities ``` * ff380e04 - (HEAD -> master, origin/master, origin/HEAD) [hash]: Implement GH frontend (#2580) (13 hours ago) [Nazarii Hnydyn] * 61bad064 - [db_migrator] Set correct CURRENT_VERSION, extend UT (#2895) (4 days ago) [Vadym Hlushko] * 6b8ee47c - [CLI][Show][BGP] Show BGP Change for no neighbor scenario (#2885) (6 days ago) [Dev Ojha] * 73d8d633 - [doc] Update Command-Reference.md, change "show bgp peer" command to "show bfd peer" (#2750) (11 days ago) [PinghaoQu] * 7bc08c28 - [db_migrator] Remove hardcoded config and migrate config from minigraph (#2887) (11 days ago) [Vaibhav Hemant Dixit] * b1aa9426 - [generate_dump]: Enhance show techsupport for Marvell platform (#2676) (11 days ago) [pavannaregundi] * 316b14c0 - Add support for secure upgrade (#2698) (2 weeks ago) [ycoheNvidia] * dc2945bc - [dns] Implement config and show commands for static DNS. (#2737) (2 weeks ago) [Oleksandr Ivantsiv] * 8414a709 - [chassis][multi asic] change acl_loader to use tcp socket for db communication (#2525) (2 weeks ago) [Arvindsrinivasan Lakshmi Narasimhan] * 0b629ba1 - Revert "[chassis][voq] Clear fabric counters queue/port (#2789)" (#2882) (3 weeks ago) [RoRonoa] * 3ba8241a - [db_migtrator] Add migration of FLEX_COUNTER_DELAY_STATUS during 1911->master upgrade + fast-reboot. Add UT. (#2839) (4 weeks ago) [Vadym Hlushko] * fceef2ed - [chassis][voq] Clear fabric counters queue/port (#2789) (4 weeks ago) [jfeng-arista] ``` #### How I did it #### How to verify it #### Description for the changelog
sonic-otn
pushed a commit
to sonic-otn/sonic-buildimage
that referenced
this issue
Sep 20, 2023
Update sonic-utilities submodule pointer to include the following: * ff380e04 [hash]: Implement GH frontend ([sonic-net#2580](sonic-net/sonic-utilities#2580)) * 61bad064 [db_migrator] Set correct CURRENT_VERSION, extend UT ([sonic-net#2895](sonic-net/sonic-utilities#2895)) * 6b8ee47c [CLI][Show][BGP] Show BGP Change for no neighbor scenario ([sonic-net#2885](sonic-net/sonic-utilities#2885)) * 73d8d633 [doc] Update Command-Reference.md, change show bgp peer command to show bfd peer ([sonic-net#2750](sonic-net/sonic-utilities#2750)) * 7bc08c28 [db_migrator] Remove hardcoded config and migrate config from minigraph ([sonic-net#2887](sonic-net/sonic-utilities#2887)) * b1aa9426 [generate_dump]: Enhance show techsupport for Marvell platform ([sonic-net#2676](sonic-net/sonic-utilities#2676)) * 316b14c0 Add support for secure upgrade ([sonic-net#2698](sonic-net/sonic-utilities#2698)) * dc2945bc [dns] Implement config and show commands for static DNS. ([sonic-net#2737](sonic-net/sonic-utilities#2737)) * 8414a709 [chassis][multi asic] change acl_loader to use tcp socket for db communication ([sonic-net#2525](sonic-net/sonic-utilities#2525)) * 0b629ba1 Revert [chassis][voq] Clear fabric counters queue/port (2789) ([sonic-net#2882](sonic-net/sonic-utilities#2882)) * 3ba8241a [db_migtrator] Add migration of FLEX_COUNTER_DELAY_STATUS during 1911->master upgrade + fast-reboot. Add UT. ([sonic-net#2839](sonic-net/sonic-utilities#2839)) * fceef2ed [chassis][voq] Clear fabric counters queue/port ([sonic-net#2789](sonic-net/sonic-utilities#2789)) Signed-off-by: dgsudharsan <sudharsand@nvidia.com>
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
Description
SNMP service start is delayed for 3 min and depends on SWSS service.
When issuing
sudo warm-rebootafter daemons reconciled state was reached (~2 min after system start) SNMP service may wake up in the middle of warm-reboot script, after SWSS was killed by warm-reboot script, and start SWSS container again. This causes warm reboot failure and system becomes in inconsistent state.See syslog messages from "Apr 5 13:24:46.321628" to "Apr 5 13:25:50.512102"
Steps to reproduce the issue:
sudo warm-rebootfew timesDescribe the results you received:
See description
Describe the results you expected:
SNMP start does not mess with warm-reboot
Additional information you deem important (e.g. issue happens only occasionally):
syslog.txt
The text was updated successfully, but these errors were encountered: