[tlm_teamd] get_dumps log because of a redundant keyspace notif handled #5

vivekrnv · 2022-01-21T10:32:32Z

What I did

Why I did it

https://github.com/Azure/sonic-swss/pull/1629 This PR originally fixed the spurious get_dumps log seen during select timeout . The same log has reappeared again, this time because of a redundant key-space notification seen when a config port channel delete is given.

Analysis

root@sonic:/home/admin# sudo config portchannel del PortChannel1111

admin@sonic:~$ redis-cli psubscribe "__keyspace@6__:*LAG*"
1) "pmessage"
2) "__keyspace@6__:*LAG*"
3) "__keyspace@6__:LAG_TABLE|PortChannel1111"
4) "hset"

Message Contents: 
{{first = "admin_status", second = "down"}, 
{first = "oper_status", second = "down"}, 
{first = "mtu", second = "9100"}, 
{first = "state", second = "ok"}}}}

An Example kvpairs when an actual SET event is seen
{{first = "admin_status", second = "up"}, 
{first = "oper_status", second = "down"}, 
{first = "mtu", second = "9100"}, 
{first = "state", second = "ok"}, 
{first = "team_device.ifinfo.ifindex", second = "175"}, 
{first = "setup.kernel_team_mode_name", second = "loadbalance"}, 
{first = "runner.active", second = "true"}, 
{first = "runner.fast_rate", second = "false"}, 
{first = "runner.fallback", second = "false"}, {first = "setup.pid", second = "1335"},
{first = "team_device.ifinfo.dev_addr", second = "1c:34:da:1c:a0:00"}

1) "pmessage"
2) "__keyspace@6__:*LAG*"
3) "__keyspace@6__:LAG_TABLE|PortChannel1111"
4) "del"

This extra notification is probably because there are two netlink msgs generated when the portchannel is deleted and teamsyncd is writing them to state_db, causing two notifications

This is causing an error log seen for tlm_teamd

Jan 21 04:19:09.049965 sonic ERR teamd#tlm_teamd: :- get_dump: Can't get dump for LAG 'PortChannel1111'. Skipping
Jan 21 04:19:09.051633 sonic NOTICE teamd#tlm_teamd: :- remove_lag: The LAG 'PortChannel1111' has been removed.
Jan 21 04:19:09.802348 sonic INFO teamd#supervisord 2022-01-21 04:19:09,801 INFO reaped unknown pid 649 (exit status 0)
Jan 21 04:19:09.843253 sonic NOTICE teamd#teammgrd: :- removeLag: Stop port channel PortChannel1111

How I verified it

Details if related

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

…_teamd_log_fix

vivekrnv · 2022-01-21T11:03:30Z

@dgsudharsan, please review.

dgsudharsan

There is a proper description of the problem but not the fix. Please add a description of the fix in 'what i did'

dgsudharsan · 2022-01-22T03:48:52Z

tlm_teamd/teamdctl_mgr.cpp

                    // This time a different lag interface errored out.
-		    m_lags_err_retry[lag_name] = 1;
+                    m_lags_err_retry[lag_name] = 1;


Can you please check if white space is introduced?

**What I did** Fix the Mem Leak by moving the raw pointers in type_maps to use smart pointers **Why I did it** ``` Indirect leak of 83776 byte(s) in 476 object(s) allocated from: #0 0x7f0a2a414647 in operator new(unsigned long) ../../../../src/libsanitizer/asan/asan_new_delete.cpp:99 #1 0x5555590cc923 in __gnu_cxx::new_allocator, std::allocator > const, referenced_object> > >::allocate(unsigned long, void const*) /usr/include/c++/10/ext/new_allocator.h:115 #2 0x5555590cc923 in std::allocator_traits, std::allocator > const, referenced_object> > > >::allocate(std::allocator, std::allocator > const, referenced_object> > >&, unsigned long) /usr/include/c++/10/bits/alloc_traits.h:460 #3 0x5555590cc923 in std::_Rb_tree, std::allocator >, std::pair, std::allocator > const, referenced_object>, std::_Select1st, std::allocator > const, referenced_object> >, std::less, std::allocator > >, std::allocator, std::allocator > const, referenced_object> > >::_M_get_node() /usr/include/c++/10/bits/stl_tree.h:584 #4 0x5555590cc923 in std::_Rb_tree_node, std::allocator > const, referenced_object> >* std::_Rb_tree, std::allocator >, std::pair, std::allocator > const, referenced_object>, std::_Select1st, std::allocator > const, referenced_object> >, std::less, std::allocator > >, std::allocator, std::allocator > const, referenced_object> > >::_M_create_node, std::allocator > const&>, std::tuple<> >(std::piecewise_construct_t const&, std::tuple, std::allocator > const&>&&, std::tuple<>&&) /usr/include/c++/10/bits/stl_tree.h:634 #5 0x5555590cc923 in std::_Rb_tree_iterator, std::allocator > const, referenced_object> > std::_Rb_tree, std::allocator >, std::pair, std::allocator > const, referenced_object>, std::_Select1st, std::allocator > const, referenced_object> >, std::less, std::allocator > >, std::allocator, std::allocator > const, referenced_object> > >::_M_emplace_hint_unique, std::allocator > const&>, std::tuple<> >(std::_Rb_tree_const_iterator, std::allocator > const, referenced_object> >, std::piecewise_construct_t const&, std::tuple, std::allocator > const&>&&, std::tuple<>&&) /usr/include/c++/10/bits/stl_tree.h:2461 #6 0x5555590e8757 in std::map, std::allocator >, referenced_object, std::less, std::allocator > >, std::allocator, std::allocator > const, referenced_object> > >::operator[](std::__cxx11::basic_string, std::allocator > const&) /usr/include/c++/10/bits/stl_map.h:501 #7 0x5555590d48b0 in Orch::setObjectReference(std::map, std::allocator >, std::map, std::allocator >, referenced_object, std::less, std::allocator > >, std::allocator, std::allocator > const, referenced_object> > >*, std::less, std::allocator > >, std::allocator, std::allocator > const, std::map, std::allocator >, referenced_object, std::less, std::allocator > >, std::allocator, std::allocator > const, referenced_object> > >*> > >&, std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator > const&) orchagent/orch.cpp:450 #8 0x5555594ff66b in QosOrch::handleQueueTable(Consumer&, std::tuple, std::allocator >, std::__cxx11::basic_string, std::allocator >, std::vector, std::allocator >, std::__cxx11::basic_string, std::allocator > >, std::allocator, std::allocator >, std::__cxx11::basic_string, std::allocator > > > > >&) orchagent/qosorch.cpp:1763 #9 0x5555594edbd6 in QosOrch::doTask(Consumer&) orchagent/qosorch.cpp:2179 #10 0x5555590c8743 in Consumer::drain() orchagent/orch.cpp:241 #11 0x5555590c8743 in Consumer::drain() orchagent/orch.cpp:238 #12 0x5555590c8743 in Consumer::execute() orchagent/orch.cpp:235 #13 0x555559090dad in OrchDaemon::start() orchagent/orchdaemon.cpp:755 #14 0x555558e9be25 in main orchagent/main.cpp:766 #15 0x7f0a299b6d09 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x23d09) ```

Vivek Reddy Karri added 2 commits January 21, 2022 10:28

tlm_teamd unnecessary log issue fixed

67fce01

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Merge branch 'master' of https://github.com/Azure/sonic-swss into tlm…

cf3d7b2

…_teamd_log_fix

oleksandrivantsiv approved these changes Jan 21, 2022

View reviewed changes

dgsudharsan reviewed Jan 22, 2022

View reviewed changes

vivekrnv closed this Apr 8, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[tlm_teamd] get_dumps log because of a redundant keyspace notif handled #5

[tlm_teamd] get_dumps log because of a redundant keyspace notif handled #5

vivekrnv commented Jan 21, 2022 •

edited

Loading

vivekrnv commented Jan 21, 2022 •

edited

Loading

dgsudharsan left a comment

dgsudharsan Jan 22, 2022

[tlm_teamd] get_dumps log because of a redundant keyspace notif handled #5

[tlm_teamd] get_dumps log because of a redundant keyspace notif handled #5

Conversation

vivekrnv commented Jan 21, 2022 • edited Loading

vivekrnv commented Jan 21, 2022 • edited Loading

dgsudharsan left a comment

Choose a reason for hiding this comment

dgsudharsan Jan 22, 2022

Choose a reason for hiding this comment

vivekrnv commented Jan 21, 2022 •

edited

Loading

vivekrnv commented Jan 21, 2022 •

edited

Loading