ovs crashes after running a few hours

[jshen28]

We are using ovs 2.9.5 on ubuntu 18.04.3 with kernel of 4.15.0-72-generic and glibc of 2.27-3ubuntu1. After running a few hours, it is observed that ovs-vswitchd is crashed. After some debugging I found

[Current thread is 1 (Thread 0x7fbba17fa700 (LWP 7640))]
(gdb) bt
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1 0x00007fbc1aa03801 in __GI_abort () at abort.c:79
#2 0x00007fbc1a9f339a in __assert_fail_base (fmt=0x7fbc1ab7a7d8 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x7fbc1b36e692 "mutex->__data.__owner == 0",
file=file@entry=0x7fbc1b36e675 "../nptl/pthread_mutex_lock.c", line=line@entry=79, function=function@entry=0x7fbc1b36e800 <PRETTY_FUNCTION.8734> "__pthread_mutex_lock") at assert.c:92
#3 0x00007fbc1a9f3412 in __GI___assert_fail (assertion=assertion@entry=0x7fbc1b36e692 "mutex->__data._owner == 0", file=file@entry=0x7fbc1b36e675 "../nptl/pthread_mutex_lock.c", line=line@entry=79,
function=function@entry=0x7fbc1b36e800 <PRETTY_FUNCTION.8734> "pthread_mutex_lock") at assert.c:101
#4 0x00007fbc1b3641c8 in GI___pthread_mutex_lock (mutex=mutex@entry=0x55f10d8e09e0 <seq_mutex>) at ../nptl/pthread_mutex_lock.c:79
#5 0x000055f10d5888f8 in ovs_mutex_lock_at (l=l@entry=0x55f10d8e09e0 <seq_mutex>, where=where@entry=0x55f10d64495e "../lib/seq.c:169") at ../lib/ovs-thread.c:76
#6 0x000055f10d5a9887 in seq_read (seq=0x55f10da0c940) at ../lib/seq.c:169
#7 0x000055f10d4d7ce6 in revalidator_sweep (revalidator=revalidator@entry=0x55f10db2b3a8, purge=purge@entry=false) at ../ofproto/ofproto-dpif-upcall.c:2560
#8 0x000055f10d4da48b in revalidator_sweep (revalidator=0x55f10db2b3a8) at ../ofproto/ofproto-dpif-upcall.c:2636
#9 udpif_revalidator (arg=0x55f10db2b3a8) at ../ofproto/ofproto-dpif-upcall.c:917
#10 0x000055f10d58958f in ovsthread_wrapper (aux=) at ../lib/ovs-thread.c:354
#11 0x00007fbc1b3616db in start_thread (arg=0x7fbba17fa700) at pthread_create.c:463
#12 0x00007fbc1aae488f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

And
(gdb) print seq_mutex
$13 = {lock = {__data = {__lock = 2, __count = 0, __owner = 0, __nusers = 0, __kind = 2, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}},
__size = "\002", '\000' <repeats 15 times>, "\002", '\000' <repeats 22 times>, __align = 2}, where = 0x55f10d61c67c ""}

It looks similar to https://www.mail-archive.com/search?l=ovs-dev@openvswitch.org&q=subject:%22%5C%5Bovs%5C-dev%5C%5D+vswitchd+crashed+when+revalidate+flows+in+ovs+2.8.2%22&o=newest&f=1 this report but it does not reach
any conclusion.

Does anyone have idea of how to fix this issue,, thank you.

[igsilya]

Hi, there are several issues in glibc in ubuntu 18.04 that were pointed to in other bug reports, e.g. #175 . Last time I checked (long ago) these wasn't backported to ubuntu 18.04.
The issue you're facing looks similar to another glibc bug: https://sourceware.org/bugzilla/show_bug.cgi?id=23275
And I have no idea if this fix found its way to ubuntu 18.04. It should be part of glibc 2.29, according to bugzilla.