ClientQueueTest.testTake sometimes causes segmentation fault

[ihsandemir]

ClientQueueTest.testTake sometimes causes segmentation fault.
Log:
https://hazelcast-l337.ci.cloudbees.com/job/cpp-linux-nightly-64-STATIC-Release/69/console
https://hazelcast-l337.ci.cloudbees.com/job/cpp-linux-nightly-64-SHARED-Release/69/console
https://hazelcast-l337.ci.cloudbees.com/job/cpp-linux-nightly-64-STATIC-Release/69/console

06:03:26 [ RUN ] ClientQueueTest.testTake
06:03:26 Apr 06, 2016 12:49:27 PM CppClientListener
06:03:26 INFO: SHUTDOWN for instance 39 is completed.
06:03:26 START command received: NEW INSTANCE OPEN
06:03:26 Apr 06, 2016 12:49:27 PM com.hazelcast.instance.DefaultAddressPicker
06:03:26 INFO: [LOCAL] [dev] [3.7-SNAPSHOT] Interfaces is disabled, trying to pick one address from TCP-IP config addresses: [127.0.0.1]
06:03:26 Apr 06, 2016 12:49:27 PM com.hazelcast.instance.DefaultAddressPicker
06:03:26 INFO: [LOCAL] [dev] [3.7-SNAPSHOT] Picked Address[127.0.0.1]:5701, using socket ServerSocket[addr=/0:0:0:0:0:0:0:0,localport=5701], bind any local is true
06:03:26 Apr 06, 2016 12:49:27 PM com.hazelcast.system
06:03:26 INFO: [127.0.0.1]:5701 [dev] [3.7-SNAPSHOT] Hazelcast 3.7-SNAPSHOT (20160405 - 68d2790) starting at Address[127.0.0.1]:5701
06:03:26 Apr 06, 2016 12:49:27 PM com.hazelcast.system
06:03:26 INFO: [127.0.0.1]:5701 [dev] [3.7-SNAPSHOT] Copyright (c) 2008-2016, Hazelcast, Inc. All Rights Reserved.
06:03:26 Apr 06, 2016 12:49:27 PM com.hazelcast.system
06:03:26 INFO: [127.0.0.1]:5701 [dev] [3.7-SNAPSHOT] Configured Hazelcast Serialization version : 1
06:03:26 Apr 06, 2016 12:49:27 PM com.hazelcast.spi.OperationService
06:03:26 INFO: [127.0.0.1]:5701 [dev] [3.7-SNAPSHOT] Backpressure is disabled
06:03:26 Apr 06, 2016 12:49:27 PM com.hazelcast.instance.Node
06:03:26 INFO: [127.0.0.1]:5701 [dev] [3.7-SNAPSHOT] Creating TcpIpJoiner
06:03:26 Apr 06, 2016 12:49:27 PM com.hazelcast.core.LifecycleService
06:03:26 INFO: [127.0.0.1]:5701 [dev] [3.7-SNAPSHOT] Address[127.0.0.1]:5701 is STARTING
06:03:26 Apr 06, 2016 12:49:27 PM com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl
06:03:26 INFO: [127.0.0.1]:5701 [dev] [3.7-SNAPSHOT] Starting with 2 generic operation threads and 2 partition operation threads.
06:03:26 Apr 06, 2016 12:49:27 PM com.hazelcast.nio.tcp.nonblocking.NonBlockingIOThreadingModel
06:03:26 INFO: [127.0.0.1]:5701 [dev] [3.7-SNAPSHOT] TcpIpConnectionManager configured with Non Blocking IO-threading model: 3 input threads and 3 output threads
06:03:26 Apr 06, 2016 12:49:27 PM com.hazelcast.nio.tcp.InitConnectionTask
06:03:26 INFO: [127.0.0.1]:5701 [dev] [3.7-SNAPSHOT] Connecting to /127.0.0.1:5702, timeout: 0, bind-any: true
06:03:26 Apr 06, 2016 12:49:27 PM com.hazelcast.nio.tcp.InitConnectionTask
06:03:26 INFO: [127.0.0.1]:5701 [dev] [3.7-SNAPSHOT] Could not connect to: /127.0.0.1:5702. Reason: SocketException[Connection refused to address /127.0.0.1:5702]
06:03:26 Apr 06, 2016 12:49:27 PM com.hazelcast.cluster.impl.TcpIpJoiner
06:03:26 INFO: [127.0.0.1]:5701 [dev] [3.7-SNAPSHOT] Address[127.0.0.1]:5702 is added to the blacklist.
06:03:26 Apr 06, 2016 12:49:27 PM com.hazelcast.nio.tcp.InitConnectionTask
06:03:26 INFO: [127.0.0.1]:5701 [dev] [3.7-SNAPSHOT] Connecting to /127.0.0.1:5704, timeout: 0, bind-any: true
06:03:26 Apr 06, 2016 12:49:27 PM com.hazelcast.nio.tcp.InitConnectionTask
06:03:26 INFO: [127.0.0.1]:5701 [dev] [3.7-SNAPSHOT] Connecting to /127.0.0.1:5705, timeout: 0, bind-any: true
06:03:26 Apr 06, 2016 12:49:27 PM com.hazelcast.nio.tcp.InitConnectionTask
06:03:26 INFO: [127.0.0.1]:5701 [dev] [3.7-SNAPSHOT] Could not connect to: /127.0.0.1:5704. Reason: SocketException[Connection refused to address /127.0.0.1:5704]
06:03:26 Apr 06, 2016 12:49:27 PM com.hazelcast.cluster.impl.TcpIpJoiner
06:03:26 INFO: [127.0.0.1]:5701 [dev] [3.7-SNAPSHOT] Address[127.0.0.1]:5704 is added to the blacklist.
06:03:26 Apr 06, 2016 12:49:27 PM com.hazelcast.nio.tcp.InitConnectionTask
06:03:26 INFO: [127.0.0.1]:5701 [dev] [3.7-SNAPSHOT] Connecting to /127.0.0.1:5703, timeout: 0, bind-any: true
06:03:26 Apr 06, 2016 12:49:27 PM com.hazelcast.nio.tcp.InitConnectionTask
06:03:26 INFO: [127.0.0.1]:5701 [dev] [3.7-SNAPSHOT] Could not connect to: /127.0.0.1:5703. Reason: SocketException[Connection refused to address /127.0.0.1:5703]
06:03:26 Apr 06, 2016 12:49:27 PM com.hazelcast.cluster.impl.TcpIpJoiner
06:03:26 INFO: [127.0.0.1]:5701 [dev] [3.7-SNAPSHOT] Address[127.0.0.1]:5703 is added to the blacklist.
06:03:26 Apr 06, 2016 12:49:27 PM com.hazelcast.nio.tcp.InitConnectionTask
06:03:26 INFO: [127.0.0.1]:5701 [dev] [3.7-SNAPSHOT] Could not connect to: /127.0.0.1:5705. Reason: SocketException[Connection refused to address /127.0.0.1:5705]
06:03:26 Apr 06, 2016 12:49:27 PM com.hazelcast.cluster.impl.TcpIpJoiner
06:03:26 INFO: [127.0.0.1]:5701 [dev] [3.7-SNAPSHOT] Address[127.0.0.1]:5705 is added to the blacklist.
06:03:27 Apr 06, 2016 12:49:28 PM com.hazelcast.cluster.impl.TcpIpJoiner
06:03:27 INFO: [127.0.0.1]:5701 [dev] [3.7-SNAPSHOT]
06:03:27
06:03:27
06:03:27 Members [1] {
06:03:27 Member [127.0.0.1]:5701 this
06:03:27 }
06:03:27
06:03:27 Apr 06, 2016 12:49:28 PM com.hazelcast.core.LifecycleService
06:03:27 INFO: [127.0.0.1]:5701 [dev] [3.7-SNAPSHOT] Address[127.0.0.1]:5701 is STARTED
06:03:28 Apr 06, 2016 12:49:28 PM com.hazelcast.nio.tcp.SocketAcceptorThread
06:03:28 INFO: [127.0.0.1]:5701 [dev] [3.7-SNAPSHOT] Accepting socket connection from /127.0.0.1:60334
06:03:28 Apr 06, 2016 12:49:28 PM com.hazelcast.nio.tcp.TcpIpConnectionManager
06:03:28 INFO: [127.0.0.1]:5701 [dev] [3.7-SNAPSHOT] Established socket connection between /127.0.0.1:5701 and /127.0.0.1:60334
06:03:28 Apr 06, 2016 12:49:28 PM com.hazelcast.client.impl.protocol.task.AuthenticationMessageTask
06:03:28 INFO: [127.0.0.1]:5701 [dev] [3.7-SNAPSHOT] Received auth from Connection [/127.0.0.1:5701 -> /127.0.0.1:60334], endpoint=null, alive=true, type=CPP_CLIENT, successfully authenticated, principal : ClientPrincipal{uuid='aa1bb8b8-3414-4f84-99d5-71cf5c69a818', ownerUuid='bf1c1eed-10b7-45a3-b6c4-22a2017f3b0a'}, owner connection : true
06:03:29 Apr 06, 2016 12:49:29 PM com.hazelcast.nio.tcp.SocketAcceptorThread
06:03:29 INFO: [127.0.0.1]:5701 [dev] [3.7-SNAPSHOT] Accepting socket connection from /127.0.0.1:60335
06:03:29 Apr 06, 2016 12:49:29 PM com.hazelcast.nio.tcp.TcpIpConnectionManager
06:03:29 INFO: [127.0.0.1]:5701 [dev] [3.7-SNAPSHOT] Established socket connection between /127.0.0.1:5701 and /127.0.0.1:60335
06:03:29 Apr 06, 2016 12:49:29 PM com.hazelcast.client.impl.protocol.task.AuthenticationMessageTask
06:03:29 INFO: [127.0.0.1]:5701 [dev] [3.7-SNAPSHOT] Received auth from Connection [/127.0.0.1:5701 -> /127.0.0.1:60335], endpoint=null, alive=true, type=CPP_CLIENT, successfully authenticated, principal : ClientPrincipal{uuid='aa1bb8b8-3414-4f84-99d5-71cf5c69a818', ownerUuid='bf1c1eed-10b7-45a3-b6c4-22a2017f3b0a'}, owner connection : false
06:03:29 Apr 06, 2016 12:49:29 PM com.hazelcast.internal.partition.InternalPartitionService
06:03:29 INFO: [127.0.0.1]:5701 [dev] [3.7-SNAPSHOT] Initializing cluster partition table arrangement...
06:03:31 Apr 06, 2016 12:49:31 PM com.hazelcast.nio.tcp.TcpIpConnection
06:03:31 INFO: [127.0.0.1]:5701 [dev] [3.7-SNAPSHOT] Connection [Address[127.0.0.1]:60334] lost. Reason: java.io.EOFException[Remote socket closed!]
06:03:31 Apr 06, 2016 12:49:31 PM com.hazelcast.nio.tcp.nonblocking.NonBlockingSocketReader
06:03:31 WARNING: [127.0.0.1]:5701 [dev] [3.7-SNAPSHOT] hz._hzInstance_40_dev.IO.thread-in-0 Closing socket to endpoint Address[127.0.0.1]:60334, Cause:java.io.EOFException: Remote socket closed!
06:03:31 Apr 06, 2016 12:49:31 PM com.hazelcast.nio.tcp.TcpIpConnection
06:03:31 INFO: [127.0.0.1]:5701 [dev] [3.7-SNAPSHOT] Connection [Address[127.0.0.1]:60335] lost. Reason: java.io.IOException[Connection reset by peer]
06:03:31 Apr 06, 2016 12:49:31 PM com.hazelcast.nio.tcp.nonblocking.NonBlockingSocketReader
06:03:31 WARNING: [127.0.0.1]:5701 [dev] [3.7-SNAPSHOT] hz._hzInstance_40_dev.IO.thread-in-1 Closing socket to endpoint Address[127.0.0.1]:60335, Cause:java.io.IOException: Connection reset by peer
06:03:31 [WARNING]
06:03:31 java.lang.reflect.InvocationTargetException
06:03:31 testLinuxSingleCase.sh: line 106: 19275 Segmentation fault (core dumped) ${BUILD_DIR}/hazelcast/test/src/${EXECUTABLE_NAME} --gtest_output="xml:CPP_Client_Test_Report.xml"
06:03:31 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
06:03:31 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
06:03:31 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
06:03:31 at java.lang.reflect.Method.invoke(Method.java:497)
06:03:31 at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293)
06:03:31 at java.lang.Thread.run(Thread.java:745)
06:03:31 Caused by: java.io.EOFException
06:03:31 at java.io.DataInputStream.readInt(DataInputStream.java:392)
06:03:31 at CppClientListener.main(CppClientListener.java:229)
06:03:31 Test FAILED. Result:139
06:03:31 ... 6 more
06:03:31 cleanup is being performed.

[ihsandemir]

When examined the core stack it is exactly the same as #76 .

Program terminated with signal 11, Segmentation fault.
#0 0x00000000006e1ba0 in hazelcast::util::LockGuard::~LockGuard() ()
(gdb) bt
#0 0x00000000006e1ba0 in hazelcast::util::LockGuard::~LockGuard() ()
#1 0x000000000075339e in hazelcast::client::connection::CallFuture::get(long) ()
#2 0x0000000000753915 in hazelcast::client::connection::CallFuture::get() ()
#3 0x00000000006ae24e in hazelcast::client::proxy::ProxyImpl::invoke(std::auto_ptrhazelcast::client::protocol::ClientMessage, int) ()
#4 0x00000000006af9a2 in hazelcast::client::proxy::IQueueImpl::offer(hazelcast::client::serialization::pimpl::Data const&, long) ()
#5 0x00000000005e927d in hazelcast::client::test::testOfferPollThread2(hazelcast::util::ThreadArgs&) ()
#6 0x00000000006e16c3 in hazelcast::util::Thread::controlledThread(void*) ()
#7 0x000000338880683d in start_thread () from /lib64/libpthread.so.0
#8 0x0000003387cd4fdd in clone () from /lib64/libc.so.6
(gdb) info threads
7 Thread 0x2b2de23a2250 (LWP 19275) 0x0000003388807c65 in pthread_join () from /lib64/libpthread.so.0
6 Thread 21095 0x0000003387cce172 in select () from /lib64/libc.so.6
5 Thread 21096 0x0000003387cce172 in select () from /lib64/libc.so.6
4 Thread 21097 0x000000338880b280 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
3 Thread 21098 0x000000338880de11 in recv () from /lib64/libpthread.so.0
2 Thread 21103 0x000000338880b280 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

• 1 Thread 21106 0x00000000006e1ba0 in hazelcast::util::LockGuard::~LockGuard() ()
  (gdb) thread 7
  [Switching to thread 7 (Thread 0x2b2de23a2250 (LWP 19275))]#0 0x0000003388807c65 in pthread_join () from /lib64/libpthread.so.0
  (gdb) bt
  #0 0x0000003388807c65 in pthread_join () from /lib64/libpthread.so.0
  test issue #1 0x00000000006e0e4e in hazelcast::util::Thread::~Thread() ()
  Added missing executeonX tests for Map. #2 0x00000000005ec744 in hazelcast::client::test::ClientQueueTest_testTake_Test::TestBody() ()
  request and response classes and related hooks of old client protocol are deleted #3 0x000000000077ee64 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test_, void (testing::Test::)(), char const) ()
  Decrement Pending packets for events also #4 0x0000000000787106 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test_, void (testing::Test::)(), char const) ()
  Changes in Map getAll/putAll implementation #5 0x0000000000774bfb in testing::Test::Run() ()
  Reconnecting to instance on same address should preserve member list correctly #6 0x0000000000774ee6 in testing::TestInfo::Run() ()
  Serialization resolution order #7 0x0000000000775007 in testing::TestCase::Run() ()
  Add tests for serialization #8 0x000000000077790d in testing::internal::UnitTestImpl::RunAllTests() ()
  Listener service should listen for events after clusterService has been started to prevent receiving duplicate events #9 0x000000000077eef4 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl_, bool (testing::internal::UnitTestImpl::)(), char const) ()
  Fixing release scripts #10 0x0000000000786ba9 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl_, bool (testing::internal::UnitTestImpl::)(), char const) ()
  New client protocol implementation #11 0x00000000007737e7 in testing::UnitTest::Run() ()
  Add All The Java Predicates To C++ #12 0x00000000005b285a in main ()

Hence closing this issue since the other shall be open.