Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

Nats client hangs without any UNSUBSCRIBE and/or Exception #319

Closed
pentolone opened this issue Jun 22, 2020 · 10 comments
Closed

Nats client hangs without any UNSUBSCRIBE and/or Exception #319

pentolone opened this issue Jun 22, 2020 · 10 comments

Comments

@pentolone
Copy link

I have this strange behaviour, after a while the client Java stops printing incoming messages.
No UNSUBSCRIBE and the nats server shows PUB and MSG with the two clients.
I am using gnatsd 2.0.2 and Java Nats client 2.6.6

Thanks in advance

` @OverRide
public synchronized void onMessage(Message msg) throws InterruptedException {
try {
String[] tmp = msg.getSubject().split(Pattern.quote("."));
if(tmp.length < minLenSub || (!tmp[1].equalsIgnoreCase(NodexxNats.connectionName) && !tmp[1].equalsIgnoreCase(NodexxConfig.bridgeModel) && !tmp[1].equalsIgnoreCase(Nats.broadCast)))
throw new IOException("Invalid Nats Subject: " + msg.getSubject());

		if(tmp[2].equalsIgnoreCase(NodexxNats.family) && tmp[3].equals(Nats.start)) {
			logger.debug("Got start message, payload " + new String(msg.getData()));
			natsConnection.sendMessage(NodexxNats.family + ".bridge." + NodexxNats.connectionName + ".start", NodexxConfig.getBridgeToNats().toString());
			return;
		}

		if(tmp[2].contentEquals(Nats.exec)) 
			nodexxCommand.exec(new String(msg.getData()));
		else 
			natsClientNodexx.manageData(msg.getSubject(), msg.getData());
	} catch (Exception e) {
		logger.fatal(e.getMessage());
		}
}`
@pentolone
Copy link
Author

TLS is no active

@matthiashanel
Copy link
Contributor

@pentolone , is this behavior repeatable? do you have server logs for the time when tis happens?

@pentolone
Copy link
Author

pentolone commented Jul 29, 2020 via email

@matthiashanel
Copy link
Contributor

@pentolone next time this happens, would you mind dumping and sharing the stack traces of your program using jstack -l

@pentolone
Copy link
Author

pentolone commented Jul 30, 2020 via email

@pentolone
Copy link
Author

pentolone commented Jul 31, 2020 via email

@matthiashanel
Copy link
Contributor

@pentolone

What is your code doing in NodexxParser.java:664?
Is it possible that your logic ends up spinning or takes unexpectedly long?

"brnodexx:3" #19 prio=5 os_prio=0 cpu=758667.71ms elapsed=5222.55s
tid=0x00007fc27497e000 nid=0x21c6 runnable  [0x00007fc23d9f9000]
   java.lang.Thread.State: RUNNABLE
at nodexx.protocol.NodexxParser.deCoupleData(NodexxParser.java:664)
at nodexx.protocol.NodexxParser.parseResponse(NodexxParser.java:525)
at
nodexx.protocol.nats.NatsClientNodexx.manageData(NatsClientNodexx.java:166)
at nodexx.nats.NodexxNatsHandler.onMessage(NodexxNatsHandler.java:48)
- locked <0x00000006155d7908> (a nodexx.nats.NodexxNatsHandler)
at io.nats.client.impl.NatsDispatcher.run(NatsDispatcher.java:98)
at java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.7
/Executors.java:515)
at java.util.concurrent.FutureTask.run(java.base@11.0.7/FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.7
/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.7
/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.7/Thread.java:834)

   Locked ownable synchronizers:
- <0x00000006156c1648> (a java.util.concurrent.ThreadPoolExecutor$Worker)

These two threads hold locks while sleeping.

"Leitfeld-Refresh-Thread (brnodexx)" #21 daemon prio=1 os_prio=0
cpu=217.76ms elapsed=5222.53s tid=0x00007fc274a35000 nid=0x21c8 waiting on
condition  [0x00007fc23d5f8000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base@11.0.7/Native Method)
at br.thread.BridgeRefreshThread.run(BridgeRefreshThread.java:118)
- locked <0x00000006155b66a8> (a br.thread.BridgeRefreshThread)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.7
/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.7
/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.7/Thread.java:834)

   Locked ownable synchronizers:
- <0x00000006155b6798> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"Leitfeld-Bridge-Retry-Thread (brnodexx)" #22 daemon prio=10 os_prio=0
cpu=593.35ms elapsed=5222.51s tid=0x00007fc274a6a800 nid=0x21c9 sleeping
[0x00007fc23d239000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(java.base@11.0.7/Native Method)
at br.thread.BridgeRetryThread.run(BridgeRetryThread.java:35)
- locked <0x00000006155b2990> (a br.thread.BridgeRetryThread)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.7
/ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.7
/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.7/Thread.java:834)

   Locked ownable synchronizers:
- <0x00000006155b2a68> (a java.util.concurrent.ThreadPoolExecutor$Worker)

@pentolone
Copy link
Author

pentolone commented Aug 25, 2020 via email

@scottf
Copy link
Contributor

scottf commented Mar 8, 2021

@pentolone Have this been resolved? We'd like to close this issue.

@pentolone
Copy link
Author

pentolone commented Mar 8, 2021 via email

@scottf scottf closed this as completed Mar 9, 2021
# for free to join this conversation on GitHub. Already have an account? # to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@scottf @matthiashanel @pentolone