NATS consumers not pulling messages from Jetstream even though consumer status shows active. Restart of client application required as workaround to resolve the issue. [v2.10.18]

[mohamedsaleem18]

Observed behavior

NATS cluster :

• 3 Nodes
• NATS version: 2.10.18

There are pending messages in the stream. Though consumers are connected to NATS, consumers are becoming stale. The workaround consists of purging the stream, deleting the affected consumers, and restarting the service they originate from. This recreates them in a healthy state to resolve the issue. This issue occurs at least once every day.
Client side NATS configuration:

nats {
  enabled = true
  address = "localhost:4222"
  bufferSize = 8
  timeout = 5000
  errorListener = "io.nats.client.impl.ErrorListenerLoggerImpl"
  streams {
    xxxStream {
      name = "lpsn32"
      subject = "xxx-response-details"
      consumerName = "xxxResponseHandler"
      handler = "xxxResponseHandler"
      batchSize = 1
      # pollingDuration is in ms
      pollingDuration = 100
    }
  }
  protocol = "stream"
}

Expected behavior

Consumer should not go stale. In case if Consumer is stale then NATS should automatically modify the consumer status as inactive or drop the consumer.

Server and client version

NATS cluster :

• 3 Nodes.
• NATS version :- 2.10.18

Host environment

VM - OS details
NAME="Oracle Linux Server"
VERSION="8.10"

Steps to reproduce

1. Publish message to stream when consumer is connected.
2. Consumer will consume the messages without any issue.
3. If there is idle time or after long time, consumer goes stale. Messages are pending in the stream but consumer are not consuming the message. NATS shows consumer as active.

Please refer description section for workaround to resolve this issue.

[derekcollison]

Could you share additional information..

1. Push or Pull consumer?
2. What client lib and version?
3. What does consumer info (NATS cli) report?
4. What does stream info report?

[mohamedsaleem18]

Push or Pull consumer? - Pull consumer
What client lib and version? - Java NATS client, jnats 2.18.1
What does consumer info (NATS cli) report? Consumer was active
What does stream info report? Messages were pending in the stream but consumers were connected as well.

[derekcollison]

Can you share the consumer info? We want to see if there are pull requests waiting or not etc..

[mohamedsaleem18]

When we encounter the issue again, I will provide the details of consumer info.