Issues post Nats Java Client Upgrade : SEVERE: exceptionOccurred, Connection:

[vishal-groww]

Observed behavior

We upgraded our nats java client from 2.12.0 to 2.15.7 and upgraded nats-server from 2.9.20 to 2.10.12. Upgrades were done in the following order

1. upgrade apps that run jetstream consumers
2. upgrade the nats-server
3. upgrade the apps that publish to jetstream. They also have consumers that responds to nats request-reply.

Step#1 and #2 went fine. But after Step3 we started seeing a flood of "SEVERE" log in the application.log

Apr 19, 2024 4:28:00 PM io.nats.client.impl.ErrorListenerLoggerImpl exceptionOccurred
SEVERE: exceptionOccurred, Connection: 45, Exception: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0

We reverted the nats-server version to 2.9.20 and then these logs stopped appearing. Client versions remain the same.
After 24 hours we started seeing the same "SEVERE" logs without any deployment. Though the applications are functional, we are trying to understand what's causing this issue. The flood of logs is also disturbing the actual log analysis.

Some more details about the project.
Framework: Micronaut 3
Jdk version : 17
Applications of nats: RequestReply, PubSub, JetStream, ObjectStore, leafnode

Expected behavior

no flooded logs as the applications are functional

Server and client version

nats-server 2.9.20
nats-client 3.15.7

Host environment

production

Steps to reproduce

upgrade nats-server to 2.10.12
upgrade nats-client to 2.15.7

[scottf]

any chance you can make a custom ErrorListener and implement

public void exceptionOccurred(final Connection conn, final Exception exp)

to give a stack trace?

Also, where/does Micronaut have an impact, if any. Is there any unusual threading model in the app or any Java 17 features being leveraged.

[vishal-groww]

Hi @scottf , I tried adding a custom ErrorListener and deployed the app, but this I didn't see any SEVERE logs. This error does not show up all the time, it is very random. I am going to let the app run with this custom exception logging and will share an update here when the exception log surfaces.

Regarding micronaut, we are not aware of any specific impact of micronaut. We aren't using any unusual threading model. We upgraded our app to Java17 recently and with the older version jnats, there were absolutely no issues.

Flood of exception logs started showing up only after the jnats version upgrade.

Meanwhile please check if I implemented the custom exception listener correctly

import io.micronaut.context.annotation.Primary;
import lombok.extern.slf4j.Slf4j;

@Slf4j
@Primary
public class CustomErrorListenerImpl implements ErrorListener {
  @Override
  public void exceptionOccurred(Connection conn, Exception exp) {
    ErrorListener.super.exceptionOccurred(conn, exp);
    log.error("Custom Exception logging {} ", conn, exp);
  }
}

I tried commenting "ErrorListener.super.exceptionOccurred(conn, exp)" and deployed the app, but still I could not observe any exception logging.

[scottf]

That looks good, as long as the logger gives a stack trace. Thanks for letting this run, seeing the trace will help immensely.

[vishal-groww]

Hi @scottf
We found the root cause of the IndexOutOfBoundsException. It was coming from one of our DTO Mapper classes. I will share the steps I followed to get the detailed exception logged and why I did not succeed in the first attempt.

• adding a custom implementation ErrorListener alone does not bind it with the interface during app start up. @primary annotation in micronaut framework is intended to inform the dependency injection system, which implementation of an interface to pick. Unfortunately, in this scenario, things are not that straight forward
• In-order to override the default errorListener, we need to create a custom implementation of the ErrorListener and provide the custom implementation reference to the io.nats.client.Option class which is an input to the Nats.connect() method. This is a very important step and I missed it, due to which my custom error logger was not working

@io.micronaut.context.annotation.Factory
public class Factory {
  @Bean
  @SneakyThrows
  @Primary
  public Connection connection(ErrorListener errorListener) {
    return Nats.connect(new Options.Builder()
        .server("nats://localhost:4222")
        .errorListener(errorListener)
        .build());
  }

  @Bean
  public ErrorListener errorListener() {
    return new CustomErrorListenerImpl();
  }

}

Thanks for the help. Closing the issue.

[scottf]

Glad you figure it out. Is there anything I could do to make the error listener better out of the box?

[vishal-groww]

Instead of just logging the Exception message, perhaps we could log the stack of exception so that it becomes readable & helpful in debugging the issue. wdyt ?