RestTemplateEurekaHttpClient not closing HttpClient on shutdown

[BenEfrati]

Spring Cloud 2021.0.0
Spring Boot 2.6.8

Bug Description:
RestTemplateEurekaHttpClient is not closing HttpClient on shutdown. This leads to TCP CLOSE_WAIT connections to eureka server.

EurekaClient will shutdown when an exception occurs on an http request, but not shutdown HttpClient.

This bug is related to

spring-cloud-netflix/spring-cloud-netflix-eureka-client/src/main/java/org/springframework/cloud/netflix/eureka/http/RestTemplateEurekaHttpClient.java

Line 203 in 27ac337

// Nothing to do

In case of exception here:
https://github.com/Netflix/eureka/blob/ed0da19ca1c049c87e3dbf75b6015c1861d5c2d0/eureka-client/src/main/java/com/netflix/discovery/shared/transport/decorator/RedirectingEurekaHttpClient.java#L96
new HttpClient will be created without closing the existing one - this causes CLOSE_WAIT connections

This supplier creates new CloseableHttpClient for every call to

spring-cloud-netflix/spring-cloud-netflix-eureka-client/src/main/java/org/springframework/cloud/netflix/eureka/http/RestTemplateTransportClientFactory.java

Line 103 in 27ac337

public EurekaHttpClient newClient(EurekaEndpoint serviceUrl) {

public class DefaultEurekaClientHttpRequestFactorySupplier implements EurekaClientHttpRequestFactorySupplier {

	@Override
	public ClientHttpRequestFactory get(SSLContext sslContext, @Nullable HostnameVerifier hostnameVerifier) {
		HttpClientBuilder httpClientBuilder = HttpClients.custom();
		if (sslContext != null) {
			httpClientBuilder = httpClientBuilder.setSSLContext(sslContext);
		}
		if (hostnameVerifier != null) {
			httpClientBuilder = httpClientBuilder.setSSLHostnameVerifier(hostnameVerifier);
		}
		CloseableHttpClient httpClient = httpClientBuilder.build();
		HttpComponentsClientHttpRequestFactory requestFactory = new HttpComponentsClientHttpRequestFactory();
		requestFactory.setHttpClient(httpClient);
		return requestFactory;
	}

}

so in case of shutdown, currentEurekaClient shutdown don't closes connections:

spring-cloud-netflix/spring-cloud-netflix-eureka-client/src/main/java/org/springframework/cloud/netflix/eureka/http/RestTemplateEurekaHttpClient.java

Line 203 in 27ac337

// Nothing to do

possible solution will be trying to close the HttpClient:
shutdown could be

public void shutdown() {
        Optional.of(unwrapRequestFactoryIfNecessary(restTemplate.getRequestFactory()))
                .filter(HttpComponentsClientHttpRequestFactory.class::isInstance)
                .map(HttpComponentsClientHttpRequestFactory.class::cast)
                .ifPresent(requestFactory-> {
                    try {
                        requestFactory.destroy();
                    } catch (Exception e) {
                    }
                });
    }

unwrapRequestFactoryIfNecessary
https://github.com/spring-projects/spring-boot/blob/47516b50c39bd6ea924a1f6720ce6d4a71088651/spring-boot-project/spring-boot/src/main/java/org/springframework/boot/web/client/RestTemplateBuilder.java#L746

spring-projects/spring-boot#31075

As you can see process open files increases over time until GC occurred

We can also create EurekaClientHttpRequestFactorySupplier which return the same ClientHttpRequestFactory, but it not a stable solution since we can't control eureka client code, maybe in case of exceptions connections not closes (hence not returns to PoolingHttpClientConnectionManager) - this can lead to no available connections in pool. In that case, restart is required

Original Issue:
#4062

[BenEfrati]

spring-projects/spring-boot#31075 (comment)

the piece code instantiating the request factory should be in charge of closing resources properly

[OlgaMaciaszek]

Hello, @BenEfrati , thanks for reporting it. Looks like a bug.

[sdzx3783]

Recently, I encountered this problem in the production environment. I found that the HttpClient was created many times. The HttpClient connection pool did not enable the thread for clearing expired connections. The problem was solved by sharing one HttpClient and enabling the IdleConnectionEvaluator thread. The code is as follows:

@Configuration
public class EurekaConfig {

    private final AtomicReference<CloseableHttpClient> ref = new AtomicReference<>();

    @Bean
    public EurekaClientHttpRequestFactorySupplier defaultEurekaClientHttpRequestFactorySupplier() {
        return (sslContext, hostnameVerifier) -> {
            HttpComponentsClientHttpRequestFactory requestFactory = new HttpComponentsClientHttpRequestFactory();
            HttpClientBuilder httpClientBuilder = HttpClients
                    .custom()
                    .evictExpiredConnections()
                    .evictIdleConnections(30L, TimeUnit.SECONDS);
            if (sslContext != null) {
                httpClientBuilder = httpClientBuilder.setSSLContext(sslContext);
            }
            if (hostnameVerifier != null) {
                httpClientBuilder = httpClientBuilder.setSSLHostnameVerifier(hostnameVerifier);
            }
            if (ref.get() == null) {
                ref.compareAndSet(null, httpClientBuilder.build());
            }
            requestFactory.setHttpClient(ref.get());
            return requestFactory;
        };
    }
}
`

[OlgaMaciaszek]

Hello @sdzx3783 - thanks for your comment. Would you like to submit a PR with a fix?

[sdzx3783]

I provide a temporary solution without changing the source code

[dangkhoaphung]

Hi @OlgaMaciaszek , may I know which version will include the fix?

[OlgaMaciaszek]

@dangkhoaphung if the issue is added to an active backlog, it will be assigned a milestone. We're now prioritising tasks for the upcoming release and will circle back to non-blocking issues later on. However, feel free to create a PR, and we will definitely review it.

[OlgaMaciaszek]

Initial fix reverted due to #4275. Reopening.

[impactCn]

Spring Cloud eureka client 4.1.3
Spring Boot 3.3.1

hi, I upgraded to 4.1.3 and still have a lot of close_wait problems.
@OlgaMaciaszek

[impactCn]

Spring Cloud eureka client 4.1.3 Spring Boot 3.3.1

hi, I upgraded to 4.1.3 and still have a lot of close_wait problems. @OlgaMaciaszek

Is there any plan to fix this issue in which version?

[OlgaMaciaszek]

Yes, @impactCn. the initial fix had to be reverted due to regression. We'll need to work on another fix, however it's not yet assigned to a new backlog. We some higher priority task right now, but I will try to look into it next month.

[impactCn]

@OlgaMaciaszek I have a temporary solution to solve this problem, using webflux's webclient as the caller. I don't seem to have this problem in the test environment. I'll test it in the production environment tomorrow and take a look.

[OlgaMaciaszek]

Sounds good @impactCn. Please take a look at that regression linked above as well to see if your solution is not doing the same. If it works fine, please feel free to create a PR or share the workaround.

[impactCn]

eureka:
  client:
    webclient:
      enabled: true

@OlgaMaciaszek
I used the above configuration and just observed it in the production environment for 30 minutes. There is no close_wait problem.
And, I'll put it to tomorrow and see if there will be any problems with close_wait.

[OlgaMaciaszek]

Ah, ok - that's a different thing - it's just using WebClient instead of HttpClient, but please verify if all your use-cases work, as the scope of integration (i.e. TransportFactories implementation) was not full for WebClient.