ClassNotFoundException is thrown when loading protocol resolvers from ForkJoinPool task

[raestio]

Hi, we have upgraded Spring Boot app version from 3.3.3 to 3.3.4 and now the SSL bundles cannot be loaded. See the full stack trace of the exception below.

I've found there have been some changes related to SSL bundles in #42119. I can see the key stores are being lazily loaded but not sure how (and if) it can change the class loader that doesn't see Base64ProtocolResolver.

We have integration @SpringBootTest tests that use the SSL bundles and they are successfully passing. So most probably the app must be compiled and packaged as we see those exceptions only when we run the app (built as a JAR) in Docker container. But I could still be missing something.

When I downgrade to 3.3.3, it works OK as expected.

I don't have a sample application that reproduces the issue yet.

The app runs on reactive stack.

java.lang.IllegalStateException: Unable to create key store: Could not load store from 'classpath:{CERT}.p12'
	at org.springframework.boot.ssl.jks.JksSslStoreBundle.createKeyStore(JksSslStoreBundle.java:96)
	at org.springframework.boot.ssl.jks.JksSslStoreBundle.lambda$new$0(JksSslStoreBundle.java:59)
	at org.springframework.util.function.SingletonSupplier.get(SingletonSupplier.java:106)
	at org.springframework.boot.ssl.jks.JksSslStoreBundle.getKeyStore(JksSslStoreBundle.java:65)
	at org.springframework.boot.ssl.DefaultSslManagerBundle.getKeyManagerFactory(DefaultSslManagerBundle.java:45)
	at org.springframework.boot.autoconfigure.web.reactive.function.client.ReactorClientHttpConnectorFactory$SslConfigurer.customizeSsl(ReactorClientHttpConnectorFactory.java:91)
	at org.springframework.util.function.ThrowingConsumer$1.acceptWithException(ThrowingConsumer.java:83)
	at org.springframework.util.function.ThrowingConsumer.accept(ThrowingConsumer.java:60)
	at org.springframework.util.function.ThrowingConsumer$1.accept(ThrowingConsumer.java:88)
	at reactor.netty.http.client.HttpClient.secure(HttpClient.java:1430)
	at org.springframework.boot.autoconfigure.web.reactive.function.client.ReactorClientHttpConnectorFactory$SslConfigurer.configure(ReactorClientHttpConnectorFactory.java:84)
	at org.springframework.boot.autoconfigure.web.reactive.function.client.ReactorNettyHttpClientMapper.lambda$of$1(ReactorNettyHttpClientMapper.java:65)
	at java.base/java.util.function.Function.lambda$andThen$1(Unknown Source)
	at java.base/java.util.function.Function.lambda$andThen$1(Unknown Source)
	at org.springframework.http.client.reactive.ReactorClientHttpConnector.createHttpClient(ReactorClientHttpConnector.java:125)
	at org.springframework.http.client.reactive.ReactorClientHttpConnector.connect(ReactorClientHttpConnector.java:142)
	at org.springframework.web.reactive.function.client.ExchangeFunctions$DefaultExchangeFunction.exchange(ExchangeFunctions.java:102)
	at org.springframework.web.reactive.function.client.DefaultWebClient$ObservationFilterFunction.filter(DefaultWebClient.java:737)
	at org.springframework.web.reactive.function.client.ExchangeFilterFunction.lambda$apply$2(ExchangeFilterFunction.java:73)
	at org.springframework.web.reactive.function.client.DefaultWebClient$DefaultRequestBodyUriSpec.lambda$exchange$11(DefaultWebClient.java:457)
	at reactor.core.publisher.MonoDeferContextual.subscribe(MonoDeferContextual.java:47)
	at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:76)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:165)
	at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onNext(FluxPeekFuseable.java:210)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
	at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2571)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:171)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:171)
	at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.request(FluxPeekFuseable.java:144)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.request(MonoFlatMap.java:194)
	at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.request(FluxPeekFuseable.java:144)
	at reactor.core.publisher.MonoFlatMap$FlatMapInner.onSubscribe(MonoFlatMap.java:291)
	at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onSubscribe(FluxPeekFuseable.java:178)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(MonoFlatMap.java:117)
	at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onSubscribe(FluxPeekFuseable.java:178)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:96)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:96)
	at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:55)
	at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:76)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:165)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
	at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2571)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:171)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.request(MonoFlatMap.java:194)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.request(FluxMapFuseable.java:360)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.request(MonoPeekTerminal.java:139)
	at reactor.core.publisher.FluxOnErrorReturn$ReturnSubscriber.request(FluxOnErrorReturn.java:153)
	at reactor.core.publisher.FluxFlatMap$FlatMapInner.onSubscribe(FluxFlatMap.java:968)
	at reactor.core.publisher.FluxOnErrorReturn$ReturnSubscriber.onSubscribe(FluxOnErrorReturn.java:95)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onSubscribe(MonoPeekTerminal.java:152)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onSubscribe(FluxMapFuseable.java:265)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(MonoFlatMap.java:117)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onSubscribe(FluxMapFuseable.java:96)
	at reactor.core.publisher.MonoJust.subscribe(MonoJust.java:55)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4576)
	at reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:202)
	at reactor.core.publisher.MonoFlatMap.subscribeOrReturn(MonoFlatMap.java:53)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4560)
	at reactor.core.publisher.FluxFlatMap$FlatMapMain.onNext(FluxFlatMap.java:430)
	at reactor.core.publisher.FluxFilter$FilterSubscriber.onNext(FluxFilter.java:113)
	at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onNext(FluxMap.java:224)
	at reactor.core.publisher.FluxUsingWhen$UsingWhenSubscriber.onNext(FluxUsingWhen.java:348)
	at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.innerNext(FluxConcatMapNoPrefetch.java:259)
	at reactor.core.publisher.FluxConcatMap$ConcatMapInner.onNext(FluxConcatMap.java:865)
	at reactor.core.publisher.FluxConcatMap$WeakScalarSubscription.request(FluxConcatMap.java:480)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2367)
	at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.onNext(FluxConcatMapNoPrefetch.java:202)
	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
	at reactor.core.publisher.FluxUsingWhen$UsingWhenSubscriber.onNext(FluxUsingWhen.java:348)
	at reactor.core.publisher.FluxFlatMap$FlatMapMain.tryEmit(FluxFlatMap.java:547)
	at reactor.core.publisher.FluxFlatMap$FlatMapInner.onNext(FluxFlatMap.java:988)
	at reactor.core.publisher.FluxUsingWhen$UsingWhenSubscriber.onNext(FluxUsingWhen.java:348)
	at reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.onNext(FluxContextWriteRestoringThreadLocals.java:118)
	at oracle.r2dbc.impl.OracleReactiveJdbcAdapter$1.onNext(OracleReactiveJdbcAdapter.java:783)
	at oracle.r2dbc.impl.AsyncLock$UsingConnectionSubscriber.onNext(AsyncLock.java:481)
	at reactor.core.publisher.StrictSubscriber.onNext(StrictSubscriber.java:89)
	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
	at reactor.core.publisher.FluxContextWriteRestoringThreadLocals$ContextWriteRestoringThreadLocalsSubscriber.onNext(FluxContextWriteRestoringThreadLocals.java:118)
	at org.reactivestreams.FlowAdapters$FlowToReactiveSubscriber.onNext(FlowAdapters.java:211)
	at oracle.jdbc.driver.PhasedPublisher$PhasedSubscription.lambda$emitNextItem$0(PhasedPublisher.java:403)
	at oracle.jdbc.driver.PhasedPublisher.handleOnNext(PhasedPublisher.java:267)
	at oracle.jdbc.driver.PhasedPublisher$PhasedSubscription.lambda$emitNextItem$1(PhasedPublisher.java:401)
	at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
Caused by: java.lang.IllegalStateException: Could not load store from 'classpath:{CERT}.p12'
	at org.springframework.boot.ssl.jks.JksSslStoreBundle.loadKeyStore(JksSslStoreBundle.java:125)
	at org.springframework.boot.ssl.jks.JksSslStoreBundle.createKeyStore(JksSslStoreBundle.java:91)
	... 88 common frames omitted
Caused by: java.lang.IllegalArgumentException: Unable to instantiate factory class [org.springframework.boot.io.Base64ProtocolResolver] for factory type [org.springframework.core.io.ProtocolResolver]
	at org.springframework.core.io.support.SpringFactoriesLoader$FailureHandler.lambda$throwing$0(SpringFactoriesLoader.java:647)
	at org.springframework.core.io.support.SpringFactoriesLoader$FailureHandler.lambda$handleMessage$3(SpringFactoriesLoader.java:671)
	at org.springframework.core.io.support.SpringFactoriesLoader.instantiateFactory(SpringFactoriesLoader.java:231)
	at org.springframework.core.io.support.SpringFactoriesLoader.load(SpringFactoriesLoader.java:206)
	at org.springframework.core.io.support.SpringFactoriesLoader.load(SpringFactoriesLoader.java:142)
	at org.springframework.boot.io.ApplicationResourceLoader.<init>(ApplicationResourceLoader.java:53)
	at org.springframework.boot.io.ApplicationResourceLoader.<init>(ApplicationResourceLoader.java:41)
	at org.springframework.boot.ssl.jks.JksSslStoreBundle.loadKeyStore(JksSslStoreBundle.java:119)
	... 89 common frames omitted
Caused by: java.lang.ClassNotFoundException: org.springframework.boot.io.Base64ProtocolResolver
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source)
	at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Unknown Source)
	at java.base/java.lang.Class.forName(Unknown Source)
	at org.springframework.util.ClassUtils.forName(ClassUtils.java:304)
	at org.springframework.core.io.support.SpringFactoriesLoader.instantiateFactory(SpringFactoriesLoader.java:224)
	... 94 common frames omitted

[wilkinsona]

Thanks for the report. The ApplicationResourceLoader is being created in such a way that the thread context class loader will be used to load its protocol resolvers. It looks like this is ClassLoaders$AppClassLoader which, if you're running your application with java -jar or java -cp … org.springframework.boot.loader.launch.JarLauncher won't have org.springframework.boot.io.Base64ProtocolResolver on its classpath. It sounds like you're using java -jar in your Docker container. Can you confirm that's the case?

In the absence of a sample that reproduces the problem, you could test the above theory by patching line 119 of JksSslStoreBundle:

Resource resource = new ApplicationResourceLoader(getClass().getClassLoader()).getResource(location);

[raestio]

Thank you for the hint. Our Dockerfile looked like this:

docs Spring Boot 3.2.0 - Dockerfiles

...
COPY --from=builder application/dependencies/ ./
COPY --from=builder application/spring-boot-loader/ ./
COPY --from=builder application/snapshot-dependencies/ ./
COPY --from=builder application/application/ ./
ENTRYPOINT ["java", "org.springframework.boot.loader.launch.JarLauncher"]

So java org.springframework.boot.loader.launch.JarLauncher.

But I have updated the Dockerfile based on the latest docs Spring Boot 3.3.4 - Dockerfiles / Source and the issue is gone, works OK. Thank you for the help!

Now I'm thinking, I can see the docs for the tools mode were updated in version 3.3.0 - #40094 - which I missed. I believe this feature https://github.com/spring-projects/spring-boot/wiki/Spring-Boot-3.3-Release-Notes#cds-support was the related one. Does it mean the changes made for the CDS feature caused some incompatibility with the launching the extracted JAR in an older way before 3.3.0 as I had it before? Because now I'm still running the extracted JAR but with java -jar application.jar (as stated in the latest docs) and it works.
I'm just trying to piece together the points from your comment and find out why it works. I guess a different class loader is used.

[wilkinsona]

I guess a different class loader is used.

Yes, that's right. Since 3.3, the extracted jar uses a Class-Path manifest header to reference all of the application's dependencies so everything's loaded by the app class loader. As you suspected, this was primarily for CDS where taking Spring Boot's custom class loader out of the picture significantly improves the hit rate against the CDS archive and increases the startup time benefits.

[wilkinsona]

There are several places in 3.3.x that are susceptible to this problem:

spring-boot/spring-boot-project/spring-boot/src/main/java/org/springframework/boot/convert/StringToFileConverter.java

Line 37 in da1edd3

Resource resource = new ApplicationResourceLoader().getResource(source);

spring-boot/spring-boot-project/spring-boot/src/main/java/org/springframework/boot/logging/java/JavaLoggingSystem.java

Line 106 in da1edd3

Resource resource = new ApplicationResourceLoader().getResource(location);

spring-boot/spring-boot-project/spring-boot/src/main/java/org/springframework/boot/logging/log4j2/Log4J2LoggingSystem.java

Line 280 in da1edd3

Resource resource = new ApplicationResourceLoader().getResource(location);

spring-boot/spring-boot-project/spring-boot/src/main/java/org/springframework/boot/logging/logback/LogbackLoggingSystem.java

Line 254 in da1edd3

Resource resource = new ApplicationResourceLoader().getResource(location);

spring-boot/spring-boot-project/spring-boot/src/main/java/org/springframework/boot/ssl/jks/JksSslStoreBundle.java

Line 119 in da1edd3

Resource resource = new ApplicationResourceLoader().getResource(location);

spring-boot/spring-boot-project/spring-boot/src/main/java/org/springframework/boot/ssl/pem/PemContent.java

Line 122 in da1edd3

Resource resource = new ApplicationResourceLoader().getResource(content);

spring-boot/spring-boot-project/spring-boot-autoconfigure/src/main/java/org/springframework/boot/autoconfigure/ssl/BundleContentProperty.java

Line 73 in da1edd3

return new ApplicationResourceLoader().getResource(this.value);

In addition, on main, there is also the following:

spring-boot/spring-boot-project/spring-boot-autoconfigure/src/main/java/org/springframework/boot/autoconfigure/couchbase/CouchbaseAutoConfiguration.java

Line 120 in f908fcd

Resource resource = new ApplicationResourceLoader().getResource(jks.getLocation());

I think we should probably change each of them to use getClass().getClassLoader() as the resource loader's class loader but I'd like to review them with the rest of the team in case I'm overlooking something.

[philwebb]

Looking into this and I wonder if we should have a ApplicationResourceLoader constructor that allows us to specify a different classloader for the SpringFactoriesLoader? I think that might help bring StringToFileConverter at least closer to the older code.

For this failure specifically, I feel like we somehow need to set the classloader that gets used on the actual SslBundle.