WebClient sendForm force multipart instead of urlencoded when deployed on Openshift

[joGrim65]

Questions

The question is a bit strange.
I compiled my app with vertx 4.5.10.
Naturally, it works fine on local dev env (Eclipse).
I built a Docker image (openjdk-17) and run it in a Docker container, both local and on a remote server. It worked fine.
The same image deployed instead on an Openshift 4.x platform gives me errors when WebClient tries to do a "sendForm(Map)", with the following error:

{"error.stacktrace":"java.lang.IllegalArgumentException\n\tat io.vertx.ext.web.client.impl.MultipartFormUpload.run(MultipartFormUpload.java:142)\n\tat io.vertx.ext.web.client.impl.HttpContext.lambda$handleCreateRequest$3(HttpContext.java:464)\n\tat io.vertx.core.impl.future.FutureImpl$4.onSuccess(FutureImpl.java:176)\n\tat io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:66)\n\tat io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:259)\n\tat io.vertx.core.Promise.complete(Promise.java:66)\n\tat io.vertx.ext.web.client.impl.HttpContext.handleSendRequest(HttpContext.java:564)\n\tat io.vertx.ext.web.client.impl.HttpContext.execute(HttpContext.java:375)\n\tat io.vertx.ext.web.client.impl.HttpContext.next(HttpContext.java:362)\n\tat io.vertx.ext.web.client.impl.HttpContext.fire(HttpContext.java:329)\n\tat io.vertx.ext.web.client.impl.HttpContext.sendRequest(HttpContext.java:232)\n\tat io.vertx.ext.web.client.impl.HttpContext.lambda$handleCreateRequest$6(HttpContext.java:496)\n\tat io.vertx.core.impl.future.FutureImpl$4.onSuccess(FutureImpl.java:176)\n\tat io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:66)\n\tat io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:259)\n\tat io.vertx.core.http.impl.HttpClientImpl.lambda$null$3(HttpClientImpl.java:392)\n\tat io.vertx.core.impl.future.FutureImpl$4.onSuccess(FutureImpl.java:176)\n\tat io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:66)\n\tat io.vertx.core.impl.future.FutureImpl.addListener(FutureImpl.java:231)\n\tat io.vertx.core.impl.future.FutureImpl.onComplete(FutureImpl.java:199)\n\tat io.vertx.core.http.impl.Http1xClientConnection.createStream(Http1xClientConnection.java:1269)\n\tat io.vertx.core.http.impl.HttpClientImpl.lambda$doRequest$4(HttpClientImpl.java:372)\n\tat io.vertx.core.net.impl.pool.Endpoint.lambda$getConnection$0(Endpoint.java:52)\n\tat io.vertx.core.http.impl.ClientHttpEndpointBase.lambda$requestConnection$0(ClientHttpEndpointBase.java:46)\n\tat io.vertx.core.http.impl.SharedClientHttpStreamEndpoint$Request.handle(SharedClientHttpStreamEndpoint.java:162)\n\tat io.vertx.core.http.impl.SharedClientHttpStreamEndpoint$Request.handle(SharedClientHttpStreamEndpoint.java:123)\n\tat io.vertx.core.impl.ContextImpl.emit(ContextImpl.java:328)\n\tat io.vertx.core.impl.ContextImpl.emit(ContextImpl.java:321)\n\tat io.vertx.core.net.impl.pool.SimpleConnectionPool$LeaseImpl.emit(SimpleConnectionPool.java:714)\n\tat io.vertx.core.net.impl.pool.SimpleConnectionPool$ConnectSuccess$2.run(SimpleConnectionPool.java:337)\n\tat io.vertx.core.net.impl.pool.Task.runNextTasks(Task.java:43)\n\tat io.vertx.core.net.impl.pool.CombinerExecutor.submit(CombinerExecutor.java:91)\n\tat io.vertx.core.net.impl.pool.SimpleConnectionPool.execute(SimpleConnectionPool.java:244)\n\tat io.vertx.core.net.impl.pool.SimpleConnectionPool.lambda$connect$2(SimpleConnectionPool.java:256)\n\tat io.vertx.core.http.impl.SharedClientHttpStreamEndpoint.lambda$connect$2(SharedClientHttpStreamEndpoint.java:102)\n\tat io.vertx.core.impl.future.FutureImpl$4.onSuccess(FutureImpl.java:176)\n\tat io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:66)\n\tat io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:259)\n\tat io.vertx.core.impl.future.Composition$1.onSuccess(Composition.java:62)\n\tat io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:66)\n\tat io.vertx.core.impl.future.FutureImpl.addListener(FutureImpl.java:231)\n\tat io.vertx.core.impl.future.Composition.onSuccess(Composition.java:43)\n\tat io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:66)\n\tat io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:259)\n\tat io.vertx.core.Promise.complete(Promise.java:66)\n\tat io.vertx.core.net.impl.NetClientImpl.lambda$connected$9(NetClientImpl.java:343)\n\tat io.vertx.core.net.impl.VertxHandler.setConnection(VertxHandler.java:82)\n\tat io.vertx.core.net.impl.VertxHandler.handlerAdded(VertxHandler.java:88)\n\tat io.netty.channel.AbstractChannelHandlerContext.callHandlerAdded(AbstractChannelHandlerContext.java:1130)\n\tat io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:608)\n\tat io.netty.channel.DefaultChannelPipeline.addLast(DefaultChannelPipeline.java:222)\n\tat io.netty.channel.DefaultChannelPipeline.addLast(DefaultChannelPipeline.java:194)\n\tat io.vertx.core.net.impl.NetClientImpl.connected(NetClientImpl.java:345)\n\tat io.vertx.core.net.impl.NetClientImpl.lambda$connectInternal2$3(NetClientImpl.java:307)\n\tat io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:270)\n\tat io.vertx.core.net.impl.ChannelProvider$1.userEventTriggered(ChannelProvider.java:123)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:398)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:376)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:368)\n\tat io.netty.handler.ssl.SslHandler.setHandshakeSuccess(SslHandler.java:1938)\n\tat io.netty.handler.ssl.SslHandler.wrapNonAppData(SslHandler.java:997)\n\tat io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1509)\n\tat io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1336)\n\tat io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1385)\n\tat io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)\n\tat io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1407)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:918)\n\tat io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.base/java.lang.Thread.run(Thread.java:840)","location.line_number":"?","location.method":"?","@timestamp":"2024-10-31T14:40:41.666+01:00","error.type":"java.lang.IllegalArgumentException","application":"r4c-gateway","location.file":"?","level":"ERROR","thread_name":"vert.x-eventloop-thread-1","location.class":"?","logger_name":"io.vertx.core.impl.ContextImpl","message":"Unhandled exception"}

Version

4.5.10, 4.5.11

Context

The exception happens when WebClient.sendForm(Map) is used.
The map contains a bunch of parameters (less than 10).
The service I try to call is exposed as @consume("application/x-www-form-urlencoded") and is never reached.
It seems the client is considering the form as "multipart" instead of "application/x-www-form-urlencoded".
I tryed forcing "content-type: application/x-www-form-urlencoded" header before "sendForm", but I had the same error.
All is working in my app, but not webclient call using "sendForm".
Could be something related to "netty native libs"?

The involved code was working fine since the early versions 4.0.x of vert.x and on Openshift 3 and 4.
The last working and deployed one was using vert.x 4.5.7.

I tried recompiling my app with 4.5.8... and it works.
It seems something related to 4.5.10 (I have not tried 4.5.9).
Recently I tried 4.5.11, hoping the problem would be solved... but it still happens.

Do you have a reproducer?

no

Extra

The starting image is "ubi9-openjdk17-jre:17" (on that, the app built on vertx 4.5.7 was perfectly working in many Openshift production env)

[tsegismont]

See eclipse-vertx/vert.x#5381 (comment)

Please provide more info about the case.

[joGrim65]

This is the snippet of code I use for a POST:

private Future sendPostOrPut(HttpMethod method, Object data) {

// get endpoitnurl to use
String endpoint_url = getEndpointUrl(encodePathParameter(pathParm));
logger.info("Forwarding URL: "+endpoint_url);
HttpRequest<Buffer> cliRequest = client
                   .requestAbs(method,endpoint_url);

// add headers, if any (generally "Authorization")
if(headersMap != null) addHeaders(cliRequest);

Promise<String> promise = Promise.promise();
Future<String> future = promise.future();
// send data
logger.info("Sending...");

try {
  Map<String, Object> options = new HashMap<String, Object>();
       
  // payload as String (generally as Json)
  if(data instanceof String) {
    ...
  }
  // payload as JsonObject
  else if(data instanceof JsonObject) {
    ...
  }
  // body payload as Form (application/x-www-form-urlencoded)
  else if(data instanceof Map) {
    MultiMap form = MultiMap.caseInsensitiveMultiMap();
    form.addAll((Map)data);
    Future<HttpResponse<Buffer>> futureResponse = cliRequest.putHeader(HEADER_CONTENT_TYPE,HEADER_VALUE_FORM_URLENCODED)
                                                            .sendForm(form);
    handleRequestResult(futureResponse, endpoint_url, promise, options);
  }
  // payload as Json (from a generic POJO)
  else {
    ...
  }
  
}
catch(Exception e) {
  logger.error("Error in sending", e);
  promise.fail(e);
}
return future;

}

Note: I added "putHeader" later, to see if forcing it the proble disappear...

Simply I call it with a variable number of string parameters (from 2 to 6) and I call a service which consume them as "x-www-form-urlencoded".
Nothing else... and until vert.x 4.5.8 all worked fine on all the running env where I deployed.
The Openshift env where I have the problem, is the same where a 4.5.7 built app was working fine.
I have only 1 POD active for the involved app, and only 1 vert.x instance.
My suspect is on "netty native", because could be the only layer could impact differently on various target deploy environments... but this is only my opinion... I don't know the code behind...
Are there other info I can supply to help you to investigate?

[tsegismont]

Yes, I don't believe invoking it's a problem related to headers set before invoking sendForm.

As far as I understand, the problem happens because the context captured here:

vertx-web/vertx-web-client/src/main/java/io/vertx/ext/web/client/impl/HttpContext.java

Line 406 in 6020284

context = client.vertx().getOrCreateContext();

is not the same as the context captured in the HttpClient when invoking:

vertx-web/vertx-web-client/src/main/java/io/vertx/ext/web/client/impl/HttpContext.java

Line 493 in 6020284

client.request(requestOptions)

To my knowledge, this could happen only if there are client interceptors. Can you show how the WebClient is created? Is it session aware? OAuth2 WebClient?

[joGrim65]

The WebClient is set in the following way:

WebClient client = WebClient.create(Vertx.currentContext().owner(), options);

Some remarks:

• The client management was working fine since at least 3 years
  In that period, the only changes in that code were due only on deprecation to be replaced, if any
• The application works fine locally (Eclipse + Windows 10/11) on different laptops.
• The application is deployed building a docker image.
  The same image is used to be run both in a pure Docker environment (Windows and Linux) and Openshift 4.x one.
  Both Docker and Openshift environments used for test were unchanged in the last 2 years
• The application worked fine in all the environments until vert.x 4.5.10
• Inside the application there is no context change
• The services invoked by WebClient are all stateless authorized by a JWT token
• Inside the application there is no use of "multipart/form" (the most are "application/json" and only a bunch of them use "application/x-www-form-urlencoded"... the ones failing under Openshift)

Considering the above remarks, it's very difficult for me thinking to a problem at application level...