Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Max frame length of 65536 has been exceeded - Quarkus 3 #43381

Closed
dometec opened this issue Sep 19, 2024 · 16 comments · Fixed by #43435
Closed

Max frame length of 65536 has been exceeded - Quarkus 3 #43381

dometec opened this issue Sep 19, 2024 · 16 comments · Fixed by #43435
Labels
area/websockets kind/bug Something isn't working
Milestone

Comments

@dometec
Copy link

dometec commented Sep 19, 2024

Describe the bug

WebSocket Server ignores Max Frame Size after upgrade from Quarkus 1.x to 3.14.4.
In application.properties I set:

quarkus.websocket.max-frame-size=2097152

and get the exception:


2024-09-18 18:45:18,450 INFO  [com.vim.ngv.wss.TunnelWSS] (executor-thread-3) WebSocket tunnel error, session: lUIS0BLIvL7sX4SNrHaQrjlQCd3ld1B5W_yJT2FQ, error: Max frame length of 65536 has been exceeded..: io.netty.handler.codec.http.websocketx.CorruptedWebSocketFrameException: Max frame length of 65536 has been exceeded.
        at io.netty.handler.codec.http.websocketx.WebSocket08FrameDecoder.protocolViolation(WebSocket08FrameDecoder.java:427)
        at io.netty.handler.codec.http.websocketx.WebSocket08FrameDecoder.decode(WebSocket08FrameDecoder.java:288)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
...

Expected behavior

No exception, or a message with my max frame size setting in "Max frame length of 65536 has been exceeded".

Actual behavior

No response

How to Reproduce?

No response

Output of uname -a or ver

Linux localhost.localdomain 6.4.4-100.fc37.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Jul 19 17:06:05 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Output of java -version

openjdk version "21.0.1" 2023-10-17 LTS

Quarkus version or git rev

3.14.4

Build tool (ie. output of mvnw --version or gradlew --version)

Apache Maven 3.9.6

Additional information

I think this method miss to take the quarkus.websocket.* configuration...

private static HttpServerOptions createHttpServerOptions(
HttpBuildTimeConfig buildTimeConfig, HttpConfiguration httpConfiguration,
LaunchMode launchMode, List<String> websocketSubProtocols) {
if (!httpConfiguration.hostEnabled) {
return null;
}
// TODO other config properties
HttpServerOptions options = new HttpServerOptions();
int port = httpConfiguration.determinePort(launchMode);
options.setPort(port == 0 ? RANDOM_PORT_MAIN_HTTP : port);
HttpServerOptionsUtils.applyCommonOptions(options, buildTimeConfig, httpConfiguration, websocketSubProtocols);
return options;
}

@dometec dometec added the kind/bug Something isn't working label Sep 19, 2024
@geoand
Copy link
Contributor

geoand commented Sep 19, 2024

Thanks for reporting.

Going forward, all our efforts around WebSockets are focused on WebSockets Next. If you could try and see if the problem exists with that stack, it would be great.

cc @mkouba

@mkouba
Copy link
Contributor

mkouba commented Sep 19, 2024

Hm, the quarkus.websocket.max-frame-size is a WebSocket client config property so I'm not really sure it ever worked on the server.

The corresponding configuration properties in WS Next are quarkus.websockets-next.server.max-message-size and quarkus.websockets-next.client.max-message-size.

@geoand geoand added the triage/needs-feedback We are waiting for feedback. label Sep 19, 2024
@dometec
Copy link
Author

dometec commented Sep 19, 2024

@geoand ok I'll try.
@mkouba not in Quarkus 1.x see https://github.com/quarkusio/quarkus/pull/7861/files#top

@mkouba
Copy link
Contributor

mkouba commented Sep 19, 2024

@geoand ok I'll try. @mkouba not in Quarkus 1.x see https://github.com/quarkusio/quarkus/pull/7861/files#top

Interesting, it was removed (maybe unintentionally) in the "split" PR #7861.

In any case, the development of the original WS extension is discontinued. We're looking forward to your feedback about WS next! ;-)

@dometec
Copy link
Author

dometec commented Sep 23, 2024

@geoand , I can't try websocket-next right now due to #43434.
@mkouba submit a PR, I understand that the extension is discontinued, but until the new is in the experimental phase, I would like to fix the problem found.

Thanks!

@geoand geoand removed the triage/needs-feedback We are waiting for feedback. label Sep 23, 2024
@quarkus-bot quarkus-bot bot added this to the 3.16 - main milestone Sep 23, 2024
@etiennedv-mathu
Copy link

I have some feedback to give on quarkus-websockets-next.

We are running into the same issue.

Our application.properties has the following max-message-size's set:

quarkus.websockets-next.server.max-message-size=1048576
quarkus.websockets-next.client.max-message-size=1048576

Yet, we are still seeing the following error on our platform:

io.netty.handler.codec.http.websocketx.CorruptedWebSocketFrameException: Max frame length of 65536 has been exceeded.
	at io.netty.handler.codec.http.websocketx.WebSocket08FrameDecoder.protocolViolation(WebSocket08FrameDecoder.java:427)
	at io.netty.handler.codec.http.websocketx.WebSocket08FrameDecoder.decode(WebSocket08FrameDecoder.java:288)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1407)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:918)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:1583)

I am going to dig into the library now to see where this 65536 value is coming from in quarkus.websockets-next

@mkouba
Copy link
Contributor

mkouba commented Feb 12, 2025

Hi @etiennedv-mathu, the 65536 value comes from Vertx, see io.vertx.core.http.HttpServerOptions.DEFAULT_MAX_WEBSOCKET_FRAME_SIZE. A message could be assembled from multiple frames.

I think that we should add the quarkus.websockets-next.server.max-frame-size config property.

@geoand
Copy link
Contributor

geoand commented Feb 12, 2025

I think that we should add the quarkus.websockets-next.server.max-frame-size config property.

+1

@etiennedv-mathu
Copy link

Hmmm, are there any workarounds you can think of?

@mkouba
Copy link
Contributor

mkouba commented Feb 12, 2025

Hmmm, are there any workarounds you can think of?

Yes! You can provide your own HttpServerOptionsCustomizer (see also the one from WS next) and set the max frame size directly with io.vertx.core.http.HttpServerOptions.setMaxWebSocketFrameSize(int).

mkouba added a commit to mkouba/quarkus that referenced this issue Feb 12, 2025
- add quarkus.websockets-next.server.max-frame-size and
quarkus.websockets-next.client.max-frame-size config properties
- related to
quarkusio#43381 (comment)
@mkouba
Copy link
Contributor

mkouba commented Feb 12, 2025

@etiennedv-mathu FYI: #46233

@etiennedv-mathu
Copy link

Thank you very much.

@etiennedv-mathu
Copy link

@mkouba it doesn't look like adding a WebSocketHttpServerOptionsCustomizer has any effect.

Heres the WebSocketHttpServerOptionsCustomizer I setup:

package za.co.mathu.bear.connectors.ferret.planner;

import io.vertx.core.http.HttpServerOptions;
import jakarta.inject.Singleton;
import io.quarkus.vertx.http.HttpServerOptionsCustomizer;
import lombok.extern.slf4j.Slf4j;

@Slf4j
@Singleton
public class WebSocketHttpServerOptionsCustomizer  implements HttpServerOptionsCustomizer {
    @Override
    public void customizeHttpServer(HttpServerOptions options) {
        customize(options);
    }

    @Override
    public void customizeHttpsServer(HttpServerOptions options) {
        customize(options);
    }

    private void customize(HttpServerOptions options) {
        log.info("WebSocketHttpServerOptionsCustomizer.customize: Setting MaxWebSocketFrameSize");
        options.setMaxWebSocketFrameSize(1024*1024);
    }
}

From the logs we can see that even though it sets the MaxWebSocketFrameSize, it has no effect:

__  ____  __  _____   ___  __ ____  ______ 
 --/ __ \/ / / / _ | / _ \/ //_/ / / / __/ 
 -/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \   
--\___\_\____/_/ |_/_/|_/_/|_|\____/___/   
2025-02-12 15:09:55,098 INFO  [za.co.mat.bea.con.fer.pla.WebSocketHttpServerOptionsCustomizer] (main) WebSocketHttpServerOptionsCustomizer.customize: Setting MaxWebSocketFrameSize
2025-02-12 15:09:55,099 INFO  [za.co.mat.bea.con.fer.pla.WebSocketHttpServerOptionsCustomizer] (main) WebSocketHttpServerOptionsCustomizer.customize: Setting MaxWebSocketFrameSize
2025-02-12 15:09:55,223 INFO  [io.quarkus] (main) application-assembly 1.0.0-SNAPSHOT on JVM (powered by Quarkus 3.15.1) started in 2.489s. Listening on: http://0.0.0.0:8081
2025-02-12 15:09:55,225 INFO  [io.quarkus] (main) Profile prod activated. 
2025-02-12 15:09:55,226 INFO  [io.quarkus] (main) Installed features: [cdi, rest, rest-client, rest-client-jackson, rest-jackson, security, smallrye-context-propagation, smallrye-health, smallrye-jwt, vertx, websockets-next]
2025-02-12 15:10:09,308 INFO  [za.co.mat.bea.mar.uti.WebSocketHandler] (executor-thread-1) WebSocketHandler.onOpen
2025-02-12 15:10:09,490 INFO  [za.co.mat.bea.con.fer.pla.soc.FerretPlannerLessonWebSocketConnector$LessonEndpoint] (executor-thread-1) FerretPlannerLessonWebSocketConnector.LessonEndpoint.onOpen
2025-02-12 15:10:09,600 ERROR [za.co.mat.bea.con.fer.pla.soc.FerretPlannerLessonWebSocketConnector$LessonEndpoint] (executor-thread-1) FerretPlannerLessonWebSocketConnector.LessonEndpoint.onError: io.netty.handler.codec.http.websocketx.CorruptedWebSocketFrameException: Max frame length of 65536 has been exceeded.
	at io.netty.handler.codec.http.websocketx.WebSocket08FrameDecoder.protocolViolation(WebSocket08FrameDecoder.java:427)
	at io.netty.handler.codec.http.websocketx.WebSocket08FrameDecoder.decode(WebSocket08FrameDecoder.java:288)
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1407)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:918)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:1583)

2025-02-12 15:10:09,607 INFO  [za.co.mat.bea.con.fer.pla.soc.FerretPlannerLessonWebSocketConnector$LessonEndpoint] (executor-thread-2) FerretPlannerLessonWebSocketConnector.LessonEndpoint.onClose
2025-02-12 15:10:09,612 ERROR [za.co.mat.bea.con.fer.pla.soc.FerretPlannerLessonWebSocketConnector$LessonEndpoint] (executor-thread-3) FerretPlannerLessonWebSocketConnector.LessonEndpoint.onError: io.vertx.core.http.HttpClosedException: Connection was closed

2025-02-12 15:10:15,783 INFO  [za.co.mat.bea.mar.uti.WebSocketHandler] (executor-thread-2) WebSocketHandler.onClose

@mkouba
Copy link
Contributor

mkouba commented Feb 12, 2025

@mkouba it doesn't look like adding a WebSocketHttpServerOptionsCustomizer has any effect.

This is weird. I've tried to modify the test introduced in #46233 and it seems to work just fine. Could you provide more details about your app, endpoint class, etc.?

@etiennedv-mathu
Copy link

Sure:

For context: This app is is acting as a business logic layer. FerretPlanner is sending this app updates on the lesson model. The exception occurs when this app tries to access lessons that are too big. The initial message includes the entire Lesson model.

The following class is where we actually open the web socket.

@Slf4j
@Dependent
public abstract class WebSocketConnectionCreator<Connector, Model> {
    @Inject
    WebSocketConnector<Connector> connector;

    public abstract void forwardMessage(Model model, WebSocketClientConnection connection);

    public WebSocketClientConnection openStream(URI uri, JsonWebToken bearerToken, String pathParameterName, String pathParameter) throws UpgradeRejectedException {
        log.debug("WebSocketClientConnection.openStream Attempting to open a WebSocket connection");

        WebSocketClientConnection connection = connector
                .baseUri(uri)
                .pathParam(pathParameterName, pathParameter)
                .addHeader("Authorization", String.format("Bearer %s", bearerToken.getRawToken()))
                .connectAndAwait();

        log.debug("WebSocketClientConnection.openStream Connected to WebSocket");
        return connection;
    }
}

The following class is a "API" so to speak. it captures the messages and. sends them upwards via the event bus. This is also where the error is happening.

package za.co.mathu.bear.connectors.ferret.planner.socket;

import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import io.quarkus.websockets.next.*;
import io.vertx.core.http.UpgradeRejectedException;
import jakarta.enterprise.context.ApplicationScoped;
import jakarta.enterprise.context.RequestScoped;
import jakarta.inject.Inject;
import lombok.extern.slf4j.Slf4j;
import org.eclipse.microprofile.config.inject.ConfigProperty;
import org.eclipse.microprofile.jwt.JsonWebToken;
import org.eclipse.microprofile.rest.client.annotation.RegisterProvider;
import za.co.mathu.bear.connectors.ferret.shared.models.StreamMessage;
import za.co.mathu.bear.connectors.ferret.sockets.WebSocketConnectionCreator;
import za.co.mathu.bear.connectors.ferret.planner.FerretPlannerException;
import za.co.mathu.bear.connectors.ferret.planner.FerretPlannerExceptionMapper;
import za.co.mathu.bear.connectors.ferret.planner.models.Lessons;
import za.co.mathu.bear.connectors.ferret.planner.socket.events.LessonEventBus;

import java.net.URI;
import java.nio.charset.StandardCharsets;

@RegisterProvider(FerretPlannerExceptionMapper.class)
public interface FerretPlannerLessonWebSocketConnector {
    @Slf4j
    @ApplicationScoped
    @WebSocketClient(path = "/lesson/stream/{lessonId}")
    class LessonEndpoint {
        @Inject
        FerretPlannerLessonSocketCreator connector;

        @Inject
        private ObjectMapper mapper;


        @OnOpen
        void onOpen (WebSocketClientConnection connection) throws FerretPlannerException {
            log.info("FerretPlannerLessonWebSocketConnector.LessonEndpoint.onOpen");
        }

        @OnClose
        void onClose(WebSocketClientConnection connection) throws FerretPlannerException {
            log.info("FerretPlannerLessonWebSocketConnector.LessonEndpoint.onClose");
        }

        @OnError
        void onError(WebSocketClientConnection connection, Throwable throwable) throws FerretPlannerException {
            log.error("FerretPlannerLessonWebSocketConnector.LessonEndpoint.onError", throwable);

            throw new FerretPlannerException()
                    .setMessage(throwable.getMessage());
        }

        @OnBinaryMessage
        void onBinaryMessage(WebSocketClientConnection connection, byte[] message) throws FerretPlannerException {
            log.debug("FerretPlannerLessonWebSocketConnector.LessonEndpoint.onBinaryMessage");

            String messageString = new String(message, StandardCharsets.UTF_8);
            handleMessage(connection, messageString);
        }

        @OnTextMessage
        void onTextMessage(WebSocketClientConnection connection, String message) throws FerretPlannerException {
            log.debug("FerretPlannerLessonWebSocketConnector.LessonEndpoint.onStringMessage");

            handleMessage(connection, message);
        }

        private void handleMessage(WebSocketClientConnection connection, String message) throws FerretPlannerException {
            try {
                StreamMessage.StreamPayload payload = mapper.readValue(message, StreamMessage.StreamPayload.class);
                Lessons.LessonStream lessonStream;

                if (payload.getType().equals(StreamMessage.MessageFields.DATA)) {
                    Lessons.Lesson lesson = mapper.convertValue(payload.getMessage(), Lessons.Lesson.class);

                    lessonStream = new Lessons.LessonStream()
                            .setType(payload.getType())
                            .setPayload(lesson);
                } else {
                    lessonStream = new Lessons.LessonStream()
                            .setType(payload.getType())
                            .setMessage(payload.getMessage().toString());
                }
                connector.forwardMessage(lessonStream, connection);
            } catch (JsonProcessingException e) {
                throw new FerretPlannerException()
                        .setMessage(e.getMessage());
            }
        }
    }

    @Slf4j
    @RequestScoped
    class FerretPlannerLessonSocketCreator extends WebSocketConnectionCreator<LessonEndpoint, Lessons.LessonStream> {
        @ConfigProperty(name = "quarkus.rest-client.ferret-planner-api.url")
        URI uri;

        @Inject
        LessonEventBus eventBus;

        @Override
        public void forwardMessage(Lessons.LessonStream lessonStream, WebSocketClientConnection connection) {
            eventBus.publish("Lesson-message", lessonStream, connection);
        }

        public WebSocketClientConnection openStream(JsonWebToken bearerToken, String lessonId) throws FerretPlannerException {
            try {
                return super.openStream(uri, bearerToken, "lessonId", lessonId);
            } catch (UpgradeRejectedException e) {
                throw new FerretPlannerException()
                        .setStatus(e.getStatus())
                        .setMessage(e.getMessage());
            }
        }
    }
}

@mkouba
Copy link
Contributor

mkouba commented Feb 13, 2025

Sure:

For context: This app is is acting as a business logic layer. FerretPlanner is sending this app updates on the lesson model. The exception occurs when this app tries to access lessons that are too big. The initial message includes the entire Lesson model.

The following class is where we actually open the web socket.
...

Ah ok, so it's a WS next client. In that case, a HttpServerOptionsCustomizer has no effect and I'm afraid that I don't have a workaround for you because there's no way to customize the WebSocketClientOptions used for a WebSocketConnector. The good news is that #46233 will be backported and quarkus.websockets-next.client.max-frame-size should fix your problem.

I've also created #46244 so that it's possible to customize the WebSocketConnectOptions and WebSocketClientOptions used.

gsmet pushed a commit to gsmet/quarkus that referenced this issue Feb 14, 2025
- add quarkus.websockets-next.server.max-frame-size and
quarkus.websockets-next.client.max-frame-size config properties
- related to
quarkusio#43381 (comment)

(cherry picked from commit 1f82857)
gsmet pushed a commit to gsmet/quarkus that referenced this issue Feb 18, 2025
- add quarkus.websockets-next.server.max-frame-size and
quarkus.websockets-next.client.max-frame-size config properties
- related to
quarkusio#43381 (comment)

(cherry picked from commit 1f82857)
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
area/websockets kind/bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants