cause=io.netty.handler.codec.http2.Http2Exception: Header size exceeded max allowed size (10240) #2935

cr22rc · 2017-04-21T12:54:32Z

Please answer these questions before submitting your issue.

1.2.0

java.runtime.name = OpenJDK Runtime Environment
12:22:18 java.runtime.version = 1.8.0_121-8u121-b13-0ubuntu1.16.04.2-b13
12:22:18 java.specification.name = Java Platform API Specification
12:22:18 java.specification.vendor = Oracle Corporation
12:22:18 java.specification.version = 1.8
12:22:18 java.vendor = Oracle Corporation
12:22:18 java.vendor.url = http://java.oracle.com/
12:22:18 java.vendor.url.bug = http://bugreport.sun.com/bugreport/
12:22:18 java.version = 1.8.0_121
12:22:18 java.vm.info = mixed mode
12:22:18 java.vm.name = OpenJDK 64-Bit Server VM
12:22:18 java.vm.specification.name = Java Virtual Machine Specification
12:22:18 java.vm.specification.vendor = Oracle Corporation
12:22:18 java.vm.specification.version = 1.8
12:22:18 java.vm.vendor = Oracle Corporation
12:22:18 java.vm.version = 25.121-b13
12:22:18 line.separator = \

grpc async call There really isn't any special options set on the channelbuilder
except usePlaintext it does have that set to true but a the moment not sure why.

The scenario this is running on works for all systems I'v but on these vms.
Ubuntu vm 16.04.

Is there some way to increase that size.
I can't imagine that the packet size would be any different between these system .. it's running the same test. Is there any other reason this exception might happen?

cr22rc · 2017-04-21T12:56:19Z

00:04:13 2017-04-21 00:04:13 DEBUG Peer:144 - sendProposalAsync name: peer1, url: grpc://localhost:7056
00:04:13 2017-04-21 00:04:13 DEBUG NettyClientHandler:147 - 
00:04:13 ----------------OUTBOUND--------------------
00:04:13 [id: 0xa601ff3c, L:/127.0.0.1:57166 - R:localhost/127.0.0.1:7056] SETTINGS: ack=false, settings={ENABLE_PUSH=0, MAX_CONCURRENT_STREAMS=0, INITIAL_WINDOW_SIZE=1048576, MAX_HEADER_LIST_SIZE=8192}
00:04:13 ------------------------------------
00:04:13 2017-04-21 00:04:13 DEBUG NettyClientHandler:147 - 
00:04:13 ----------------OUTBOUND--------------------
00:04:13 [id: 0xa601ff3c, L:/127.0.0.1:57166 - R:localhost/127.0.0.1:7056] WINDOW_UPDATE: streamId=0, windowSizeIncrement=983041
00:04:13 ------------------------------------
00:04:13 2017-04-21 00:04:13 DEBUG NettyClientHandler:147 - 
00:04:13 ----------------INBOUND--------------------
00:04:13 [id: 0xa601ff3c, L:/127.0.0.1:57166 - R:localhost/127.0.0.1:7056] SETTINGS: ack=false, settings={}
00:04:13 ------------------------------------
00:04:13 2017-04-21 00:04:13 DEBUG NettyClientHandler:147 - 
00:04:13 ----------------OUTBOUND--------------------
00:04:13 [id: 0xa601ff3c, L:/127.0.0.1:57166 - R:localhost/127.0.0.1:7056] SETTINGS: ack=true
00:04:13 ------------------------------------
00:04:13 2017-04-21 00:04:13 DEBUG NettyClientHandler:147 - 
00:04:13 ----------------INBOUND--------------------
00:04:13 [id: 0xa601ff3c, L:/127.0.0.1:57166 - R:localhost/127.0.0.1:7056] WINDOW_UPDATE: streamId=0, windowSizeIncrement=983025
00:04:13 ------------------------------------
00:04:13 2017-04-21 00:04:13 DEBUG NettyClientHandler:147 - 
00:04:13 ----------------INBOUND--------------------
00:04:13 [id: 0xa601ff3c, L:/127.0.0.1:57166 - R:localhost/127.0.0.1:7056] SETTINGS: ack=true
00:04:13 ------------------------------------
00:04:13 2017-04-21 00:04:13 DEBUG NettyClientHandler:147 - 
00:04:13 ----------------OUTBOUND--------------------
00:04:13 [id: 0xa601ff3c, L:/127.0.0.1:57166 - R:localhost/127.0.0.1:7056] HEADERS: streamId=3, headers=GrpcHttp2OutboundHeaders[:authority: localhost:7056, :path: /protos.Endorser/ProcessProposal, :method: POST, :scheme: http, content-type: application/grpc, te: trailers, user-agent: grpc-java-netty/1.2.0, grpc-accept-encoding: gzip, grpc-census-bin: ], streamDependency=0, weight=16, exclusive=false, padding=0, endStream=false
00:04:13 ------------------------------------
00:04:13 2017-04-21 00:04:13 DEBUG NettyClientHandler:147 - 
00:04:13 ----------------OUTBOUND--------------------
00:04:13 [id: 0xa601ff3c, L:/127.0.0.1:57166 - R:localhost/127.0.0.1:7056] DATA: streamId=3, padding=0, endStream=true, length=14657, bytes=000000393c0af0710ad1060a5e080310011a0c08fd98e5c70510c0faafb7022a4038343436323166363436633037373864353432666238373861626139626563...
00:04:13 ------------------------------------
00:04:14 2017-04-21 00:04:13 DEBUG NettyClientHandler:147 - 
00:04:14 ----------------OUTBOUND--------------------
00:04:14 [id: 0xa601ff3c, L:/127.0.0.1:57166 - R:localhost/127.0.0.1:7056] GO_AWAY: lastStreamId=0, errorCode=1, length=45, bytes=4865616465722073697a65206578636565646564206d617820616c6c6f7765642073697a652028313032343029
00:04:14 ------------------------------------
00:04:14 2017-04-21 00:04:13 DEBUG Http2ConnectionHandler:836 - [id: 0xa601ff3c, L:/127.0.0.1:57166 - R:localhost/127.0.0.1:7056] Sent GOAWAY: lastStreamId '0', errorCode '1', debugData 'Header size exceeded max allowed size (10240)'. Forcing shutdown of the connection.
00:04:14 2017-04-21 00:04:14 ERROR Chain:2105 - Sending proposal to peer1 failed because of gRPC failure=Status{code=INTERNAL, description=null, cause=io.netty.handler.codec.http2.Http2Exception: Header size exceeded max allowed size (10240)
00:04:14 	at io.netty.handler.codec.http2.Http2Exception.connectionError(Http2Exception.java:85)
00:04:14 	at io.netty.handler.codec.http2.Http2CodecUtil.headerListSizeExceeded(Http2CodecUtil.java:264)
00:04:14 	at io.netty.handler.codec.http2.DefaultHttp2FrameReader$HeadersBlockBuilder.headerSizeExceeded(DefaultHttp2FrameReader.java:701)
00:04:14 	at io.netty.handler.codec.http2.DefaultHttp2FrameReader$HeadersBlockBuilder.addFragment(DefaultHttp2FrameReader.java:716)
00:04:14 	at io.netty.handler.codec.http2.DefaultHttp2FrameReader$2.processFragment(DefaultHttp2FrameReader.java:485)
00:04:14 	at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readHeadersFrame(DefaultHttp2FrameReader.java:495)
00:04:14 	at io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:253)
00:04:14 	at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:160)
00:04:14 	at io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41)
00:04:14 	at io.grpc.netty.FixedHttp2ConnectionDecoder.decodeFrame(FixedHttp2ConnectionDecoder.java:119)
00:04:14 	at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:341)
00:04:14 	at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:401)
00:04:14 	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411)
00:04:14 	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248)
00:04:14 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
00:04:14 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
00:04:14 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)
00:04:14 	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
00:04:14 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
00:04:14 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
00:04:14 	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
00:04:14 	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:129)
00:04:14 	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:642)
00:04:14 	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:565)
00:04:14 	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:479)
00:04:14 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441)
00:04:14 	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
00:04:14 	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
00:04:14 	at java.lang.Thread.run(Thread.java:745)

carl-mastrangelo · 2017-04-21T17:10:44Z

You can set this, but you will have to use NettyChannelBuilder. It is currently not a stable API, so you may need to change in the future as we modify it.

cr22rc · 2017-04-23T15:05:41Z

@carl-mastrangelo I'm really new to grpc can you give a quick outline of how to do that ? Option on channelBuilder?
Or a pointer to an example if available ?
Thank you!

cr22rc · 2017-04-23T18:45:04Z

Another question can you tell if this issue is with the outbound request or the inbound ?
I suspect inbound but still new to this.

denisgaebler · 2017-04-24T08:39:38Z

Are you connecting bigendian and littleendian systems, e.g. are you using a hyperledger on s390x?
Then there is an endianess problem fixed with netty 4.1.10, but it cannot be used because grpc-java only supports 4.1.8.

cr22rc · 2017-04-24T11:47:43Z

All on a local box. Client connecting to servers running in docker. hopefully no big/little endian issue there :)

cr22rc · 2017-04-24T11:48:36Z

From that trance is it due to outbound or inbound ?

carl-mastrangelo · 2017-04-24T15:49:47Z

@cr22rc See http://www.grpc.io/grpc-java/javadoc/io/grpc/netty/NettyChannelBuilder.html#maxHeaderListSize-int-

cr22rc · 2017-04-26T07:42:37Z

Thanks what had me confused about that was is states "ListSize" but I'll look into it.
Still looking at the trace provided is there any way to know for sure if it inbound/outbound having the issue?

ejona86 · 2017-04-26T16:07:28Z

@cr22rc, you're right, the documentation wasn't very clear... unless you were already intimately familiar with HTTP/2. I created #2948 to try to make it better.

cvkumar · 2017-06-08T23:15:54Z

Hey, I am getting a similar, but not quite the same issue. I am uses gRPC-java 1.3.0 When I build the a channel I set the max header size to be very high.

originChannel = NettyChannelBuilder.forAddress(this.configuration.getHost(), this.configuration.getPort()).maxHeaderListSize(100000)
          .sslContext(GrpcSslContexts.forClient().build()).build();

However, when I am building a lot of streams stub.send(streamObserver); I eventually hit the following error.

WARNING: Publish stream closing because failed with error: Status{code=INTERNAL, description=null, ****cause=io.netty.handler.codec.http2.Http2Exception$HeaderListSizeException: Header size exceeded max allowed size (8192)****
	at io.netty.handler.codec.http2.Http2Exception.headerListSizeError(Http2Exception.java:169)
	at io.netty.handler.codec.http2.Http2CodecUtil.headerListSizeExceeded(Http2CodecUtil.java:252)
	at io.netty.handler.codec.http2.internal.hpack.Encoder.encodeHeadersEnforceMaxHeaderListSize(Encoder.java:128)
	at io.netty.handler.codec.http2.internal.hpack.Encoder.encodeHeaders(Encoder.java:112)
	at io.netty.handler.codec.http2.DefaultHttp2HeadersEncoder.encodeHeaders(DefaultHttp2HeadersEncoder.java:69)
	at io.netty.handler.codec.http2.DefaultHttp2FrameWriter.writeHeadersInternal(DefaultHttp2FrameWriter.java:435)
	at io.netty.handler.codec.http2.DefaultHttp2FrameWriter.writeHeaders(DefaultHttp2FrameWriter.java:200)
	at io.netty.handler.codec.http2.Http2OutboundFrameLogger.writeHeaders(Http2OutboundFrameLogger.java:60)
	at io.netty.handler.codec.http2.DefaultHttp2ConnectionEncoder.writeHeaders(DefaultHttp2ConnectionEncoder.java:188)
	at io.netty.handler.codec.http2.DecoratingHttp2FrameWriter.writeHeaders(DecoratingHttp2FrameWriter.java:52)
	at io.netty.handler.codec.http2.StreamBufferingEncoder.writeHeaders(StreamBufferingEncoder.java:157)
	at io.netty.handler.codec.http2.StreamBufferingEncoder.writeHeaders(StreamBufferingEncoder.java:141)
	at io.grpc.netty.NettyClientHandler.createStream(NettyClientHandler.java:426)
	at io.grpc.netty.NettyClientHandler.write(NettyClientHandler.java:240)
	at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:739)
	at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:731)
	at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:817)
	at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:724)
	at io.netty.channel.DefaultChannelPipeline.write(DefaultChannelPipeline.java:1022)
	at io.netty.channel.AbstractChannel.write(AbstractChannel.java:291)
	at io.grpc.netty.WriteQueue.flush(WriteQueue.java:127)
	at io.grpc.netty.WriteQueue.access$000(WriteQueue.java:47)
	at io.grpc.netty.WriteQueue$1.run(WriteQueue.java:59)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:445)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
	at java.lang.Thread.run(Thread.java:745)
}

It seems that io.grpc.netty.NettyClientHandler.createStream has an Encoder class io.netty.handler.codec.http2.internal.hpack.Encoder.encodeHeaders(Encoder.java:112) which I can't change the maximum header size allowed.

Is there a way for me to change the maximum header size for creation of streams, not just channels?

P.s. Still a little new to this gRPC-java thing

ejona86 · 2017-06-09T15:30:15Z

@cvkumar, maxHeaderListSize(100000) configures the receiving limit. In your case you are sending, which is determined by the remote side.

cvkumar · 2017-06-14T21:29:17Z

@ejona86 I see. So you mean the server side is running into this error? That is interesting.... To give you some context, we are trying to update our gRPC java version from 1.0.0 to 1.3.0. But when we do some of our tests break with this error. Do you know of a reason why upgrading our version would cause this?

ejona86 · 2017-06-19T20:19:16Z

@cvkumar, I don't know why upgrading would trigger the behavior difference. There were changes (like #2350), but I can't pin the exact behavior change.

ejona86 · 2017-07-27T19:14:30Z

Closing. Unknown why upgrading from 1.0 to 1.3 would trigger the behavior, but the limit was already in place in 1.0 and the breaking code is exceeding the maximum message size.

JLofgren · 2018-03-27T17:26:19Z

I think @cvkumar was onto something here. I am also seeing this error using grpc 1.10 with netty-codec-http:4.1.17.Final
This error is coming from the client side. As his stack trace points out: io.netty.handler.codec.http2.internal.hpack.Encoder.encodeHeadersEnforceMaxHeaderListSize (which is now at io.netty.handler.codec.http2.internal.hpack.Encoder.encodeHeadersEnforceMaxHeaderListSize) is enforcing a client-side limit on the header size.

It appears that Encoder has a constructor param called 'ignoreMaxHeaderListSize' that would eliminate this error. How do we set this option using [EDIT] ~~NettyServerBuilder~~ NettyChannelBuilder or otherwise?

JLofgren · 2018-03-27T21:49:12Z

I should add that my error is occurring when my client is connecting through an Envoy proxy. This error does not occur when I connect directly to the Java server.

I am gathering that perhaps there is some negotiation between the client and server that establishes an agreed-upon maxHeaderSize. Is that correct or am I off track?

carl-mastrangelo · 2018-03-28T00:34:50Z

There is some negotiation of SETTINGS_MAX_HEADER_LIST_SIZE, but you would need to check the SETTINGS frame to see if there are different.

JLofgren · 2018-03-29T01:32:43Z

After doing a bit more research and debugging, I'm not sure how to proceed. I run a distributed Java server application with Envoy proxies fronting the servers. We have clients in Java, Python, and Go, but this particular error in occurring only in a Java client. We have configured the server application to initialize the channel with NettyChannelBuilder.maxHeaderListSize(PLENTY_LARGE).

As @carl-mastrangelo suggested, I debugged the client and looked at the settings frame that is coming from the server. When I connect with my Java client directly to the java server, I see in the settings frame from the server that SETTINGS_MAX_HEADER_LIST_SIZE is set to PLENTY_LARGE. The client works fine in this case. However, when I connect through an Envoy proxy, Envoy sends no value for SETTINGS_MAX_HEADER_LIST_SIZE.

The HTTP/2 spec says this about SETTINGS_MAX_HEADER_LIST_SIZE:

For any given request, a lower limit than what is advertised MAY be enforced. The initial value of this setting is unlimited.

Netty chooses an initial default header size of 8192 bytes, but it is enforcing this limit on BOTH the sending and receiving side of the client - even when the server sends no value for SETTINGS_MAX_HEADER_LIST_SIZE. If the server has not sent any value for this setting, then the client should not place an arbitrary limit on the header size!

So, how can I tell the client to stop enforcing a limit on header size when the server has not specified any limit?

JLofgren · 2018-03-31T12:51:32Z

My issue is being addressed in #4284 . Thanks!

ejona86 mentioned this issue Jun 1, 2017

netty: Clarify max header list size configuration #2948

Merged

ejona86 closed this as completed Jul 27, 2017

wenbozhu added the usability label Aug 3, 2017

lock bot locked as resolved and limited conversation to collaborators Sep 28, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cause=io.netty.handler.codec.http2.Http2Exception: Header size exceeded max allowed size (10240) #2935

cause=io.netty.handler.codec.http2.Http2Exception: Header size exceeded max allowed size (10240) #2935

cr22rc commented Apr 21, 2017 •

edited

Loading

cr22rc commented Apr 21, 2017 •

edited by ejona86

Loading

carl-mastrangelo commented Apr 21, 2017

cr22rc commented Apr 23, 2017

cr22rc commented Apr 23, 2017

denisgaebler commented Apr 24, 2017

cr22rc commented Apr 24, 2017

cr22rc commented Apr 24, 2017

carl-mastrangelo commented Apr 24, 2017

cr22rc commented Apr 26, 2017 •

edited

Loading

ejona86 commented Apr 26, 2017

cvkumar commented Jun 8, 2017

ejona86 commented Jun 9, 2017

cvkumar commented Jun 14, 2017

ejona86 commented Jun 19, 2017

ejona86 commented Jul 27, 2017

JLofgren commented Mar 27, 2018 •

edited

Loading

JLofgren commented Mar 27, 2018 •

edited

Loading

carl-mastrangelo commented Mar 28, 2018

JLofgren commented Mar 29, 2018

JLofgren commented Mar 31, 2018

cause=io.netty.handler.codec.http2.Http2Exception: Header size exceeded max allowed size (10240) #2935

cause=io.netty.handler.codec.http2.Http2Exception: Header size exceeded max allowed size (10240) #2935

Comments

cr22rc commented Apr 21, 2017 • edited Loading

cr22rc commented Apr 21, 2017 • edited by ejona86 Loading

carl-mastrangelo commented Apr 21, 2017

cr22rc commented Apr 23, 2017

cr22rc commented Apr 23, 2017

denisgaebler commented Apr 24, 2017

cr22rc commented Apr 24, 2017

cr22rc commented Apr 24, 2017

carl-mastrangelo commented Apr 24, 2017

cr22rc commented Apr 26, 2017 • edited Loading

ejona86 commented Apr 26, 2017

cvkumar commented Jun 8, 2017

ejona86 commented Jun 9, 2017

cvkumar commented Jun 14, 2017

ejona86 commented Jun 19, 2017

ejona86 commented Jul 27, 2017

JLofgren commented Mar 27, 2018 • edited Loading

JLofgren commented Mar 27, 2018 • edited Loading

carl-mastrangelo commented Mar 28, 2018

JLofgren commented Mar 29, 2018

JLofgren commented Mar 31, 2018

cr22rc commented Apr 21, 2017 •

edited

Loading

cr22rc commented Apr 21, 2017 •

edited by ejona86

Loading

cr22rc commented Apr 26, 2017 •

edited

Loading

JLofgren commented Mar 27, 2018 •

edited

Loading

JLofgren commented Mar 27, 2018 •

edited

Loading