cause=io.netty.handler.codec.http2.Http2Exception: Header size exceeded max allowed size (10240)

[cr22rc]

Please answer these questions before submitting your issue.

1.2.0

java.runtime.name = OpenJDK Runtime Environment
12:22:18 java.runtime.version = 1.8.0_121-8u121-b13-0ubuntu1.16.04.2-b13
12:22:18 java.specification.name = Java Platform API Specification
12:22:18 java.specification.vendor = Oracle Corporation
12:22:18 java.specification.version = 1.8
12:22:18 java.vendor = Oracle Corporation
12:22:18 java.vendor.url = http://java.oracle.com/
12:22:18 java.vendor.url.bug = http://bugreport.sun.com/bugreport/
12:22:18 java.version = 1.8.0_121
12:22:18 java.vm.info = mixed mode
12:22:18 java.vm.name = OpenJDK 64-Bit Server VM
12:22:18 java.vm.specification.name = Java Virtual Machine Specification
12:22:18 java.vm.specification.vendor = Oracle Corporation
12:22:18 java.vm.specification.version = 1.8
12:22:18 java.vm.vendor = Oracle Corporation
12:22:18 java.vm.version = 25.121-b13
12:22:18 line.separator = \

grpc async call There really isn't any special options set on the channelbuilder
except usePlaintext it does have that set to true but a the moment not sure why.

The scenario this is running on works for all systems I'v but on these vms.
Ubuntu vm 16.04.

Is there some way to increase that size.
I can't imagine that the packet size would be any different between these system .. it's running the same test. Is there any other reason this exception might happen?

[cr22rc]

00:04:13 2017-04-21 00:04:13 DEBUG Peer:144 - sendProposalAsync name: peer1, url: grpc://localhost:7056
00:04:13 2017-04-21 00:04:13 DEBUG NettyClientHandler:147 - 
00:04:13 ----------------OUTBOUND--------------------
00:04:13 [id: 0xa601ff3c, L:/127.0.0.1:57166 - R:localhost/127.0.0.1:7056] SETTINGS: ack=false, settings={ENABLE_PUSH=0, MAX_CONCURRENT_STREAMS=0, INITIAL_WINDOW_SIZE=1048576, MAX_HEADER_LIST_SIZE=8192}
00:04:13 ------------------------------------
00:04:13 2017-04-21 00:04:13 DEBUG NettyClientHandler:147 - 
00:04:13 ----------------OUTBOUND--------------------
00:04:13 [id: 0xa601ff3c, L:/127.0.0.1:57166 - R:localhost/127.0.0.1:7056] WINDOW_UPDATE: streamId=0, windowSizeIncrement=983041
00:04:13 ------------------------------------
00:04:13 2017-04-21 00:04:13 DEBUG NettyClientHandler:147 - 
00:04:13 ----------------INBOUND--------------------
00:04:13 [id: 0xa601ff3c, L:/127.0.0.1:57166 - R:localhost/127.0.0.1:7056] SETTINGS: ack=false, settings={}
00:04:13 ------------------------------------
00:04:13 2017-04-21 00:04:13 DEBUG NettyClientHandler:147 - 
00:04:13 ----------------OUTBOUND--------------------
00:04:13 [id: 0xa601ff3c, L:/127.0.0.1:57166 - R:localhost/127.0.0.1:7056] SETTINGS: ack=true
00:04:13 ------------------------------------
00:04:13 2017-04-21 00:04:13 DEBUG NettyClientHandler:147 - 
00:04:13 ----------------INBOUND--------------------
00:04:13 [id: 0xa601ff3c, L:/127.0.0.1:57166 - R:localhost/127.0.0.1:7056] WINDOW_UPDATE: streamId=0, windowSizeIncrement=983025
00:04:13 ------------------------------------
00:04:13 2017-04-21 00:04:13 DEBUG NettyClientHandler:147 - 
00:04:13 ----------------INBOUND--------------------
00:04:13 [id: 0xa601ff3c, L:/127.0.0.1:57166 - R:localhost/127.0.0.1:7056] SETTINGS: ack=true
00:04:13 ------------------------------------
00:04:13 2017-04-21 00:04:13 DEBUG NettyClientHandler:147 - 
00:04:13 ----------------OUTBOUND--------------------
00:04:13 [id: 0xa601ff3c, L:/127.0.0.1:57166 - R:localhost/127.0.0.1:7056] HEADERS: streamId=3, headers=GrpcHttp2OutboundHeaders[:authority: localhost:7056, :path: /protos.Endorser/ProcessProposal, :method: POST, :scheme: http, content-type: application/grpc, te: trailers, user-agent: grpc-java-netty/1.2.0, grpc-accept-encoding: gzip, grpc-census-bin: ], streamDependency=0, weight=16, exclusive=false, padding=0, endStream=false
00:04:13 ------------------------------------
00:04:13 2017-04-21 00:04:13 DEBUG NettyClientHandler:147 - 
00:04:13 ----------------OUTBOUND--------------------
00:04:13 [id: 0xa601ff3c, L:/127.0.0.1:57166 - R:localhost/127.0.0.1:7056] DATA: streamId=3, padding=0, endStream=true, length=14657, bytes=000000393c0af0710ad1060a5e080310011a0c08fd98e5c70510c0faafb7022a4038343436323166363436633037373864353432666238373861626139626563...
00:04:13 ------------------------------------
00:04:14 2017-04-21 00:04:13 DEBUG NettyClientHandler:147 - 
00:04:14 ----------------OUTBOUND--------------------
00:04:14 [id: 0xa601ff3c, L:/127.0.0.1:57166 - R:localhost/127.0.0.1:7056] GO_AWAY: lastStreamId=0, errorCode=1, length=45, bytes=4865616465722073697a65206578636565646564206d617820616c6c6f7765642073697a652028313032343029
00:04:14 ------------------------------------
00:04:14 2017-04-21 00:04:13 DEBUG Http2ConnectionHandler:836 - [id: 0xa601ff3c, L:/127.0.0.1:57166 - R:localhost/127.0.0.1:7056] Sent GOAWAY: lastStreamId '0', errorCode '1', debugData 'Header size exceeded max allowed size (10240)'. Forcing shutdown of the connection.
00:04:14 2017-04-21 00:04:14 ERROR Chain:2105 - Sending proposal to peer1 failed because of gRPC failure=Status{code=INTERNAL, description=null, cause=io.netty.handler.codec.http2.Http2Exception: Header size exceeded max allowed size (10240)
00:04:14 	at io.netty.handler.codec.http2.Http2Exception.connectionError(Http2Exception.java:85)
00:04:14 	at io.netty.handler.codec.http2.Http2CodecUtil.headerListSizeExceeded(Http2CodecUtil.java:264)
00:04:14 	at io.netty.handler.codec.http2.DefaultHttp2FrameReader$HeadersBlockBuilder.headerSizeExceeded(DefaultHttp2FrameReader.java:701)
00:04:14 	at io.netty.handler.codec.http2.DefaultHttp2FrameReader$HeadersBlockBuilder.addFragment(DefaultHttp2FrameReader.java:716)
00:04:14 	at io.netty.handler.codec.http2.DefaultHttp2FrameReader$2.processFragment(DefaultHttp2FrameReader.java:485)
00:04:14 	at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readHeadersFrame(DefaultHttp2FrameReader.java:495)
00:04:14 	at io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:253)
00:04:14 	at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:160)
00:04:14 	at io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41)
00:04:14 	at io.grpc.netty.FixedHttp2ConnectionDecoder.decodeFrame(FixedHttp2ConnectionDecoder.java:119)
00:04:14 	at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:341)
00:04:14 	at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:401)
00:04:14 	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411)
00:04:14 	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248)
00:04:14 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
00:04:14 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
00:04:14 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)
00:04:14 	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
00:04:14 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
00:04:14 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
00:04:14 	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
00:04:14 	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:129)
00:04:14 	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:642)
00:04:14 	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:565)
00:04:14 	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:479)
00:04:14 	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441)
00:04:14 	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
00:04:14 	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
00:04:14 	at java.lang.Thread.run(Thread.java:745)

[carl-mastrangelo]

You can set this, but you will have to use NettyChannelBuilder. It is currently not a stable API, so you may need to change in the future as we modify it.

[cr22rc]

@carl-mastrangelo I'm really new to grpc can you give a quick outline of how to do that ? Option on channelBuilder?
Or a pointer to an example if available ?
Thank you!

[cr22rc]

Another question can you tell if this issue is with the outbound request or the inbound ?
I suspect inbound but still new to this.

[denisgaebler]

Are you connecting bigendian and littleendian systems, e.g. are you using a hyperledger on s390x?
Then there is an endianess problem fixed with netty 4.1.10, but it cannot be used because grpc-java only supports 4.1.8.

[cr22rc]

All on a local box. Client connecting to servers running in docker. hopefully no big/little endian issue there :)

[cr22rc]

From that trance is it due to outbound or inbound ?

[carl-mastrangelo]

@cr22rc See http://www.grpc.io/grpc-java/javadoc/io/grpc/netty/NettyChannelBuilder.html#maxHeaderListSize-int-

[cr22rc]

Thanks what had me confused about that was is states "ListSize" but I'll look into it.
Still looking at the trace provided is there any way to know for sure if it inbound/outbound having the issue?

[ejona86]

@cr22rc, you're right, the documentation wasn't very clear... unless you were already intimately familiar with HTTP/2. I created #2948 to try to make it better.

[cvkumar]

Hey, I am getting a similar, but not quite the same issue. I am uses gRPC-java 1.3.0 When I build the a channel I set the max header size to be very high.

originChannel = NettyChannelBuilder.forAddress(this.configuration.getHost(), this.configuration.getPort()).maxHeaderListSize(100000)
          .sslContext(GrpcSslContexts.forClient().build()).build();

However, when I am building a lot of streams stub.send(streamObserver); I eventually hit the following error.

WARNING: Publish stream closing because failed with error: Status{code=INTERNAL, description=null, ****cause=io.netty.handler.codec.http2.Http2Exception$HeaderListSizeException: Header size exceeded max allowed size (8192)****
	at io.netty.handler.codec.http2.Http2Exception.headerListSizeError(Http2Exception.java:169)
	at io.netty.handler.codec.http2.Http2CodecUtil.headerListSizeExceeded(Http2CodecUtil.java:252)
	at io.netty.handler.codec.http2.internal.hpack.Encoder.encodeHeadersEnforceMaxHeaderListSize(Encoder.java:128)
	at io.netty.handler.codec.http2.internal.hpack.Encoder.encodeHeaders(Encoder.java:112)
	at io.netty.handler.codec.http2.DefaultHttp2HeadersEncoder.encodeHeaders(DefaultHttp2HeadersEncoder.java:69)
	at io.netty.handler.codec.http2.DefaultHttp2FrameWriter.writeHeadersInternal(DefaultHttp2FrameWriter.java:435)
	at io.netty.handler.codec.http2.DefaultHttp2FrameWriter.writeHeaders(DefaultHttp2FrameWriter.java:200)
	at io.netty.handler.codec.http2.Http2OutboundFrameLogger.writeHeaders(Http2OutboundFrameLogger.java:60)
	at io.netty.handler.codec.http2.DefaultHttp2ConnectionEncoder.writeHeaders(DefaultHttp2ConnectionEncoder.java:188)
	at io.netty.handler.codec.http2.DecoratingHttp2FrameWriter.writeHeaders(DecoratingHttp2FrameWriter.java:52)
	at io.netty.handler.codec.http2.StreamBufferingEncoder.writeHeaders(StreamBufferingEncoder.java:157)
	at io.netty.handler.codec.http2.StreamBufferingEncoder.writeHeaders(StreamBufferingEncoder.java:141)
	at io.grpc.netty.NettyClientHandler.createStream(NettyClientHandler.java:426)
	at io.grpc.netty.NettyClientHandler.write(NettyClientHandler.java:240)
	at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:739)
	at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:731)
	at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:817)
	at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:724)
	at io.netty.channel.DefaultChannelPipeline.write(DefaultChannelPipeline.java:1022)
	at io.netty.channel.AbstractChannel.write(AbstractChannel.java:291)
	at io.grpc.netty.WriteQueue.flush(WriteQueue.java:127)
	at io.grpc.netty.WriteQueue.access$000(WriteQueue.java:47)
	at io.grpc.netty.WriteQueue$1.run(WriteQueue.java:59)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:445)
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
	at java.lang.Thread.run(Thread.java:745)
}

It seems that io.grpc.netty.NettyClientHandler.createStream has an Encoder class io.netty.handler.codec.http2.internal.hpack.Encoder.encodeHeaders(Encoder.java:112) which I can't change the maximum header size allowed.

Is there a way for me to change the maximum header size for creation of streams, not just channels?

P.s. Still a little new to this gRPC-java thing

[ejona86]

@cvkumar, maxHeaderListSize(100000) configures the receiving limit. In your case you are sending, which is determined by the remote side.

[cvkumar]

@ejona86 I see. So you mean the server side is running into this error? That is interesting.... To give you some context, we are trying to update our gRPC java version from 1.0.0 to 1.3.0. But when we do some of our tests break with this error. Do you know of a reason why upgrading our version would cause this?

[ejona86]

@cvkumar, I don't know why upgrading would trigger the behavior difference. There were changes (like #2350), but I can't pin the exact behavior change.