Querier error returned as compressed binary data

[leth]

If I run a certain query in graph mode, I am told: Error: rpc error: code = Code(413) desc = exceeded maximum number of samples in a query (100000)
If I run the same query in table mode, instead I am told: Server error (rpc error: code = Code(500) desc = ����������4�M �0�����V-���*ЕW��1��B��{ ��w��g3�<Śo�� U�����^)N�ܨٯ�-N��|�!�H̸���<�FDZ�n���H~[RO�=ݨ(w�Ou�a��xt�a:~��������������+W����� )

[leth]

This issue started after 6024681. @tomwilkie Perhaps this was introduced in #910 as it was the next build to release a new querier image?

[leth]

I'm not sure what fixed this, but it seems to no longer happen on the latest releases!

[leth]

Turns out we didn't test what we thought we had tested!

[leth]

Can confirm this is still an issue on the latest image

[bboreham]

It's working on 6024681c and broken on 57b71bbf, which is the first commit that produced a Querier image after the working one.

57b71bbf is from #910 which changed httpgrpc.pb.go.

[bboreham]

@leth gave me a tip which revealed what the corruption is - the code generating the error message is writing to a compressedResponseWriter, while ErrorFromHTTPResponse which pulls it out into gRPC is assuming it can use those bytes directly.

@tomwilkie does that raise any thoughts as to how we should avoid this?

[tomwilkie]

Can I look tomorrow? In Amsterdam for monitorama....

…

On Wed, 5 Sep 2018 at 19:17, Bryan Boreham ***@***.***> wrote: @leth <https://github.com/leth> gave me a tip which revealed what the corruption is - the code generating the error message is writing to a compressedResponseWriter, while ErrorFromHTTPResponse which pulls it out into gRPC is assuming it can use those bytes directly. @tomwilkie <https://github.com/tomwilkie> does that raise any thoughts as to how we should avoid this? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#938 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAbGhQOOIIRSdQ1LQmABFHvAbESwseL2ks5uYAcQgaJpZM4WD0aS> .

[bboreham]

So this was actually caused by #898 not #910, and the reason I was looking at the wrong commit is that a unit test failed on the merge for #910 https://circleci.com/gh/weaveworks/cortex/2700

[bboreham]

@tomwilkie yeah it can wait. Maybe I'll just delete httpgrpc 😉

[tomwilkie]

Meh, what did httpgrpc ever do for us?

…

On Wed, 5 Sep 2018 at 19:34, Bryan Boreham ***@***.***> wrote: @tomwilkie <https://github.com/tomwilkie> yeah it can wait. Maybe I'll just delete httpgrpc 😉 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#938 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAbGhaB49dUzsWw4s4EL4xEAH1rt44Maks5uYAsqgaJpZM4WD0aS> .