Skip to content

GRPC + nginx fails with "protocol_error" - not transparent #2942

Closed
@afirth

Description

@afirth

Bug Report

What is the issue?

setup: grpcurl ---TLS---> nginx+l5d2 ---plaintext---> fortune-teller+l5d2

works fine until I linkerd inject

How can it be reproduced?

deploy fortune-teller to cluster
deploy nginx-ingress to cluster
add TLS to fortune-teller ingress
use grpcurl from outside to verify that the ingress works
linkerd inject the ingress control and fortune-teller
no longer works (502 bad gateway)

  • traffic comes into nginx
  • nginx passes traffic to linkerd proxy sidecar@nginx
  • proxy@nginx vomits errors
  • no traffic received by proxy@service backend
    (see trace logs below)

@olix0r may have more to add here

Logs, error output, etc

logs from the proxy@nginx
https://gist.github.com/afirth/ca016f72fc98a05e3d8145911d70f3a8

linkerd check output

linkerd check
kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version
√ is running the minimum kubectl version

linkerd-existence
-----------------
√ control plane namespace exists
√ controller pod is running
√ can initialize the client
√ can query the control plane API

linkerd-api
-----------
√ control plane pods are ready
√ control plane self-check
√ [kubernetes] control plane can talk to Kubernetes
√ [prometheus] control plane can talk to Prometheus
√ no invalid service profiles

linkerd-version
---------------
√ can determine the latest version
√ cli is up-to-date

control-plane-version
---------------------
√ control plane is up-to-date
√ control plane and cli versions match

Status check results are √

Environment

linkerd version
Client version: stable-2.3.2
Server version: stable-2.3.2

kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.2", GitCommit:"66049e3b21efe110454d67df4fa62b08ea79a19b", GitTreeState:"clean", BuildDate:"2019-05-16T16:23:09Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.7-gke.10", GitCommit:"8d9b8641e72cf7c96efa61421e87f96387242ba1", GitTreeState:"clean", BuildDate:"2019-04-12T22:59:24Z", GoVersion:"go1.10.8b4", Compiler:"gc", Platform:"linux/amd64"}

cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.2 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.2 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

Additional context

nothing in proxy logs at server

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions