Compability with Teleport #72
Comments
|
Thanks for raising this @tchellomello , this is an interesting issue. I don't have teleport to test this out. Is there any specific version of the kubectl client that supports exec calls with teleport? I'm using the exec subcommand in the client-go restclient library but maybe it needs additional parameters. Can you get me the verbose logs from a successful exec call so I can inspect the URL? |
|
Hello @gabeduke, for a reproducer you can install a Teleport instance pretty easy by following the doc here -> https://goteleport.com/docs/deploy-a-cluster/helm-deployments/custom/ So I tried to do a bit of troubleshooting and I was exactly on this path to compare the I've added a few debug calls to the iexec and this is what I'm getting back: Looking at the teleport server, I see that the stream channel is open, but it looks the Here is what I get via regular I'm planning to attach a I'll share any findings I get on my end. Thanks! |
|
Doing a little bit of debug, here is what I found so far: ❯ go build -gcflags="all=-N -l" kubectl-iexec.go
❯ dlv exec ./kubectl-iexec -- --log-level=trace postfix
bType 'help' for list of commands.
(dlv) break exec
Breakpoint 1 set at 0x2465592 for github.com/gabeduke/kubectl-iexec/pkg/iexec.exec() ./pkg/iexec/iexec.go:151
(dlv) c
DEBU[0000] iexec config values... Naked=false Namespace=default Vim Mode=false containerFilter= podFilter=postfix remote command="[/bin/sh]"
DEBU[0000] total pods discovered... namespace=default pods=43
INFO[0000] Get all pods for podFilter... SearchFilter=postfix
INFO[0000] Found pod... PodName=postfix-6b89f86fb-4b277 index=23
INFO[0000] Exec to pod... container=postfix namespace=default pod=postfix-6b89f86fb-4b277
> github.com/gabeduke/kubectl-iexec/pkg/iexec.exec() ./pkg/iexec/iexec.go:151 (hits goroutine(1):1 total:1) (PC: 0x2465592)
146: return err
147: }
148: return nil
149: }
150:
=> 151: func exec(restCfg *rest.Config, pod corev1.Pod, container corev1.Container, cmd []string) error {
152: client, err := kubernetes.NewForConfig(restCfg)
153: if err != nil {
154: return errors.Wrap(err, "unable to get kubernetes client config")
155: }
156:
(dlv) args
restCfg = ("*k8s.io/client-go/rest.Config")(0xc0003eed80)
pod = k8s.io/api/core/v1.Pod {TypeMeta: (*"k8s.io/apimachinery/pkg/apis/meta/v1.TypeMeta")(0xc000157d98), ObjectMeta: (*"k8s.io/apimachinery/pkg/apis/meta/v1.ObjectMeta")(0xc000157db8), Spec: (*"k8s.io/api/core/v1.PodSpec")(0xc000157eb0),...+1 more}
container = k8s.io/api/core/v1.Container {Name: "postfix", Image: "docker.io/REDACTED", Command: []string len: 1, cap: 1, [...],...+19 more}
cmd = []string len: 1, cap: 1, [...]
~r0 = error nil
### --> container information seems accurate
(dlv) print container
k8s.io/api/core/v1.Container {
Name: "postfix",
Image: "docker.io/xxxxxx",
Command: []string len: 1, cap: 1, [
"/vault/vault-env",
#### --> same for the pod object
(dlv) print pod
k8s.io/api/core/v1.Pod {
TypeMeta: k8s.io/apimachinery/pkg/apis/meta/v1.TypeMeta {Kind: "", APIVersion: ""},
ObjectMeta: k8s.io/apimachinery/pkg/apis/meta/v1.ObjectMeta {
Name: "postfix-6b89f86fb-4b277",
GenerateName: "postfix-6b89f86fb-",
Namespace: "default",
### -> the restCfg object
(dlv) print restCfg
*k8s.io/client-go/rest.Config {
Host: "https://teleport.REDACTED:3026",
APIPath: "",
ContentConfig: k8s.io/client-go/rest.ContentConfig {
AcceptContentTypes: "",
ContentType: "",
GroupVersion: *k8s.io/apimachinery/pkg/runtime/schema.GroupVersion nil,
NegotiatedSerializer: k8s.io/apimachinery/pkg/runtime.NegotiatedSerializer nil,},
Username: "",
Password: "",
BearerToken: "",
BearerTokenFile: "",
Impersonate: k8s.io/client-go/rest.ImpersonationConfig {
UserName: "",
Groups: []string len: 0, cap: 0, nil,
Extra: map[string][]string nil,},
AuthProvider: *k8s.io/client-go/tools/clientcmd/api.AuthProviderConfig nil,
AuthConfigPersister: k8s.io/client-go/rest.AuthProviderConfigPersister nil,
ExecProvider: *k8s.io/client-go/tools/clientcmd/api.ExecConfig {
Command: "/usr/bin/tsh",
Args: []string len: 5, cap: 8, [
"kube",
"credentials",
"--kube-cluster=teleport.REDACTED",
"--teleport-cluster=teleport.REDACTED",
"--proxy=teleport.REDACTED:443",
],
Env: []k8s.io/client-go/tools/clientcmd/api.ExecEnvVar len: 0, cap: 0, nil,
APIVersion: "client.authentication.k8s.io/v1beta1",
InstallHint: "",
ProvideClusterInfo: false,
Config: k8s.io/apimachinery/pkg/runtime.Object nil,
InteractiveMode: "IfAvailable",
StdinUnavailable: false,
StdinUnavailableMessage: "",},
TLSClientConfig: k8s.io/client-go/rest.TLSClientConfig {
Insecure: false,
ServerName: "",
CertFile: "",
KeyFile: "",
CAFile: "",
CertData: []uint8 len: 0, cap: 0, nil,
KeyData: []uint8 len: 0, cap: 0, nil,
CAData: []uint8 len: 1322, cap: 1323, [45,45,45,45,45,66,69,71,73,78,32,67,69,82,84,73,70,73,67,65,84,69,45,45,45,45,45,10,77,73,73,68,111,106,67,67,65,111,113,103,65,119,73,66,65,103,73,81,76,57,68,67,119,72,102,119,85,110,88,78,99,121,122,85,...+1258 more],
NextProtos: []string len: 0, cap: 0, nil,},
UserAgent: "",
DisableCompression: false,
Transport: net/http.RoundTripper nil,
WrapTransport: nil,
QPS: 0,
Burst: 0,
RateLimiter: k8s.io/client-go/util/flowcontrol.RateLimiter nil,
WarningHandler: k8s.io/client-go/rest.WarningHandler nil,
Timeout: 0,
Dial: nil,
Proxy: nil,}So moving on the code and inspecting the > github.com/gabeduke/kubectl-iexec/pkg/iexec.exec() ./pkg/iexec/iexec.go:172 (PC: 0x2465984)
167: Stdout: true,
168: Stderr: true,
169: TTY: true,
170: }, scheme.ParameterCodec)
171:
=> 172: log.WithFields(log.Fields{
173: "URL": req.URL(),
174: }).Debug("Request")
175:
176: exec, err := remotecommand.NewSPDYExecutor(restCfg, "POST", req.URL())
177: if err != nil {
(dlv) print req
*k8s.io/client-go/rest.Request {
c: *k8s.io/client-go/rest.RESTClient {
base: *(*"net/url.URL")(0xc000617440),
versionedAPIPath: "/api/v1",
content: (*"k8s.io/client-go/rest.ClientContentConfig")(0xc0003a6e78),
createBackoffMgr: k8s.io/client-go/rest.readExpBackoffConfig,
rateLimiter: k8s.io/client-go/util/flowcontrol.RateLimiter(*k8s.io/client-go/util/flowcontrol.tokenBucketRateLimiter) ...,
warningHandler: k8s.io/client-go/rest.WarningHandler nil,
Client: *(*"net/http.Client")(0xc00015b560),},
warningHandler: k8s.io/client-go/rest.WarningHandler nil,
rateLimiter: k8s.io/client-go/util/flowcontrol.RateLimiter(*k8s.io/client-go/util/flowcontrol.tokenBucketRateLimiter) *{
limiter: *(*"golang.org/x/time/rate.Limiter")(0xc000112f00),
clock: k8s.io/client-go/util/flowcontrol.Clock(k8s.io/client-go/util/flowcontrol.realClock) *(*"k8s.io/client-go/util/flowcontrol.Clock")(0xc0000ea588),
qps: 5,},
backoff: k8s.io/client-go/rest.BackoffManager(*k8s.io/client-go/rest.NoBackoff) *{},
timeout: 0,
verb: "POST",
pathPrefix: "/api/v1",
subpath: "",
params: net/url.Values [
"stdin": ["true"],
"stdout": ["true"],
"stderr": ["true"],
"tty": ["true"],
"container": ["postfix"],
"command": ["/bin/sh"],
],
headers: net/http.Header [
"Accept": [
"application/json, */*",
],
],
namespace: "default",
namespaceSet: true,
resource: "pods",
resourceName: "postfix-6b89f86fb-4b277",
subresource: "exec",
err: error nil,
body: io.Reader nil,
retry: k8s.io/client-go/rest.WithRetry(*k8s.io/client-go/rest.withRetry) *{maxRetries: 10, attempts: 0},}At the > github.com/gabeduke/kubectl-iexec/pkg/iexec.exec() ./pkg/iexec/iexec.go:177 (PC: 0x2465c07)
172: log.WithFields(log.Fields{
173: "URL": req.URL(),
174: }).Debug("Request")
175:
176: exec, err := remotecommand.NewSPDYExecutor(restCfg, "POST", req.URL())
=> 177: if err != nil {
178: return errors.Wrap(err, "unable to execute remote command")
179: }
180:
181: fd := int(os.Stdin.Fd())
182:
(dlv) print exec.url.Path
"/api/v1/namespaces/default/pods/postfix-6b89f86fb-4b277/exec"
(dlv) print exec.url.RawPath
""
(dlv) print exec.url.RawQuery
"command=%2Fbin%2Fsh&container=postfix&stderr=true&stdin=true&std...+17 more"Then I saw this: > github.com/gabeduke/kubectl-iexec/pkg/iexec.exec() ./pkg/iexec/iexec.go:185 (PC: 0x2465d15)
180:
181: fd := int(os.Stdin.Fd())
182:
183: // Put the terminal into raw mode to prevent it echoing characters twice.
184: oldState, err := term.MakeRaw(fd)
=> 185: if err != nil {
186: return errors.Wrap(err, "unable to init terminal")
187: }
188:
189: termWidth, termHeight, _ := term.GetSize(fd)
190: termSize := remotecommand.TerminalSize{Width: uint16(termWidth), Height: uint16(termHeight)}
(dlv) print oldState
*golang.org/x/term.State nil
(dlv) print err
error(syscall.Errno) golang.org/x/sys/unix.ENOTTY (25)
> github.com/gabeduke/kubectl-iexec/pkg/iexec.(*Iexec).Do() ./pkg/iexec/iexec.go:145 (PC: 0x246550b)
140: "container": container.Name,
141: "namespace": r.config.Namespace,
142: }).Info("Exec to pod...")
143:
144: err = exec(r.restConfig, pod, container, r.config.RemoteCmd)
=> 145: if err != nil {
146: return err
147: }
148: return nil
149: }
150:
(dlv) print err
error(*github.com/pkg/errors.withStack) *{
error: error(*github.com/pkg/errors.withMessage) *{
cause: error(syscall.Errno) *(*error)(0xc00012a2a0),
msg: "unable to init terminal",},
stack: *github.com/pkg/errors.stack len: 11, cap: 32, [38165805,38163707,40890360,40886766,38437233,38441243,38438424,40890921,40891031,12678739,12885153],}Then finally get into the |
|
It looks like it is failing when it attempts to spawn a new shell. I'm assuming there is some sort of callback that is being lost. I spent some time trying to set up a teleport cluster but wasn't able to get it working locally. I'll give it another shot when I get some free time. |
|
@gabeduke interesting you saying that because on the Teleport's side I see the timeout. teleport-proxy-7f8c9c4cc6-xc4jj teleport 2023-07-18T16:01:37Z DEBU [PROXY:PRO] Exec request failed error:[
teleport-proxy-7f8c9c4cc6-xc4jj teleport ERROR REPORT:
teleport-proxy-7f8c9c4cc6-xc4jj teleport Original Error: trace.aggregate timed out waiting for client to create streams
teleport-proxy-7f8c9c4cc6-xc4jj teleport Stack Trace:
teleport-proxy-7f8c9c4cc6-xc4jj teleport github.com/gravitational/teleport/lib/kube/proxy/remotecommand.go:179 github.com/gravitational/teleport/lib/kube/proxy.createSPDYStreams
teleport-proxy-7f8c9c4cc6-xc4jj teleport github.com/gravitational/teleport/lib/kube/proxy/remotecommand.go:105 github.com/gravitational/teleport/lib/kube/proxy.createRemoteCommandProxy
teleport-proxy-7f8c9c4cc6-xc4jj teleport github.com/gravitational/teleport/lib/kube/proxy/forwarder.go:1768 github.com/gravitational/teleport/lib/kube/proxy.(*Forwarder).exec
teleport-proxy-7f8c9c4cc6-xc4jj teleport github.com/gravitational/teleport/lib/kube/proxy/forwarder.go:684 github.com/gravitational/teleport/lib/kube/proxy.(*Forwarder).withAuth.func1
teleport-proxy-7f8c9c4cc6-xc4jj teleport github.com/gravitational/teleport/lib/httplib/httplib.go:117 github.com/gravitational/teleport/lib/httplib.MakeHandlerWithErrorWriter.func1
teleport-proxy-7f8c9c4cc6-xc4jj teleport github.com/julienschmidt/httprouter@v1.3.0/router.go:399 github.com/julienschmidt/httprouter.(*Router).ServeHTTP
teleport-proxy-7f8c9c4cc6-xc4jj teleport go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.40.0/handler.go:213 go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*Handler).ServeHTTP
teleport-proxy-7f8c9c4cc6-xc4jj teleport github.com/gravitational/teleport/lib/kube/proxy/forwarder.go:412 github.com/gravitational/teleport/lib/kube/proxy.(*Forwarder).ServeHTTP
teleport-proxy-7f8c9c4cc6-xc4jj teleport github.com/gravitational/teleport/lib/auth/middleware.go:696 github.com/gravitational/teleport/lib/auth.(*Middleware).ServeHTTP
teleport-proxy-7f8c9c4cc6-xc4jj teleport github.com/gravitational/oxy@v0.0.0-20221029012416-9fbf4c444680/ratelimit/tokenlimiter.go:118 github.com/gravitational/oxy/ratelimit.(*TokenLimiter).ServeHTTP
teleport-proxy-7f8c9c4cc6-xc4jj teleport github.com/gravitational/oxy@v0.0.0-20221029012416-9fbf4c444680/connlimit/connlimit.go:75 github.com/gravitational/oxy/connlimit.(*ConnLimiter).ServeHTTP
teleport-proxy-7f8c9c4cc6-xc4jj teleport github.com/gravitational/teleport/lib/httplib/httplib.go:94 github.com/gravitational/teleport/lib/httplib.MakeTracingHandler.func1
teleport-proxy-7f8c9c4cc6-xc4jj teleport net/http/server.go:2122 net/http.HandlerFunc.ServeHTTP
teleport-proxy-7f8c9c4cc6-xc4jj teleport go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@v0.40.0/handler.go:213 go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*Handler).ServeHTTP
teleport-proxy-7f8c9c4cc6-xc4jj teleport net/http/server.go:2936 net/http.serverHandler.ServeHTTP
teleport-proxy-7f8c9c4cc6-xc4jj teleport net/http/server.go:1995 net/http.(*conn).serve
teleport-proxy-7f8c9c4cc6-xc4jj teleport runtime/asm_amd64.s:1598 runtime.goexit
teleport-proxy-7f8c9c4cc6-xc4jj teleport User Message: timed out waiting for client to create streams] pid:7.1 proxy/forwarder.go:1725 |
|
@gabeduke I work with @tchellomello and we were diving deeper into this issue. In an attempt to understand the difference between The The The The cachedCreds that were dumped were slightly different but I think this is expected during teleports communications. Both commands authenticate and are the same in the kubeconfig which is used to create the restConfig but I believe there is a cert generated for the communication. Another difference was the agent string: I did not see the agent in the The only other noticeable difference was in the the transport
The I walked the communications down the following path: github.com/gabeduke/kubectl-iexec/vendor/k8s.io/client-go/tools/remotecommand/v4.go L 55 Then the createStreams calls happen: Which calls createStreams from v3 and then to v2 github.com/gabeduke/kubectl-iexec/vendor/k8s.io/client-go/tools/remotecommand/v2.go L 57 github.com/gabeduke/kubectl-iexec/vendor/k8s.io/apimachinery/pkg/util/httpstream/spdy/connection.go 137 github.com/gabeduke/kubectl-iexec/vendor/github.com/moby/spdystream/connection.go This now goes down a few more layers into write and io.Writer. The last place I could track this was here:
This fails and then returns. The error observed here is |
# for free
to join this conversation on GitHub.
Already have an account?
# to comment
I would be great to see the
iexecplugin working with Teleport.Whenever trying to connect proxied via Teleport, it does not move along.
The text was updated successfully, but these errors were encountered: