InternalError when accessing kubernetes cluster with kubernetes service #5031
Comments
|
@gclawes how did you configure the |
|
Created a long-lived token with |
|
@gclawes did you restart/recreate the |
|
I did not restart the |
|
Thanks. Can you show the output of It may be related to #5008, due to a |
|
I think I'm encountering the same problem. Currently 3 kubernetes clusters are connected to the main teleport server, but 1 cluster now always returns Forbidden when trying to connect. I'm sure it worked initially but at some point it stopped working. Restarting the kube-agent or the main teleport server did nothing. By executing your command, I see now that the "bad" cluster is returned multiple times: # tctl get kube_service | grep "\- name"
- name: aks-ndt-sales-westeurope-production
- name: aks-ndt-shared-westeurope-production
- name: aks-ndt-shared-westeurope-production
- name: nephroflow
- name: aks-ndt-internal-westeurope-production
- name: aks-ndt-shared-westeurope-production
- name: aks-ndt-shared-westeurope-production
- name: aks-ndt-shared-westeurope-production
- name: aks-ndt-shared-westeurope-production
- name: aks-ndt-shared-westeurope-production
- name: aks-ndt-shared-westeurope-productionThe other ones are only registered once, and are also working perfectly. |
|
I show the same: Though I set this k3s cluster up a week ago, I don't fully recall if I re-installed the I guess I should clarify my response above, I did not reset the I'll try to find time to re-do the whole setup and replicate the issue, but it may take a day or two. |
|
@gclawes @pcallewaert yep, it all lines up now. When a kube-agent restarts for any reason (upgrade or automatically rescheduled by k8s), it registers as a new You can temporarily fix this by removing all You have to run Here's a command to remove them all: $ for uuid in $(tctl get kube_service | grep ' name:' | awk '{print $2}'); do tctl rm kube_service/$uuid; done |
|
Cool, I'll give that a shot. Regarding the token for setting up the kubernetes service:
|
|
TL;DR: you can use dynamic tokens with
Tokens are used by teleport instances (any instances, not just
In general, you can use both static and dynamic tokens for any teleport service. The difference is that dynamic tokens expire after 30min. The caveat with the kube-agent doesn't have persistence, because I couldn't find a portable way to do so in Kubernetes. All the persistent disk options are cloud provider specific. |
|
Ah, ok, so what I did with Could persistence be done by creating a kubernetes If teleport creates a |
|
Yeah, we could definitely teach teleport to use the k8s API for storage. |
|
The fixes mentioned above are now in 5.0.1 |
Description
What happened:
On the initial install of the 5.0.0 kubernetes_service in a cluster
kubectlcommands work without issue. After the teleport cert expires andtsh loginis re-ran,kubectlreceives an InternalError.The following errors appear in the
proxycomponent:Running
tsh kube login <cluster name>does not fix the issue:Restarting the proxy component does resolve the issue.
Teleport is an open-source teleport 5.0.0 configured with separate auth/proxy components:
What you expected to happen:
kubectlworks as intendedHow to reproduce it (as minimally and precisely as possible):
tsh --proxy=<proxy> login.tsh kube login <cluster name>kubectlcommands sucessfullytsh loginkubectlcommands fail with aboveInternalErrormessagekubectlcommands succeed*NOTE: I have not tested this with a certificate ttl shorter than the default of 12hr.
Environment
Teleport version (use
teleport version): 5.0.0Tsh version (use
tsh version): 5.0.0OS (e.g. from
/etc/os-release): 5.0.0Where are you running Teleport? (e.g. AWS, GCP, Dedicated Hardware): Dedicated hardware (see above)
The text was updated successfully, but these errors were encountered: