Provide an easy option to setup OpenSSH Config.

[benarent]

Feature Request

Using Teleport with OpenSSH still requires a diving into the docs and knowing a lot about Teleport, if they don't want to use an SSH agent.

The output would be repopulated with the tsh profile.

$ tsh config

    #
    # Begin generated Teleport configuration for teleport-7.asteroid.earth:443 from `tsh config`
    #
    
    # Common flags for all teleport-7.asteroid.earth hosts
    Host *.teleport-7.asteroid.earth teleport-7.asteroid.earth
        UserKnownHostsFile "/Users/benarent/.tsh/known_hosts"
    
    # Flags for all teleport-7.asteroid.earth hosts except the proxy
    Host *.teleport-7.asteroid.earth !teleport-7.asteroid.earth
        Port 3022
        ProxyCommand ssh -p 3023 teleport-7.asteroid.earth -s proxy:$(echo %h | cut -d '.' -f 1):%p
    
    # End generated Teleport configuration

Motivation

This was an idea from @webvictim and it'll help reduce support load and make Windows setup much easier.

Who's it for?

OSS User, Pro, Enterprise

[webvictim]

We should also support the --cluster flag to this so people can easily add aliases pointing to trusted clusters (using the appropriate 'different' ProxyCommand format) as well.

On a slightly different note, I also have a script I wrote (https://gist.github.com/webvictim/6a306267cad85c024d93641985acfa0b) that goes into ProxyCommand that automatically checks whether your Teleport certificate is expired and re-runs tsh login with appropriate parameters to get a new certificate if it's needed when you run ssh server. It supports passing the proxy and username currently - I could fairly easily modify it to also support a trusted cluster if we wanted. Alternatively, we could modify to tsh to build this functionality into the client itself - basically a way to optionally relogin if the certificate is expired rather than needing to parse output or anything else.

Also, I've been meaning to write a guide for a while on how to use HostKeyAlias along with the output of tctl auth export --type=host to automatically match hostnames in an SSH config. Maybe I'll finally get around to that...

[webvictim]

Similar to #4989

[duckfez]

Coming from the Okta ASA / ScaleFT world they have something similar to this that works like so:

# To use ScaleFT proxycommand, add this configuration block to your $HOME/.ssh/config
Match exec "/usr/local/bin/sft resolve -q  %h"
    ProxyCommand "/usr/local/bin/sft" proxycommand  %h
    UserKnownHostsFile "/Users/dwaddle/Library/Application Support/ScaleFT/proxycommand_known_hosts"

[klizhentas]

tentatively adding this to 7.0

[timothyb89]

After discussing with @russjones I've put together a prototype implementation in a branch implementing what I believe may be the best path forward: eaaa19c.

(It is pretty rough at the moment, with a number of unsupported edge cases and outstanding TODOs.)

---

As already discussed, there are a few different paths to implement this. The sft approach is definitely interesting (it works with all current and future clusters/profiles with no additional configuration!) but potentially slows down all ssh invocations to run the match query. A safer way forward iterates on the original suggestion:

Host *.foo.example.com !proxy.foo.example.com
    ProxyCommand ssh -p 3023 proxy.foo.example.com -s proxy:$(echo %h | cut -d '.' -f 1):3022

Of note, this:

• Reliably excludes the proxy server from being matched by the wildcard directly (which could cause a recursive ssh proxy if the proxy and nodes fall under the same domain)
• Trims the cluster name off the incoming node name (ssh alice@node0.foo.example.com results in a proxy to node0)

Unfortunately it still has some drawbacks:

• Only works on Unix (subshell syntax plus cut is unlikely to work on Windows)
• Requires that all cluster nodes use the same port (in this case 3022)
• Does not check or refresh certificates automatically, leading to unhelpful errors if they've expired
• Must be repeated for different clusters/profiles

To work around these drawbacks, I'd like to introduce an extra helper subcommand to tsh which we can use as a ProxyCommand wrapper, alongside the export functionality already discussed.

In the prototype branch, users can generate an SSH config snippet using tsh export ssh and copy/redirect it into their ~/.ssh/config. This config snippet then proxies all connections under the cluster domain through tsh export proxy. The proxy helper implements the same logic as the example snippet above but should be more portable and resolves server ports properly.

If this looks like a sane direction, there's a number of further improvements that could be made:

• Automatic certificate refresh per @webvictim's example
• Specify IdentityFile to avoid need for ssh-agent
• Support for leaf clusters
• If possible, set known hosts to avoid unnecessary prompts
• If possible, automatically select the proper proxy for users with multiple profiles
• Write configuration directly to a tsh-owned config file (e.g. ~/.tsh/ssh_config) which can be updated automatically, and direct users to include it (Include ~/.tsh/ssh_config)

[klizhentas]

Question:

@timothyb89 how much the slowdown the match exec really is? I think it's so interesting and simple, compared to additional subcommands and patterns.

[klizhentas]

Note on CLI/UX:

• Existing tsh commands are tsh app config and tsh db config. @r0mant what's the plan with our config commands, are we going to have tsh config --format to consolidate? Otherwise we'd support tsh ssh config to match other commands

[klizhentas]

Random thought (feel free to ignore)

What if we use the gcloud strategy?

I ran: gcloud compute config-ssh and here is what happened:

gcloud compute config-ssh
Updating project ssh metadata...⠼                                                
Updating project ssh metadata...done.                                                                                                                     
You should now be able to use ssh/scp with your instances.
For example, try running:

  $ ssh gke-houston-default-pool-dc969126-nr65.us-central1-a.kubeadm-167321

 ~/.ssh/config 
# Google Compute Engine Section
#
# The following has been auto-generated by "gcloud compute config-ssh"
# to make accessing your Google Compute Engine virtual machines easier.
#
# To remove this blob, run:
#
#   gcloud compute config-ssh --remove
#
# You can also manually remove this blob by deleting everything from
# here until the comment that contains the string "End of Google Compute
# Engine Section".
#
# You should not hand-edit this section, unless you are deleting it.
#
Host gke-houston-default-pool-dc969126-nr65.us-central1-a.kubeadm-167321
    HostName 1.2.3.4
    IdentityFile /home/sasha/.ssh/google_compute_engine
    UserKnownHostsFile=/home/sasha/.ssh/google_compute_known_hosts
    HostKeyAlias=compute.7416332556765778597
    IdentitiesOnly=yes
    CheckHostIP=no

# End of Google Compute Engine Section

[r0mant]

I think having subcommands tsh app config, tsh db config, tsh ssh config etc. makes sense since we seem to be going the direction of providing relevant commands for each access type. I'm not sure though if we can easily do ssh subcommands though since it would probably conflict with tsh ssh (ssh'ing into a node) but that's probably solvable.

[timothyb89]

@timothyb89 how much the slowdown the match exec really is? I think it's so interesting and simple, compared to additional subcommands and patterns.

I’d like to look into this more and try to quantify the impact, the concern is that the check would be executed for all ssh calls, not just those getting proxied via Teleport. A poor network connection might effectively break ssh for users, for instance. Maybe we could mitigate this by being cautious with timeouts and caching?

Existing tsh commands are tsh app config and tsh db config. @r0mant what's the plan with our config commands, are we going to have tsh config --format to consolidate? Otherwise we'd support tsh ssh config to match other commands

I’m very much open to revising the command naming, my placeholders are more than a little clunky. I mainly wanted to avoid creating any conflicts for the proof of concept.

What if we use the gcloud strategy?

I definitely like parts of this UX, I think we could avoid mutating the user’s own ssh config by writing our own config and including it. With gcloud do you occasionally need to re-run that command to account for added or removed instances?

[timothyb89]

Cataloging a couple more observations from today:

• The default known_hosts UX is pretty rough after keys rotate, I think the suggestion from Feature request: Make tsh ssh on Windows generate an ephemeral SSH config file and use an installed OpenSSH client to connect #4989 is important to implement
• Testing nodes with Match exec ... adds roughly 100ms to each ssh call on my toy 3-node localhost cluster.

I've updated my branch to add a third subcommand to test if a hostname is a Teleport node (used to produce the above measurement). The following ssh client config now works as an alternative to what's generated by tsh export ssh:

Match exec "tsh export test 'foo.example.com' '%h' 2>&1 > /dev/null"
    ProxyCommand "tsh" export proxy "proxy.foo.example.com" "foo.example.com" "%h"

(as before, these are placeholder subcommands/flags)

I'm a bit wary of the added runtime cost, especially given it affects all ssh calls. I don't see a gRPC API to fetch a single node by hostname, so I expect the cost will increase with larger clusters unless we add a more efficient query.

---

Measurement notes: I ran the following using different configurations:

for i in {0..20}; do gtime -f '%e' ssh node000.foo.example.com exit; done

...and averaged the results:

• With wildcard host blocks (Host *.foo.example.com): .15s
• With trivial Match exec: 0.25s

Similarly, connecting to a non-Teleport host (just localhost):

• ssh localhost without Match exec: .12s
• ssh localhost with Match exec for teleport: .19s

[timothyb89]

After speaking with @russjones again, our proposal is to start with a simpler solution and iterate on it over time, particularly as @smallinsky's work on SNI routing progresses.

We'd like to start with a single new subcommand, tsh config ssh, which will generate the previously suggested Host config:

Host *.foo.example.com !proxy.foo.example.com
    ProxyCommand ssh -p 3023 proxy.foo.example.com -s proxy:$(echo %h | cut -d '.' -f 1):3022

... with a small addendum to use ~/.tsh/known_hosts for host key verification. This may require a change to tsh's own formatting of ~/.tsh/known_hosts, but we believe it should be possible to make compatible with both Teleport and OpenSSH.

We'd prefer to hold off on introducing any additional subcommands; a test command of some sort to use in tandem with Match exec ... may come later, but we'll need to look into maintaining a local store of hosts. We'll invoke ssh directly for the ProxyCommand rather than using a wrapper, though I suspect this may prevent compatibility with Windows until a tsh connect command is ready.

[klizhentas]

@timothyb89 sounds good, but a quick nitpick on the new command naming, I think tsh ssh config would be more consistent with tsh app config and tsh db config we have already. Would be also helpful to have tsh config --remove to remove the section from generated config similarly to how google console does it.

[klizhentas]

@timothyb89 chatted to @russjones - it seems like tsh config ssh is not going to work, so I agree with tsh config assuming we would also port tsh config app and tsh config db while hiding the legacy tsh app config and tsh db config cc @r0mant

[timothyb89]

@klizhentas I agree that it would be more consistent, however I'm not sure it's possible to implement that without potentially introducing a conflict. config seems to be a valid Teleport node name, so a special case/reserved word might be problematic. I'm certainly open to suggestions for ways we could make it work, though. - already addressed, whoops!

As for removal, my plan was to leave the editing the file entirely up to the user - we'll output the config snippet to stdout which the user is free to insert into their ~/.ssh/config as desired (redirecting, copy and paste, etc), which might imply they'd need to remove it themselves as well. This is the approach ASA uses.

[r0mant]

@timothyb89 chatted to @russjones - it seems like tsh config ssh is not going to work, so I agree with tsh config assuming we would also port tsh config app and tsh config db while hiding the legacy tsh app config and tsh db config cc @r0mant

@klizhentas Hmm, that might lead to a bit confusing UX IMHO. Right now we've been designing all k8s, application and database access related commands to be in their respective subcommand "namespaces" (tsh app ... and tsh db ...) which works out quite nicely from UX perspective. Same for teleport subcommands i.e. there are teleport app start and teleport db start to simplify starting respective agents.

With the proposed change it would be kind of weird if the users will need to run tsh config db but tsh db XXX for everything else (tsh db ls/#/logout/env/connect, teleport db start, etc.). Unless we want to invert those as well, which would I think pollute the global tsh command namespace and may not even be possible to implement for the same reason (e.g. "tsh login db" would probably not be possible since "db" could be a valid cluster name).

I think the way we've been approaching it so far, is that since SSH is our oldest protocol, commands related to it don't really have any "subcommand" namespace, while new access types we're adding (kube/app/db) are grouped under respective subcommands. So for example what we have now is:

$ tsh login  # retrieve cluster (effectively, SSH) credentials
$ tsh (kube/app/db) login  # retrieve k8s/app/db credentials

$ tsh ls  # shows SSH servers
$ tsh (kube/app/db) ls   # shows K8s/app/db servers

By this analogy, just leaving tsh config command to be SSH specific and keeping tsh app/db config commands seems consistent with the current UX. Is this approach not desired?

[timothyb89]

@klizhentas Any thoughts on @r0mant's suggestion? I'm happy to implement either option.

[timothyb89]

I've just posted a pull request with an initial implementation here: #7437

[klizhentas]

@timothyb89 let's go with @r0mant suggestion cc @russjones