Hostname used withing Host entries in ssh config

[ghost]

On Debian 10 "buster", amd64, using rex from backports, version "(R)?ex 1.12.2".
Those points where discussed on IRC.

Describe the bug

Rex uses the Hostname sub-key of a Host section in ~/.ssh/config

How to reproduce it

Steps to reproduce the behavior:

1. Create 1 proxy machine "proxy" with 2 NICs and a running ssh-server instance;
   1.1 connect 1 NIC so that it can be reached from LAN, and give it, for example, IP 192.168.1.200/16
   1.2 connect the other one so that it can not be reached from LAN and give it IP 10.0.0.1/16
   1.3 create a user "bouncer", with no password and nologin shell;
   1.4 give "bouncer" a file ~bouncer/.ssh/authorized_keys filled with a public key "vmproxy";
2. Create 1 target machine "target" which only NIC is only reachable by "proxy";
   2.1 create a normal user "user"
   2.2 give "user" a file ~user/.ssh/authorized_keys filled with a public key "vmlan";
   2.3 give NIC the IP 10.0.0.10/16
3. on host, create a ~/.ssh/config file such as:

Host proxy
  Hostname 192.168.1.200
  User bouncer
  IdentityFile ~/.ssh/vmproxy

Host target
  Hostname 10.0.0.10
  User user
  IdentityFile ~/.ssh/vmlan
  ProxyJump proxy

4. connect to proxy via ssh. It should work.
5. connect to target via ssh. It should also work.
6. run rex -H target -u u user -e say run uptime <= will fail
7. on host, modify a ~/.ssh/config file such as:

Host proxy
  Hostname 192.168.1.200
  User bouncer
  IdentityFile ~/.ssh/vmproxy

Host target
  Hostname 10.0.0.10
Host 10.0.0.10
  User user
  IdentityFile ~/.ssh/vmlan
  ProxyJump proxy

8. run rex -H target -u u user -e say run uptime <= will work

Expected behavior

step 6 should behave the same as step 8

Circumstances

• Rex version: (R)?ex 1.12.2
• Perl version: This is perl 5, version 28, subversion 1 (v5.28.1) built for x86_64-linux-gnu-thread-multi
  (with 65 registered patches, see perl -V for more detail)
• OS running rex: Debian 10 buster, amd64
• OS managed by rex: same.
• How rex was installed: apt-get

Debug log

�[31m[2020-12-13 21:23:18] DEBUG - This is Rex version: 1.12.2
�[0m�[31m[2020-12-13 21:23:18] DEBUG - Command Line Parameters
�[0m�[31m[2020-12-13 21:23:18] DEBUG - 	e = say
�[0m�[31m[2020-12-13 21:23:18] DEBUG - 	u = user
�[0m�[31m[2020-12-13 21:23:18] DEBUG - 	H = sfiles
�[0m�[31m[2020-12-13 21:23:18] DEBUG - 	d = 1
�[0m�[31m[2020-12-13 21:23:18] DEBUG - Creating lock-file (Rexfile.lock)
�[0m�[31m[2020-12-13 21:23:18] DEBUG - Loading Rexfile
�[0m�[31m[2020-12-13 21:23:18] DEBUG - Creating new distribution class of type: Base
�[0m�[31m[2020-12-13 21:23:18] DEBUG - new distribution class of type Rex::TaskList::Base created.
�[0m�[31m[2020-12-13 21:23:18] DEBUG - Initializing Logger from parameters found in Rexfile
�[0m�[31m[2020-12-13 21:23:18] DEBUG - Executing command line code
�[0m�[31m[2020-12-13 21:23:18] DEBUG - 	say
�[0m�[31m[2020-12-13 21:23:18] DEBUG - Returning existing distribution class of type: Rex::TaskList::Base
�[0m�[31m[2020-12-13 21:23:18] DEBUG - Creating task: eval-line
�[0m�[31m[2020-12-13 21:23:18] DEBUG - 	server: sfiles
�[0m�[31m[2020-12-13 21:23:18] DEBUG - Found Net::OpenSSH and Net::SFTP::Foreign - using it as default
�[0m�[31m[2020-12-13 21:23:18] DEBUG - Returning existing distribution class of type: Rex::TaskList::Base
�[0m�[31m[2020-12-13 21:23:18] DEBUG - Returning existing distribution class of type: Rex::TaskList::Base
�[0m�[31m[2020-12-13 21:23:18] DEBUG - Returning existing distribution class of type: Rex::TaskList::Base
�[0m�[32m[2020-12-13 21:23:18] INFO - Running task eval-line on sfiles
�[0m�[31m[2020-12-13 21:23:18] DEBUG - Checking for a private key in .ssh/config
�[0m�[31m[2020-12-13 21:23:18] DEBUG - Rex::Group::Entry::Server (private_key): returning ~/.ssh/vmlan
�[0m�[31m[2020-12-13 21:23:18] DEBUG - Checking for a public key in .ssh/config
�[0m�[31m[2020-12-13 21:23:18] DEBUG - Rex::Group::Entry::Server (public_key): returning ~/.ssh/vmlan.pub
�[0m�[31m[2020-12-13 21:23:18] DEBUG - $VAR1 = '';

�[0m�[31m[2020-12-13 21:23:18] DEBUG - Auth-Information inside Task:
�[0m�[31m[2020-12-13 21:23:18] DEBUG - public_key => [[~/.ssh/vmlan.pub]]
�[0m�[31m[2020-12-13 21:23:18] DEBUG - port => [[]]
�[0m�[31m[2020-12-13 21:23:18] DEBUG - sudo_password => [[**********]]
�[0m�[31m[2020-12-13 21:23:18] DEBUG - password => [[%s]]
�[0m�[31m[2020-12-13 21:23:18] DEBUG - auth_type => [[try]]
�[0m�[31m[2020-12-13 21:23:18] DEBUG - user => [[user]]
�[0m�[31m[2020-12-13 21:23:18] DEBUG - sudo => [[]]
�[0m�[31m[2020-12-13 21:23:18] DEBUG - private_key => [[~/.ssh/vmlan]]
�[0m�[31m[2020-12-13 21:23:18] DEBUG - Using Net::OpenSSH for connection
�[0m�[31m[2020-12-13 21:23:18] DEBUG - Using user: user
�[0m�[31m[2020-12-13 21:23:18] DEBUG - Connecting to 10.0.1.10:22 (user)
�[0m�[31m[2020-12-13 21:23:18] DEBUG - get_openssh_opt()
�[0m�[31m[2020-12-13 21:23:18] DEBUG - $VAR1 = {};

�[0m�[31m[2020-12-13 21:23:18] DEBUG - OpenSSH: key_auth or not defined: 10.0.1.10:22 - user
�[0m�[31m[2020-12-13 21:23:18] DEBUG - OpenSSH options: 
�[0m�[31m[2020-12-13 21:23:18] DEBUG - $VAR1 = [
          '10.0.1.10',
          'user',
          'user',
          'port',
          22,
          'master_opts',
          [
            '-o',
            'ConnectTimeout=2',
            '-o',
            'LogLevel=QUIET'
          ],
          'default_ssh_opts',
          $VAR1->[6]
        ];

�[0m�[31m[2020-12-13 21:23:18] DEBUG - OpenSSH constructor options: 
�[0m�[31m[2020-12-13 21:23:18] DEBUG - $VAR1 = {};

�[0m�[31m[2020-12-13 21:23:18] DEBUG - Trying following auth types:
�[0m�[31m[2020-12-13 21:23:18] DEBUG - $VAR1 = [
          'key',
          'pass'
        ];

�[0m�[33m[2020-12-13 21:23:31] WARN - Can't authenticate against 10.0.1.10 (unable to establish master SSH connection: master process exited unexpectedly)
�[0m�[31m[2020-12-13 21:23:31] DEBUG - Connections in queue: 1
�[0m�[31m[2020-12-13 21:23:31] DEBUG - Destroying all cached os information
�[0m�[31m[2020-12-13 21:23:31] DEBUG - Waiting for children to finish
�[0m�[31m[2020-12-13 21:23:31] DEBUG - Need to reinitialize connections.
�[0m�[31m[2020-12-13 21:23:31] DEBUG - Returning existing distribution class of type: Rex::TaskList::Base
�[0m�[31m[2020-12-13 21:23:31] ERROR - 1 out of 1 task(s) failed:
�[0m�[31m[2020-12-13 21:23:31] ERROR - 	eval-line failed on sfiles
�[0m�[31m[2020-12-13 21:23:31] ERROR - 		Couldn't authenticate against sfiles. It may be caused by one or more of:
�[0m�[31m[2020-12-13 21:23:31] ERROR - 		 - wrong username, password, key or passphrase
�[0m�[31m[2020-12-13 21:23:31] ERROR - 		 - changed remote host key
�[0m�[31m[2020-12-13 21:23:31] ERROR - 		 at /usr/share/perl5/Rex/TaskList/Base.pm line 342.
�[0m�[31m[2020-12-13 21:23:31] DEBUG - Removing lockfile
�[0m�[31m[2020-12-13 21:23:31] DEBUG - Returning existing distribution class of type: Rex::TaskList::Base

[ferki]

Thanks for the patience, and for the highly detailed report! 👍 Providing a workaround example for others is also highly appreciated ❤️

I couldn't do a fully detailed examination of the problem yet, but I still wanted to make some notes for whoever looking at this issue next (probably me ^^):

• I assume that the IP address mismatch between the sample ssh config (10.0.0.10) and the provided debug log (10.0.1.10) is only due to a typo.
• I believe that the behavior might be slightly different depending on the SSH backend module in use (Net::SSH2 vs Net::OpenSSH). The latter uses the OpenSSH binary under the hood, so that should explain why it can pick up the correct bits from ssh config when using the workaround (which I expect to still fail with Net::SSH2 - FIXME).
• For the same reason, I believe the correct fix would be to skip ssh config parsing when Net::OpenSSH backend is used (as proposed on Skip ssh config parsing when Net::OpenSSH backend is used #1229), but it might also be possible to improve the current approach before that bigger change.