Terraform provisioners cannot use Teleport as a bastion host.

[donovan]

What happened:

I am attempting to use terraform file and remote-exec provisioners to provision EC2 instances behind a teleport proxy. I am making use of the terraform provisioner connection which supports connecting through a Bastion Host with SSH

When I run terraform apply the connection hangs indefinitely when it attempts
to use the file or remote-exec provisioner:

null_resource.argo_instance: Destroying... (ID: 5184140358547470671)
null_resource.argo_instance: Destruction complete after 0s
null_resource.argo_instance: Creating...
  triggers.%:                    "" => "1"
  triggers.cluster_instance_ids: "" => "i-028b7f6a34e285916"
null_resource.argo_instance: Provisioning with 'file'...
null_resource.argo_instance: Still creating... (10s elapsed)
null_resource.argo_instance: Still creating... (20s elapsed)

What you expected to happen:

I expect to be able to use a terraform file or remote-exec provisioner to provision EC2 instances behind a teleport bastion host.

How to reproduce it (as minimally and precisely as possible):

In AWS (or other cloud) create a teleport proxy with a public ip. Then create a node_instance with a private ip running a teleport node. Ensure that the proxy can connect to 3022 on the node. Assert that standard teleport features are working such that you can connect to the node via the proxy:

$ tsh --user foo ssh ubuntu@node.example.net
ubuntu@node:~$ hostname -f
node.example.net
ubuntu@node:~$ whoami
ubuntu
ubuntu@node:~$ logout
the connection was closed on the remote side on  18 Apr 19 10:44 NZST

$ tsh ls
Node Name         Address         Labels
------------------------------ --------------- ----------------
node.example.net  10.9.0.199:3022 codename=bionic distro=Ubuntu
proxy.example.net 127.0.0.1:3022  codename=bionic distro=Ubuntu

Install terraform.

Create the following terraform file:

resource "null_resource" "node_instance" {
  connection {
    type         = "ssh"
    user         = "ubuntu"
    agent        = true
    #host         = "node.example.net"
    host         = "10.9.0.199"
    port         = 3022
    #bastion_host = "proxy.example.net"
    bastion_host = "1.2.3.4" # real public ip goes here
    bastion_user = "foo"
    bastion_port = 3023
  }

  provisioner "file" {
    source      = "test.txt"
    destination = "/home/ubuntu/test.txt"
  }

  provisioner "remote-exec" {
    inline = ["touch /home/ubuntu/foo"]
  }
}

enable the null provider:

$ cd terraform/teleport-testing/
$ ls
provider-null.tf  teleport-testing.tf  test.txt
$ cat provider-null.tf
provider "null" {
  version = "2.1.0"
}
$ terraform init
...

now run terraform with debug logging:

$ TF_LOG=TRACE terraform apply --state=terraform.tfstate . 2> "debug-$( date +%s ).log"

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  + null_resource.node_instance
      id: <computed>


Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

null_resource.node_instance: Creating...
null_resource.node_instance: Provisioning with 'file'...
null_resource.node_instance: Still creating... (10s elapsed)
null_resource.node_instance: Still creating... (20s elapsed)
^CInterrupt received.
Please wait for Terraform to exit or data loss may occur.
Gracefully shutting down...
stopping operation...

Now comment out the file provisioner in teleport-testing.tf and try again (using the remote-exec provisioner).

$ TF_LOG=TRACE terraform apply --state=terraform.tfstate . 2> "debug-$( date +%s ).log"
null_resource.node_instance: Refreshing state... (ID: 239947275875864365)

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement

Terraform will perform the following actions:

-/+ null_resource.node_instance (tainted) (new resource required)
      id: "239947275875864365" => <computed> (forces new resource)


Plan: 1 to add, 0 to change, 1 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

null_resource.node_instance: Destroying... (ID: 239947275875864365)
null_resource.node_instance: Destruction complete after 0s
null_resource.node_instance: Creating...
null_resource.node_instance: Provisioning with 'remote-exec'...
null_resource.node_instance (remote-exec): Connecting to remote host via SSH...
null_resource.node_instance (remote-exec):   Host: 10.9.0.199
null_resource.node_instance (remote-exec):   User: ubuntu
null_resource.node_instance (remote-exec):   Password: false
null_resource.node_instance (remote-exec):   Private key: false
null_resource.node_instance (remote-exec):   SSH Agent: true
null_resource.node_instance (remote-exec):   Checking Host Key: false
null_resource.node_instance (remote-exec): Using configured bastion host...
null_resource.node_instance (remote-exec):   Host: 1.2.3.4
null_resource.node_instance (remote-exec):   User: foo
null_resource.node_instance (remote-exec):   Password: false
null_resource.node_instance (remote-exec):   Private key: false
null_resource.node_instance (remote-exec):   SSH Agent: true
null_resource.node_instance (remote-exec):   Checking Host Key: false
null_resource.node_instance (remote-exec): Connecting to remote host via SSH...
null_resource.node_instance (remote-exec):   Host: 10.9.0.199
null_resource.node_instance (remote-exec):   User: ubuntu
null_resource.node_instance (remote-exec):   Password: false
null_resource.node_instance (remote-exec):   Private key: false
null_resource.node_instance (remote-exec):   SSH Agent: true
null_resource.node_instance (remote-exec):   Checking Host Key: false

Environment:

• Teleport v3.2.2 git:v3.2.2-0-gfbd6b388 go1.11.5
• Terraform v0.11.13
• Terraform provider.null v2.1.0
• Terraform provider.aws v2.3.0
• Ubuntu 18.04.2 LTS (client, bastion and node_instance):

Relevant Debug Logs If Applicable

Here are the relevant terrafrom logs:

File provisioner log:

2019-04-18T11:15:00.715+1200 [DEBUG] plugin.terraform: file-provisioner (internal) 2019/04/18 11:15:00 [DEBUG] connecting to TCP connection for SSH
2019-04-18T11:15:00.715+1200 [DEBUG] plugin.terraform: file-provisioner (internal) 2019/04/18 11:15:00 [DEBUG] Connecting to bastion: 1.2.3.4:3023
2019-04-18T11:15:01.369+1200 [DEBUG] plugin.terraform: file-provisioner (internal) 2019/04/18 11:15:01 [DEBUG] Connecting via bastion (1.2.3.4:3023) to host: 10.9.0.199:3022
2019-04-18T11:15:01.426+1200 [DEBUG] plugin.terraform: file-provisioner (internal) 2019/04/18 11:15:01 [ERROR] connection error: ssh: rejected: unknown channel type (unknown channel type: direct-tcpip)
2019-04-18T11:15:01.426+1200 [DEBUG] plugin.terraform: file-provisioner (internal) 2019/04/18 11:15:01 [WARN] retryable error: ssh: rejected: unknown channel type (unknown channel type: direct-tcpip)
2019-04-18T11:15:01.426+1200 [DEBUG] plugin.terraform: file-provisioner (internal) 2019/04/18 11:15:01 [INFO] sleeping for 8s

remote-exec provisioner log:

2019-04-18T11:27:06.436+1200 [DEBUG] plugin.terraform: remote-exec-provisioner (internal) 2019/04/18 11:27:06 [DEBUG] connecting to TCP connection for SSH
2019-04-18T11:27:06.436+1200 [DEBUG] plugin.terraform: remote-exec-provisioner (internal) 2019/04/18 11:27:06 [DEBUG] Connecting to bastion: 1.2.3.4:3023
2019-04-18T11:27:06.827+1200 [DEBUG] plugin.terraform: remote-exec-provisioner (internal) 2019/04/18 11:27:06 [DEBUG] Connecting via bastion (1.2.3.4:3023) to host: 10.9.0.199:3022
2019-04-18T11:27:06.879+1200 [DEBUG] plugin.terraform: remote-exec-provisioner (internal) 2019/04/18 11:27:06 [ERROR] connection error: ssh: rejected: unknown channel type (unknown channel type: direct-tcpip)
2019-04-18T11:27:06.879+1200 [DEBUG] plugin.terraform: remote-exec-provisioner (internal) 2019/04/18 11:27:06 [WARN] retryable error: ssh: rejected: unknown channel type (unknown channel type: direct-tcpip)
2019-04-18T11:27:06.879+1200 [DEBUG] plugin.terraform: remote-exec-provisioner (internal) 2019/04/18 11:27:06 [INFO] sleeping for 1s

Discussion

This seems to be related to #2543. I can replicate the error by trying to connect with SSH using ProxyJump (-J):

$ ssh -J foo@1.2.3.4:3023 ubuntu@10.9.0.199:3022
channel 0: open failed: unknown channel type: unknown channel type: direct-tcpip
stdio forwarding failed
ssh_exchange_identification: Connection closed by remote host

I am going to revert to using SSHD for provisioning but this is annoying to have to manage this in addition to teleport. Are there any suggestions as to how to get this working with teleport or is it a case of waiting for ProxyJump to be implemented? I don't see any connection options that I can change on the terraform side.

Notes

• The public ip 1.2.3.4, private ip, proxy and node fqdns and username foo have been changed.
• The same behaviour occurs if I use the proxy and node fqdns rather than ips in the provisioner connection.

[kontsevoy]

@klizhentas I think we must prioritize this higher.

[zmb3]

@jakule is this still an issue?