Testnet: expanding existing deployment results in network error

[coesensbert]

Terraform Version

• provider registry.terraform.io/threefoldtech/grid v0.1.32

Affected Resource(s)

Please list the resources as a list, for example:

• full vm's

Terraform Configuration Files

terraform {
  required_providers {
    grid = {
      source = "threefoldtech/grid"
    }
  }
}

provider "grid" {
}

resource "grid_network" "net1" {
    nodes = [186, 188, 189, 190]
    ip_range = "10.30.0.0/16"
    name = "tfopstestnetworkk"
    description = "threefold ops testnet networkk"
    add_wg_access = true
}
resource "grid_deployment" "d1" {
  node = 186
  network_name = grid_network.net1.name
  ip_range = lookup(grid_network.net1.nodes_ip_range, 186, "")
  disks {
    name = "pub02data"
    size = 100
  }
    vms {
    name = "tfchaintestpub02"
    description ="Threefold TFchain Testnet public node 02"
    flist = "https://hub.grid.tf/tf-official-vms/ubuntu-20.04-lts.flist"
    cpu = 4
    publicip = true
    publicip6 = true
    memory = 8192
#    rootfs_size = 100000
    entrypoint = "/init.sh"
    mounts {
        disk_name = "pub02data"
        mount_point = "/data"
    }
    planetary = true
    env_vars = {
      SSH_KEY ="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDYNeJXJV2FNEwuQz6e0jkKeqRbKWwftBKq+sjSTqa2x"
    }
  }
}
resource "grid_deployment" "d2" {
  node = 188
  network_name = grid_network.net1.name
  ip_range = lookup(grid_network.net1.nodes_ip_range, 188, "")
  disks {
    name = "pub03data"
    size = 100
  }
   vms {
    name = "tfchaintestpub03"
    description ="Threefold TFchain Testnet public node 03"
    flist = "https://hub.grid.tf/tf-official-vms/ubuntu-20.04-lts.flist"
    cpu = 4
    publicip = true
    publicip6 = true
    memory = 8192
#    rootfs_size = 100000
    entrypoint = "/init.sh"
    mounts {
        disk_name = "pub03data"
        mount_point = "/data"
    }
    planetary = true
    env_vars = {
      SSH_KEY ="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDYNeJXJV2FNEwuQz6e0jkKeqRbKWwftBKq+sjSTqa2x"
    }
  }
}
resource "grid_deployment" "d3" {
  node = 189
  network_name = grid_network.net1.name
  ip_range = lookup(grid_network.net1.nodes_ip_range, 189, "")
  disks {
    name = "pub04data"
    size = 100
  }
   vms {
    name = "tfchaintestpub04"
    description ="Threefold TFchain Testnet public node 04"
    flist = "https://hub.grid.tf/tf-official-vms/ubuntu-20.04-lts.flist"
    cpu = 4
    publicip = true
    publicip6 = true
    memory = 8192
#    rootfs_size = 100000
    entrypoint = "/init.sh"
    mounts {
        disk_name = "pub04data"
        mount_point = "/data"
    }
    planetary = true
    env_vars = {
      SSH_KEY ="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDYNeJXJV2FNEwuQz6e0jkKeqRbKWwftBKq+sjSTqa2x"
    }
  }
}
resource "grid_deployment" "d4" {
  node = 190
  network_name = grid_network.net1.name
  ip_range = lookup(grid_network.net1.nodes_ip_range, 190, "")
  disks {
    name = "data"
    size = 100
  }
    vms {
    name = "tfchaintestgraphql02"
    description ="testnet Graphql 02"
    flist = "https://hub.grid.tf/tf-official-vms/ubuntu-20.04-lts.flist"
    cpu = 4
    publicip = true
    publicip6 = true
    memory = 16384
#    rootfs_size = 25000
    mounts {
        disk_name = "data"
        mount_point = "/data"
    }
    planetary = true
    env_vars = {
      SSH_KEY ="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDYNeJXJV2FNEwuQz6e0jkKeqRbKWwftBKq+sjSTqa2x"
    }
  }
}
output "wg_config" {
    value = grid_network.net1.access_wg_config
}
output "node1_vm1_ip" {
    value = grid_deployment.d1.vms[0].ip
}
output "public_ip" {
    value = grid_deployment.d1.vms[0].computedip
}
output "public_ip6" {
    value = grid_deployment.d1.vms[0].computedip6
}
output "ygg_ip" {
    value = grid_deployment.d1.vms[0].ygg_ip
}

Actual Behavior

This error:

grid_network.net1: Modifying... [id=94446cd5-cafc-4847-970d-2d4cd3283cf0]
grid_network.net1: Still modifying... [id=94446cd5-cafc-4847-970d-2d4cd3283cf0, 10s elapsed]
grid_network.net1: Still modifying... [id=94446cd5-cafc-4847-970d-2d4cd3283cf0, 20s elapsed]
grid_network.net1: Still modifying... [id=94446cd5-cafc-4847-970d-2d4cd3283cf0, 30s elapsed]
grid_network.net1: Still modifying... [id=94446cd5-cafc-4847-970d-2d4cd3283cf0, 40s elapsed]
grid_network.net1: Still modifying... [id=94446cd5-cafc-4847-970d-2d4cd3283cf0, 50s elapsed]
grid_network.net1: Still modifying... [id=94446cd5-cafc-4847-970d-2d4cd3283cf0, 1m0s elapsed]
grid_network.net1: Modifications complete after 1m7s [id=94446cd5-cafc-4847-970d-2d4cd3283cf0]
╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for grid_deployment.d4 to include new values learned so far during apply, provider "registry.terraform.io/threefoldtech/grid" produced an invalid new value for .ip_range: was cty.StringVal(""), but now cty.StringVal("10.30.7.0/24").
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

Trying a second time always works.

Steps to Reproduce

1. deploy a vm with network
2. add a second vm to the existing deployment

Extra info

First deployment works, error comes up consistently when adding to the deployment.

• Deploy one vm with internal network -> works
• Deploy second vm within the same deployment (so extending the main.tf) -> above error
• Doing an apply again after the error makes the deployment work the second time. This behavior (terraform apply again after the error) is also consistent.

[OmarElawady]

Yeah, this limitation is documented here, but it looks like terraform supports updating the computed value during planning if the to-be-known value is known at planning time
hashicorp/terraform-plugin-sdk#818 (comment)

[coesensbert]

Had the error again while expanding from a 2 to a 3 vm deployment

grid_network.net1: Modifying... [id=20778380-9ce2-4231-9992-840b352c5300]
grid_network.net1: Still modifying... [id=20778380-9ce2-4231-9992-840b352c5300, 10s elapsed]
grid_network.net1: Still modifying... [id=20778380-9ce2-4231-9992-840b352c5300, 20s elapsed]
grid_network.net1: Still modifying... [id=20778380-9ce2-4231-9992-840b352c5300, 30s elapsed]
grid_network.net1: Still modifying... [id=20778380-9ce2-4231-9992-840b352c5300, 40s elapsed]
grid_network.net1: Still modifying... [id=20778380-9ce2-4231-9992-840b352c5300, 50s elapsed]
grid_network.net1: Modifications complete after 50s [id=20778380-9ce2-4231-9992-840b352c5300]
╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for grid_deployment.d3 to include new values learned so far during apply, provider "registry.terraform.io/threefoldtech/grid" produced an invalid new value for .ip_range: was
│ cty.StringVal(""), but now cty.StringVal("10.41.5.0/24").

[OmarElawady]

Closing as duplicate of #13