Unable (difficult) to use with GCP's KMS on Terraform Cloud

[multani]

Hello,

I was interested to use the sops provider on Terraform Cloud to decrypt secrets encrypted via GCP's KMS, but it's not possible to have it work correctly at the moment:

• sops uses the Google Cloud SDK, which reads its configuration from the GOOGLE_APPLICATION_CREDENTIALS environment variable.
• The GOOGLE_APPLICATION_CREDENTIALS variable must contain the path to a service account private key file
• In Terraform Cloud, we can setup environment variables, but we can't write files directly in the workspace.
• So the sops provider (sops actually) is never able to find the credentials file and to authenticate against GCP :(

I understand this is more a sops / Google Cloud SDK issue, rather than the sops provider, but since this mostly affects restricted environment where we'd like to run Terraform, I wanted to raise this here.

There are a few possible alternatives at the moment:

• Commit the credentials file directly into the repository used by Terraform Cloud and export GOOGLE_APPLICATION_CREDENTIALS to point to this local file. This is, for obvious reason, a huge no-no :)

• I haven't tried it, but it might be possible to work around the limitation with local_file, the credentials in a Terraform Cloud variable and a dependency between the local_file and the data source. Something like this:

  # Considering the following Terrafom Cloud variables:
  #
  # * var.google_credentials: the content of the credentials file, as a Terraform
  #   Cloud HCL secret
  #
  # * `GOOGLE_APPLICATION_CREDENTIALS`: a Terraform Cloud environment variable
  #   containing a path name, like `gcp-creds.json`
  
  resource "local_file" "google_credentials" {
    filename          = "gcp-creds.json" # Must be the same as $GOOGLE_APPLICATION_CREDENTIALS
    content           = var.google_credentials
    sensitive_content = true
  }
  
  provider "sops" {}
  
  data "sops_file" "secrets" {
    source_file = "secrets.yaml"
    depends_on  = [local_file.google_credentials]
  }

  I haven't tried: it's a bit ugly, but that may work is sops initialization is
  lazy :)

• Support initializing sops with an alternative variable, like the Terraform provider itself is doing using GOOGLE_CREDENTIALS.

  Without some help from sops, I'm not sure if that's doable though.

Any other idea on how to use the sops provider with Terraform Cloud and GCP?

[carlpett]

Hm, tricky. One "solution" is to run with self-hosted Terraform agents on GCE (then you will implicitly have service accounts that can be given permissions on the key), but that requires an Enterprise license, so might not be within reach for everyone.
As you point out, any other solution requires a bit of help from sops itself, I'm not that keen on the provider starting to write files with secrets, to be honest.

[multani]

One "solution" is to run with self-hosted Terraform agents on GCE (then you will implicitly have service accounts that can be given permissions on the key), but that requires an Enterprise license, so might not be within reach for everyone.

That would indeed be the best, but the price to access these self-hosted agents from Terraform Cloud is completely outside our price range so it definitely won't be an option for us any time soon 😞

As you point out, any other solution requires a bit of help from sops itself.

I put all my hopes in getsops/sops#953 for now, let's see how that goes 😁

[multani]

getsops/sops#953 + https://github.com/carlpett/terraform-provider-sops/releases/tag/v0.7.1 work like a charm on Terraform Cloud now :)