Problem using WIF authentication with google terraform provider

[jketcham]

TL;DR

I'm not sure if this is an issue with the auth action or with the google terraform provider (or something else), so apologies if this is not the correct place.

When I use the auth action to setup WIF authentication to google cloud so I can run terraform, I get this error immediately as terraform tries to initialize the backend (with state stored in Cloud Storage):

Initializing the backend...

Error: storage.NewClient() failed: dialing: google: error getting credentials using GOOGLE_APPLICATION_CREDENTIALS environment variable: unknown credential type: "external_account"

This should be reproducible with the given GitHub Actions YAML below and any terraform config that uses Cloud Storage as it's state backend.

---

A quick search turned up these issues that also dealt with WIF and experienced this unknown credential type: "external_account" error:

• Cannot create GCP KMS when using Google Workload Identity Federation getsops/sops#969
• Flakybot does not appear to accept external_account permissions googleapis/repo-automation-bots#2752

Expected behavior

Terraform is able to initialize module directory without error.

Observed behavior

Terraform returns error when initializing module directory.

Action YAML

name: 'apply terraform configs'

on:
  push:
    paths:
    - terraform/**
    branches:
    - main
  pull_request:
    paths:
    - terraform/**

env:
  GCP_PROJECT_ID: xxx
  ...snip...
  TF_IN_AUTOMATION: 1
  TF_REGISTRY_CLIENT_TIMEOUT: 60s

jobs:
  terraform:
    name: Terraform
    runs-on: self-hosted # <-- Using https://github.com/actions-runner-controller/actions-runner-controller default image
    permissions:
      contents: 'read'
      id-token: 'write'

    steps:
    - uses: actions/checkout@v2
      with:
        fetch-depth: 0

    - uses: actions/setup-node@v2
      with:
        node-version: '14'

    - uses: hashicorp/setup-terraform@v1
      with:
        terraform_version: 0.14.8

    - uses: google-github-actions/auth@v0
      with:
        workload_identity_provider: 'projects/yyy/locations/global/workloadIdentityPools/pool-ci-cd/providers/github-actions'
        service_account: 'terraform@xxx.iam.gserviceaccount.com'

    ...snip...

    - name: Terraform Format
      run: ...terraform fmt -check collected modules...

    - name: Terraform Init # <-- Error happens here
      run: ...terraform init collected modules...

    ...snip...

Additional information

I have WIF setup with the following terraform config I've already applied locally (and this is the same terraform config that the workflow is trying to init):

terraform {
  required_version = ">= 0.14"

  required_providers {
    google = {
      source  = "hashicorp/google"
      version = ">= 4.4.0"
    }
  }

  backend "gcs" {
    bucket = "xxx-terraform-state"
    prefix = "envs/prod/iam"
  }
}

provider "google" {
  project = "xxx"
}

provider "google-beta" {
  project = "xxx"
}

resource "google_iam_workload_identity_pool" "ci" {
  provider = google-beta

  workload_identity_pool_id = "pool-ci-cd"
  display_name              = "CI/CD"
  description               = "Identity pool for continuous integration service accounts."
  disabled                  = false
}

resource "google_iam_workload_identity_pool_provider" "github_actions" {
  provider = google-beta

  workload_identity_pool_id          = google_iam_workload_identity_pool.ci.workload_identity_pool_id
  workload_identity_pool_provider_id = "github-actions"
  display_name                       = "GitHub Actions"
  description                        = "Provider used by GitHub Action workflows."
  disabled                           = false

  attribute_mapping = {
    "google.subject"       = "assertion.sub"
    "attribute.actor"      = "assertion.actor"
    "attribute.repository" = "assertion.repository"
  }

  oidc {
    issuer_uri = "https://token.actions.githubusercontent.com"
  }
}

resource "google_service_account" "terraform" {
  account_id   = "terraform"
  display_name = "terraform"
  description  = "Used by terraform to apply changes"
}

resource "google_service_account_iam_member" "terraform_sa_workload_identity" {
  service_account_id = google_service_account.terraform.name
  role               = "roles/iam.workloadIdentityUser"
  member             = "principalSet://iam.googleapis.com/${google_iam_workload_identity_pool.ci.name}/attribute.repository/xxx/yyy"
}

---

This issue builds off my comment here: google-github-actions/setup-gcloud#440 (comment)

Thanks!

[sethvargo]

Per https://registry.terraform.io/providers/hashicorp/google/latest/docs/guides/provider_reference#authentication, the provider supports discovery and authentication via WIF, but it looks like it might not be supported for authenticating to the remote state endpoint. I think this is a Terraform issue, since we are correctly exporting GOOGLE_APPLICATION_CREDENTIALS and the documentation notes it should be supported.

[jketcham]

Ok, I can move this to the google terraform provider repo. Thank you.

[bharathkkb]

@jketcham Remote state is part of terraform core, not the provider and I found an open issue hashicorp/terraform#29656

[jketcham]

Ahhh there it is, thanks for the info @bharathkkb

[markbecker-work]

FYI for those running into this issue, one way around it is to generate an access token and then do:
- run: echo "GOOGLE_OAUTH_ACCESS_TOKEN=${{ steps.auth.outputs.access_token }}" >> $GITHUB_ENV

Terraform will then use the access token to auth instead of the credentials file: https://registry.terraform.io/providers/hashicorp/google/latest/docs/guides/provider_reference#access_token