bug: vault_kv_secret_v2 Permission denied for prefix/metadata/my/path/here

[kiwimato]

Terraform Version

$ terraform -v
Terraform v1.4.6
on linux_amd64

Affected Resource(s)

• vault_kv_secret_v2

Terraform Configuration Files

resource "vault_kv_secret_v2" "test" {
  mount           = "prefix"
  name             = "my/path/here"
  cas                 = 1
  delete_all_versions = true
  data_json = jsonencode(
    {
      bam     = "bam",
    }
  )
}

Debug Output

Actual request after setting TF_LOG=DEBUG:

2023-06-02T18:31:56.141+0200 [INFO]  provider.terraform-provider-vault_v3.15.2_x5: 2023/06/02 18:31:56 [DEBUG] Reading metadata for KVV2 secret at prefix/metadata/my/path/here: timestamp=2023-06-02T18:31:56.140+0200
2023-06-02T18:31:56.141+0200 [INFO]  provider.terraform-provider-vault_v3.15.2_x5: 2023/06/02 18:31:56 [DEBUG] Vault API Request Details:
---[ REQUEST ]---------------------------------------
GET /v1/prefix/metadata/my/path/here HTTP/1.1
Host: redacted.system
User-Agent: Go-http-client/1.1
X-Vault-Request: true
X-Vault-Token: redacted
Accept-Encoding: gzip

Response:

---[ RESPONSE ]--------------------------------------
HTTP/2.0 403 Forbidden
Content-Length: 60
Cache-Control: no-store
Content-Type: application/json
Date: Fri, 02 Jun 2023 16:31:56 GMT
Strict-Transport-Security: max-age=15724800; includeSubDomains

Expected Behavior

The secrets get created in Vault without Terraform popping out any errors.

Actual Behavior

The Vault secrets ARE getting created, however, the command fails afterwards with the error below.
After creation it also fails on terraform plan, I assume it tries to read data from the wrong URL after creation.
I also tried using a data instead to read whatever was created there and it gives the same error, so it might confirm it's a problem reading it.

│ Error: Error making API request.
│ 
│ URL: GET https://redacted.system/v1/prefix/metadata/my/path/here
│ Code: 403. Errors:
│ 
│ * 1 error occurred:
│ 	* permission denied

Tried debugging it, and it seems even with my admin credentials the path containing metadata doesn't exist but the one with data does:

$ vault read prefix/data/my/path/here
Key         Value
---         -----
data        map[redacted]
metadata    map[created_time:2023-06-02T15:47:09.015986611Z custom_metadata:<nil> deletion_time: destroyed:false version:1]

$ vault read prefix/metadata/my/path/here
Error reading prefix/metadata/my/path/here: Error making API request.

URL: GET https://redacted.system/v1/prefix/metadata/my/path/here
Code: 403. Errors:

* 1 error occurred:
	* permission denied

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. terraform plan - no errors, just says it wants to create the secret.
2. terraform apply - errors out with permission denied
3. terraform plan - errors out with permission denied

Important Factoids

None that I know of.

References

I assume it could be related to #1719
cc @vinay-gopalan

[fairclothjm]

@kiwimato Hello, can you please confirm that your policy allows reading metadata? Based on the 403 error given for vault read prefix/data/my/path/here this seems likely.

See https://developer.hashicorp.com/vault/docs/secrets/kv/kv-v2#acl-rules

[YohannHammad]

Hello, I have a similar problem with this configuration :

[...]
resource "vault_kv_secret_v2" "input-queue" {
  mount                      = local.vault_mount
  name                       = "/XXXXX/${var.environment}/input-queue"
  cas                        = 1
  delete_all_versions        = false
  data_json                  = jsonencode(
    {
      name       = aws_sqs_queue.XXXX-input-queue.name,
    }
  )
}
[...]

And this policy :

[...]
path "app/metadata/XXXXX/XXX/*" {
    capabilities = ["read", "delete"]
}
[...]

The error :

│ Error: error writing custom metadata to /app/metadata//XXXXX/XXXXXX/input-queue, err=Error making API request.
│ 
│ URL: PUT https://XXXXXXXX/v1/app/metadata/XXXXX/XXXXXX/input-queue
│ Code: 403. Errors:
│ 
│ * 1 error occurred:
│ 	* permission denied
│ 
│ 
│ 
│   with vault_kv_secret_v2.input-queue,
│   on deployment.tf line 79, in resource "vault_kv_secret_v2" "input-queue":
│   79: resource "vault_kv_secret_v2" "input-queue" {
│ 

This is weird because I'm not trying to write metadata: there is no custom_metadata key in the resource.

[gage-russell]

Hello, I have a similar problem with this configuration :

[...]
resource "vault_kv_secret_v2" "input-queue" {
mount = local.vault_mount
name = "/XXXXX/${var.environment}/input-queue"
cas = 1
delete_all_versions = false
data_json = jsonencode(
{
name = aws_sqs_queue.XXXX-input-queue.name,
}
)
}
[...]
And this policy :

[...]
path "app/metadata/XXXXX/XXX/*" {
    capabilities = ["read", "delete"]
}
[...]

The error :

│ Error: error writing custom metadata to /app/metadata//XXXXX/XXXXXX/input-queue, err=Error making API request.
│ 
│ URL: PUT https://XXXXXXXX/v1/app/metadata/XXXXX/XXXXXX/input-queue
│ Code: 403. Errors:
│ 
│ * 1 error occurred:
│ 	* permission denied
│ 
│ 
│ 
│   with vault_kv_secret_v2.input-queue,
│   on deployment.tf line 79, in resource "vault_kv_secret_v2" "input-queue":
│   79: resource "vault_kv_secret_v2" "input-queue" {
│ 

This is weird because I'm not trying to write metadata: there is no custom_metadata key in the resource.

@YohannHammad I am experiencing the same issue. Error writing custom metadata but I am not trying to write custom metadata. I wonder if the terraform provider is automatically writing some custom metadata anytime the secret is updated?