S3 Versioning set to false and object_lock_configuration set to Enabled = bucket creation fail

[TomGudman]

Description

If versioning is false and object_lock is enabled, terraform/aws will complain that it cannot change the versioning status because the lock is in place. Commenting the versioning block fixes the issue.

Versions

• Terraform: v0.14.7 and Terraform 0.15.1

• Terragrunt v0.28.7 and Terragrunt 0.29.1

• AMD64 arch

• provider registry.terraform.io/hashicorp/aws v3.37.0

• terraform-aws-modules/s3-bucket/aws 2.1.0

Reproduction

1. Have a versioning block with an object_lock

╷
│ Error: Error putting S3 versioning: InvalidBucketState: An Object Lock configuration is present on this bucket, so the versioning state cannot be changed.
│ 	status code: 409, request id: 7309TRQPXQRXJZKH, host id: DaKsKeOtji12fJgyyh4Z/CEZAAX2oVIhW4tGbi3G+2mlJWD6I96qmF2WTWHZdE4n48wsMSfVt5k=
│ 
│   with module.s3_bucket.aws_s3_bucket.this[0],
│   on .terraform/modules/s3_bucket/main.tf line 5, in resource "aws_s3_bucket" "this":
│    5: resource "aws_s3_bucket" "this" {
│ 

Code Snippet to Reproduce

We use terragrunt as an interface to terraform but I am mainly only passing on the variables from terragrunt to terraform so that would be :

  versioning = {                                                                                                                                                                                                      
    enabled = false
  }

  [..]

  object_lock_configuration = {
    object_lock_enabled = "Enabled"
    rule = {
      default_retention = {
        mode = "GOVERNANCE", 
        days = "1" 
      }
    }
  }

However the below is how I called the s3 module with

• var.versioning = false
• and var.object_lock_configuration like the above

module "s3_bucket" {
  source = "terraform-aws-modules/s3-bucket/aws"

  create_bucket = var.create_bucket
  tags          = var.tags
  bucket        = var.bucket
  bucket_prefix = var.bucket_prefix
  acl           = var.acl
  attach_policy = var.attach_policy
  policy        = var.policy

  versioning = {                                           ###  Commenting this block                                                                                                                                                         
    enabled = var.versioning                        ###  allows to go through the
  }                                                               ###   S3 bucket creation.

  lifecycle_rule = var.lifecycle_rule

  # S3 bucket-level Public Access Block configuration
  block_public_acls       = var.block_public_acls
  block_public_policy     = var.block_public_policy
  ignore_public_acls      = var.ignore_public_acls
  restrict_public_buckets = var.restrict_public_buckets

  # Encryption by default
  server_side_encryption_configuration = {
    rule = {
      apply_server_side_encryption_by_default = {
        sse_algorithm = "AES256"
      }
    }
  }
  object_lock_configuration = var.object_lock_configuration
}

Expected behavior

I expect the bucket to be created successfully with versioning set to false (instead of being absent) and obviously the lock enabled.

Actual behavior

Impossible to create a bucket with object_lock_configuration and versioning set to false without an error.

Workaround

Comment the versioning {} block but unfortunately we use terragrunt to reuse this s3 module and feed it variables. We don't want to create separate terragrunt modules one with versioning variable and no object_lock for and one without versioning but with object_lock. It's possible but annoying.

Additional context

My terragrunt output showing

$ terragrunt apply

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # module.s3_bucket.aws_s3_bucket.this[0] will be created
  + resource "aws_s3_bucket" "this" {
      + acceleration_status         = (known after apply)
      + acl                         = "private"
      + arn                         = (known after apply)
      + bucket                      = "da-broken-bucket"
      + bucket_domain_name          = (known after apply)
      + bucket_regional_domain_name = (known after apply)
      + force_destroy               = false
      + hosted_zone_id              = (known after apply)
      + id                          = (known after apply)
      + region                      = (known after apply)
      + request_payer               = (known after apply)
      + tags                        = {
          + "Environment" = "Development"
        }
      + website_domain              = (known after apply)
      + website_endpoint            = (known after apply)

      + lifecycle_rule {
          + abort_incomplete_multipart_upload_days = 7
          + enabled                                = true
          + id                                     = "Delete logs after X days"

          + expiration {
              + days = 30
            }
        }
      + lifecycle_rule {
          + abort_incomplete_multipart_upload_days = 7
          + enabled                                = true
          + id                                     = "Move files to lesser tier after X days"

          + transition {
              + days          = 180
              + storage_class = "GLACIER"
            }
          + transition {
              + days          = 60
              + storage_class = "STANDARD_IA"
            }
        }

      + object_lock_configuration {
          + object_lock_enabled = "Enabled"

          + rule {
              + default_retention {
                  + days = 1
                  + mode = "GOVERNANCE"
                }
            }
        }

      + server_side_encryption_configuration {
          + rule {
              + apply_server_side_encryption_by_default {
                  + sse_algorithm = "AES256"
                }
            }
        }

      + versioning {
          + enabled    = false
          + mfa_delete = false
        }
    }

  # module.s3_bucket.aws_s3_bucket_public_access_block.this[0] will be created
  + resource "aws_s3_bucket_public_access_block" "this" {
      + block_public_acls       = true
      + block_public_policy     = true
      + bucket                  = (known after apply)
      + id                      = (known after apply)
      + ignore_public_acls      = true
      + restrict_public_buckets = true
    }

Plan: 2 to add, 0 to change, 0 to destroy.
Changes to Outputs:
  + s3_bucket_arn  = (known after apply)
  + s3_bucket_name = (known after apply)

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

module.s3_bucket.aws_s3_bucket.this[0]: Creating...
╷
│ Error: Error putting S3 versioning: InvalidBucketState: An Object Lock configuration is present on this bucket, so the versioning state cannot be changed.
│ 	status code: 409, request id: 7309TRQPXQRXJZKH, host id: DaKsKeOtji12fJgyyh4Z/CEZAAX2oVIhW4tGbi3G+2mlJWD6I96qmF2WTWHZdE4n48wsMSfVt5k=
│ 
│   with module.s3_bucket.aws_s3_bucket.this[0],
│   on .terraform/modules/s3_bucket/main.tf line 5, in resource "aws_s3_bucket" "this":
│    5: resource "aws_s3_bucket" "this" {
│ 
╵
ERRO[0020] Hit multiple errors:
Hit multiple errors:
exit status 1 

[antonbabenko]

Hi, @TomGudman !

This one is hard. There is a dedicated note about Terragrunt in README.

There was a similar issue with cors_rule argument - #79

We can make similar changes to this module for all other arguments with type = any if necessary.

What do you think? Can you verify that such a solution would work for you?

[TomGudman]

Hi Anton The issue you are referencing is a terragrunt syntax issue. Whereas it seems my problem is more an ordering issue or incompatibility issue; As per AWS error message (and not terragrunt/terraform). BUT I will try, I will be surprised if that work That said, thanks for the reply and reference to the jsonencode() trick because I am pretty sure I met this problem on a different topic 2 months ago.

…

On 30 Apr 2021, at 17:51, Anton Babenko ***@***.***> wrote:  Hi, @TomGudman ! This one is hard. There is a dedicated note about Terragrunt in README. There was a similar issue with cors_rule argument - #79 We can make similar changes to this module for all other arguments with type = any if necessary. What do you think? Can you verify that such a solution would work for you? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[TomGudman]

For reference the error is :

Error putting S3 versioning: InvalidBucketState: An Object Lock configuration is present on this bucket, so the versioning state cannot be changed. status code: 409

That's AWS telling terraform (terragrunt in my case) that Versioning (despite being set to false) cannot be changed because of the Object Lock state.

Again, happy to try the JSON thing but I am skeptical.

[TomGudman]

As expected jsonencode() does not fix the AWS error.

Terragrunt inputs

inputs
{
  bucket = "foobar"
  versioning = false
  tags = {
    Environment = "${local.ENVIRONMENT_NAME}"
  }

  # Enable WORM 
  # Ref: https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html
  object_lock_configuration = jsonencode({
    object_lock_enabled = "Enabled"
    rule = {
      default_retention = {
        mode = "GOVERNANCE", # FIXME: Switch to COMPLIANCE once working
        days = "${local.WORM_RETENTION}" 
      }
    }
  })

  # Enable data lifecycle : automatically delete or decrease tiers 
  lifecycle_rule = jsonencode([ 
    {
      id      = "Delete logs after X days"
      enabled = true
      prefix  = ""
      abort_incomplete_multipart_upload_days = 7
      expiration = {
        days = "${local.RETENTION_DAYS}"
      }
    },
    {
      id      = "Move files to lesser tier after X days"
      enabled = true
      prefix  = ""
      abort_incomplete_multipart_upload_days = 7
      transition = [
        {
          days          = 60 
          storage_class = "STANDARD_IA"
        },
        {
          days          = 180 
          storage_class = "GLACIER"
        },
      ]
    }
  ])
}

Terragrunt module calling the terraform-aws-s3-bucket module

module "s3_bucket" {
  source = "terraform-aws-modules/s3-bucket/aws"

  create_bucket = var.create_bucket
  tags          = var.tags
  bucket        = var.bucket
  bucket_prefix = var.bucket_prefix
  acl           = var.acl
  attach_policy = var.attach_policy
  policy        = var.policy

  versioning = {                            # if I comment this
    enabled = var.versioning        # then it will create the
  }                                                # object lock

  lifecycle_rule = var.lifecycle_rule

  # S3 bucket-level Public Access Block configuration
  block_public_acls       = var.block_public_acls
  block_public_policy     = var.block_public_policy
  ignore_public_acls      = var.ignore_public_acls
  restrict_public_buckets = var.restrict_public_buckets

  # Encryption by default
  server_side_encryption_configuration = {
    rule = {
      apply_server_side_encryption_by_default = {
        sse_algorithm = "AES256"
      }
    }
  }
  object_lock_configuration = var.object_lock_configuration
}

Terragrunt output

$ terragrunt apply
Acquiring state lock. This may take a few moments...
module.s3_bucket.aws_s3_bucket.this[0]: Refreshing state... [id=foobar]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # module.s3_bucket.aws_s3_bucket.this[0] will be created
  + resource "aws_s3_bucket" "this" {
      + acceleration_status         = (known after apply)
      + acl                         = "private"
      + arn                         = (known after apply)
      + bucket                      = "foobar"
      + bucket_domain_name          = (known after apply)
      + bucket_regional_domain_name = (known after apply)
      + force_destroy               = false
      + hosted_zone_id              = (known after apply)
      + id                          = (known after apply)
      + region                      = (known after apply)
      + request_payer               = (known after apply)
      + tags                        = {
          + "Environment" = "Development"
        }
      + website_domain              = (known after apply)
      + website_endpoint            = (known after apply)

      + lifecycle_rule {
          + abort_incomplete_multipart_upload_days = 7
          + enabled                                = true
          + id                                     = "Delete logs after X days"

          + expiration {
              + days = 30
            }
        }
      + lifecycle_rule {
          + abort_incomplete_multipart_upload_days = 7
          + enabled                                = true
          + id                                     = "Move files to lesser tier after X days"

          + transition {
              + days          = 180
              + storage_class = "GLACIER"
            }
          + transition {
              + days          = 60
              + storage_class = "STANDARD_IA"
            }
        }

      + object_lock_configuration {
          + object_lock_enabled = "Enabled"

          + rule {
              + default_retention {
                  + days = 1
                  + mode = "GOVERNANCE"
                }
            }
        }

      + server_side_encryption_configuration {
          + rule {
              + apply_server_side_encryption_by_default {
                  + sse_algorithm = "AES256"
                }
            }
        }

      + versioning {
          + enabled    = false
          + mfa_delete = false
        }
    }

  # module.s3_bucket.aws_s3_bucket_public_access_block.this[0] will be created
  + resource "aws_s3_bucket_public_access_block" "this" {
      + block_public_acls       = true
      + block_public_policy     = true
      + bucket                  = (known after apply)
      + id                      = (known after apply)
      + ignore_public_acls      = true
      + restrict_public_buckets = true
    }

Plan: 2 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + s3_bucket_arn  = (known after apply)
  + s3_bucket_name = (known after apply)

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

module.s3_bucket.aws_s3_bucket.this[0]: Creating...
╷
│ Error: Error putting S3 versioning: InvalidBucketState: An Object Lock configuration is present on this bucket, so the versioning state cannot be changed.
│ 	status code: 409, request id: R3TW0HJ2C2FSCSSC, host id: FGrrxywcXQFLgc+Vxg9mKgtiab8TYmK9HiIN3eMWmyYciGLuFBNa92qKeiyqcxTDYcgUWiD/GWo=
│ 
│   with module.s3_bucket.aws_s3_bucket.this[0],
│   on .terraform/modules/s3_bucket/main.tf line 5, in resource "aws_s3_bucket" "this":
│    5: resource "aws_s3_bucket" "this" {
│ 
╵
Releasing state lock. This may take a few moments...
ERRO[0066] Hit multiple errors:
Hit multiple errors:
exit status 1 

[TomGudman]

Actually versioning is a REQUIREMENT for Object Lock... Therefore you cannot disable it. The documentation isn't clear. But the AWS console helped me confirm the behaviour. Before enabling object lock, versioning was set to disable. Once enabled, versioning is set to enabled and the option is greyed out. I have not tested yet but once confirmed, I will update the ticket.

[ghost]

I am also having this exact same error even with the main.tf having versioning enabled
here is my code

`Terraform v1.0.1
on darwin_amd64

• provider registry.terraform.io/hashicorp/aws v3.48.0
  `

resource "aws_s3_bucket" "enabled_via_object" {
bucket = "test-bucket"
acl = "private"
versioning {
enabled = true
}
object_lock_configuration = {
object_lock_enabled = "Enabled"
}
}

terraform init
terraform plan -out tf.plan
terraform show -json tf.plan > tf.json

it fails to create the plan
terraform plan -out tf.plan

╷
│ Error: Unsupported argument
│
│ on main.tf line 9, in resource "aws_s3_bucket" "enabled_via_object":
│ 9: object_lock_configuration = {
│
│ An argument named "object_lock_configuration" is not expected here. Did you mean to define a block of type "object_lock_configuration"?
`

[github-actions]

This issue has been automatically marked as stale because it has been open 30 days
with no activity. Remove stale label or comment or this issue will be closed in 10 days

[github-actions]

This issue was automatically closed because of stale in 10 days

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.