(core): file asset publishing role is not used while performing cdk diff with larger templates

[mrlikl]

Describe the bug

cdk diff is not using file asset publish role when uploading larger templates to S3 while creating the change set. It is using the user/role that is configured in the CLI. If the user/role (configured in CLI) lacks permissions then AccessDenied error is thrown and fallback to template differences.

Expected Behavior

File asset publish role to be used to upload templates for cdk diff

Current Behavior

fail: Access Denied
Failed to publish one or more assets. See the error messages above for more information.

Reproduction Steps

1. Create IAM user or role with deny all S3 actions.

2. Configure your CLI with the above role/user.

3. Create a CDK project with one SQS queue

// create a stack with one sqs queue
new sqs.Queue(this, `Queue`, {});

4. Deploy

5. Add more resources that will create a larger template to the stack and execute cdk diff -v

// add 300 more sqs queue's to the same stack and perform cdk diff -v
for (let i = 0; i < 300; i++) {
      new sqs.Queue(this, `Queue${i}`, {
      });
    }   

Stack trace -

Stack CdktsStack
[23:46:13] Retrieved account ID xxxxxxxxxxxx from disk cache
[23:46:13] Assuming role 'arn:aws:iam::xxxxxxxxxxxx:role/cdk-hnb659fds-lookup-role-xxxxxxxxxxxx-us-east-1'.
[23:46:18] Retrieved account ID xxxxxxxxxxxx from disk cache
[23:46:20] Retrieved account ID xxxxxxxxxxxx from disk cache
[23:46:20] Assuming role 'arn:aws:iam::xxxxxxxxxxxx:role/cdk-hnb659fds-deploy-role-xxxxxxxxxxxx-us-east-1'.
[23:46:21] Waiting for stack CDKToolkit to finish creating or updating...
[23:46:22] Storing template in S3 at: https://cdk-hnb659fds-assets-xxxxxxxxxxxx-us-east-1.s3.us-east-1.amazonaws.com/cdk/CdktsStack/e6cbc1ec64bd0bf5f83ee188026e9c010633551b8252f036384f6dddb29932f0.yml
[23:46:22] [0%] start: Publishing e6cbc1ec64bd0bf5f83ee188026e9c010633551b8252f036384f6dddb29932f0:current
[23:46:23] [0%] check: Check s3://cdk-hnb659fds-assets-xxxxxxxxxxxx-us-east-1/cdk/CdktsStack/e6cbc1ec64bd0bf5f83ee188026e9c010633551b8252f036384f6dddb29932f0.yml
[23:46:32] [0%] upload: Upload s3://cdk-hnb659fds-assets-xxxxxxxxxxxx-us-east-1/cdk/CdktsStack/e6cbc1ec64bd0bf5f83ee188026e9c010633551b8252f036384f6dddb29932f0.yml
[100%] fail: Access Denied
[23:46:36] Failed to publish one or more assets. See the error messages above for more information.
Could not create a change set, will base the diff on template differences (run again with -v to see the reason)

The IAM user configured in my CLI does not have S3 permissions.

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.138.0

Framework Version

No response

Node.js Version

20.10

OS

macos

Language

TypeScript

Language Version

No response

Other information

No response

[pahud]

I am not sure if I get it correct.

Can you elaborate about the steps with some real code snippets or commands and share the full error message when you cdk diff -v?

Reproduction Steps
Create IAM user or role with deny all S3 actions.

Configure an environment with the role/user

[mrlikl]

Apologies ! I have updated the issue with more detailed steps and stack trace @pahud !

[JoHuang]

I got the same issue after upgrading from 2.86.0 to 2.147.0 .

[sfriedlizuehlke]

I use the following additional permissions assigned to the identity executing the cdk commands as a workaround:

[
    {
        "effect": "ALLOW",
        "actions": [
            "s3:GetBucketLocation",
            "s3:ListBucket",
            "s3:GetEncryptionConfiguration"
        ],
        "resources": [
            "arn:aws:s3:::cdk-assets-${account}-${region}"
        ]
    },
    {
        "effect": "ALLOW",
        "actions": [
            "s3:PutObject",
            "s3:GetObject"
        ],
        "resources": [
            "arn:aws:s3:::cdk-assets-${account}-${region}/*"
        ]
    }
]

[pahud]

probably related to #29718 which was fixed in 2.136.0 but this issue is still relevant in 2.138.0 and this report was at 2.147.0.

Will discuss this with the team.

[pahud]

OK I can reproduce this issue.

Looks like it would roll back to the current CDK CLI execution principal with permission denied on getBucketLocation operation.

aws-cdk/packages/cdk-assets/lib/private/handlers/files.ts

Lines 68 to 69 in 0875ef9

	case BucketOwnership.SOMEONE_ELSES_OR_NO_ACCESS:
	throw new Error(`Bucket named '${destination.bucketName}' exists, but not in account ${await account()}. Wrong account?`);

Tested with CDK 2.148.0.

[mwebber]

@pahud I get this with 2.148.0, here is the output from cdk diff -v as requested:

[15:27:03] outdir: cdk.out
[15:27:03] env: {
  CDK_DEFAULT_REGION: 'us-east-1',
  CDK_DEFAULT_ACCOUNT: '311111111112',
  CDK_OUTDIR: 'cdk.out',
  CDK_CLI_ASM_VERSION: '36.0.0',
  CDK_CLI_VERSION: '2.148.0'
}
Stack my-stack-name
[15:27:14] Retrieved account ID 311111111112 from disk cache
[15:27:14] Assuming role 'arn:aws:iam::311111111112:role/cdk-hxxxxxxxs-lookup-role-311111111112-us-east-1'.
[15:27:14] Retrieved account ID 311111111112 from disk cache
[15:27:14] Retrieved account ID 311111111112 from disk cache
[15:27:14] Assuming role 'arn:aws:iam::311111111112:role/cdk-hxxxxxxxs-deploy-role-311111111112-us-east-1'.
[15:27:14] Waiting for stack CDKToolkit to finish creating or updating...
[15:27:15] Storing template in S3 at: https://cdk-hxxxxxxxs-assets-311111111112-us-east-1.s3.us-east-1.amazonaws.com/cdk/my-stack-name/e02bf760edf8662e11000c2b56091214b60a0729b5382d4ddd8be0aeb7504f90.yml
[15:27:15] [0%] start: Publishing e02bf760edf8662e11000c2b56091214b60a0729b5382d4ddd8be0aeb7504f90:current
[15:27:15] [0%] check: Check s3://cdk-hxxxxxxxs-assets-311111111112-us-east-1/cdk/my-stack-name/e02bf760edf8662e11000c2b56091214b60a0729b5382d4ddd8be0aeb7504f90.yml
[15:27:15] [0%] upload: Upload s3://cdk-hxxxxxxxs-assets-311111111112-us-east-1/cdk/my-stack-name/e02bf760edf8662e11000c2b56091214b60a0729b5382d4ddd8be0aeb7504f90.yml
[100%] fail: Access Denied
[15:27:15] Failed to publish one or more assets. See the error messages above for more information.

[mwebber]

The workaround from @sfriedlizuehlke above works well, with one small change to the S3 bucket name pattern
arn:aws:s3:::cdk-assets-${account}-${region}/*: there is a short hash between cdk and assets

I used this (CDK/Python)

iam.PolicyStatement(
    actions=[
        "s3:GetBucketLocation",
        "s3:ListBucket",
        "s3:GetEncryptionConfiguration",
        ],
    resources=["arn:aws:s3:::cdk-*-assets-*"],
),
iam.PolicyStatement(
        actions=["s3:GetObject", "s3:PutObject"],
        resources=["arn:aws:s3:::cdk-*-assets-*/*"],
),

[sumupitchayan]

@mrlikl you mention that the IAM user configured in your CLI does not have S3 permissions, but from your stack trace it looks like the CLI is assuming the CDK Deploy Role:

[23:46:20] Assuming role 'arn:aws:iam::xxxxxxxxxxxx:role/cdk-hnb659fds-deploy-role-xxxxxxxxxxxx-us-east-1'.
[23:46:21] Waiting for stack CDKToolkit to finish creating or updating...
[23:46:22] Storing template in S3 at: https://cdk-hnb659fds-assets-xxxxxxxxxxxx-us-east-1.s3.us-east-1.amazonaws.com/cdk/CdktsStack/e6cbc1ec64bd0bf5f83ee188026e9c010633551b8252f036384f6dddb29932f0.yml

Can you confirm whether or not you changed S3 permissions in the Deploy Role or if you are using a custom bootstrap template?

[sumupitchayan]

@mrlikl I am unable to reproduce this bug, so I am wondering if you are using a custom bootstrap template where you restricted permissions on any of the roles?

I wrote a CLI integ test with the example you provided and created a sample app manually, I also created an IAM role that had no permissions for any s3 actions - both times I ran cdk diff -v, it worked successfully.

Also want to note that the S3 permissions on your assumed IAM role when calling cdk diff or any other CDK command should not matter, as cdk diff assumes the LookupRole from the bootstrap template, not whatever role is configured from your CLI.

[mrlikl]

@sumupitchayan I am still able to reproduce the issue. Did a fresh bootstrap now and confirmed again that the below steps would reproduce the behavior.

1. Create IAM user or role with AdministratorAccess policy and a inline policy to deny all S3 actions.

2. Configure an environment to execute CDK CLI to use the created IAM principal (e.g. configure access key and secret as default profile, attach the created role to EC2 instance and so on)

3. Execute cdk init --language=typescript to initialize CDK project

4. Define the following resource and deploy it

    new sqs.Queue(this, `Queue, {
    });

5. Define many resources by the following code to generate large template.

    for (let i = 0; i < 300; i++) {
      new sqs.Queue(this, `Queue${i}`, {
      });
    }

6. Execute cdk diff -v

verbose logs:

Stack CdktsStack
[12:27:06] Retrieved account ID 123456789123 from disk cache
[12:27:06] Assuming role 'arn:aws:iam::123456789123:role/cdk-hnb659fds-lookup-role-123456789123-us-east-1'.
[12:27:10] Retrieved account ID 123456789123 from disk cache
[12:27:11] Retrieved account ID 123456789123 from disk cache
[12:27:11] Assuming role 'arn:aws:iam::123456789123:role/cdk-hnb659fds-deploy-role-123456789123-us-east-1'.
[12:27:12] Waiting for stack CDKToolkit to finish creating or updating...
[12:27:13] Storing template in S3 at: https://cdk-hnb659fds-assets-123456789123-us-east-1.s3.us-east-1.amazonaws.com/cdk/CdktsStack/f6771e02c39180c5684fb2696dbd1458ddf69bb849e083a5a7c6c9dc802f8e2e.yml
[12:27:13] [0%] start: Publishing f6771e02c39180c5684fb2696dbd1458ddf69bb849e083a5a7c6c9dc802f8e2e:current
[12:27:14] [0%] check: Check s3://cdk-hnb659fds-assets-123456789123-us-east-1/cdk/CdktsStack/f6771e02c39180c5684fb2696dbd1458ddf69bb849e083a5a7c6c9dc802f8e2e.yml
[12:27:15] Call failed: getBucketLocation({"Bucket":"cdk-hnb659fds-assets-123456789123-us-east-1"}) => User: arn:aws:iam::123456789123:user/mac-cli is not authorized to perform: s3:GetBucketLocation on resource: "arn:aws:s3:::cdk-hnb659fds-assets-123456789123-us-east-1" with an explicit deny in an identity-based policy (code=AccessDenied)
[12:27:15] Retrieved account ID 123456789123 from disk cache
[100%] fail: Bucket named 'cdk-hnb659fds-assets-123456789123-us-east-1' exists, but not in account 123456789123. Wrong account?
[12:27:15] Failed to publish one or more assets. See the error messages above for more information.
Could not create a change set, will base the diff on template differences (run again with -v to see the reason)

The diff will show the difference, but the change set type diff creation fails. Adding s3:GetBucketLocation to the CLI role would resolve the issue.

Internal TT: P126676177

[mrlikl]

@sumupitchayan

Can you confirm whether or not you changed S3 permissions in the Deploy Role or if you are using a custom bootstrap template?

The FilePublishingRole already has the permission for the failing s3 call, so instead of deploy role shouldn't the file asset publishing role be used here ?

[sumupitchayan]

@sumupitchayan

Can you confirm whether or not you changed S3 permissions in the Deploy Role or if you are using a custom bootstrap template?

The FilePublishingRole already has the permission for the failing s3 call, so instead of deploy role shouldn't the file asset publishing role be used here ?

@mrlikl nevermind yes you are right, I was able to reproduce the issue. Will put out a PR fix soon.

[github-actions]

Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.

[github-actions]

Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.