Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

(cli): (cdk fails with "Profile <profile name> did not include credential process) #25870

Closed
kirnberger1980 opened this issue Jun 6, 2023 · 12 comments · Fixed by #31702
Closed
Assignees
Labels
bug This issue is a bug. effort/medium Medium work item – several days of effort p1 package/tools Related to AWS CDK Tools or CLI

Comments

@kirnberger1980
Copy link

kirnberger1980 commented Jun 6, 2023

Describe the bug

We want to use the CDK CLI with Gitlab Runner, which is running on EKS in Account A. cdk deploy is executed for Account B (Cross-Account Deployment).

cdk deploy --profile AccountB

CLI Command fails with:

Unable to determine the default AWS account (ProcessCredentialsProviderFailure): Profile <AccountB> did not include credential process
....
Could not assume arn:aws:iam::xxxxxxx:role/cdk-xxxxxxxx-lookup-role-xxxxxxx-eu-central-1, proceeding anyway.

We are using IRSA (IAM Role for Service Accounts), further called Role A, for the Gitlab Runner in Account A.
In Account B there is an IAM Role (Role B), which can be assumed by Role A. RoleB can assume all cdk-provisioned IAM Roles in that Account (created by cdk bootstrap).

Profile file (stored in /configmaps/awsconfig):

    [default]
    region = eu-central-1
    role_arn = arn:aws:iam::AccountA:role/RoleA
    web_identity_token_file=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
    output = json


    [profile AccountB]
    role_arn = arn:aws:iam::AccountB:role/RoleB
    source_profile = default
    output = json
    region = eu-central-1

We have set environment variables AWS_PROFILE and AWS_CONFIG_FILE:

AWS_PROFILE=AccountB
AWS_CONFIG_FILE=/configmaps/awsconfig

AWS CLI is working fine with above profile file and the environment variables:

aws sts get-caller-identity
{
    "UserId": "AROA2SFT2XXXXXXXXX:botocore-session-168600000",
    "Account": "00000000000",
    "Arn": "arn:aws:sts::00000000000:assumed-role/RoleB/botocore-session-168600000"
}

Assuming the cdk-lookup Role from RoleB with AWS CLI works fine.

aws sts assume-role --role-arn arn:aws:iam::xxxxxxx:role/cdk-xxxxxxxx-lookup-role-xxxxxxx-eu-central-1 --role-session-name test123
{
    "Credentials": {
        "AccessKeyId": "<somekey>",
        "SecretAccessKey": "<somesecret>",
        "SessionToken": "<nice jwt>",
        "Expiration": "2023-06-06T15:29:36+00:00"
    },
    "AssumedRoleUser": {
        "AssumedRoleId": "<some otherkey:test123",
        "Arn": "arn:aws:iam::xxxxxxx:role/cdk-xxxxxxxx-lookup-role-xxxxxxx-eu-central-1"
    }
}

By the way: cdk also seems not to respect AWS_PROFILE environment variable as it is using the default role (it could be, that the session token by IRSA is used), when NOT using --profile.

Expected Behavior

We expect that AWS CDK CLI is acting identically to AWS CLI when using AWS profiles for Cross-Account Deployments. Environment Variable AWS_PROFILE should be respected.

Current Behavior

The CDK CLI fails that the cdk-Lookup-Role cannot be assumed, because "Profile did not include credential process". When you don't specify --profile, then default-Profile (or session token created by OIDC Provider of IAM) is used.

Reproduction Steps

You need to run a pod in EKS:

  • configure IRSA (RoleA) for the pod in the same account (Account A)
  • configure a RoleB in Account B, which can be assumed by RoleA
  • give permissions to RoleB to assume cdk-provisioned roles
  • configure an AWS_CONFIG_FILE like described above.
  • install CDK, NODE etc on the pod
  • Configure env variables AWS_PROFILE and AWS_CONFIG_FILE
  • Run cdk deploy

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.78.0

Framework Version

2.78.0

Node.js Version

v18.16.0

OS

node:18-alpine

Language

Typescript

Language Version

3.9.7

Other information

No response

@kirnberger1980 kirnberger1980 added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Jun 6, 2023
@github-actions github-actions bot added the package/tools Related to AWS CDK Tools or CLI label Jun 6, 2023
@pahud
Copy link
Contributor

pahud commented Jun 6, 2023

Thanks for reporting.

  1. Can you run this in the pod from Account A?
aws --profile AccountB sts get-caller-identity
  1. Can you share a minimal CDK code that provisions this environment so I can try reproduce it in my account?

@pahud pahud self-assigned this Jun 6, 2023
@pahud pahud added investigating This issue is being investigated and/or work is in progress to resolve the issue. p2 effort/medium Medium work item – several days of effort response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. and removed needs-triage This issue or PR still needs to be triaged. labels Jun 6, 2023
@kirnberger1980
Copy link
Author

kirnberger1980 commented Jun 7, 2023

Thanks for the quick reply.

  1. Here is the result of the aws cli response:
{
    "UserId": "AXXXXXXXXXX:botocore-session-1686146634",
    "Account": "<AccountB>",
    "Arn": "arn:aws:sts::<AccountB>:assumed-role/RoleB/botocore-session-1686146634"
}

This is, what we expected, as setting env variable AWS_PROFILE or using --profile should return the identical response.

  1. Here is some sample code tested, where we could reproduce the issue:

bin-File:

#!/usr/bin/env node
import 'source-map-support/register';
import * as cdk from 'aws-cdk-lib';
import { CdkStack } from '../lib/cdk-stack';

const app = new cdk.App();
new CdkStack(app, 'CdkStack', {
  env: { account: "<AccountB>",region: "eu-central-1" }
});

lib-File:

import * as cdk from 'aws-cdk-lib';
import { Construct } from 'constructs';

export class CdkStack extends cdk.Stack {
  constructor(scope: Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

  const vpc = cdk.aws_ec2.Vpc.fromLookup(this, "MyVPC", {
    vpcName: "MyVpcName"
  });

  new cdk.aws_ec2.SecurityGroup(this, "MyNewSecurityGroup", {
    vpc,
    description: "This will secure my resources"
  });
  }
}

Kind Regards,
Marcell

@github-actions github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Jun 7, 2023
@kirnberger1980
Copy link
Author

Any news on this issue?

@github-actions github-actions bot added p1 and removed p2 labels Jan 28, 2024
Copy link

This issue has received a significant amount of attention so we are automatically upgrading its priority. A member of the community will see the re-prioritization and provide an update on the issue.

@jumpinjan
Copy link

I am facing a similar issue. An update would be greatly appreciated.

@elixxx
Copy link

elixxx commented Feb 8, 2024

Whats going on with this issue? We are facing the same issue but sounds like a common workflow!

@megakoresh
Copy link

megakoresh commented Feb 19, 2024

Is this a recent thing? I just now saw this. CDK worked totally fine then suddenly it's now failing with this error and in the verbose log I see profile xxx did not include credential process. I have the profile name exported in the AWS_PROFILE variable. I reset the terminal and ran aws sso login again, but nothing helped. Removal of cdk.out folda also didn't help. It's as if it broke as a result of some change in the cloud since it broke today after 16:00 and before that worked just fine.

UPDATE: I managed to fix this by updating the awscliv2 to the latest version, doing aws sso logout then aws sso login and then it worked. So might be something to do with token timings, and not CDK per se. Still, strange. AWS CLI commands worked just fine.

@megakoresh
Copy link

Updating: this seems to have something to do with timing. This happens if the session expires. Once it does, CDK will break and no amount of logging out and back in will help. Only waiting seems to do the trick. I have also tried clearing context, but still nothing. After waiting for hours, and then logging out and back in the problem goes away until the next day, but that is hardly a "solution". Is anyone looking into this?

@sturman
Copy link

sturman commented Apr 4, 2024

I think the problem is in web_identity_token_file.
In my case with AWS_PROFILE=oidc environment variable set and the following content in ~/.aws/config file

[profile oidc]
role_arn=arn:aws:iam::012345678901:role/gitlab-deploy
web_identity_token_file=/tmp/web_identity_token

command aws sts get-caller-identity works as expected.
But npx cdk diff -vv fails with

...
[16:42:45] Determining if we're on an EC2 instance.
[16:42:45] Does not look like an EC2 instance.
[16:42:45] Toolkit stack: CDKToolkit
[16:42:45] Setting "CDK_DEFAULT_REGION" environment variable to eu-central-1
[16:42:45] Resolving default credentials
[16:42:45] Unable to determine the default AWS account (ProcessCredentialsProviderFailure): Profile oidc did not include credential process
[16:42:45] context: {
...

So I ended up assuming role and setting AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN environment variables with command

export $(printf "AWS_ACCESS_KEY_ID=%s AWS_SECRET_ACCESS_KEY=%s AWS_SESSION_TOKEN=%s"
$(aws sts assume-role-with-web-identity
--role-arn ${ROLE_ARN}
--role-session-name "GitLabRunner-${CI_PROJECT_ID}-${CI_PIPELINE_ID}"
--web-identity-token ${GITLAB_OIDC_TOKEN}
--duration-seconds 3600
--query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]'
--output text))

@pahud pahud removed their assignment Aug 7, 2024
@pahud pahud removed the investigating This issue is being investigated and/or work is in progress to resolve the issue. label Aug 7, 2024
@otaviomacedo otaviomacedo self-assigned this Oct 2, 2024
@TheRealAmazonKendra
Copy link
Contributor

I'm pretty sure the issue here is that our current credential chain does not support web_identity_token_file inside an ini file, so the credential chain here fails on profileCredentials because of the presence of web_identity_token_file and then again on SsoCredentials because none are setup, as expected. The last credential provider is ProcessCredentials, which will fail here, also as expected. Since it is the last in the chain, that's the error that actually gets surfaced, which is super misleading.

We are currently in the process of upgrading from sdkv2 to sdkv3, at which point web_identity_token_file will be supported and you should no longer experience this error. You can follow the progress in #31702, if you'd like.

Copy link

Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.

1 similar comment
Copy link

Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Nov 12, 2024
otaviomacedo pushed a commit that referenced this issue Nov 12, 2024
This PR updates the CDK CLI to use the AWS SDK V3 instead of V2.

### Manual Test Cases for Authorization
All tests were run verbosely so that I could manually check the
credentials being used from the CLI output.

- [x] No credentials setup and no default profile fails as expected
(established to ensure nothing was unintentionally setup)
- [x] Explicitly provided profile sourced from config file, tested with
both `--profile` and `AWS_PROFILE`
  - [x] `aws_access_key_id` and `aws_secret_access_key`
  - [x] `aws_access_key_id` and `aws_secret_access_key` and `region`
- [x] `source_profile` and `role_arn` that points to another profile
with `aws_access_key_id` and `aws_secret_access_key` in both config and
credentials
  - [x] `source_profile` and `role_arn` that points to Environment
- [x] `source_profile` that and `role_arn` that points to SSO profile in
both config and credentials
- [x] SSO both using all fields in `[profile]` and split between
`[sso-session]` and `[profile]`
  - [x] `credential_source` is Environment
  - [x] `credential_process`
- [x] Explicitly provided profile sourced from  credentials file
  - [x] `aws_access_key_id` and `aws_secret_access_key`
  - [x] `aws_access_key_id` and `aws_secret_access_key` and `region`
- [x] `source_profile` and `role_arn` that points to another profile
with `aws_access_key_id` and `aws_secret_access_key` in both config and
credentials
  - [x] `source_profile` and `role_arn` that points to Environment
- [x] `source_profile` that and `role_arn` that points to SSO profile in
both config and credentials
  - [x]  `credential_source` is Environment
  - [x] `credential_source` is EcsContainer (integ tests in CodeBuild)
  - [x] `credential_process`
- [x] Default profile (in both config file and credentials file)
- [x] `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment
variables set (cleared after test)
- [x] `AMAZON_ACCESS_KEY_ID` and `AMAZON_SECRET_ACCESS_KEY` environment
variables set (cleared after test)
  - [x] `aws_access_key_id` and `aws_secret_access_key`
  - [x] `aws_access_key_id` and `aws_secret_access_key` and `region`
- [x] `source_profile` and `role_arn` that points to a profile with
`aws_access_key_id` and `aws_secret_access_key`
- [x] `source_profile` and `role_arn` that points to Environment (this
will use `fromEnv` prior to getting to looking for `source_profile` in
the Ini file)
  - [x] SSO profile is setup as default
  - [x] `credential_process`

Closes #25870, #26292, #20956, #24744, #27265, 20896.

### Reason for this change

The AWS SDK V2 is now in maintenance mode.

----

*By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache-2.0 license*

---------

Co-authored-by: Otavio Macedo <288203+otaviomacedo@users.noreply.github.com>
Co-authored-by: Rico Huijbers <rix0rrr@gmail.com>

(cherry picked from commit 5bc0662)
# for free to subscribe to this conversation on GitHub. Already have an account? #.
Labels
bug This issue is a bug. effort/medium Medium work item – several days of effort p1 package/tools Related to AWS CDK Tools or CLI
Projects
None yet
Development

Successfully merging a pull request may close this issue.

8 participants