Dynamic Host Catalog creation fails credential validation for AWS GovCloud

[BoswellB35]

Describe the bug
When attempting to create a Dynamic Host Catalog using AWS GovCloud credentials, the process fails with the following credential validation error:

host_catalogs.(Service).createPluginInRepo: unable to create host catalog: plugin.(Repository).CreateCatalog: in scope: p_8jkl5MaznR: db.DoTx: plugin.(Repository).CreateCatalog: unknown, unknown: error #0: rpc error: code = Unknown desc = error during credential validation: error validating credentials: InvalidClientTokenId: The security token included in the request is invalid. status code: 403, request id: 563721ac-224d-41aa-a58d-eece96f3b4ed

The AWS credentials, Boundary deployment, and target systems all exist within the same AWS GovCloud account. I have previously tested this feature in a matching deployment in a standard AWS account and did not encounter this issue.

I suspect this has to do with the endpoint that the plugin is using to validate credentials. GovCloud has different endpoints for some services which the host catalog plugin may not be aware of, for example see the IAM endpoints

To Reproduce
Steps to reproduce the behavior:

1. In an AWS GovCloud account, create a user with ec2:DescribeInstances permissions, and generate an Access Key for that user
2. Ensure the IAM credentials are valid by using them with the aws cli to run the aws ec2 describe-instances command against a GovCloud region (i.e. us-gov-east-1)
3. In Boundary (either the API or Web UI), attempt to create a Dynamic Host Catalog using the same access key and an AWS GovCloud region

Expected behavior
I would expect this process to create a Dynamic Host Catalog without error that can then see instances in the corresponding AWS GovCloud account.

Additional context
I am currently using Boundary version 0.8.1 running in Docker on Ubuntu EC2 Instances. I can create a matching issue in the boundary-plugin-host-aws repo if that would be more appropriate for this.

[sswati0]

Thanks for reporting the issue. We are looking into it.

[jefferai]

awsutil supports setting custom endpoints but it seems for some reason very few AWS client options are actually exposed in boundary-plugin-host-aws -- probably we need to instrument a few more through there.

[jefferai]

Do you know if EC2 has other endpoints as well? I've gone through a bunch of the documentation and for things like EC2 I don't see govcloud-specific endpoints, but I might be looking in the wrong place. I want to make sure that if I add support for specifying endpoints I'm covering the full spectrum.

[BoswellB35]

As far as I know the endpoints for EC2 are based exclusively on the region (e.g. ec2.us-west-1.amazonaws.com vs. ec2.us-gov-east-1.amazonaws.com). This seems to be the standard for most services; however there are some, such as IAM, that do not support regions. See the comment at the bottom of the "regional endpoints" section here: AWS Service Endpoints.

IAM in particular only uses different endpoints for govcloud and fips, per this page. (e.g. iam.amazonaws.com vs. iam.us-gov.amazonaws.com). As such this issue seems to be the exception, not the rule. I can't find a page that specifically lists non-region based endpoints, but as best I can tell IAM is the only one relevant to Boundary.

[jefferai]

@BoswellB35 I believe that if the region is set that it should pick the right endpoints automatically. However, I found that while we are setting the region for the EC2 client we are not setting the region for the IAM client. I'm going to add that logic anyways as it's clearly the right thing to do, but would you be able to give it a test if I send you a build? I don't have GovCloud access currently.

[jefferai]

@BoswellB35 Let me know if this doesn't fix things for you, but my understanding is that properly setting the region should handle the endpoints transparently.

[BoswellB35]

I've tested this build in our environment and was successfully able to create a dynamic host catalog, so this seems to have fixed our issue. Thank you!