AWS-Auth configmap is mis-configured

[MarvinCai]

Getting following error from EKS console:
Your worker nodes do not have access to the cluster. Verify if the node instance role is present and correctly configured in the aws-auth ConfigMap.
Node IAM Role ARN is
arn:aws:iam::311022431024:role/StreamNative/sn-rxrevu-prod-us-east-120211109000743895600000009 But in aws-auth configmap there is:

➜  /Users/marvincai  k get configmap aws-auth -nkube-system -oyaml
apiVersion: v1
data:
  mapAccounts: |
    []
  mapRoles: |
    - "groups":
      - "system:bootstrappers"
      - "system:nodes"
      "rolearn": "arn:aws:iam::311022431024:role/sn-rxrevu-prod-us-east-120211109000743895600000009"
      "username": "system:node:{{EC2PrivateDNSName}}"
    - "groups":
      - "system:masters"
      "rolearn": "arn:aws:iam::311022431024:role/aws-reserved/sso.amazonaws.com/us-west-2/AWSReservedSSO_AdministratorAccess_d24587bd48be062e"
      "username": "management-admin"
    - "groups":
      - "system:masters"
      "rolearn": "arn:aws:iam::311022431024:role/gcp_eks_admin"
      "username": "gcp"
  mapUsers: |

The 2 role ARNs doesn't match.

[jrsdav]

The problem is the EKS API reports on the external configuration state of the cluster, i.e. the parameters contained in our API request to create an EKS node group (as a refresher, you create an EKS cluster first, then create a node group(s) second. They are separate API actions).

The issue is the IAM role we told the EKS API our node group would be using contains a path, but due to a problem the IAM integration in EKS, IAM roles are only recognized in the aws-auth configmap if they don’t contain a path.
kubernetes-sigs/aws-iam-authenticator#153

So the terraform module we use to create an EKS node group accounts for this by stripping out any paths contained in a role arn before adding it to the aws-auth configmap.
https://github.com/terraform-aws-modules/terraform-aws-eks/blob/56e93d77de58f311f1d1d7051f40bf77e7b03524/aws_auth.tf#L49

And for a bit of background, we recently added paths to our IAM resources to improve how we’re scoping our vendor access in customer’s managed accounts.

So TL;DR:
Don’t trust the EKS API for the cluster status! Check the cluster itself…