unknown field "permissionsBoundary" under add-ons section while trying to create cluster via config file.

[cooldudesam]

What were you trying to accomplish?

I want to enable add-ons via eksctl while provisiong cluster. Right now since only vpc-cni is available from AWS would like to test it add-on so, eksctl creates this add-on, IAM role and service account to annotate it under kube-system namespace.

What happened?

Iam using the filed permissionsBoundary under add-ons section and as per eksctl schema doc it says you can add permissionsBoundary under addons section but when i do that i get below error

eksctl schema https://eksctl.io/usage/schema/

% eksctl create cluster  --config-file=/tmp/eksctl-cluster/us-east-1/mw-eksctltest-c0/cluster_config.yaml --without-nodegroup
Error: loading config file "/tmp/eksctl-cluster/us-east-1/mw-eksctltest-c0/cluster_config.yaml": error unmarshaling JSON: while decoding JSON: json: unknown field "permissionsBoundary"

Here is how iam giving this in my cluster config file

iam:
  serviceRolePermissionsBoundary: "arn:aws:iam::xxxxxxxxx:policy/CreatedRolesPermissionsBoundary"
  withOIDC: true

addons:
- name: vpc-cni
  version: 1.7.5 # optional
  attachPolicyARNs: #optional
  - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
  permissionsBoundary: "arn:aws:iam::xxxxxxxxxxxx:policy/CreatedRolesPermissionsBoundary"

Is anyone facing the similar issue? Can we get this issue fixed so we can use the feature in the addons section?

[aclevername]

@cooldudesam what version of eksctl are you using? This only just got added in 0.39.0

[cooldudesam]

Thanks for quick reply!

I was using 0.38.0 eksctl version. I installed the latest 0.40.0 eksctl version and this time i did not see the issue unknown field "permissionsBoundary" under add-ons section. But the cluster created and add-on stacks got completed succesfully but in the below stdout i see it threw an error about vpc-cni version and i dont see the add-on being added under the EKS cluster section.

Since the below stdout failed and that could be the reason why it did not create any serviceaccount for vpc-cni and annotate it? OR do i have to explicitly add serviceaccount section in my clusterconfig file?

Let me try removing the vpc-cni version and re-try again.

2021-03-08 18:17:44 [ℹ]  eksctl version 0.40.0
2021-03-08 18:17:44 [ℹ]  using region us-east-1
2021-03-08 18:39:50 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "mw-eksctltestui-c0" to succeed
2021-03-08 18:40:08 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "mw-eksctltestui-c0" to succeed
2021-03-08 18:40:11 [ℹ]  daemonset "kube-system/aws-node" restarted
2021-03-08 18:40:11 [ℹ]  creating role using provided policies ARNs
2021-03-08 18:40:12 [ℹ]  deploying stack "eksctl-mw-eksctltestui-c0-addon-vpc-cni"
2021-03-08 18:40:12 [ℹ]  waiting for CloudFormation stack "eksctl-mw-eksctltestui-c0-addon-vpc-cni"
2021-03-08 18:40:31 [ℹ]  waiting for CloudFormation stack "eksctl-mw-eksctltestui-c0-addon-vpc-cni"
2021-03-08 18:40:47 [ℹ]  waiting for CloudFormation stack "eksctl-mw-eksctltestui-c0-addon-vpc-cni"
2021-03-08 18:40:48 [ℹ]  creating addon
2021-03-08 18:40:48 [!]  1 error(s) occurred and cluster hasn't been created properly, you may wish to check CloudFormation console
2021-03-08 18:40:48 [ℹ]  to cleanup resources, run 'eksctl delete cluster --region=us-east-1 --name=mw-eksctltestui-c0'
2021-03-08 18:40:48 [✖]  failed to create addon "vpc-cni": InvalidParameterException: Addon version specified is not supported
{
  RespMetadata: {
    StatusCode: 400,
    RequestID: "9fde0b01-3170-492c-a226-daf56212aee0"
  },
  AddonName: "vpc-cni",
  ClusterName: "mw-eksctltestui-c0",
  Message_: "Addon version specified is not supported"
}

[cooldudesam]

Thanks much! After removing the version field i was able to create the cluster along with the add-on and it did create service account as well. I can see the add-on being added in EKS console.

2021-03-08 20:21:51 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "mw-eksctltestui-c0" to succeed
2021-03-08 20:22:10 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "mw-eksctltestui-c0" to succeed
2021-03-08 20:22:28 [ℹ]  waiting for requested "EndpointAccessUpdate" in cluster "mw-eksctltestui-c0" to succeed
2021-03-08 20:22:30 [ℹ]  daemonset "kube-system/aws-node" restarted
2021-03-08 20:22:31 [ℹ]  creating role using provided policies ARNs
2021-03-08 20:22:31 [ℹ]  deploying stack "eksctl-mw-eksctltestui-c0-addon-vpc-cni"
2021-03-08 20:22:31 [ℹ]  waiting for CloudFormation stack "eksctl-mw-eksctltestui-c0-addon-vpc-cni"
2021-03-08 20:22:51 [ℹ]  waiting for CloudFormation stack "eksctl-mw-eksctltestui-c0-addon-vpc-cni"
2021-03-08 20:22:52 [ℹ]  creating addon
2021-03-08 20:22:53 [ℹ]  successfully created addon
2021-03-08 20:22:53 [ℹ]  waiting for the control plane availability...
2021-03-08 20:22:53 [✔]  saved kubeconfig as "/tmp/kubeprompt/kubeconfig.261852329.yaml"
2021-03-08 20:22:53 [ℹ]  no tasks
2021-03-08 20:22:53 [✔]  all EKS cluster resources for "mw-eksctltestui-c0" have been created
2021-03-08 20:22:54 [ℹ]  kubectl command should work with "/tmp/kubeprompt/kubeconfig.261852329.yaml", try 'kubectl --kubeconfig=/tmp/kubeprompt/kubeconfig.261852329.yaml get nodes'
2021-03-08 20:22:54 [✔]  EKS cluster "mw-eksctltestui-c0" in "us-east-1" region is ready

I have one question - If i have an existing eksctl cluster created using the config file where the add-on is not enabled. Could i just update the config file with setting OIDC to "true" and adding add-ons section and just upgrade the cluster. Would that take care of creating the OIDC Identity provider and service account and everything.

[aclevername]

I have one question - If i have an existing eksctl cluster created using the config file where the add-on is not enabled. Could i just update the config file with setting OIDC to "true" and adding add-ons section and just upgrade the cluster. Would that take care of creating the OIDC Identity provider and service account and everything.

No, we don't currently have a way of updating the cluster spec as a whole, we have a issue open for adding support for eksctl apply #2774.

At the moment you would need to run eksclt utils associate-iam-oidc-provider to enable OIDC, and then run eksctl create addon to create the addon

[cooldudesam]

Thanks for the response. Having support for eksctl apply similar to kubectl apply would help in updating the configuration since we are applying using the config file to provision cluster and managed nodegroups.

Could I just update the cluster config.yaml file with the add-on specifications, including the permission boundary spec something like this

iam:
withOIDC: true

addons:

• name: vpc-cni
  attachPolicyARNs:
  • arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
    permissionsBoundary: "arn:aws:iam::XXXXXXXXX:policy/PermissionBoundary

I have used the below commands to create add-on and enable OIDC provider but passing the updated cluster config file.

eksclt utils associate-iam-oidc-provider --config-file=cluster_config.yaml
eksctl create addon to create the addon --config-file=cluster_config.yaml

Thanks for all the help!