Vault EC2 authentication slow

[agy]

Overview

We're using Vault with AWS EC2 authentication enabled and binding to the IAM instance profile ARN. This was working well for us until we deployed to our production environment and found 95p of logins went from ~1.5s to ~6s (spiking to over 20s). This causes some of our applications using Vault to timeout and require re-auth. The main difference between environments are the number on instances running in each.

Issue

AWS EC2 authentication takes a long time requiring clients to require to retry.

What I expect

Running the commands from aws CLI takes on the order of 0.5s. I expect that the login should be near that value.

See PR: #2766

Investigation

In order to simulate Vault's EC2 authentication we instrumented the process and worked out that the below EC2 API call was taking the majority of the time.

From https://github.com/hashicorp/vault/blob/master/builtin/credential/aws/path_login.go#L172-L181

	status, err := ec2Client.DescribeInstances(&ec2.DescribeInstancesInput{
		Filters: []*ec2.Filter{
			&ec2.Filter{
				Name: aws.String("instance-id"),
				Values: []*string{
					aws.String(instanceID),
				},
			},
		},
	})

I then created a simple test case:

package main

import (
        "fmt"
        "github.com/aws/aws-sdk-go/aws"
        "github.com/aws/aws-sdk-go/aws/session"
        "github.com/aws/aws-sdk-go/service/ec2"
)

func main() {
        var instanceID string
        instanceID  = "i-XXXXXXXXXXXXXX"
        sess, err := session.NewSession(&aws.Config{
                Region: aws.String("us-east-1"),
        })
        svc := ec2.New(sess)
        params := &ec2.DescribeInstancesInput{
                Filters: []*ec2.Filter{
                        &ec2.Filter{
                                Name: aws.String("instance-id"),
                                Values: []*string{
                                        aws.String(instanceID),
                                },
                        },
                },
        }

        resp, err := svc.DescribeInstances(params)
        if err != nil {
                panic(err)
        }

        fmt.Println(resp)
}

This confirmed that the login runs were taking ~6s to complete.

The AWS SDK documentation for ec2.DescribeInstances shows that it can take filters and/or instance IDs as parameters.

According to AWS EC2 API documentation, this is expected behaviour:

If you do not specify instance IDs, Amazon EC2 returns information for all relevant instances.

I created a modified test case to use instance IDs instead of filters:

package main

import (
	"fmt"
	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/aws/session"
	"github.com/aws/aws-sdk-go/service/ec2"
)

func main() {
	var instanceID string
        instanceID  = "i-XXXXXXXXXXXXXX"
	sess, err := session.NewSession(&aws.Config{
		Region: aws.String("us-east-1"),
	})
	svc := ec2.New(sess)
	params := &ec2.DescribeInstancesInput{
		InstanceIds: []*string{
			aws.String(instanceID),
		},
	}

	resp, err := svc.DescribeInstances(params)
	if err != nil {
		panic(err)
	}

	fmt.Println(resp)
}

The API call now takes ~0.3s to complete.

[agy]

Referencing PR: #2766

[vishalnayak]

Fixed by #2766