aws-exporter

Standalone exporter to export AWS CloudWatch Metrics and Logs as prometheus metrics. This exporter uses AWS APIs and fetches both metadata and metric data

License

This software is available under Apache 2.0 License

Configuration

The exporter needs to be configured to extract metrics for one or more AWS Service types. AWS Service types map to a namespace in CloudWatch.

Supported CloudWatch namespaces

• AWS/ApplicationELB
• AWS/ApiGateway
• AWS/AppSync
• AWS/Billing
• AWS/CloudFront
• AWS/DocDB
• AWS/DynamoDB
• AWS/EBS
• AWS/Elasticache
• AWS/EC2
• AWS/EC2Spot
• AWS/ECS
• ECS/ContainerInsights
• AWS/EFS
• AWS/ELB
• AWS/ElasticMapReduce
• AWS/ES
• AWS/FSx
• AWS/GameLift
• Glue
• AWS/Kinesis
• AWS/NetworkFirewall
• AWS/NATGateway
• AWS/Lambda
• LambdaInsights
• AWS/NetworkELB
• AWS/Redshift
• AWS/RDS
• AWS/Route53Resolver
• AWS/S3
• AWS/SES
• AWS/SQS
• AWS/TransitGateway
• AWS/VPN
• AWS/AutoScaling
• AWS/Kafka
• AWS/Firehose
• AWS/SNS
• AWS/States
• AWS/WAFV2

Scrape Interval

The exporter scrapes metrics periodically at a frequency determined by the scrapeInterval. In each scrape the exporter requests an aggregate statistic with period of scrapeInterval using the GetMetricData API. The unit for time interval configuration is second. The default scrape interval is 60 seconds. The time interval can beconfigured in the following different ways

• Global This will apply to all metrics across all namespaces
• Namespace This will apply to all metrics for a specific namespace
• Metric This will apply to a specific metric in a namespace

ECS Tasks Service Discovery

The exporter can discover ECS Tasks and generate scrape targets. The target port and metric scrape url path will be automatically detected from the docker labels PROMETHEUS_EXPORTER_PORT and PROMETHEUS_EXPORTER_PATH. If the PROMETHEUS_EXPORTER_PATH is not specified it will be defaulted to /metrics. If the PROMETHEUS_EXPORTER_PORT is not specified, and the task has only one container which exposes only one port, this port will be used. If the container has multiple ports exposed or if there are multiple containers in the task then task target ports need to be specified in the scrape configuration yaml as follows

- ecsTaskScrapeConfigs
  - containerDefinitionName: model-builder
    containerPort: 8080
    metricPath: /model-builder/actuator/prometheus

The metricPath is still optional. If not specified the default value of /metrics will be used

Exporter internal metrics

The exporter also exports the following metrics to enable monitoring itself

Metric Name	Description
aws_exporter_milliseconds_sum	AWS API Latency Counter
aws_exporter_milliseconds_count	AWS API Count
aws_exporter_interval_seconds	The scrape interval metric for each namespace

Running it locally

$ cp conf/cloudwatch_scrape_config_sample.yml ./cloudwatch_scrape_config.yml 
$ ./bin/aws-exporter

The exporter listens on port 8010 by default. The metrics can be scraped from http://localhost:8010/aws-exporter/actuator/prometheus. Here is a sample output of metrics

# HELP aws_sqs_number_of_messages_deleted_sum 
# TYPE aws_sqs_number_of_messages_deleted_sum gauge
aws_sqs_number_of_messages_deleted_sum{d_queue_name="lamda-sqs-poc-input-queue",region="us-west-2",tag_asserts_aws_resource="tag_for_discovery",} 0.0 1633625100000
aws_sqs_number_of_messages_deleted_sum{d_queue_name="lamda-sqs-poc-output-queue",region="us-west-2",tag_asserts_aws_resource="tag_for_discovery",} 0.0 1633625100000
aws_sqs_number_of_messages_deleted_sum{d_queue_name="aws-lambda-poc-destination-queue",region="us-west-2",} 0.0 1633625220000
aws_sqs_number_of_messages_deleted_sum{d_queue_name="lamda-sqs-poc-dl-queue",region="us-west-2",tag_asserts_aws_resource="tag_for_discovery",} 0.0 1633625220000

# HELP aws_lambda_errors_sum 
# TYPE aws_lambda_errors_sum gauge
aws_lambda_errors_sum{d_function_name="first-lambda-function",region="us-west-2",tag_asserts_aws_resource="tag_for_discovery",} 0.0 1633625220000
aws_lambda_errors_sum{region="us-west-2",} 0.0 1633625220000
aws_lambda_errors_sum{d_function_name="first-lambda-function",d_resource="first-lambda-function:version2",region="us-west-2",tag_asserts_aws_resource="tag_for_discovery",} 0.0 1633625220000
aws_lambda_errors_sum{d_executed_version="2",d_function_name="first-lambda-function",d_resource="first-lambda-function:version2",region="us-west-2",tag_asserts_aws_resource="tag_for_discovery",} 0.0 1633625220000
aws_lambda_errors_sum{d_function_name="first-lambda-function",d_resource="first-lambda-function",region="us-west-2",tag_asserts_aws_resource="tag_for_discovery",} 0.0 1633625220000

# HELP aws_lambda_concurrent_executions_sum 
# TYPE aws_lambda_concurrent_executions_sum gauge
aws_lambda_concurrent_executions_sum{d_function_name="first-lambda-function",region="us-west-2",tag_asserts_aws_resource="tag_for_discovery",} 18.0 1633625220000
aws_lambda_concurrent_executions_sum{region="us-west-2",} 18.0 1633625220000
aws_lambda_concurrent_executions_sum{d_function_name="first-lambda-function",d_resource="first-lambda-function:version1",region="us-west-2",tag_asserts_aws_resource="tag_for_discovery",} 2.0 1633625220000
aws_lambda_concurrent_executions_sum{d_executed_version="2",d_function_name="first-lambda-function",d_resource="first-lambda-function:version2",region="us-west-2",tag_asserts_aws_resource="tag_for_discovery",} 4.0 1633625220000
aws_lambda_concurrent_executions_sum{d_function_name="first-lambda-function",d_resource="first-lambda-function",region="us-west-2",tag_asserts_aws_resource="tag_for_discovery",} 4.0 1633625220000

AWS IAM Permissions

There are two roles required for the AWS Exporter. An ECS Execution Role and an ECS Task role. Please refer to this CloudFormation template https://github.com/asserts/downloads/blob/main/cloudformation/aws-integration/ecs/iam-roles-user-access-key.yaml to see the permissions required in these two roles.