ecs_airflow

Diagram

Requirements

Requires Terraform and an AWS account.

Tools/Services Used

• Docker
• Terraform
• Apache Airflow
• AWS
• ECS

Short Description

A project that sets up an AWS ECS infrastructure and hosts an Apachi Airflow running inside a docker container.

Process Description

The entire process is automated using Terraform. Below are the steps

1. A VPC is created
2. 2 Subnets are created inside the above VPC
1. An internet gateway is created
2. A route table is created
3. The above Internet Gateway is associated to the above subnets(making them public), using the route table
3. A Security Group is created to be used with ECS Instances
4. ECS Service Role and ECS Instance Role are created to insure necessary permissions for the ECS service
5. An Application Load Balancer is created to act as a "starting point" for the requests to the airflow docker container
6. A Listener is created, along with a Target Group to route all the Application Load Balancer requests to that Target Group
7. A Launch Configuration is created for the ECS Instances, along with an Autoscaling Group to use the Launch Configuration to keep the ECS Instances count equal to the desired count
8. An ECS Cluster is Created
9. A container definition is created separatly to be used while creating an ECS task
10. An ECS Task is definied using the above Container Definition
11. An ECS Service is created using the above ECS Task Definition

Execution

In order to execute, modify the varriable.tfvar file with proper variables and run
terraform apply -var-file=variables.tfvars && terraform output -json > outputs.json
In order to tear down the infrastructure, run
terraform destroy -var-file=variables.tfvars

To Do

• Automatically push the DAG's from S3
• Revisit the key_pair creation process
• Improve Parametrization and add descriptions to tfvars

Nice To Have

• Remote s3 logging

Warnings

Current configuration of this project will be using AWS services that are beyond the Free Tier!