airflow.md

gke/airflow walkthrough

A guide for using CNDI to deploy a GitOps enabled Airflow cluster on Kubernetes in Google Cloud Platform

overview 🔭

This walkthough uses cndi to customize and deploy our airflow Template to GKE. In just a few minutes we will be able to deploy a new Kubernetes cluster to GKE that has been optimally configured for Airflow, including GitOps with Secrets management, TLS and High Availibility right out-of-the-box. This framework will enable quick iteration of infrastructure, applications and manifests in a GitHub workflow you are already comfortable with.

prerequisites ✅

You will need the following things to get up and running with cndi successfully:

• A GCP account and a GCP project: cndi will deploy infrastructure into a Google Cloud Project connected to a valid billing account.

• Your GCP service account credentials: cndi will leverage a Google Cloud Service Account using a service-account-key.json credentials file to deploy resources.

• A domain name: The most convenient way to access your cluster is by attaching a domain name to the load balancer, if you provide this domain during cndi create in an upcoming step we should be able to wire it up automatically.

• (Optional if you dont have an domain name) Here's a guide of how to connect to your Google Kubernetes Cluster once its deployed and Port Forward Argocd and the Airflow Web Server

• A GitHub account: cndi helps you manage the state of your infrastructure using a GitOps workflow, so you'll need a GitHub account with a valid GitHub Personal Access Token.

• GitHub CLI: You will need to have the GitHub CLI installed on your machine. You can download it here.

• Here's a guide of how to set up your Google Cloud account including roles and permissions

download cndi ⬇️

macos and linux

Installing for macOS and Linux is the way to go if you have that option. Simply run the following:

curl -fsSL https://raw.githubusercontent.com/polyseam/cndi/main/install.sh | sh

windows

Installing for Windows should be just as easy. Here is the command to install CNDI on Windows:

irm https://raw.githubusercontent.com/polyseam/cndi/main/install.ps1 | iex

create your cndi project 📂

CNDI is designed around a GitOps workflow, so all of your cluster configuration and infrastructure will be stored as code within a git repo, let's create that now!

# cndi create <owner>/<repo> && cd <repo>

 cndi create polyseam/my-cndi-cluster && cd my-cndi-cluster

my-cndi-cluster You will first be prompted to enter the name of your cndi project

Please enter a name for your CNDI project: (my-cndi-cluster)

When you're prompted to choose a template for your project, select the airflow template.

"Pick a template"
  basic
❯ airflow 
  cnpg
  neo4j
  mssqlserver

Make sure airflow is highlighted and press Enter to confirm your selection.

Next, you'll need to decide where you want to deploy your cluster. For this project, choose gcp if you're deploying to GCP, The prompt will appear as follows:

"Where do you want to deploy your cluster?"
  aws 
  azure
❯ gcp
  dev

Ensure gke is highlighted and press Enter to proceed.

Finally, select a Kubernetes distribution for your deployment. The gke (Google Kubernetes Engine) option is for deploying on Google. The prompt will be:

Select a distribution
❯ gke

After confirming that gke is highlighted, press Enter to finalize your choice. You will then need to provide specific information at various interactive prompts. Below is a comprehensive list of the prompts used during this init process:

• Cndi Project Name: name of project
• Template: list of templates to choose from

---

• GitHub Username: a user's handle on GitHub.
• GitHub Repository URL: the url for the GitHub repository that will hold all cluster configuration
• GitHub Personal Access Token: the access token CNDI will use to access your repo for cluster creation and synchronization

---

• GCP Region: region where the infastructure is being created
• Path to GCP service account key json: path to JSON credentials file for GCP Service Account

---

• Git Username for Airflow DAG Storage: a user's handle on GitHub used to synchronize Airflow DAGs
• Git Password for Airflow DAG Storage: a personal access token used to synchronize Airflow DAGs
• Git Repo for Airflow DAG Storage: url for repo where your Airflow DAGs will be stored

---

• Domain name you want ArgoCD to be accessible on: domain where ArgoCD will be hosted
• Domain name you want Airflow to be accessible on: domain where Airflow will be hosted

---

• Email address you want to use for lets encrypt: an email for lets encrypt to use when generating certificates
• Username you want to use for airflow cnpg database: username you want to use for airflow database
• Password you want to use for airflow cnpg database: password you want to use for airflow database
• Name of the postgresql database you want to use for airflow cnpg database: name of the postgresql database you want to use for airflow cnpg database

This process will generate a cndi_config.yaml file, and cndi directory at the root of your repository containing all the necessary cluster and infrastructure resources. It will also generate a .env file that will be used to store sensitive information that we don't want to commit to our repository as source code.

The structure of the generated CNDI project will be as follows:

├── 📁 cndi
│   ├── 📁 cluster_manifests
│   │   ├── 📁 applications
│   │   │   └── airflow.application.yaml
│   │   ├── argo-ingress.yaml
│   │   ├── cert-manager-cluster-issuer.yaml
│   │   └── git-credentials-secret.yaml
│   └── 📁 terraform
│       ├── gke_cluster_airflow_nodes.tf.json
│       └── etc 
├── cndi_config.yaml
├── .env
├── .gitignore
├── .github
└── README.md

cndi create polyseam/my-cndi-cluster

Please confirm the destination directory for your CNDI project: »  (C:\Users\Taylor\polyseam\my-cndi-cluster)
Please enter a name for your CNDI project: (my-cndi-cluster) » my-cndi-cluster
Pick a template » airflow
Where do you want to deploy your cluster? » gcp
Select a distribution » gke
Please in your GCP Region: us-central1
Please enter the path GCP credentials JSON to your () »  ~/polyseam/gcp-testing.json   
Would you like ArgoCD to connect to your repo using a Git token or SSH key? » token
What is your git username? () » IamTamika
Please enter your Git Personal Access Token: () » ****************************
Please enter your Git Repo URL: () »
What email address should be used for Lets Encrypt certificate registration?  » tamika.taylor@untribe.com
Would you like to enable external-dns for automatic DNS management? (Y/n) » Yes
Please select your DNS provider (aws) » gke
Do you want to expose ArgoCD with an Ingress? (Y/n) » Yes
What hostname should ArgoCD be accessible at? » argocd.untribe.com
Do you want to expose the Airflow UI to the web? (Y/n) » Yes
What hostname should Airflow be accessible at? » airflow.untribe.com
What is the URL of the Git repository containing your Airflow DAGs? (https://github.com/polyseam/demo-dag-bag) » https://github.com/polyseam/demo-dag-bag
Do you want to use your cluster credentials for Airflows Git Sync? (Y/n) » Yes

created cndi cluster repo at https://github.com/polyseam/my-cndi-cluster

Once complete you should click on the link of the newly created repo cluster, and scroll down to the readme for more infomation about about your airflow deployment

You should now see the cluster configuration has been uploaded to GitHub:

Now, open your web browser and navigate to your project on GitHub. Click on the Actions tab, then click on the job that was triggered from your latest commit.

You will see something like the image below, which shows that GitHub has successfully run the workflow.

It is common for cndi run to take a fair amount of time, as is the case with most Terraform and cloud infrastructure deployments.

Once cndi run has been completed, at the end of the run will be a link to resource groups, where you can view resources deployed by CNDI for this project.

attach the Load Balancer to Your Domain 🌐 with ExternalDNS

Instead of manually creating an A record in your domain's DNS settings, if you enabled & configured ExternalDNS during the cndi init process then ExternalDNS will automatically create a A record in Route 53, pointing to your load balancer's public host. This process eliminates the need for manual DNS record management.If everything is working correctly you should now open the domain name you've assigned for ArgoCD in your browser to see the ArgoCD login page. The DNS changes may take a few minutes to propagate

• (Optional if you dont have an domain name) Here's a guide of how to connect to your GKE Kubernetes Cluster once its deployed and Port Forward Argocd and the Airflow Web Server

To log in, use the username admin and the password which is the value of the ARGOCD_ADMIN_PASSWORD in the .env located in your CNDI project folder

Attach the load balancer to your domain manually (Optional)

At the end of the cndi run there is also an output called resource groups, which will have public loadbalancer. Copy the IP address(public host) of the loadbalancer thats attached to your GKE instances.

• Copy public host
• Go to your custom domain,
• Create an A record to route traffic to the load balancer IP address public host for Airflow and Argocd at the domain you provided.

If everything is working correctly you should now open the domain name you've assigned for ArgoCD in your browser to see the ArgoCD login page. The DNS changes may take a few minutes to propagate.

To log in, use the username admin and the password which is the value of the ARGOCD_ADMIN_PASSWORD in the .env located in your CNDI project folder

Verify all applications and manifests in the GitHub repository are present and their status is healthy in the ArgoCD UI

verify that Airflow is accessible on the chosen domain 🧐

After setting up your Airflow application on the chosen domain, it is necessary to verify that Airflow is accessible. To do this, the user can simply go to the chosen domain and see if they can see Airflow's login page. The default username is admin and the password is admin. If the page is accessible, then the user can log in and begin using Airflow. If not, the user should go back and make sure the previous steps were done correctly.

Verify Airflow is connected to the private DAG repository 🧐

Verify that Airflow is connected to the private DAG repository. If correct, the private DAGs should be visible on the Airflow UI. If not,you should go back and make sure that the private DAG repository is properly connected to Airflow with the correct credentials:

and you are done! ⚡️

You now have a fully-configured 3-node Kubernetes cluster with TLS-enabled Airflow and ArgoCD.

modifying the cluster! 🛠️

To add another a node to the cluster:

• Go to the cndi_config.yaml
• In the infrastructure.cndi.nodes section, add a new airflow node and save the file
• Run cndi ow
• Commit changes
• Push your code changes to the repository

destroying resources in the cluster! 💣

If you just want to take down any of your individual applications:

• Delete that application or manifest from your cndi_config.yaml
• Run cndi ow
• Commit changes
• Push your code changes to the repository

If you want to take down the entire cluster run:

cndi destroy