This is my mono-repository for my home Kubernetes cluster following GitOps principles and Infrastructure as Code. The purpose of this cluster is mostly a learning exercise, but with the ultimate benefit of having something solid to work with in my home environment.
My Kubernetes Cluster is deployed using Talos Linux. It currently is running as five virtual machines located on a singular Proxmox node, but in the future I will be separating into individual Proxmox nodes. I am primarily using OpenEBS for local storage, and Minio for object storage. The Talos VMs are backed up to an external drive regularly.
- actions-runner-controller: Self-hosted Github runners.
- cert-manager: Creates SSL certificates for services in my cluster.
- cilium: Internal Kubernetes container networking interface.
- cloudflared: Enables Cloudflare secure access to certain ingresses.
- grafana: Visualization into my cluster.
- external-dns: Automatically syncs ingress DNS records to a DNS provider.
- external-secrets: Managed Kubernetes secrets using an external provider (Bitwarden).
- ingress-nginx: Kubernetes ingress controller using NGINX as a reverse proxy and load balancer.
- prometheus: Cluster monitoring and metrics.
- sops: Managed secrets for Kubernetes and Terraform which are commited to Git.
- spegel: Stateless cluster local OCI registry mirror.
- rook-ceph: Storage Orchestration for Kubernetes.
- volsync: Backup and recovery of persistent volume claims.
Flux watches my cluster in the Kubernetes folder and makes the changes to my cluster based on the state within my Git repository.
The way Flux works for my cluster is by recursively searching the kubernetes/apps folder until it finds the top most kustomization.yaml per directory and then apply all resources listed within. The kustomization.yaml file will contain a namespace and one or more ks.yaml Flux kustomizations. Within those Flux kustomzations will be HelmReleases which dictate the resources that are applied for the specific application.
Renovate watches everything within the repository looking for updates. Once an update is found it creates a Pull Request in Github, allowing me to review changes before merging them. Once these changes are merged, Flux picks them up and applies the changes.
This Git repostories contains the following directories under Kubernetes
📁 kubernetes
├── 📁 apps # applications
├── 📁 bootstrap # bootstrap procedures
├── 📁 flux # core flux configuration
└── 📁 templates # re-useable componentsWhile I try to self-host almost everything myself, there are a couple of things that I currently maintain outside of the cluster for ease. Some of these are critical services that are just plain required, while others are luxuries that won't bankrupt me.
| Service | Use | Cost |
|---|---|---|
| Bitwarden | External Password Manager, and Secrets Manager with External Secrets | $10/yr |
| Cloudflare | Domain Registration and Cloudflare Tunnels | $10/yr |
| GitHub | Hosting this repository and continuous integration/deployments | Free |
| Pushover | Kubernetes Alerts and application notifications | $5 |
| UptimeRobot | Monitoring external facing applications | Free |
| Healthchecks.io | Heartbeat Monitoring for AlertManager and Internet | Free |
Total: |
$20/yr |
I am currently using ExternalDNS to create public DNS records in Cloudflare for externally facing applications and endpoints. I use the external ingress name and external ingress annotations to determine if an application is internal or external.
For my Home DNS I am using Pi-Hole (for now). Along with Pi-Hole I am utilizing the CoreDNS plugin, k8s_gateway to be able to automatically resolve internal dns using split DNS and dnsmasq.
Massive shoutout to the home-operations group, it has been fun lurking.