Awesome Mutator

Awesome Mutator is a Kubernetes mutating webhook that dynamically modifies pod specifications based on custom rules defined in a ConfigMap. It can add or remove node selectors, tolerations, and other configurations to match specific conditions.

Features

• Dynamic Mutations: Modify pod specifications on-the-fly based on configurable rules.
• Node Selectors and Tolerations: Add or remove node selectors and tolerations to influence pod scheduling.
• ConfigMap Driven: Easily update mutation rules without changing the webhook code.
• Fail-Open Mechanism: Ensures that pod creation continues even if the webhook encounters errors.

Prerequisites

• Kubernetes Cluster: A running Kubernetes cluster.
• kubectl: Command-line tool to interact with the Kubernetes cluster.
• Docker: To build and manage container images.
• Kubernetes Python Client: Used to interact with the Kubernetes API.

Setup and Deployment

• You can pull the pre-built docker image from: https://hub.docker.com/repository/docker/lkup77/awesome_mutator/general

Apply the k8s scripts

• kubectl apply -f k8s

This will create the needed rbac, service, deployment and webhook along with a sample configuration that you can change in awesome-mutator-config-map.yaml

Note that when you first apply the k8s manifests, the webhook may not be up yet so the test pods will fail scheduling. Just delete them and retry once you've confirmed the webhook is up and running ok.

You should be able to see the startup logs as follows:

INFO:awesome_mutator:Loaded mutation rules from ConfigMap: [{'name': 'rule1', 'podSelector': 'app=myapp,environment=prod', 'removeNodeSelectors': ['disktype'], 'addNodeSelectors': {'scheduling.cast.ai/node-template': 'test-mut-nt'}}, {'name': 'remove-agentpool-for-canyon', 'podSelector': 'app=canyon', 'removeNodeSelectors': ['agentpool'], 'addNodeSelectors': {'scheduling.cast.ai/node-template': 'test-mut-nt-3'}, 'addTolerations': [{'key': 'scheduling.cast.ai/node-template', 'operator': 'Equal', 'value': '', 'effect': 'NoSchedule'}]}]
INFO:     Application startup complete.

The k8s folder also contains 2 test pods with various labels and node selectors. Once the webhook is up, it will mutate according to the rules in the config map.

Rules

• Rules stop on first match
• Rules are matched by label selector using podSelector (and operation)
• Rules will remove NodeSelecor by name using removeNodeSelectors
• Rules will add NodeSelector using details of addNodeSelectors
• Rules will add Tolerations using the details of addTolerations

An example of an applied rule from the logs:

INFO:awesome_mutator:Received mutation request
INFO:awesome_mutator:Filtered object metadata: {'name': 'test-pod-agentpool', 'namespace': 'default', 'labels': {'app': 'canyon'}, 'annotations': {'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"labels":{"app":"canyon"},"name":"test-pod-agentpool","namespace":"default"},"spec":{"containers":[{"image":"nginx","name":"nginx"}],"nodeSelector":{"agentpool":"copprfpool"}}}\n'}}
INFO:awesome_mutator:Filtered pod spec: {'volumes': [{'name': 'kube-api-access-cm4xs', 'projected': {'sources': [{'serviceAccountToken': {'expirationSeconds': 3607, 'path': 'token'}}, {'configMap': {'name': 'kube-root-ca.crt', 'items': [{'key': 'ca.crt', 'path': 'ca.crt'}]}}, {'downwardAPI': {'items': [{'path': 'namespace', 'fieldRef': {'apiVersion': 'v1', 'fieldPath': 'metadata.namespace'}}]}}], 'defaultMode': 420}}], 'containers': [{'name': 'nginx', 'image': 'nginx', 'resources': {}, 'volumeMounts': [{'name': 'kube-api-access-cm4xs', 'readOnly': True, 'mountPath': '/var/run/secrets/kubernetes.io/serviceaccount'}], 'terminationMessagePath': '/dev/termination-log', 'terminationMessagePolicy': 'File', 'imagePullPolicy': 'Always'}], 'tolerations': [{'key': 'node.kubernetes.io/not-ready', 'operator': 'Exists', 'effect': 'NoExecute', 'tolerationSeconds': 300}, {'key': 'node.kubernetes.io/unreachable', 'operator': 'Exists', 'effect': 'NoExecute', 'tolerationSeconds': 300}], 'priority': 0}
INFO:awesome_mutator:Created V1Pod object for pod: test-pod-agentpool
INFO:awesome_mutator:Creating JSON patches for pod: test-pod-agentpool
INFO:awesome_mutator:Checking if pod matches selector 'app=myapp,environment=prod': False
INFO:awesome_mutator:Checking if pod matches selector 'app=canyon': True
INFO:awesome_mutator:Pod matches rule 'remove-agentpool-for-canyon': Applying mutations
INFO:awesome_mutator:Added patch to create nodeSelector
INFO:awesome_mutator:Adding node selector 'scheduling.cast.ai/node-template: test-mut-nt-3'
INFO:awesome_mutator:Adding toleration: {'key': 'scheduling.cast.ai/node-template', 'operator': 'Equal', 'value': '', 'effect': 'NoSchedule'}
INFO:awesome_mutator:Stopping rule evaluation after applying rule 'remove-agentpool-for-canyon'
INFO:awesome_mutator:Generated patches: [{'op': 'add', 'path': '/spec/nodeSelector', 'value': {}}, {'op': 'add', 'path': '/spec/nodeSelector/scheduling.cast.ai~1node-template', 'value': 'test-mut-nt-3'}, {'op': 'add', 'path': '/spec/tolerations/-', 'value': {'key': 'scheduling.cast.ai/node-template', 'operator': 'Equal', 'value': '', 'effect': 'NoSchedule'}}]
INFO:awesome_mutator:Sending admission response
INFO:     10.60.1.2:48094 - "POST /mutate?timeout=10s HTTP/1.1" 200 OK

TODO's

• Unit Tests + Mocks
• Change to CRD's versus ConfigMap
• Create helm chart to install and operate