k8s-job-buildkite-plugin

Run a Command Step as a Kubernetes Job using a Pod Spec.

Requirements

The plugin assumes you have kubectl and kustomize available on your agent, and these have appropriate credentials with sufficient privileges to create resources on your kubernetes cluster.

There are buildkite/agent:alpine-k8s images available on Docker Hub which are the standard buildkite/agent:alpine image with the appropriate tools included as well.

You can run such an agent in a kubernetes pod with a service account and an appropriate RBAC policy like:

kind: ServiceAccount
apiVersion: v1
metadata:
  name: agent-k8s-job

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: agent-k8s-job
rules:
  - apiGroups: [""]
    resources: ["pods", "pods/log"]
    verbs: ["get", "watch", "list"]
  - apiGroups: ["batch"]
    resources: ["jobs"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: agent-k8s-job
subjects:
  - kind: ServiceAccount
    name: agent-k8s-job
roleRef:
  kind: Role
  name: agent-k8s-job
  apiGroup: rbac.authorization.k8s.io

NOTE: If you want k8s job to be deployed to specific namespace then above resources needed to be create in that k8s namespace.

Usage

You can specify a pod-spec within the plugin configuration. This takes a kubernetes pod spec that will run in a kubernetes job. It is often easier to format this with a yaml reference like below:

success-spec: &success-spec
  restartPolicy: OnFailure
  containers:
    - name: ${BUILDKITE_PIPELINE_SLUG}-${BUILDKITE_BUILD_NUMBER}-${BUILDKITE_STEP_ID}-success
      image: deftinc/winpenguin
      command: ["/app/entrypoint.sh"]

steps:
  - label: "Consulting my :crystal_ball: thats inside my :k8s: pod"
    plugins:
      - k8s-job:
          pod-spec: *success-spec

Examples

Here is a simple command step that you can run as job in k8s:

steps:
  - label: "Consulting my :crystal_ball: thats inside my :k8s: pod"
    plugins:
      - k8s-job#v1.0.0:
          pod-spec:
            containers:
              - name: ${BUILDKITE_PIPELINE_SLUG}-${BUILDKITE_BUILD_NUMBER}-${BUILDKITE_STEP_ID}-success
                image: busybox:1.28
                command: ['sh', '-c', 'echo "Hello, Kubernetes!" && sleep 30']

Options

The K8s Job plugin supports a number of different configuration options.

pod-spec (required, object)

pod-spec will be used to specify the k8s job definition

cleanup (optional, boolean)

cleanup will clean up the job and pod resources after the step runs. If set to false they will remain for 10 minutes to debug. Defaults to true.

metadata (optional, object)

Metadata object will be used to add annotations and namespace for the kubernetes job.

metadata.annotations will add metadata (annotations) to the kubernetes job. Defaults to an empty object.

metadata.namespace will add namespace to the kubernetes job. Defaults to default namespace.

mount-source (optional, boolean)

mount-source will mount the source code to /buildkite/src if set to true. This can be used for build steps that build images from source. Defaults to false.

To use this option your agent must include in the nodeName it is running on in its environment variables as BUILDKITE_AGENT_NODE_NAME. It must also include a hostPath volumes and volumeMount for the agent.

env:
  - name: BUILDKITE_AGENT_NODE_NAME
    valueFrom:
      fieldRef:
        fieldPath: spec.nodeName

volumes:
  - name: builds
    hostPath:
      path: /data/buildkite/builds
      type: DirectoryOrCreate

volumeMounts:
  - name: builds
    mountPath: "/buildkite/builds"

timeoutInSeconds (optional, number)

timeoutInSeconds will set how long the job should wait for all of the init containers and containers in the pod spec to finish before reporting an error. Defaults to 300.