Project for demonstrating fault injection with k6 and xk6-disruptor on the OpenTelemetry Demo in Kubernetes.
The scenario is that we have our system under test (SUT) provided by the most excellent OpenTelemetry Astronomy Shop. The entire system will be running within a single Kubernetes cluster on which we can inject failures to test reliability.
We'll be making use of these tools during the course of this demonstration.
- git - For accessing the sourcecode repositories.
- go - To install
xk6and build a customk6binary. - Docker - For running the examples and other tooling.
- k3d - Wrapper to run a k3s Kubernetes cluster in Docker.
- kubectl - Client for talking to Kubernetes clusters.
- helm - Client for deploying software packages into Kubernetes.
There may be others that I didn't recall as having them installed long ago. My apologies for any issues!
k6 is flexible tool allowing for custom capabilities via an extension system using xk6.
Using xk6 and extensions, you build a new k6 binary incorporating the desired functionality.
xk6-disruptor is one of those extensions which are not included in
the core k6 product by default.
🔖 Refer to the xk6 guides for the latest information if you run into problems building.
First, ensure xk6 is installed into your Go toolchain:
go install go.k6.io/xk6/cmd/xk6@latestNext we'll use xk6 to build a new k6 binary with the disruptor source code pulled directly from GitHub:
xk6 build latest \
--with github.com/grafana/xk6-disruptor👉 If you'd like additional extensions to try out, take a look at the Bundle section of the k6 documentation to review extensions and build your command line. Make sure
xk6-disruptoris selected!
Once the xk6 command finishes building, verify your custom build by checking the version.
./k6 versionYou should see output similar to the following:
$ ./k6 version
k6 v0.46.0 ((devel), go1.21.1, darwin/arm64)
Extensions:
github.com/grafana/xk6-disruptor v0.3.9, k6/x/disruptor [js]
⚠️ Make sure you use the newly builtk6from your current directory! It's possible that you also have another version ofk6on your system path.
We'll use k3d to run a local Kubernetes cluster within Docker. Once installed, use
the following command to create a three-node cluster named k6-demo-cluster.
k3d cluster create k6-demo-cluster \
--api-port 6550 \
-p "8081:80@loadbalancer" \
--agents 3👉 If you've previously created the cluster, you can start the cluster using
k3d cluster start k6-demo-clusterif not already running.
Once this is complete, you'll now have a running Kubernetes cluster on which you can use kubectl as well as other tooling
like k9s.
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
k3d-k6-demo-cluster-agent-1 Ready <none> 15s v1.27.4+k3s1
k3d-k6-demo-cluster-server-0 Ready control-plane,master 18s v1.27.4+k3s1
k3d-k6-demo-cluster-agent-2 Ready <none> 14s v1.27.4+k3s1
k3d-k6-demo-cluster-agent-0 Ready <none> 14s v1.27.4+k3s1
The Astronomy Shop provides a realistic microservices environment comprised of multiple services, written in various languages, communicating using various protocols including gRPC, Kafka messaging, and HTTP.
The recommended approach for installing the demo application is using a Helmchart.
🔖 Refer to the Kubernetes installation for the latest information if you run into problems deploying.
If not previously done, you'll need to configure your Helm client to look for deployment packages, i.e. Helmcharts, within the OpenTelemetry chart repository. This should only need to be done once.
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-chartsNext, we'll create an isolated namespace within our Kubernetes cluster, then install the open-telemetry/opentelemetry-demo
chart into it as otel-demo.
kubectl create ns otel-demo
helm install otel-demo open-telemetry/opentelemetry-demo -n otel-demoHere is an example of the output to expect.
$ helm install otel-demo open-telemetry/opentelemetry-demo -n otel-demo
NAME: otel-demo
LAST DEPLOYED: Tue Sep 19 16:26:07 2023
NAMESPACE: otel-demo
STATUS: deployed
REVISION: 1
NOTES:
=======================================================================================
██████╗ ████████╗███████╗██╗ ██████╗ ███████╗███╗ ███╗ ██████╗
██╔═══██╗╚══██╔══╝██╔════╝██║ ██╔══██╗██╔════╝████╗ ████║██╔═══██╗
██║ ██║ ██║ █████╗ ██║ ██║ ██║█████╗ ██╔████╔██║██║ ██║
██║ ██║ ██║ ██╔══╝ ██║ ██║ ██║██╔══╝ ██║╚██╔╝██║██║ ██║
╚██████╔╝ ██║ ███████╗███████╗ ██████╔╝███████╗██║ ╚═╝ ██║╚██████╔╝
╚═════╝ ╚═╝ ╚══════╝╚══════╝ ╚═════╝ ╚══════╝╚═╝ ╚═╝ ╚═════╝
- All services are available via the Frontend proxy: http://localhost:8080
by running these commands:
kubectl port-forward svc/otel-demo-frontendproxy 8080:8080
The following services are available at these paths once the proxy is exposed:
Webstore http://localhost:8080/
Grafana http://localhost:8080/grafana/
Feature Flags UI http://localhost:8080/feature/
Load Generator UI http://localhost:8080/loadgen/
Jaeger UI http://localhost:8080/jaeger/ui/
- OpenTelemetry Collector OTLP/HTTP receiver (required for browser spans to be emitted):
by running these commands:
kubectl port-forward svc/otel-demo-otelcol 4318:4318
As noted in the above output from the installation, we'll add some port-forwarding to enable access to the services from outside the Kubernetes cluster, i.e. from our desktop.
👉 This can also be accomplished by creating a Port-Forward on the services from within k9s.
# Enable access to the UI elements from the host machine browser.
kubectl port-forward -n otel-demo svc/otel-demo-frontendproxy 8080:8080With the forwarding enabled, the following links should now be accessible:
# Enable publishing Open Telemetry metrics and traces from our k6 tests.
kubectl port-forward -n otel-demo svc/otel-demo-otelcol 4318:4318By default, Locust is running generating some activity against our site. Open Grafana, looking at activity for the Recommendation Service.
Using k6, let's create some increased load for the service.
./k6 run tests/01-recommendation-spike.jsUsing the objects from xk6-disruptor, we'll inject service disruptions to the recommendation service.
./k6 run tests/02-recommendation-service-disruption.jsUsing the objects from xk6-disruptor, we'll inject pod disruptions to pods of the recommendation service.
./k6 run tests/03-recommendation-pod-disruption.js