Test: Upgrade Operator From Previous Version (using operator‐sdk)

Summary

The purpose of this test is to verify that upgrading from the previous operator version (through Openshift) works as expected. This test can be difficult to set up before the driver, operator, and operator bundle are pushed out into the world because the operator bundle is built to reference the operator's canonical repo and the operator is built to reference the driver's canonical repo. The most practical approach is to use operator-sdk to deploy the bundle from the previous version and to then use operator-sdk to upgrade to a new version of the bundle.

Prerequisites:

• A Kubernetes cluster with the prerequisites needed to Run Operator With OLM Integration Using kubectl.

• Access to the operator bundle for the previous and current version of the driver.

  • Only a limited number of test packages are retained. If needed checkout the tag for the previous version and regenerate the operator bundle for the previous version and push it to a registry you have access to (note "-bundle" will automatically be appended to IMAGE_TAG_BASE):

  export VERSION=1.5.0
  export IMAGE_TAG_BASE=ghcr.io/iamjoemccormick/test-beegfs-csi-driver-operator
  make bundle-build bundle-push
  

  IMPORTANT: Note you do not want to remake the bundle, just build and push it. If you were to do a make bundle with IMAGE_TAG_BASE then the bundle would be regenerated with the controller image specified by IMAGE_TAG_BASE, not the actual release controller image (i.e., ghcr.io/thinkparq/beegfs-csi-driver-operator).

• The K8s namespace beegfs-csi must already exist.

Steps:

1. Deploy the previous version of the operator using operator-sdk using the canonical bundle image created for it on the last release: operator-sdk run bundle <IMAGE_NAME>-bundle:<TAG> -n beegfs-csi

2. Deploy a BeegfsDriver custom resource: kubectl apply -f config/samples/beegfs_v1_beegfsdriver.yaml -n beegfs-csi

• Ensure not to accidentally specify containerImageOverrides to ensure we can test the upgrade.
• Wait for the driver to deploy successfully.

3. Deploy the current version of the operator using operator-sdk on the appropriate bundle built by GitHub actions: operator-sdk run bundle-upgrade <IMAGE_NAME>-bundle:<TAG> -n beegfs-csi

• IMPORTANT: This step cannot be done until the deployment manifests for the operator have been updated to reference the new version (even though it should not be tagged/released yet) and the updated bundle regenerated and committed. Otherwise running the bundle upgrade will fail for an error like duplicate entry "beegfs-csi-driver-operator.v1.5.0". Note it is expected the updated operator is not yet pushed/tagged to thinkparq/beegfs-csi-driver-operator:<VERSION>, the bundles built by GitHub actions override the operator image to point to a test image like ghcr.io/thinkparq/test-beegfs-csi-driver-operator:<GIT_COMMIT> (the version the operator-sdk cares about in the bundle is different than the image tag).

4. Verify the following:

• The operator pod has been recreated.
• The operator pod is running the latest image version.
• The operator pod logs indicate it updated the controller service StatefulSet and node service DaemonSet (more than 1 operator pod).

The following resources should be recreated and have their images updated to the new version, but will have failed to start because the new image is not yet externally available:

• The controller service pod has been recreated.
• The controller service StatefulSet is running the latest image version (kubectl describe statefulset -n beegfs-csi).
• All node service pods have been recreated
• The node service DaemonSet is running the latest image version (kubectl describe ds -n beegfs-csi).

5. If desired cleanup by running: operator-sdk cleanup beegfs-csi-driver-operator -n beegfs-csi