Endless Rolling Restarts when using Tolerations

[siegenthalerroger]

What did you do to encounter the bug?

Had a MongoDB resource with v0.8.0 like below (incomplete) and then upgraded to v0.8.1.

apiVersion: mongodbcommunity.mongodb.com/v1
kind: MongoDBCommunity
metadata:
  name: mongodb
  namespace: demo-mongodb
spec:
  members: 2
  type: ReplicaSet
  version: "6.0.8"
  statefulSet:
    spec:
      template:
        metadata:
          labels:
            variant: global
        spec:
          containers:
            - ... (resource configuration for agent + mongod, aswell as mongo-exporter sidecar)
          tolerations:
            - key: "optimised-for"
              operator: "Equal"
              value: "memory"
              effect: "PreferNoSchedule"

What did you expect?

The cluster to continue to operate with the newer operator.

What happened instead?

There was an endless rolling restart loop between the two replicas.
Downgrading to 0.8.0 rectified the issue.

I triaged this to be due to the operator repeatedly adding the "toleration" defined above to the StatefulSet, causing a state difference which k8s reconciled by triggering a rolling update. I suspect this bug was introduced by #1314.

Following a cleaned-up variant of the operator-reconciled mongo StatefulSet:

``` apiVersion: apps/v1 kind: StatefulSet metadata: name: mongodb namespace: demo-mongodb uid: 1922f590-e379-4bff-8362-8a7a5cbfc352 resourceVersion: '242788943' generation: 2798 creationTimestamp: '2023-08-02T11:41:33Z' ownerReferences: - apiVersion: mongodbcommunity.mongodb.com/v1 kind: MongoDBCommunity name: mongodb uid: 3bbe2f15-b9d8-4c26-8507-eb538927bcce controller: true blockOwnerDeletion: true managedFields: - manager: manager operation: Update apiVersion: apps/v1 time: '2023-08-02T11:41:33Z' fieldsType: FieldsV1 fieldsV1: f:metadata: f:ownerReferences: .: {} k:{"uid":"3bbe2f15-b9d8-4c26-8507-eb538927bcce"}: {} f:spec: f:podManagementPolicy: {} f:replicas: {} f:revisionHistoryLimit: {} f:selector: {} f:serviceName: {} f:template: f:metadata: f:labels: .: {} f:app: {} f:variant: {} f:spec: f:containers: k:{"name":"mongo-exporter"}: .: {} f:args: {} f:env: .: {} k:{"name":"MONGODB_URI"}: .: {} f:name: {} f:value: {} f:image: {} f:imagePullPolicy: {} f:name: {} f:resources: .: {} f:limits: .: {} f:cpu: {} f:memory: {} f:requests: .: {} f:cpu: {} f:memory: {} f:terminationMessagePath: {} f:terminationMessagePolicy: {} k:{"name":"mongod"}: .: {} f:args: {} f:command: {} f:env: .: {} k:{"name":"AGENT_STATUS_FILEPATH"}: .: {} f:name: {} f:value: {} f:image: {} f:imagePullPolicy: {} f:name: {} f:resources: .: {} f:limits: .: {} f:cpu: {} f:memory: {} f:requests: .: {} f:cpu: {} f:memory: {} f:securityContext: .: {} f:allowPrivilegeEscalation: {} f:readOnlyRootFilesystem: {} f:terminationMessagePath: {} f:terminationMessagePolicy: {} f:volumeMounts: .: {} k:{"mountPath":"/data"}: .: {} f:mountPath: {} f:name: {} k:{"mountPath":"/healthstatus"}: .: {} f:mountPath: {} f:name: {} k:{"mountPath":"/hooks"}: .: {} f:mountPath: {} f:name: {} k:{"mountPath":"/tmp"}: .: {} f:mountPath: {} f:name: {} k:{"mountPath":"/var/lib/mongodb-mms-automation/authentication"}: .: {} f:mountPath: {} f:name: {} k:{"mountPath":"/var/log/mongodb-mms-automation"}: .: {} f:mountPath: {} f:name: {} k:{"name":"mongodb-agent"}: .: {} f:command: {} f:env: .: {} k:{"name":"AGENT_LOG_LEVEL"}: .: {} f:name: {} f:value: {} k:{"name":"AGENT_MAX_LOG_FILE_DURATION_HOURS"}: .: {} f:name: {} f:value: {} k:{"name":"AGENT_STATUS_FILEPATH"}: .: {} f:name: {} f:value: {} k:{"name":"AUTOMATION_CONFIG_MAP"}: .: {} f:name: {} f:value: {} k:{"name":"HEADLESS_AGENT"}: .: {} f:name: {} f:value: {} k:{"name":"POD_NAMESPACE"}: .: {} f:name: {} f:valueFrom: .: {} f:fieldRef: {} f:image: {} f:imagePullPolicy: {} f:name: {} f:readinessProbe: .: {} f:exec: .: {} f:command: {} f:failureThreshold: {} f:initialDelaySeconds: {} f:periodSeconds: {} f:successThreshold: {} f:timeoutSeconds: {} f:resources: .: {} f:limits: .: {} f:cpu: {} f:memory: {} f:requests: .: {} f:cpu: {} f:memory: {} f:securityContext: .: {} f:allowPrivilegeEscalation: {} f:readOnlyRootFilesystem: {} f:terminationMessagePath: {} f:terminationMessagePolicy: {} f:volumeMounts: .: {} k:{"mountPath":"/data"}: .: {} f:mountPath: {} f:name: {} k:{"mountPath":"/opt/scripts"}: .: {} f:mountPath: {} f:name: {} k:{"mountPath":"/tmp"}: .: {} f:mountPath: {} f:name: {} k:{"mountPath":"/var/lib/automation/config"}: .: {} f:mountPath: {} f:name: {} f:readOnly: {} k:{"mountPath":"/var/lib/mongodb-mms-automation/authentication"}: .: {} f:mountPath: {} f:name: {} k:{"mountPath":"/var/log/mongodb-mms-automation"}: .: {} f:mountPath: {} f:name: {} k:{"mountPath":"/var/log/mongodb-mms-automation/healthstatus"}: .: {} f:mountPath: {} f:name: {} f:dnsPolicy: {} f:initContainers: .: {} k:{"name":"mongod-posthook"}: .: {} f:command: {} f:image: {} f:imagePullPolicy: {} f:name: {} f:resources: {} f:securityContext: .: {} f:allowPrivilegeEscalation: {} f:readOnlyRootFilesystem: {} f:terminationMessagePath: {} f:terminationMessagePolicy: {} f:volumeMounts: .: {} k:{"mountPath":"/hooks"}: .: {} f:mountPath: {} f:name: {} k:{"name":"mongodb-agent-readinessprobe"}: .: {} f:command: {} f:image: {} f:imagePullPolicy: {} f:name: {} f:resources: {} f:securityContext: .: {} f:allowPrivilegeEscalation: {} f:readOnlyRootFilesystem: {} f:terminationMessagePath: {} f:terminationMessagePolicy: {} f:volumeMounts: .: {} k:{"mountPath":"/opt/scripts"}: .: {} f:mountPath: {} f:name: {} f:restartPolicy: {} f:schedulerName: {} f:securityContext: .: {} f:fsGroup: {} f:runAsNonRoot: {} f:runAsUser: {} f:serviceAccount: {} f:serviceAccountName: {} f:terminationGracePeriodSeconds: {} f:tolerations: {} f:volumes: .: {} k:{"name":"agent-scripts"}: .: {} f:emptyDir: {} f:name: {} k:{"name":"automation-config"}: .: {} f:name: {} f:secret: .: {} f:defaultMode: {} f:secretName: {} k:{"name":"healthstatus"}: .: {} f:emptyDir: {} f:name: {} k:{"name":"hooks"}: .: {} f:emptyDir: {} f:name: {} k:{"name":"mongodb-keyfile"}: .: {} f:emptyDir: {} f:name: {} k:{"name":"tmp"}: .: {} f:emptyDir: {} f:name: {} f:updateStrategy: f:type: {} f:volumeClaimTemplates: {} - manager: kube-controller-manager operation: Update apiVersion: apps/v1 time: '2023-08-02T12:03:39Z' fieldsType: FieldsV1 fieldsV1: f:status: f:availableReplicas: {} f:collisionCount: {} f:currentRevision: {} f:observedGeneration: {} f:readyReplicas: {} f:replicas: {} f:updateRevision: {} f:updatedReplicas: {} subresource: status selfLink: /apis/apps/v1/namespaces/demo-mongodb/statefulsets/mongodb status: observedGeneration: 2798 replicas: 2 readyReplicas: 1 updatedReplicas: 1 currentRevision: mongodb-86c99f5dbd updateRevision: mongodb-fc5fc6d9d collisionCount: 0 availableReplicas: 1 spec: replicas: 2 selector: matchLabels: app: mongodb-svc template: metadata: creationTimestamp: null labels: app: mongodb-svc variant: global spec: volumes: - name: agent-scripts emptyDir: {} - name: automation-config secret: secretName: mongodb-config defaultMode: 416 - name: healthstatus emptyDir: {} - name: hooks emptyDir: {} - name: mongodb-keyfile emptyDir: {} - name: tmp emptyDir: {} initContainers: - name: mongod-posthook image: >- quay.io/mongodb/mongodb-kubernetes-operator-version-upgrade-post-start-hook:1.0.7 command: - cp - version-upgrade-hook - /hooks/version-upgrade resources: {} volumeMounts: - name: hooks mountPath: /hooks terminationMessagePath: /dev/termination-log terminationMessagePolicy: File imagePullPolicy: Always securityContext: readOnlyRootFilesystem: true allowPrivilegeEscalation: false - name: mongodb-agent-readinessprobe image: quay.io/mongodb/mongodb-kubernetes-readinessprobe:1.0.15 command: - cp - /probes/readinessprobe - /opt/scripts/readinessprobe resources: {} volumeMounts: - name: agent-scripts mountPath: /opt/scripts terminationMessagePath: /dev/termination-log terminationMessagePolicy: File imagePullPolicy: Always securityContext: readOnlyRootFilesystem: true allowPrivilegeEscalation: false containers: - name: mongo-exporter image: percona/mongodb_exporter:0.39.0 args: - '--discovering-mode' - '--collect-all' - '--collector.dbstats' - '--collector.topmetrics' - '--collector.collstats-limit=-1' - '--log.level=info' env: - name: MONGODB_URI value: mongodb://LOREM-IPSUM resources: limits: cpu: 250m memory: 50Mi requests: cpu: 100m memory: 50Mi terminationMessagePath: /dev/termination-log terminationMessagePolicy: File imagePullPolicy: Always - name: mongod image: docker.io/mongo:6.0.8 command: - /bin/sh - '-c' - |+

          #run post-start hook to handle version changes
          /hooks/version-upgrade

          # wait for config and keyfile to be created by the agent
           while ! [ -f /data/automation-mongod.conf -a -f /var/lib/mongodb-mms-automation/authentication/keyfile ]; do sleep 3 ; done ; sleep 2 ;

          # start mongod with this configuration
          exec mongod -f /data/automation-mongod.conf;

      args:
        - ''
      env:
        - name: AGENT_STATUS_FILEPATH
          value: /healthstatus/agent-health-status.json
      resources:
        limits:
          cpu: '1'
          memory: 2500Mi
        requests:
          cpu: 250m
          memory: 2500Mi
      volumeMounts:
        - name: data-volume
          mountPath: /data
        - name: healthstatus
          mountPath: /healthstatus
        - name: hooks
          mountPath: /hooks
        - name: logs-volume
          mountPath: /var/log/mongodb-mms-automation
        - name: mongodb-keyfile
          mountPath: /var/lib/mongodb-mms-automation/authentication
        - name: tmp
          mountPath: /tmp
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      imagePullPolicy: IfNotPresent
      securityContext:
        readOnlyRootFilesystem: true
        allowPrivilegeEscalation: false
    - name: mongodb-agent
      image: quay.io/mongodb/mongodb-agent:12.0.24.7719-1
      command:
        - /bin/bash
        - '-c'
        - >-
          current_uid=$(id -u)

          AGENT_API_KEY="$(cat
          /mongodb-automation/agent-api-key/agentApiKey)"

          declare -r current_uid

          if ! grep -q "${current_uid}" /etc/passwd ; then

          sed -e "s/^mongodb:/builder:/" /etc/passwd > /tmp/passwd

          echo "mongodb:x:$(id -u):$(id -g):,,,:/:/bin/bash" >> /tmp/passwd

          export NSS_WRAPPER_PASSWD=/tmp/passwd

          export LD_PRELOAD=libnss_wrapper.so

          export NSS_WRAPPER_GROUP=/etc/group

          fi

          agent/mongodb-agent
          -healthCheckFilePath=/var/log/mongodb-mms-automation/healthstatus/agent-health-status.json
          -serveStatusPort=5000
          -cluster=/var/lib/automation/config/cluster-config.json
          -skipMongoStart -noDaemonize -useLocalMongoDbTools -logFile
          /var/log/mongodb-mms-automation/automation-agent.log
          -maxLogFileDurationHrs ${AGENT_MAX_LOG_FILE_DURATION_HOURS}
          -logLevel ${AGENT_LOG_LEVEL}
      env:
        - name: AGENT_LOG_LEVEL
          value: INFO
        - name: AGENT_MAX_LOG_FILE_DURATION_HOURS
          value: '24'
        - name: AGENT_STATUS_FILEPATH
          value: >-
            /var/log/mongodb-mms-automation/healthstatus/agent-health-status.json
        - name: AUTOMATION_CONFIG_MAP
          value: mongodb-config
        - name: HEADLESS_AGENT
          value: 'true'
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
      resources:
        limits:
          cpu: 200m
          memory: 100Mi
        requests:
          cpu: 50m
          memory: 100Mi
      volumeMounts:
        - name: agent-scripts
          mountPath: /opt/scripts
        - name: automation-config
          readOnly: true
          mountPath: /var/lib/automation/config
        - name: data-volume
          mountPath: /data
        - name: healthstatus
          mountPath: /var/log/mongodb-mms-automation/healthstatus
        - name: logs-volume
          mountPath: /var/log/mongodb-mms-automation
        - name: mongodb-keyfile
          mountPath: /var/lib/mongodb-mms-automation/authentication
        - name: tmp
          mountPath: /tmp
      readinessProbe:
        exec:
          command:
            - /opt/scripts/readinessprobe
        initialDelaySeconds: 5
        timeoutSeconds: 1
        periodSeconds: 10
        successThreshold: 1
        failureThreshold: 40
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      imagePullPolicy: Always
      securityContext:
        readOnlyRootFilesystem: true
        allowPrivilegeEscalation: false
  restartPolicy: Always
  terminationGracePeriodSeconds: 30
  dnsPolicy: ClusterFirst
  serviceAccountName: mongodb-database
  serviceAccount: mongodb-database
  securityContext:
    runAsUser: 2000
    runAsNonRoot: true
    fsGroup: 2000
  schedulerName: default-scheduler
  tolerations:
    - key: optimised-for
      operator: Equal
      value: memory
      effect: PreferNoSchedule
    - key: optimised-for
      operator: Equal
      value: memory
      effect: PreferNoSchedule
    - key: optimised-for
      operator: Equal
      value: memory
      effect: PreferNoSchedule
    - key: optimised-for
      operator: Equal
      value: memory
      effect: PreferNoSchedule
    - ... quite literally thousands more copies

volumeClaimTemplates:
- kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: data-volume
creationTimestamp: null
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 30Gi
storageClassName: ssd-retained
volumeMode: Filesystem
status:
phase: Pending
- kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: logs-volume
creationTimestamp: null
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 2G
volumeMode: Filesystem
status:
phase: Pending
serviceName: mongodb-svc
podManagementPolicy: OrderedReady
updateStrategy:
type: RollingUpdate
revisionHistoryLimit: 10

</details>


**Operator Information**

 - Operator Version: v0.8.1
 - MongoDB Image used: v6.0.8

**Kubernetes Cluster Information**

 - Distribution: AKS
 - Version: 1.24
 - Image Registry location: quay.io

[mircea-cosbuc]

@siegenthalerroger Thank you for raising this, it is indeed a bug addressed with #1346 . We're releasing a fix version shortly.

[code-uri]

fix #1346 seems working fine for me after upgrading to 0.8.2 version.

[mircea-cosbuc]

@siegenthalerroger Closing this as it was addressed with the 0.8.2 version. Please re-open if you're still encountering the issue. Thank you for bringing the issue up so quickly!

[siegenthalerroger]

Also tested on my end, working as intended now :)