Upgrading from 0.7.9 to 0.8.1 causes crashlooping

[laurivosandi]

What did you do to encounter the bug?
We have couple MongoDB instances managed by MongoDB Community Kubernetes Operator. Everything works as expected with 0.7.9. Yesterday tried upgrading to 0.8.1 which resulted all instances having a crashlooping pod

What did you expect?
No surprises

What happened instead?
The very first pod in the statefulset which was being rolled out kept crashlooping. I couldn't pinpoint a specific error in the instance or operator logs.

Screenshots
Didn't manage to get there, as I rolled back to 0.7.9

Operator Information

• Operator Version: 0.8.1
• MongoDB Image used: 5.0.9 and 6.0.3 both

Kubernetes Cluster Information

• Distribution: AWS EKS
• Version: 1.25.9
• Image Registry location: quay.io

Additional context
Will attempt upgrade again in a sandbox environment to provide logs.

[laurivosandi]

This might be related to #1344 , we have dedicated nodes for persistent storage (toleration+nodeSelector)

[mircea-cosbuc]

@laurivosandi Thank you for raising this, we're preparing a release with the fix (#1346). Interesting to note, in the issue you mentioned, there was no crashloop, just the reconciler adding tolerations to the statefulset and not completing. It would be useful to see logs, to understand if it's the same issue.

[laurivosandi]

Gave 0.8.2 a try and I am getting

2023-08-14T17:46:49.437Z	ERROR	controllers/mongodb_status_options.go:104	Error deploying MongoDB ReplicaSet: error creating/updating StatefulSet: error creating/updating StatefulSet: StatefulSet.apps "logmower-mongodb" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden

Generally speaking I've come to the conclusion that database operators should manage databases directly and avoid StatefulSet altogether. Just take a look at what CloudNativePG guys are doing

[github-actions]

This issue is being marked stale because it has been open for 60 days with no activity. Please comment if this issue is still affecting you. If there is no change, this issue will be closed in 30 days.

[github-actions]

This issue was closed because it became stale and did not receive further updates. If the issue is still affecting you, please re-open it, or file a fresh Issue with updated information.