Failed to start WiredTiger under any compatibility version, Mongo v5.0.23

[james-ro-williams]

What did you do to encounter the bug?
Steps to reproduce the behavior:

1. Start the MongoDB Community Kubernetes Operator using Helm
2. Attempt to start a ReplicaSet, using a custom storage class for Amazon EFS storage

What did you expect?
The pods in the ReplicaSet to startup.

What happened instead?
The pods do not start, due to the "Failed to start up WiredTiger under any compatibility version" error.

Screenshots

Operator Information

• MongoDB Community Kubernetes Operator 0.11.0
• 5.0.23

Kubernetes Cluster Information

• AWS EKS

Additional context
Wondering if this means that Mongo 5.0.23 cannot be deployed using the operator? It's EOL, so that would make sense.

❯ k get pods
NAME                                           READY   STATUS    RESTARTS   AGE
mongodb-kubernetes-operator-558d9545b8-l7jtz   1/1     Running   0          39m
test-mongodb-0                                 0/2     Error     12         37m
                
❯ k get mdbc
NAME           PHASE   VERSION
test-mongodb           

• yaml definitions of your MongoDB Deployment(s):

apiVersion: v1
items:
- apiVersion: mongodbcommunity.mongodb.com/v1
  kind: MongoDBCommunity
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"mongodbcommunity.mongodb.com/v1","kind":"MongoDBCommunity","metadata":{"annotations":{},"name":"test-mongodb","namespace":"default"},"spec":{"additionalMongodConfig":{"storage.wiredTiger.engineConfig.journalCompressor":"zlib"},"members":1,"security":{"authentication":{"modes":["SCRAM"]}},"statefulSet":{"spec":{"securityContext":{"fsGroup":1005,"runAsUser":1005},"volumeClaimTemplates":[{"metadata":{"name":"data-volume"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"1Gi"}},"storageClassName":"efs-sc-mongo"}},{"metadata":{"name":"logs-volume"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"1Gi"}},"storageClassName":"efs-sc-mongo"}}]}},"type":"ReplicaSet","users":[{"db":"admin","name":"my-user","passwordSecretRef":{"name":"my-user-password"},"roles":[{"db":"admin","name":"clusterAdmin"},{"db":"admin","name":"userAdminAnyDatabase"}],"scramCredentialsSecretName":"my-scram"}],"version":"5.0.23"}}
    creationTimestamp: "2024-11-01T14:32:59Z"
    generation: 1
    name: test-mongodb
    namespace: default
    resourceVersion: "1177775"
    uid: c2da8054-441a-433b-9a28-304c46856c92
  spec:
    additionalMongodConfig:
      storage.wiredTiger.engineConfig.journalCompressor: zlib
    members: 1
    security:
      authentication:
        ignoreUnknownUsers: true
        modes:
        - SCRAM
    statefulSet:
      spec:
        securityContext:
          fsGroup: 1005
          runAsUser: 1005
        volumeClaimTemplates:
        - metadata:
            name: data-volume
          spec:
            accessModes:
            - ReadWriteOnce
            resources:
              requests:
                storage: 1Gi
            storageClassName: efs-sc-mongo
        - metadata:
            name: logs-volume
          spec:
            accessModes:
            - ReadWriteOnce
            resources:
              requests:
                storage: 1Gi
            storageClassName: efs-sc-mongo
    type: ReplicaSet
    users:
    - db: admin
      name: my-user
      passwordSecretRef:
        name: my-user-password
      roles:
      - db: admin
        name: clusterAdmin
      - db: admin
        name: userAdminAnyDatabase
      scramCredentialsSecretName: my-scram
    version: 5.0.23
kind: List
metadata:
  resourceVersion: ""

• The Pod logs:

Defaulted container "mongod" out of: mongod, mongodb-agent, mongod-posthook (init), mongodb-agent-readinessprobe (init)
2024-11-01T15:10:55.291Z        INFO    versionhook/main.go:33  Running version change post-start hook
2024-11-01T15:10:55.291Z        INFO    versionhook/main.go:40  Waiting for agent health status...
2024-11-01T15:10:56.292Z        INFO    versionhook/main.go:76  Pod should not be deleted, mongod started
{"t":{"$date":"2024-11-01T15:10:58.316+00:00"},"s":"I",  "c":"NETWORK",  "id":4915701, "ctx":"-","msg":"Initialized wire specification","attr":{"spec":{"incomingExternalClient":{"minWireVersion":0,"maxWireVersion":13},"incomingInternalClient":{"minWireVersion":0,"maxWireVersion":13},"outgoing":{"minWireVersion":0,"maxWireVersion":13},"isInternalClient":true}}}
{"t":{"$date":"2024-11-01T15:10:58.318+00:00"},"s":"I",  "c":"CONTROL",  "id":23285,   "ctx":"main","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
{"t":{"$date":"2024-11-01T15:10:58.318+00:00"},"s":"W",  "c":"ASIO",     "id":22601,   "ctx":"main","msg":"No TransportLayer configured during NetworkInterface startup"}
{"t":{"$date":"2024-11-01T15:10:58.318+00:00"},"s":"I",  "c":"NETWORK",  "id":4648601, "ctx":"main","msg":"Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize."}
{"t":{"$date":"2024-11-01T15:10:58.382+00:00"},"s":"W",  "c":"ASIO",     "id":22601,   "ctx":"main","msg":"No TransportLayer configured during NetworkInterface startup"}
{"t":{"$date":"2024-11-01T15:10:58.382+00:00"},"s":"W",  "c":"ASIO",     "id":22601,   "ctx":"main","msg":"No TransportLayer configured during NetworkInterface startup"}
{"t":{"$date":"2024-11-01T15:10:58.382+00:00"},"s":"I",  "c":"REPL",     "id":5123008, "ctx":"main","msg":"Successfully registered PrimaryOnlyService","attr":{"service":"TenantMigrationDonorService","ns":"config.tenantMigrationDonors"}}
{"t":{"$date":"2024-11-01T15:10:58.382+00:00"},"s":"I",  "c":"REPL",     "id":5123008, "ctx":"main","msg":"Successfully registered PrimaryOnlyService","attr":{"service":"TenantMigrationRecipientService","ns":"config.tenantMigrationRecipients"}}
{"t":{"$date":"2024-11-01T15:10:58.382+00:00"},"s":"I",  "c":"CONTROL",  "id":5945603, "ctx":"main","msg":"Multi threading initialized"}
{"t":{"$date":"2024-11-01T15:10:58.382+00:00"},"s":"I",  "c":"CONTROL",  "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":1,"port":27017,"dbPath":"/data","architecture":"64-bit","host":"test-mongodb-0"}}
{"t":{"$date":"2024-11-01T15:10:58.382+00:00"},"s":"W",  "c":"CONTROL",  "id":20720,   "ctx":"initandlisten","msg":"Memory available to mongo process is less than total system memory","attr":{"availableMemSizeMB":476,"systemMemSizeMB":7930}}
{"t":{"$date":"2024-11-01T15:10:58.382+00:00"},"s":"I",  "c":"CONTROL",  "id":23403,   "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"5.0.23","gitVersion":"3367195a14d0ba2734d2ba2719294fb974ad0834","openSSLVersion":"OpenSSL 1.1.1k  FIPS 25 Mar 2021","modules":[],"allocator":"tcmalloc","environment":{"distmod":"rhel80","distarch":"x86_64","target_arch":"x86_64"}}}}
{"t":{"$date":"2024-11-01T15:10:58.382+00:00"},"s":"I",  "c":"CONTROL",  "id":51765,   "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"Red Hat Enterprise Linux release 8.10 (Ootpa)","version":"Kernel 6.1.112-122.189.amzn2023.x86_64"}}}
{"t":{"$date":"2024-11-01T15:10:58.382+00:00"},"s":"I",  "c":"CONTROL",  "id":21951,   "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"config":"/data/automation-mongod.conf","net":{"bindIp":"0.0.0.0","port":27017},"replication":{"replSetName":"test-mongodb"},"security":{"authorization":"enabled","keyFile":"/var/lib/mongodb-mms-automation/authentication/keyfile"},"setParameter":{"authenticationMechanisms":"SCRAM-SHA-256"},"storage":{"dbPath":"/data","wiredTiger":{"engineConfig":{"journalCompressor":"zlib"}}}}}}
{"t":{"$date":"2024-11-01T15:10:58.390+00:00"},"s":"I",  "c":"STORAGE",  "id":22315,   "ctx":"initandlisten","msg":"Opening WiredTiger","attr":{"config":"create,cache_size=256M,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=zlib),builtin_extension_config=(zstd=(compression_level=6)),file_manager=(close_idle_time=600,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress,compact_progress],"}}
{"t":{"$date":"2024-11-01T15:10:58.956+00:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":17,"message":"[1730473858:956440][1:0x7fabc735ffc0], connection: __posix_open_file, 815: /data/WiredTiger.wt: handle-open: open: File exists"}}
{"t":{"$date":"2024-11-01T15:10:58.986+00:00"},"s":"I",  "c":"STORAGE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"unexpected file WiredTiger.wt found, renamed to WiredTiger.wt.36"}}
{"t":{"$date":"2024-11-01T15:10:58.991+00:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":1,"message":"[1730473858:991693][1:0x7fabc735ffc0], connection: __posix_open_file, 815: /data/WiredTiger.wt: handle-open: open: Operation not permitted"}}
{"t":{"$date":"2024-11-01T15:10:59.031+00:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":17,"message":"[1730473859:31003][1:0x7fabc735ffc0], connection: __posix_open_file, 815: /data/WiredTiger.wt: handle-open: open: File exists"}}
{"t":{"$date":"2024-11-01T15:10:59.038+00:00"},"s":"I",  "c":"STORAGE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"unexpected file WiredTiger.wt found, renamed to WiredTiger.wt.37"}}
{"t":{"$date":"2024-11-01T15:10:59.044+00:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":1,"message":"[1730473859:44444][1:0x7fabc735ffc0], connection: __posix_open_file, 815: /data/WiredTiger.wt: handle-open: open: Operation not permitted"}}
{"t":{"$date":"2024-11-01T15:10:59.083+00:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":17,"message":"[1730473859:83369][1:0x7fabc735ffc0], connection: __posix_open_file, 815: /data/WiredTiger.wt: handle-open: open: File exists"}}
{"t":{"$date":"2024-11-01T15:10:59.091+00:00"},"s":"I",  "c":"STORAGE",  "id":22430,   "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":"unexpected file WiredTiger.wt found, renamed to WiredTiger.wt.38"}}
{"t":{"$date":"2024-11-01T15:10:59.099+00:00"},"s":"E",  "c":"STORAGE",  "id":22435,   "ctx":"initandlisten","msg":"WiredTiger error","attr":{"error":1,"message":"[1730473859:99878][1:0x7fabc735ffc0], connection: __posix_open_file, 815: /data/WiredTiger.wt: handle-open: open: Operation not permitted"}}
{"t":{"$date":"2024-11-01T15:10:59.106+00:00"},"s":"W",  "c":"STORAGE",  "id":22347,   "ctx":"initandlisten","msg":"Failed to start up WiredTiger under any compatibility version. This may be due to an unsupported upgrade or downgrade."}
{"t":{"$date":"2024-11-01T15:10:59.106+00:00"},"s":"F",  "c":"STORAGE",  "id":28595,   "ctx":"initandlisten","msg":"Terminating.","attr":{"reason":"1: Operation not permitted"}}
{"t":{"$date":"2024-11-01T15:10:59.106+00:00"},"s":"F",  "c":"-",        "id":23091,   "ctx":"initandlisten","msg":"Fatal assertion","attr":{"msgid":28595,"file":"src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp","line":688}}
{"t":{"$date":"2024-11-01T15:10:59.106+00:00"},"s":"F",  "c":"-",        "id":23092,   "ctx":"initandlisten","msg":"\n\n***aborting after fassert() failure\n\n"}

[github-actions]

This issue is being marked stale because it has been open for 60 days with no activity. Please comment if this issue is still affecting you. If there is no change, this issue will be closed in 30 days.

[james-ro-williams]

Still affecting me.