[BUG] Blob Storage CSI driver not upgraded in all AKS 1.29 clusters in norwayeast

[Laffs2k5]

Describe the bug

We are still observing #4242 in our AKS 1.29 clusters in Norway East.

The cause of the issue is a bug introduced in Azure Blob Storage CSI driver for Kubernetes version 1.23.4.

The bug is fixed in driver versions:

• v1.23.5, ref. [release-1.23] fix: create private endpoint failure kubernetes-sigs/blob-csi-driver#1408
• v1.24.1, ref. [release-1.24] chore: refine refine private endpoint to reduce error messages kubernetes-sigs/blob-csi-driver#1301

AKS releases shows that the Blob Storage CSI driver was bumped to 1.24.1 for AKS 1.28 as part of AKS v20240428. The AKS Release tracker shows Norway East currently have v20240513 deployed.

It is unclear from AKS releases what version to expect to get for the Blob Storage CSI driver in AKS clusters 1.29. It was indicated in #4242 in a comment that upgrading to AKS 1.29 would resolve the issue, but that is not the case for all our clusters.

We have 3 AKS clusters in Norway East, all running AKS 1.29. One of the clusters has the Blob Storage CSI driver version 1.24.1 running, while the other two have the Blob Storage CSI driver version 1.23.4 running. It's the latter two clusters that are experiencing the issue.

To Reproduce

1. Have the private DNS zone privatelink.blob.core.windows.net already existing in the resource group of the AKS vnet.
2. Assign the AKS identity Contributor role in the resource group hosting the AKS vnet.
3. Have the blob storage CSI driver enabled: az aks update --enable-blob-driver
4. Deploy sc + pvc + deployment to AKS (see examples of our declarations under Additional context below)

Expected behavior

1. Storage account + blob is created
2. Private DNS zone is updated with a record for the blob
3. The AKS deployment provisions successfully

Screenshots

Not applicable.

Environment

• AKS 1.29.4 (cluster kubernetesVersion and node pool orchestratorVersion, all 3 clusters)
• Node image version AKSUbuntu-2204gen2containerd-202405.27.0 (all 3 clusters)
• Azure Blob Storage CSI driver for Kubernetes version 1.23.4 (for clusters with issues)

Additional context

custom StorageClass

apiVersion: storage.k8s.io/v1
kind: StorageClass
allowVolumeExpansion: true
metadata:
  labels:
  name: sc-blob-storage-test-application-pr-203
mountOptions:
  - '-o allow_other'
  - '--file-cache-timeout-in-seconds=120'
  - '--use-attr-cache=true'
  - '--cancel-list-on-mount-seconds=10'
  - '-o attr_timeout=120'
  - '-o entry_timeout=120'
  - '-o negative_timeout=120'
  - '--log-level=LOG_WARNING'
  - '--cache-size-mb=1000'
parameters:
  allowBlobPublicAccess: 'false'
  containerName: test-application-pr-203
  matchTags: 'true'
  networkEndpointType: privateEndpoint
  protocol: fuse2
  skuName: Standard_LRS
  tags: >-
    CreatedBy=Azure Kubernetes Service,ApplicationName=AKS application
    storage,Description=Storage account for ephemeral environments in AKS
provisioner: blob.csi.azure.com
reclaimPolicy: Delete
volumeBindingMode: Immediate

PersistentVolumeClaim

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    volume.beta.kubernetes.io/storage-class: sc-blob-storage-test-application-pr-203
  name: pvc-blob-storage-test-application-pr-203
  namespace: test-application-pr-203
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Gi
  storageClassName: sc-blob-storage-test-application-pr-203

Deployment (not a complete example, just the important parts)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-application-pr-203-deployment
  namespace: test-application-pr-203
spec:
  replicas: 1
  selector:
    matchLabels:
      app: test-application-pr-203-app
  template:
    metadata:
      labels:
        app: test-application-pr-203-app
        app.kubernetes.io/name: test-application-backend
        app.kubernetes.io/part-of: test-application
        app.kubernetes.io/version: pr-203-2024.04.25.53165
        azure.workload.identity/use: 'false'
    spec:
      containers:
        - env:
            - name: AZURE_PERSISTENT_STORAGE_MOUNT_PATH
              value: /mnt/azure-blob-storage
          image: <our ACR>/test-application-pr-203:pr-203-2024.04.25.53165
          name: test-application-pr-203-container
          volumeMounts:
            - mountPath: /mnt/azure-blob-storage
              name: blob-storage
              readOnly: false
      volumes:
        - name: blob-storage
          persistentVolumeClaim:
            claimName: pvc-blob-storage-test-application-pr-203

[andyzhangx]

v1.24.1 has a critical bug (token expire issue), we have released v1.23.5 to fix this issue: https://github.com/kubernetes-sigs/blob-csi-driver/releases/tag/v1.23.5, the rollout on aks would still take a few weeks, just email me your cluster fqdn, I could upgrade your cluster with v1.23.5 in advance if you cannot wait, thanks.

[microsoft-github-policy-service]

Thanks for reaching out. I'm closing this issue as it was marked with "Answer Provided" and it hasn't had activity for 2 days.