SubPath unmount fails with "directory not empty" after smb-server Service IP changes

[drigz]

What happened:

After the smb-server Service IP changes, I am unable to delete pods that have mounted smb volumes. The pods get stuck in "Terminating", and the kubelet logs contain:

Feb 03 18:40:58  kubelet[862425]: W0203 18:40:58.568146  862425 mount_helper_common.go:33] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/ddc7ea95-c27f-4b3c-9136-cb6acea5f0f6/volume-subpaths/volume/volume/1
Feb 03 18:41:19  kubelet[862425]: W0203 18:41:19.048156  862425 mount_helper_common.go:33] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/ddc7ea95-c27f-4b3c-9136-cb6acea5f0f6/volume-subpaths/volume/volume/2
Feb 03 18:41:19  kubelet[862425]: E0203 18:41:19.048366  862425 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/smb.csi.k8s.io^volume podName:ddc7ea95-c27f-4b3c-9136-cb6acea5f0f6 nodeName:}" failed. No retries permitted until 2021-02-03 18:43:21.048305415 +0100 CET m=+12623.734651277 (durationBeforeRetry 2m2s). Error: "error cleaning subPath mounts for volume \"volume\" (UniqueName: \"kubernetes.io/csi/smb.csi.k8s.io^volume\") pod \"ddc7ea95-c27f-4b3c-9136-cb6acea5f0f6\" (UID: \"ddc7ea95-c27f-4b3c-9136-cb6acea5f0f6\") : error deleting /var/lib/kubelet/pods/ddc7ea95-c27f-4b3c-9136-cb6acea5f0f6/volume-subpaths/volume/volume: remove /var/lib/kubelet/pods/ddc7ea95-c27f-4b3c-9136-cb6acea5f0f6/volume-subpaths/volume/volume: directory not empty"

The "path does not exist" warning is wrong:

$ sudo ls /var/lib/kubelet/pods/ddc7ea95-c27f-4b3c-9136-cb6acea5f0f6/volume-subpaths/volume/volume/1
ls: cannot access '/var/lib/kubelet/pods/ddc7ea95-c27f-4b3c-9136-cb6acea5f0f6/volume-subpaths/volume/volume/1': Host is down

What you expected to happen:

Pods can be deleted and recreated.

How to reproduce it:

1. Create a volume based on smb-server.yaml.
2. Mount the volume in a pod.
3. Delete and recreate the smb-server service to change its IP. (I think this can happen other ways, I just did this to reproduce an error we encountered on another cluster)
4. Restart the smb-server process. (not sure why this is required, maybe to break existing connections?)
5. Try to delete the pod.

Anything else we need to know?:

#64 seems similar but the error is different.
kubernetes/kubernetes#97031 seems to be a similar codepath (and the fix may be the same?) but with a different cause.

Environment:

• CSI Driver version: 0.4.0
• Kubernetes version (use kubectl version): 1.18.10
• OS (e.g. from /etc/os-release): Debian testing
• Kernel (e.g. uname -a): 5.7.17
• Install tools: manually copied YAMLs from this repo
• Others:

[drigz]

I also have a question: if changing the service IP addresses breaks existing mounts, would it be better to specify a clusterIP in the smb-server Service?

[drigz]

Note: if you try to create a new pod instead of deleting an old one, you get the MountVolume.MountDevice failed for volume "workcell-spec" : stat /var/lib/kubelet/plugins/kubernetes.io/csi/pv/workcell-spec/globalmount: host is down error described in #164. I have also seen that error in our live clusters, although I don't if it correlated with an IP change in those cases.

[andyzhangx]

looks like it's related to #164, need upstream fix, will take a look later.