smb mount fails with mount error(5): Input/output error

[vignesh-hbk]

What happened:

I have issues mounting a NetApp SMB share onto my ubuntu 22.04 machine using csi-smb-driver. The storage has AD auth enabled and with smbclient, I am able to connect and authenticate to the storage. However when I try to mount using CIFS or CSI-smb-driver, I am running into issues.

Smb logs :
E0120 09:59:29.001256 1 utils.go:81] GRPC error: rpc error: code = Internal desc = volume(smb-01-vol) mount "//xxxx/yyy" on "/var/lib/kubelet/plugins/kubernetes.io/csi/smb.csi.k8s.io/9b400ffc4472d1e8195452d738fac44a1ff7719ad65879f69fda6f55599971b1/globalmount" failed with mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t cifs -o domain=fmee, //xxx/yyy /var/lib/kubelet/plugins/kubernetes.io/csi/smb.csi.k8s.io/9b400ffc4472d1e8195452d738fac44a1ff7719ad65879f69fda6f55599971b1/globalmount
Output: mount error(5): Input/output error
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs) and kernel log messages (dmesg)
dmesg log
[ 39.528383] kernel: FS-Cache: Netfs 'cifs' registered for caching
[ 39.638853] kernel: Key type cifs.spnego registered
[ 39.638869] kernel: Key type cifs.idmap registered
[ 39.639378] kernel: CIFS: Attempting to mount \xxx\yyy
[ 39.960848] kernel: kauditd_printk_skb: 53 callbacks suppressed
[ 39.960852] kernel: audit: type=1400 audit(1737111830.818:65): apparmor="DENIED" operation="capable" profile="/usr/sbin/cupsd" pid=1180 comm="cupsd" capability=12 capname="net_admin"
[ 40.188282] kernel: CIFS: VFS: \xxx\yyy Send error in SessSetup = -5
[ 40.189072] kernel: CIFS: VFS: cifs_mount failed w/return code = -5
[ 40.746703] kernel: audit: type=1400 audit(1737111831.602:66): apparmor="DENIED" operation="capable" profile="/usr/sbin/cups-browsed" pid=1259 comm="cups-browsed" capability=23 capname="sys_nice"
[ 44.913797] kernel: loop12: detected capacity change from 0 to 8 (edited)

What you expected to happen:

Mount to be successful

How to reproduce it:

Try mounting a SMB share on Ubuntu 22.04 using csi-smb driver.

Anything else we need to know?:

I am able to connect and authenticate and see the shares using smbclient on the same server. But when attempting to mount using CIFS or smb-driver, it fails with the below error.

Environment:

• CSI Driver version: v1.11.0
• Kubernetes version (use kubectl version): v1.31.3+k3s1
• OS (e.g. from /etc/os-release): Ubuntu 22.04

[vignesh-hbk]

PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy

[vignesh-hbk]

Could someone help with this?

[andyzhangx]

does manual mount work on the node?

mount -t cifs -o domain=fmee, //xxx/yyy /mountpath

[vignesh-hbk]

@andyzhangx - no - it does not. See the below error.
Output: mount error(5): Input/output error
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs) and kernel log messages (dmesg).

However, when we use smbclient, we are able to authenticate and list the shares etc.

[andyzhangx]

what is smbclient?

[vignesh-hbk]

@andyzhangx - It's an ftp-like client to access SMB/CIFS resources on servers - https://www.samba.org/samba/docs/4.9/man-html/smbclient.1.html

[vignesh-hbk]

@andyzhangx - Could you help with this issue? Please let me know if you need further info. Thanks.

[Akash-HBK]

@andyzhangx : Could you please help us with an input on this to proceed further in order to mount the storage on the Ubuntu server using CSI-SMB driver?

[andyzhangx]

smb csi driver only uses standard smb driver on linux, and I see smbclient is an unmaintained version, not sure how to leverage that smbclient in smb csi driver.

[vignesh-hbk]

@andyzhangx - I mentioned smbclient only to indicate that there are no connectivity or authentication issues. The core issue is that we are unable to mount the smb drive using the csi-smb driver.

[andyzhangx]

point is this smb csi driver only uses mount -t cifs command to mount file share.