Failed to create filesystem

[pdhamdhere]

Guest: Ubuntu
Plugin & ESX: 0.11.2

docker volume create -d vmdk test
Error response from daemon: create test: VolumeDriver.Create: Failed to create filesystem on /dev/disk/by-path/pci-0000:0b:00.0-scsi-0:0:0:0: exit status 1. Output = mke2fs 1.42.13 (17-May-2015)
The file /dev/disk/by-path/pci-0000:0b:00.0-scsi-0:0:0:0 does not exist and no size was specified.

Plugin Log

vi /var/log/docker-volume-vsphere.log 

2017-02-23 23:30:36.775775667 +0000 UTC [INFO] No config file found. Using defaults.
2017-02-23 23:30:36.779514188 +0000 UTC [INFO] Starting plugin driver=vmdk log_level=debug config="/etc/docker-volume-vsphere.conf"
2017-02-23 23:30:36.779532548 +0000 UTC [INFO] Plugin options - target= project= host= port=1019
2017-02-23 23:30:36.779587945 +0000 UTC [INFO] Getting volume data from unix:///var/run/docker.sock
2017-02-23 23:30:36.881349688 +0000 UTC [DEBUG] Docker info: version=1.13.1, root=/var/lib/docker, OS=Ubuntu 16.04.1 LTS
2017-02-23 23:30:36.885583808 +0000 UTC [DEBUG] Found 1 running or paused containers
2017-02-23 23:30:36.892998512 +0000 UTC [DEBUG]   Mounts for [/ng_nginx.1.l5ofaz04vxbv57yptgv2e3v20]
2017-02-23 23:30:36.893479953 +0000 UTC [INFO] Discovered 0 volumes in use.
2017-02-23 23:30:36.893535381 +0000 UTC [INFO] Docker VMDK plugin started mock_esx=false version="VMDK Volume Driver v0.3" port=1019
2017-02-23 23:30:36.893624281 +0000 UTC [INFO] Going into ServeUnix - Listening on Unix socket address="/run/docker/plugins/vmdk.sock"
2017-02-23 23:30:36.893795439 +0000 UTC [DEBUG] root group found. gid: 0
2017-02-23 23:34:00.878984855 +0000 UTC [DEBUG] vmdkOps.List
2017-02-23 23:34:24.853644321 +0000 UTC [DEBUG] vmdkOps.Get name=test
2017-02-23 23:34:25.140046239 +0000 UTC [DEBUG] vmdkOp.Create name=test
2017-02-23 23:34:26.075956163 +0000 UTC [INFO] Attaching volume and creating filesystem name=test fstype=ext4
2017-02-23 23:34:26.076045762 +0000 UTC [DEBUG] vmdkOps.Attach name=test
2017-02-23 23:34:30.730551151 +0000 UTC [ERROR] Create filesystem failed, removing the volume name=test error="Failed to create filesystem on /dev/disk/by-path/pci-0000:0b:00.0-scsi-0:0:0:0: exit status 1. Output = mke2fs 1.42.13 (17-May-2015)\nThe file /dev/disk/by-path/pci-0000:0b:00.0-scsi-0:0:0:0 does not exist and no size was specified.\n"
2017-02-23 23:34:30.741866376 +0000 UTC [DEBUG] vmdkOps.Detach name=test
2017-02-23 23:34:31.26366184 +0000 UTC [DEBUG] vmdkOps.Remove name=test
2017-02-23 23:35:14.532690695 +0000 UTC [DEBUG] vmdkOps.List
2017-02-23 23:36:24.684071814 +0000 UTC [DEBUG] vmdkOps.Get name=test2
2017-02-23 23:36:24.864055681 +0000 UTC [DEBUG] vmdkOp.Create name=test2
2017-02-23 23:36:25.700283584 +0000 UTC [INFO] Attaching volume and creating filesystem name=test2 fstype=ext4
2017-02-23 23:36:25.700372296 +0000 UTC [DEBUG] vmdkOps.Attach name=test2
2017-02-23 23:36:26.747723996 +0000 UTC [DEBUG] vmdkOps.Detach name=test2
2017-02-23 23:36:27.248259567 +0000 UTC [INFO] Volume and filesystem created name=test2 fstype=ext4

I ran into this only once. I thought this should not happen with #893 CC/ @govint

Will bump the priority if run into this again.

[pdhamdhere]

This is 100% reproducible on fresh Ubuntu installation. First volume creation fails but subsequent volume creation works fine.

[govint]

Tried this out but not reproducing. CI also isn't showing this yet.

@pdhamdhere are there service and plugin logs for the failure that I can check.

[pdhamdhere]

Tried this out but not reproducing. CI also isn't showing this yet.

This is only reproducible with fresh ubuntu installation on first volume create command.

@pdhamdhere are there service and plugin logs for the failure that I can check.

Plugin logs are copy-pasted in first comment. Sorry, forgot to collect ESX logs. I'll try to repro tomorrow.

[govint]

Ok, got a repro with an Ubuntu VM just now. I'll root cause this one.

[msterin]

#1058 resolves this. We should not see the Exceeded timeout while waiting for device attach to complete nor filesystem create failure on pvscsi attach anymore.

Closing