[Windows Plugin] Failed to mount volume with short name when it is already mounted with long name previously

[ashahi1]

Failed to mount volume with short name when it is already mounted with long name previously and vice versa.

Test Name: TestLongVolumeName
https://github.com/vmware/docker-volume-vsphere/blob/ee3ef339cb0703492e6d02f96fb4cb8a242065a1/tests/e2e/restart_test.go

Steps:

1. Create a volume.
2. Run a container using short volume name - successful.
3. Tried to start another container using long volume name - failed. Error: C:\Program Files\Docker\docker.exe: Error response from daemon: error while mounting volume 'C:\Windows\system32\config\systempr ofile\AppData\Local\docker-volume-vsphere\mounts\test1234@sharedvmfs-0\': VolumeDriver.Mount: Add-PartitionAccessPath : The requ ested access path is already in use.

===========

Detailed steps and their output are as follows:

1. Create a volume - test1234

PS C:\Users\root> docker volume create -d vsphere --name test1234
test1234
PS C:\Users\root> docker volume ls
DRIVER              VOLUME NAME
vsphere             test1234@sharedvmfs-0
vsphere             testwin@sharedvmfs-0

2. Run a container using short volume name

PS C:\Users\root> docker run -d -v test1234:C:\vol microsoft/nanoserver powershell ping -t google.com
ccefc8ac026d2b6c3a5b2eb2bd96de0742d1c738193d0e72ab603f1421ee4f53
PS C:\Users\root>

3. Run another container using long volume name - Failed

PS C:\Users\root> docker run -d -v test1234@sharedvmfs-0:C:\vol microsoft/nanoserver powershell ping -t google.com
3a625a8a635ed0225e0ebef444097775465fd546d367e2459483f92ac22a7051
C:\Program Files\Docker\docker.exe: Error response from daemon: error while mounting volume 'C:\Windows\system32\config\systempr
ofile\AppData\Local\docker-volume-vsphere\mounts\test1234@sharedvmfs-0\': VolumeDriver.Mount: Add-PartitionAccessPath : The requ
ested access path is already in use.
Activity ID: {4c5c84b0-f8be-46c9-8871-6f9986d53ef5}
At line:1 char:45
+ ... se;         Add-PartitionAccessPath -DiskNumber 2 -PartitionNumber 1  ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidArgument: (StorageWMI:ROOT/Microsoft/.../
   MSFT_Partition) [Add-PartitionAccessPath], CimException
    + FullyQualifiedErrorId : StorageWMI 42002,Add-PartitionAccessPath

:               Set-Disk -Number 2 -IsReadOnly $false;          Add-PartitionAccessPath -DiskNumber 2 -PartitionNumber 1 -Access
Path "C:\Windows\system32\config\systemprofile\AppData\Local\docker-volume-vsphere\mounts\test1234@sharedvmfs-0\";.

Logs:
vmdk_ops.txt
docker-volume-vsphere.txt

[ashahi1]

@govint Can you please take a look at this issue?

[ashahi1]

Hit the same issue when I write a message from host1 to a file on the volume and try to read the content from host2 from same file on the volume.

PS C:\Users\root> docker run --rm -it -v xyz@sharedvmfs-0:C:\vol microsoft/nanoserver powershell 'type C:\vol\testABC.txt'
C:\Program Files\Docker\docker.exe: Error response from daemon: error while mounting volume 'C:\Windows\system32\config\systempr
ofile\AppData\Local\docker-volume-vsphere\mounts\xyz@sharedvmfs-0\': VolumeDriver.Mount: Add-PartitionAccessPath : Not Supported
Activity ID: {c229db8c-eb5d-4f48-bce7-3459c11f197c}
At line:1 char:45
+ ... se;         Add-PartitionAccessPath -DiskNumber 2 -PartitionNumber 1  ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (StorageWMI:ROOT/Microsoft/...
   /MSFT_Partition) [Add-PartitionAccessPath], CimException
    + FullyQualifiedErrorId : StorageWMI 1,Add-PartitionAccessPath

:               Set-Disk -Number 2 -IsReadOnly $false;          Add-PartitionAccessPath -DiskNumber 2 -PartitionNumber 1 -Access
Path "C:\Windows\system32\config\systemprofile\AppData\Local\docker-volume-vsphere\mounts\xyz@sharedvmfs-0\";.
PS C:\Users\root>

[shuklanirdesh82]

In response to #2006 (comment)

Hit the same issue when I write a message from host1 to a file on the volume and try to read the content from host2 from same file on the volume.

It is not reproducible and not an issue. I have executed following steps and am confirming that the scenario is working as expected (let's ignore the comment #2006 (comment))

testbed setup: 2 windows VM registered to one ESX

steps

1. VM1: create volume and invoke "docker volume ls" 
2. ESX: invoke "esxcli storage guestvol volume ls" to make sure volume is created correctly and properties are valid
3. VM2: invoke "docker volume ls" and validates its visibility
4. VM1: attach volume and write some data > detach volume
5. VM2: attach the same volume and read the written data by VM1
6. VM2: append some data > detach volume
7. VM1: mount the volume again and validates written data by VM2
8. VM1: detach volume > delete volume

[shuklanirdesh82]

in response to #2006 (comment)

if the volume is mounted with short name subsequent mount request with short volume is not having issue and vice versa for the long volume name.

The reported issue is observed when the volume is mounted with short name and the next mout request comes with long volume name.

one question I have here is: what happens when there is two volume present with the same name created on the different datastore?

root@sc-rdops-vm02-dhcp-52-237:~# docker volume ls
DRIVER              VOLUME NAME
vsphere:latest      a@sharedVmfs-0
vsphere:latest      a@vsanDatastore

@govint can you please help for the workflow mentioned at #2006 (comment)?

[shuklanirdesh82]

one question I have here is: what happens when there is two volume present with the same name created on the different datastore?

root@sc-rdops-vm02-dhcp-52-237:~# docker volume ls
DRIVER              VOLUME NAME
vsphere:latest      a@sharedVmfs-0
vsphere:latest      a@vsanDatastore

Discussed with @govint offline and updating the thread here.

Answer: If there are volumes with the same name on different DS then the long name has to be given, it is a known issue and we need to document this one. (Thanks @govint for the explanation)

@ashahi1 we need to document this one, can you please add this information to your ongoing PR.

[govint]

Root cause is fs_windows.go:GetMountInfo() is parsing the result of "Get-Partition" (powershell) and parses the result and seems to end up stripping all of the returned mounted partitions. When the next mount request of the same volume comes, the code misses the fact that the volume is already mounted and tries to mount again (always uses the @ format to mount) and gets a mount error.

The failed mount results in the volume getting detached. Meaning,. the container thats already using the volume just lost its access to the volume and data.

Will post the fix tomorrow for review.

[govint]

Another issue is how windows passes the volume name to docker and hence the plugin. The datastore name gets converted to lower case and so the plugin doesn't realize the volume is already mounted and tries mounting again.

For example,

First we mount a volume "mvol2" and then try to mount it again,
2017-12-04 13:05:48.9853312 +0000 GMT [INFO] Mounting volume name=mvol2
2017-12-04 13:05:49.1412432 +0000 GMT [INFO] Already mounted, skipping mount. name="mvol2@sharedVmfs-0" refcount=2

then we try to mount it with the datastore name attached,
2017-12-04 13:48:05.5566709 +0000 GMT [INFO] Mounting volume name="mvol2@sharedvmfs-0" <------ the datastore name is in lower case and hence refcounting didn't catch that the same volume is being mounted and proceed to mount the volume.
2017-12-04 13:48:22.0746727 +0000 GMT [INFO] List mounts script executedstdout="1 \n1 \\?\Volume{2a973c43-d3bf-11e7-a94d-0050568059aa}\\n3 \n3 \\?\Volume{999fbcf2-d543-11e7-a955-cd80bb9bb971}\\n2 C:\Windows\System32\config\systemprofile\AppData\Local\docker-volume-vsphere\mounts\mvol2@sharedVmfs-0\ \\?\Volume{4bfb0fac-0000-0000-0000-010000000000}\\n0 \\?\Volume{5362f858-0000-0000-0000-100000000000}\\n0 C:\ \\?\Volume{5362f858-0000-0000-0000-501f00000000}\\n"

[govint]

Looks like the approach used in #1712 wasn't quiet complete, case insensitive checks in the ESX service is one thing, the plugin can fail within itself even before reaching the ESX service. This issue is one example. The refcnt module is unable to recognize the volume name with lowercase datastore name and allows a duplicate mount to get thru.

[shuklanirdesh82]

Looks like the approach used in #1712 wasn't quiet complete, case insensitive checks in the ESX service is one thing, the plugin can fail within itself even before reaching the ESX service. This issue is one example. The refcnt module is unable to recognize the volume name with lowercase datastore name and allows a duplicate mount to get thru.

CC'ing Shaomin here.

/CC @shaominchen