VMDK Driver - VolumeDriver.Mount: Server rejected

[brockz]

Hello

i have 2 Worker hosts (photon os based) running in swarm mode.
When i create a Service with an mounted Volume via vmdk Driver and the Docker Host fails . The vmdk volume doesnt get unmounted. The Instance is getting restarted on the other Worker but the volume is not mounted:
cswy7i9te7688920nh6wvjbu3 _ nginx.1 nginx id-nb-pht-wrk01 Shutdown Rejected 24 minutes ago "VolumeDriver.Mount: Server re…"

[pdhamdhere]

@brockz We will need some more details about error and your setup. Can you please share some more details;

• Which release you are using?
• Setup: Can you please share more details on Swarm setup, VMs & ESX Hosts.
• Command that you used to create volume and/or start service
• Logs from Docker Hosts (/var/log/docker-volume-vsphere.log)
• Logs from ESX Host (/var/log/vmware/vmdk_ops.log)

[brockz]

Hello pdhamdhere.
Here are the Infos.

vSphere:
4 Node vSphere Cluster running ESX 6.0.0 Build 4600944
vCenter 6.0.0 4541948
vSan Datastore

Docker/Swarm:
4 VMS running Photon OS 1.0, Revision 2 (2 Swarm Manager and 2 Swarm Worker=
docker -v
Docker version 1.12.1, build 23cf638

Building Swarm:
Manager Node1: docker swarm init --advertise-addr 10.10.105.221
Manager Node2: docker swarm join --token xxxxxxxxxxxxxxxxxxx 10.10.105.221:2377
Worker Node1: docker swarm join --token xxxxxxxxx 10.10.105.221:2377
Worker Node2: docker swarm join --token xxxxxxxxx 10.10.105.221:2377

Create Volume:
docker volume create --driver=vmdk --name=MyVolume -o size=10gb

Created Service:
docker service create --name nginx --mount type=volume,source=MyVolume@shrk-vSAN01,target=/usr/share/nginx/html -p 80:80 nginx

Service started on Worker 2 succesfully:
root@id-nb-pht-ma01 [ ~ ]# docker service ps nginx
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR
cof9w87ki3xrtxckxn5btw3t0 nginx.1 nginx id-nb-pht-wrk02 Running Running 3 minutes ago

when i kill Worker 02 (vsphere poeroff vm id-nb-pht-wrk02) Service get restarted at worker 1 as expected but doesnt mount the volume:
cswy7i9te7688920nh6wvjbu3 _ nginx.1 nginx id-nb-pht-wrk01 Shutdown Rejected 24 minutes ago "VolumeDriver.Mount: Server re…"

What i can see from vSphere Side the Volume (virtualdisk) is still attached at worker2 and not attached at worker1 where the Service is now running.

I zipped all Log Files here:
docker-volume-vsphere_worker2.zip

Br Stefan

[pdhamdhere]

Thanks @brockz We have root caused the issue you are running into and working on a patch which will be available soon.

[msterin]

The issue is fixed in both master and release-0.11 branches.
We will create and publish a release 0.11.2 with the fix tonight ( @shuklanirdesh82 ).

@brockz - with this fix, the failover will work as expected. However, the VM which was force-poweroffed (Worker2 in your case) will fail to poweron without manual intervention, because it will try to attach MyVolume.vmdk , which is already attached to worker1. To poweron Worker2 back, you'd need to manually (UI or vim-cmd or script) detach MyVolume.vmdk

[brockz]

Great Work!

Running as expected !
Thanks

BR Stefan