Perform manual validation for Docker Swarm Mode interop identified tests

[lipingxue]

This is to track Docker Swarm Mode(introduced in docker 1.12 release) interop related issue.

[govint]

ESX service needs to be made cluster aware. Currently there are multiple places where we have no way to identify if a VM really exists or not as the VM may not be on the same node. #934, #929 show how a VM (not necessarily deleted) but running on a different node can't be identified as existing,. We can have the UUID of the VM in a tenant's VM list and not know if the VM exists or not.

There has to be a way to determine the following:

1. Include the datacenter in VC to which the ESX host belongs and make it part of input to the ESX service.
2. Provide an api to contact VC to figure if the VM (UUID) exists in the datacenter to which the host belongs.
3. Use (2) in all cases where we need to determine if a VM actually exists (VM list per tenant, handling stale attach in vmdk_ops.py).

[tusharnt]

@lipingxue can you please add your analysis here? We can then close this investigation issue.

[lipingxue]

The following are two action items identified after trying out Docker Swarm Mode with vSphere docker volume service.

• Add Swarm related test in CI/CD test
  In CI/CD test, we can add the test for stateful service fail-over in Swarm mode. This is the test we need:
  1. Create a Swarm cluster, with three VMs (one master, two workers)
  2. Deploy a stateful service (redis for example) with persistent mode on worker1, login to redis-cli to set some key/value pair, which will be stored in redis.
  3. Kill worker1, wait for a while, check the service to make sure it restarts successfully on worker2. Login to redis-cli, and read the value with given key which was set in step 2, and the value should be same as the values set in step2.

• Understand what is the customer need to use Swarm with persistent storage (fail-over only or have the need to scale-up if needed)
  Current Swarm mode can only support fail-over case well (--replica 1).
  Swarm mode cannot support scale(--replica > 1) for stateful services well due to its own limitation (no task-level volume support), so not much things we can do to from vSphere volume service point of view.

[shuklanirdesh82]

#927 (comment)

@ashahi1 I'm assigning to you to perform manual validation for both photon and ubuntu distros.

[ashahi1]

Testing the fail-over scenario for docker swarm.

---

1. Created a swarm of three nodes - one master and two worker nodes. All three VMs are on the same host and shared-vmfs datastore

root@sc-rdops-vm02-dhcp-52-237:~# docker node ls
ID                           HOSTNAME                   STATUS  AVAILABILITY  MANAGER STATUS
2vixtu2afs45tx73xgn3qu4z7 *  sc-rdops-vm02-dhcp-52-237  Ready   Active        Leader
3fgwmxj57tz6b6f3e4o6uw5q7    sc-rdops-vm02-dhcp-52-237  Ready   Active
6np11t99e82iqp3cxenj87mfu    sc-rdops-vm02-dhcp-52-237  Ready   Active
root@sc-rdops-vm02-dhcp-52-237:~#

2. Created a volume from master - redis-data

root@sc-rdops-vm02-dhcp-52-237:~# docker volume create --driver=vsphere --name=redis-data -o size=1GB
redis-data
root@sc-rdops-vm02-dhcp-52-237:~# docker volume ls
DRIVER              VOLUME NAME
vsphere             Vol1_200mb@sharedVmfs-0
vsphere             redis-data@sharedVmfs-0

3. Created the redis service from the master with volume mounted

docker service create --mount type=volume,target=/data,source=redis-data -p 6379:6379 --constraint node.role==worker redis redis-server --appendonly yes


[root@sc-rdops-vm18-dhcp-54-69:/tmp] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py ls
Volume      Datastore     VM-Group  Capacity  Used  Filesystem  Policy  Disk Format  Attached-to      Access      Attach-as               Created By     Created Date
----------  ------------  --------  --------  ----  ----------  ------  -----------  ---------------  ----------  ----------------------  -------------  ------------------------
Vol1_200mb  sharedVmfs-0  _DEFAULT  100MB     15MB  ext4        N/A     thin         detached         read-write  independent_persistent  ubuntu-Master  Thu Mar 16 22:32:15 2017
redis-data  sharedVmfs-0  _DEFAULT  1GB       54MB  ext4        N/A     thin         ubuntu-worker-2  read-write  independent_persistent  ubuntu-Master  Thu Mar 16 23:28:09 2017

[root@sc-rdops-vm18-dhcp-54-69:/tmp]

4. Verified service started on worker-2

docker service ls

root@sc-rdops-vm02-dhcp-52-237:~# docker service ls
ID            NAME         REPLICAS  IMAGE  COMMAND
463qb2vp1m1k  gracious_keller  1/1       redis  redis-server --appendonly yes
root@sc-rdops-vm02-dhcp-52-237:~#

5. Updated the docker service

docker service update --restart-condition on-failure gracious_keller

6. Entered data to redis

root@sc-rdops-vm02-dhcp-52-237:~# redis-cli
127.0.0.1:6379> set hello world
OK
127.0.0.1:6379> get hello
"world"
127.0.0.1:6379> set India Delhi
OK
127.0.0.1:6379> get India
"Delhi"
127.0.0.1:6379>

7. Shutdown the worker-2. Service started on worker-1

root@sc-rdops-vm02-dhcp-52-237:~# docker node ls
ID                           HOSTNAME                   STATUS  AVAILABILITY  MANAGER STATUS
2vixtu2afs45tx73xgn3qu4z7 *  sc-rdops-vm02-dhcp-52-237  Ready   Active        Leader
3fgwmxj57tz6b6f3e4o6uw5q7    sc-rdops-vm02-dhcp-52-237  Ready   Active
6np11t99e82iqp3cxenj87mfu    sc-rdops-vm02-dhcp-52-237  Down    Active


[root@sc-rdops-vm18-dhcp-54-69:/tmp] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py ls
Volume      Datastore     VM-Group  Capacity  Used  Filesystem  Policy  Disk Format  Attached-to      Access      Attach-as               Created By     Created Date
----------  ------------  --------  --------  ----  ----------  ------  -----------  ---------------  ----------  ----------------------  -------------  ------------------------
Vol1_200mb  sharedVmfs-0  _DEFAULT  100MB     15MB  ext4        N/A     thin         detached         read-write  independent_persistent  ubuntu-Master  Thu Mar 16 22:32:15 2017
redis-data  sharedVmfs-0  _DEFAULT  1GB       55MB  ext4        N/A     thin         ubuntu-worker-1  read-write  independent_persistent  ubuntu-Master  Thu Mar 16 23:28:09 2017

[root@sc-rdops-vm18-dhcp-54-69:/tmp]

8. Verified the data from worker-1

root@sc-rdops-vm02-dhcp-52-237:~# redis-cli
127.0.0.1:6379> get hello
"world"
127.0.0.1:6379> get India
"Delhi"
127.0.0.1:6379> set USA Washington D.C.
(error) ERR syntax error
127.0.0.1:6379> set USA WashingtonD.C.
OK
127.0.0.1:6379> get USA
"WashingtonD.C."
127.0.0.1:6379>

[ashahi1]

@lipingxue

Liping, I have tested the fail-over scenario for docker swarm. I wrote data from one of the workers(worker-2), killed that worker and then I was able to read data from first worker (worker-1).

Please take a look at the posted output and re-open the issue if we need to do anymore testing.