Skip to content
This repository has been archived by the owner on Nov 9, 2020. It is now read-only.

Identify tests for multi-esx scenario #957

Closed
5 tasks done
shuklanirdesh82 opened this issue Feb 23, 2017 · 10 comments
Closed
5 tasks done

Identify tests for multi-esx scenario #957

shuklanirdesh82 opened this issue Feb 23, 2017 · 10 comments
Assignees
Milestone

Comments

@shuklanirdesh82
Copy link
Contributor

shuklanirdesh82 commented Feb 23, 2017

Multi-Esx tests

Testbed setup:

  1. 2 ESX hosts (E1 & E2)
  2. E1 is having access to local-vmfs, vsan, shared-vmfs ( the same for E2)
  3. 2 VMs per each datastore from ESX (e.g. 2 vms on local/vsan/shared-vmfs backed by E1 i.e. total 6 vms)

Note: atleast one vm is mandatory (more than 1 is recommended for complex scenario testing

for our test scenario

- vm1_shared_E1 & vm11_shared_E1 created on shared vmfs backed by E1
- vm2_shared_E2 & vm22_shared_E2 created on shared vmfs backed by E2
- vm3_vsan_E1 & vm33_vsan_E1 created on vsan backed by E1
- vm4_vsan_E2 & vm44_vsan_E2 created on vsan backed by E2
- vm5_local_E1 & vm55_local_E1 created on local vmfs backed by E1
- vm6_local_E2 & vm66_local_E2 created on local vmfs backed by E2
  • Scenario 1 (with fresh install + _DEFAULT tenant behavior)

Steps:

1. create one volume from each vms and make sure each has the accessibility to volumes created on the datastore backed by ESX where vm is resided (means no visibility to volumes created on local datastore backed by E2)

e.g. "docker volume ls" on vm1_shared_E1 should has access to volumes created by following vms

- vm1_shared_E1 
- vm11_shared_E1
- vm2_shared_E2  
- vm22_shared_E2
- vm3_vsan_E1 
- vm33_vsan_E1
- vm4_vsan_E2 
- vm44_vsan_E2
- vm5_local_E1 
- vm55_local_E1 

2. verifies attach/detach from above vms one by one (make sure vms can manipulate data)
3. verifies vm6_local_E2 is not having any access volumes created by vm5_local_E1 & vm55_local_E1
4. Perform volume delete from some vms which has access to created volumes

Note: Scenario 2 is not valid anymore by taking account of the current level of code. After introducing cross-esx multi tenancy, there are already testcase planned to exercise respective workflow for singlemode or multi-mode

- [x] Scenario 2 (user created tenant to check tenancy is per ESX)
Steps:

1. create tenant T1 from E1 & E2
2. add vm33_vsan_E1 to T1 on ESX E1
3. add vm44_vsan_E2 to T1 on ESX E2
4. invoke "docker volume ls" from vm33_vsan_E1 & vm44_vsan_E2 makes sure no volumes are visible after moving to T1
5. make sure volumes created by above both VMs are still visible from other docker_hosts which are part of _DEFAULT tenant
6. create volume from above vms with short names and validates tenancy access is per ESX and above vms can't see each others Volumes
7. on E1: set default datastore for T1 as vsan datastore
8. on E2: set default datastore for T1 as shared vmfs
9.  create volume from above vms with short names and validates tenancy access is per ESX and above vms can't see each others Volumes
10. make sure other vms, are part of _DEFAULT, will not have access to such volumes and vice versa
11. deletes created volumes

  • Scenario 3 (vsan policy)
    Steps:

1. create a valid vsan policy from E1 and E2
2. create two volumes from vm3_vsan_E1 one with policy created on E1 and another with E2
3. #2 for vm4_vsan_E2
4. perform attach/detach for all 4 volumes from both vms and perform data manipulation as well
5. on vm3_vsan_E1: remove volumes created by vm4_vsan_E2
6. on vm4_vsan_E2: remove volumes created by vm3_vsan_E1

  • Scenario 4 (fail-over for docker host created on vsan)
    Steps:

1. create a valid vsan policy from E1 and E2
2. create two volumes from vm3_vsan_E1 one with policy created on E1 and another with E2
3. #2 for vm4_vsan_E2
4. perform attach/detach for all 4 volumes from both vms and perform data manipulation as well
5. on vm3_vsan_E1: attach volume created by vm3_vsan_E1
6. on E1: kill the vm vm3_vsan_E1 to test fail over
7. on E2: try to attach the same volume which was in attached state on vm3_vsan_E1
8. make sure attach operation succeeds
9. perform data manipulation
10. perform detach and remove the volume

  • Scenario 5 ( fail-over for docker host created on shared vmfs)
    Steps: Perform above steps for vm1_shared_E1 & vm2_shared_E2

  • Scenario 6 (upgrade testing from vspherDockerVolume 0.11 or earlier => 0.11.2 or latest)
    Steps:

Perform steps mentioned in scenario 1 on the system which is having vmdk plugin & driver 0.11 or earlier version
update vmdk plugin & driver (Note: plugin is not mandatory)
follow the upgrade steps
verify the pre-upgrade setup is intact and perform attach/detach/creation/deletion.

Additional Tests:

Steps for issue 987:
1. Create a volume.
2. Run a container and mount the volume.
3. While the container is running, reboot esx.
4. After esx comes back up, power-on the vm.
5. After the vm comes back up, do 'docker volume ls' to see the volumes are same as they were prior to esx reboot
@shuklanirdesh82 shuklanirdesh82 added this to the 0.12 milestone Feb 23, 2017
@shuklanirdesh82 shuklanirdesh82 self-assigned this Feb 23, 2017
@shuklanirdesh82
Copy link
Contributor Author

shuklanirdesh82 commented Feb 24, 2017

Sanity testing (_DEFAULT tenant)

  • Testbed setup 1
- 2 ESXs E1 & E2 having shared datastore (shared vmfs or vsan pick one)
- VM1 on E1; VM2 on E2
  • Testbed setup 2
- 2 ESXs E1 & E2 having **2** shared datastore (shared vmfs or vsan pick one)
- VM1 on E1<->SDS1; VM2 on E2<->SDS2

Note:

  • Need to perform following 3 basic scenarios on above testbed flavors.

  • Need to perform cross verification with Admin CLI outcome

  • Fail over scenario

on VM1: create volume
on VM1: list volume 
on VM2: list volume
on VM1: start container -> write some data
on VM1: kill VM1
on VM2: start container
on VM2: verify data
  • Basic create delete scenario
VM1: create volume > list volume
VM2: list volume > delete volume
  • Basic create/attach volume scenario
VM1: create volume vol1> list volume
VM2: create volume vol2> list volume
VM1: start container by mounting to vol2 > write some data > stop container
VM2: start container by mounting to vol1 > write some data > stop container
VM1: start container by mounting to vol1 > verify data > stop container
VM2: start container by mounting to vol2 > verify data > stop container
VM1/VM2: perform cleanup (rm container/volume)

Tracked at #991

  • Basic create/attach volume scenario
VM1: create volume vol1> list volume > docker volume inspect vol1
VM2: list volume > docker volume inspect vol1
VM1: start container by mounting to vol1 > write some data > leave it attached/running
VM2: docker volume inspect vol1 (it should be in attached state)
ESX: make sure vmdkops_admin output is in sync with volume inspect
VM2: start container by mounting to vol1 > expect an error (docker: Error response from daemon: VolumeDriver.Mount: Failed to add disk 'scsi1:0'. disk /vmfs/volumes/sharedVmfs-0/dockvols/11111111-1111-1111-1111-111111111111/abc.vmdk already attached to VM=422f7a41-10e5-0937-30ad-bd9bffa54565.)
VM2: docker volume inspect vol1 (make sure output stays the same as before)
ESX: make sure vmdkops_admin output is in sync with volume inspect
VM1: exit/detach container > docker volume inspect vol1
ESX: make sure vmdkops_admin output is in sync with volume inspect
VM2: docker volume inspect vol1
VM1/VM2: perform cleanup (rm container/volume)

//CC @pdhamdhere @tusharnt

@shuklanirdesh82
Copy link
Contributor Author

@ashahi1 I'm assigning this to you.

Please get started with sanity testing then complex scenario testing.

@ashahi1
Copy link
Contributor

ashahi1 commented Feb 28, 2017

Testbed setup:

2 ESX hosts (E1 & E2)
E1 is having access to local-vmfs, vsan, shared-vmfs ( the same for E2)

2 VMs per each datastore from ESX (e.g. 2 vms on local/vsan/shared-vmfs backed by E1 i.e. total 6 vms)

Scenario 1:

  1. create one volume from each vms
root@photon-xVmYMbyTn [ ~ ]# docker volume create --driver=vmdk --name=Vol6Esx1Vsan2 -o size=200mb
Vol6Esx1Vsan2
root@photon-xVmYMbyTn [ ~ ]# docker volume ls
DRIVER              VOLUME NAME
vmdk                Vol1Esx1Local1@local.1-0
vmdk                Vol2Esx1Local2@local.1-0
vmdk                Vol3Esx1Shared1@sharedVmfs-0
vmdk                Vol4Esx1Shared2@sharedVmfs-0
vmdk                Vol5Esx1Vsan11@vsanDatastore
vmdk                Vol6Esx1Vsan22@vsanDatastore
root@photon-xVmYMbyTn [ ~ ]#

Admin cli command to list volumes from ESX1:

[root@sc2-rdops-vm03-dhcp-107-129:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py ls
Volume           Datastore      Created By VM    Created                   Attached To VM (name/..  Policy          Capacity  Used  Disk Format  Filesystem Type  Access      Attach As
---------------  -------------  ---------------  ------------------------  -----------------------  --------------  --------  ----  -----------  ---------------  ----------  ----------------------
Vol3Esx1Shared1  sharedVmfs-0   VM1-129-Shared1  Mon Feb 27 06:31:22 2017  detached                 N/A             200MB     14MB  thin         ext4             read-write  independent_persistent
Vol4Esx1Shared2  sharedVmfs-0   VM1-129-Shared2  Mon Feb 27 06:32:31 2017  detached                 N/A             200MB     14MB  thin         ext4             read-write  independent_persistent
Vol1Esx1Local1   local.1-0      VM1-129-Local1   Mon Feb 27 06:20:54 2017  detached                 N/A             200MB     19MB  thin         ext4             read-write  independent_persistent
Vol2Esx1Local2   local.1-0      VM1-129-Local2   Mon Feb 27 06:22:27 2017  detached                 N/A             200MB     19MB  thin         ext4             read-write  independent_persistent
Vol5Esx1Vsan11   vsanDatastore  VM1-129-Vsan1    Mon Feb 27 06:57:34 2017  detached                 [VSAN default]  200MB     96MB  thin         ext4             read-write  independent_persistent
Vol6Esx1Vsan22   vsanDatastore  VM1-129-Vsan2    Mon Feb 27 06:58:18 2017  detached                 [VSAN default]  200MB     96MB  thin         ext4             read-write  independent_persistent

[root@sc2-rdops-vm03-dhcp-107-129:~]

Mounted all the volumes(vol1-to-vol6) from vm1:

root@photon-xVmYMbyTn [ ~ ]# docker run -it --volume-driver=vmdk -v Vol1Esx1Local1@local.1-0:/vol1 -v Vol2Esx1Local2@local.1-0:/vol2 -v Vol3Esx1Shared1@sharedVmfs-0:/vol3 -v 

Vol4Esx1Shared2@sharedVmfs-0:/vol4 -v Vol5Esx1Vsan11@vsanDatastore:/vol5 -v Vol6Esx1Vsan22@vsanDatastore:/vol6 --name ub ubuntu
root@c82184ad5d64:/# ls
bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var  vol1  vol2  vol3  vol4  vol5  vol6
root@c82184ad5d64:/# cat vol1/hi
I'm Vol1 from vm1
root@c82184ad5d64:/# cat vol2/hi
I'm Vol2 from vm1
root@c82184ad5d64:/# cat vol3/hi
cat: vol3/hi: No such file or directory
root@c82184ad5d64:/# echo "I'm Vol3 from vm1" > vol3/hi
root@c82184ad5d64:/# cat vol3/hi
I'm Vol3 from vm1
root@c82184ad5d64:/# cat vol4/hi
cat: vol4/hi: No such file or directory
root@c82184ad5d64:/# echo "I'm Vol4 from vm1" > vol4/hi
root@c82184ad5d64:/# cat vol4/hi
I'm Vol4 from vm1
root@c82184ad5d64:/# cat vol5/hi
cat: vol5/hi: No such file or directory
root@c82184ad5d64:/# echo "I'm Vol5 from vm1" > vol5/hi
root@c82184ad5d64:/# cat vol5/hi
I'm Vol5 from vm1
root@c82184ad5d64:/# echo "I'm Vol6 from vm1" > vol6/hi
root@c82184ad5d64:/# cat vol6/hi
I'm Vol6 from vm1
root@c82184ad5d64:/# exit
exit
root@photon-xVmYMbyTn [ ~ ]# docker stop ub
ub
root@photon-xVmYMbyTn [ ~ ]# docker rm ub
ub
root@photon-xVmYMbyTn [ ~ ]#

Stopped the contianer and mounted all the volumes on second container started on another vm

root@photon-xVmYMbyTn [ ~ ]# docker run -it --volume-driver=vmdk -v Vol1Esx1Local1@local.1-0:/vol1 -v Vol2Esx1Local2@local.1-0:/vol2 -v Vol3Esx1Shared1@sharedVmfs-0:/vol3 -v 

Vol4Esx1Shared2@sharedVmfs-0:/vol4 -v Vol5Esx1Vsan11@vsanDatastore:/vol5 -v Vol6Esx1Vsan22@vsanDatastore:/vol6 --name ub ubuntu
Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu

8aec416115fd: Pull complete
695f074e24e3: Pull complete
946d6c48c2a7: Pull complete
bc7277e579f0: Pull complete
2508cbcde94b: Pull complete
Digest: sha256:71cd81252a3563a03ad8daee81047b62ab5d892ebbfbf71cf53415f29c130950
Status: Downloaded newer image for ubuntu:latest
root@9826f2448424:/# ls
bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var  vol1  vol2  vol3  vol4  vol5  vol6
root@9826f2448424:/# cat vol1/hi
I'm Vol1 from vm1
root@9826f2448424:/# cat vol2/hi
I'm Vol2 from vm1
root@9826f2448424:/# cat vol3/hi
I'm Vol3 from vm1
root@9826f2448424:/# cat vol4/hi
I'm Vol4 from vm1
root@9826f2448424:/# cat vol5/hi
I'm Vol5 from vm1
root@9826f2448424:/# cat vol6/hi
I'm Vol6 from vm1
root@9826f2448424:/#

From ESX1:

[root@sc2-rdops-vm03-dhcp-107-129:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py ls
Volume           Datastore      Created By VM    Created                   Attached To VM (name/..  Policy          Capacity  Used  Disk Format  Filesystem Type  Access      Attach As
---------------  -------------  ---------------  ------------------------  -----------------------  --------------  --------  ----  -----------  ---------------  ----------  ----------------------
Vol3Esx1Shared1  sharedVmfs-0   VM1-129-Shared1  Mon Feb 27 06:31:22 2017  VM1-129-Vsan2            N/A             200MB     19MB  thin         ext4             read-write  independent_persistent
Vol4Esx1Shared2  sharedVmfs-0   VM1-129-Shared2  Mon Feb 27 06:32:31 2017  VM1-129-Vsan2            N/A             200MB     19MB  thin         ext4             read-write  independent_persistent
Vol1Esx1Local1   local.1-0      VM1-129-Local1   Mon Feb 27 06:20:54 2017  VM1-129-Vsan2            N/A             200MB     19MB  thin         ext4             read-write  independent_persistent
Vol2Esx1Local2   local.1-0      VM1-129-Local2   Mon Feb 27 06:22:27 2017  VM1-129-Vsan2            N/A             200MB     19MB  thin         ext4             read-write  independent_persistent
Vol5Esx1Vsan11   vsanDatastore  VM1-129-Vsan1    Mon Feb 27 06:57:34 2017  VM1-129-Vsan2            [VSAN default]  200MB     96MB  thin         ext4             read-write  independent_persistent
Vol6Esx1Vsan22   vsanDatastore  VM1-129-Vsan2    Mon Feb 27 06:58:18 2017  VM1-129-Vsan2            [VSAN default]  200MB     96MB  thin         ext4             read-write  independent_persistent

[root@sc2-rdops-vm03-dhcp-107-129:~]

Docker volume ls from vm on ESX-2:

root@photon-xVmYMbyTn [ ~ ]# docker volume ls
DRIVER              VOLUME NAME
vmdk                Vol1Esx2Local1@local.0-0
vmdk                Vol2Esx2Local2@local.0-0
vmdk                Vol3Esx1Shared1@sharedVmfs-0
vmdk                Vol3Esx2Shared1@sharedVmfs-0
vmdk                Vol4Esx1Shared2@sharedVmfs-0
vmdk                Vol4Esx2Shared2@sharedVmfs-0
vmdk                Vol5Esx1Vsan11@vsanDatastore
vmdk                Vol5Esx2Vsan11@vsanDatastore
vmdk                Vol6Esx1Vsan22@vsanDatastore
vmdk                Vol6Esx2Vsan22@vsanDatastore
root@photon-xVmYMbyTn [ ~ ]#

From Admin cli from ESX-2:

[root@sc2-rdops-vm03-dhcp-117-65:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py ls
Volume           Datastore      Created By VM    Created                  Attached To VM (name/..  Policy          Capacity  Used   Disk Format  Filesystem Type  Access      Attach As
---------------  -------------  ---------------  -----------------------  -----------------------  --------------  --------  -----  -----------  ---------------  ----------  ----------------------
Vol3Esx1Shared1  sharedVmfs-0   VM1-129-Shared1  Mon Feb 27 06:31:22 2..  detached                 N/A             200MB     19MB   thin         ext4             read-write  independent_persistent
Vol4Esx1Shared2  sharedVmfs-0   VM1-129-Shared2  Mon Feb 27 06:32:31 2..  detached                 N/A             200MB     19MB   thin         ext4             read-write  independent_persistent
Vol3Esx2Shared1  sharedVmfs-0   VM2-65-Shared1   Mon Feb 27 07:47:43 2..  detached                 N/A             400MB     23MB   thin         ext4             read-write  independent_persistent
Vol4Esx2Shared2  sharedVmfs-0   VM2-65-Shared2   Mon Feb 27 07:49:20 2..  detached                 N/A             400MB     23MB   thin         ext4             read-write  independent_persistent
Vol1Esx2Local1   local.0-0      VM2-65-Local2    Mon Feb 27 07:38:55 2..  detached                 N/A             400MB     23MB   thin         ext4             read-write  independent_persistent
Vol2Esx2Local2   local.0-0      VM2-65-Local1    Mon Feb 27 07:40:04 2..  detached                 N/A             400MB     23MB   thin         ext4             read-write  independent_persistent
Vol5Esx1Vsan11   vsanDatastore  VM1-129-Vsan1    Mon Feb 27 06:57:34 2..  detached                 [VSAN default]  200MB     96MB   thin         ext4             read-write  independent_persistent
Vol6Esx1Vsan22   vsanDatastore  VM1-129-Vsan2    Mon Feb 27 06:58:18 2..  detached                 [VSAN default]  200MB     96MB   thin         ext4             read-write  independent_persistent
Vol5Esx2Vsan11   vsanDatastore  VM2-65-Vsan1     Mon Feb 27 08:03:04 2..  detached                 [VSAN default]  400MB     144MB  thin         ext4             read-write  independent_persistent
Vol6Esx2Vsan22   vsanDatastore  VM2-65-Vsan2     Mon Feb 27 08:04:31 2..  detached                 [VSAN default]  400MB     144MB  thin         ext4             read-write  independent_persistent

[root@sc2-rdops-vm03-dhcp-117-65:~]

Docker volume ls from vm on ESX-1:

root@photon-xVmYMbyTn [ ~ ]# docker volume ls
DRIVER              VOLUME NAME
vmdk                Vol1Esx1Local1@local.1-0
vmdk                Vol2Esx1Local2@local.1-0
vmdk                Vol3Esx1Shared1@sharedVmfs-0
vmdk                Vol3Esx2Shared1@sharedVmfs-0
vmdk                Vol4Esx1Shared2@sharedVmfs-0
vmdk                Vol4Esx2Shared2@sharedVmfs-0
vmdk                Vol5Esx1Vsan11@vsanDatastore
vmdk                Vol5Esx2Vsan11@vsanDatastore
vmdk                Vol6Esx1Vsan22@vsanDatastore
vmdk                Vol6Esx2Vsan22@vsanDatastore
root@photon-xVmYMbyTn [ ~ ]#

From ESX-1:

[root@sc2-rdops-vm03-dhcp-107-129:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py ls
Volume           Datastore      Created By VM    Created                  Attached To VM (name/..  Policy          Capacity  Used   Disk Format  Filesystem Type  Access      Attach As
---------------  -------------  ---------------  -----------------------  -----------------------  --------------  --------  -----  -----------  ---------------  ----------  ----------------------
Vol3Esx1Shared1  sharedVmfs-0   VM1-129-Shared1  Mon Feb 27 06:31:22 2..  detached                 N/A             200MB     19MB   thin         ext4             read-write  independent_persistent
Vol4Esx1Shared2  sharedVmfs-0   VM1-129-Shared2  Mon Feb 27 06:32:31 2..  detached                 N/A             200MB     19MB   thin         ext4             read-write  independent_persistent
Vol3Esx2Shared1  sharedVmfs-0   VM2-65-Shared1   Mon Feb 27 07:47:43 2..  detached                 N/A             400MB     23MB   thin         ext4             read-write  independent_persistent
Vol4Esx2Shared2  sharedVmfs-0   VM2-65-Shared2   Mon Feb 27 07:49:20 2..  detached                 N/A             400MB     23MB   thin         ext4             read-write  independent_persistent
Vol1Esx1Local1   local.1-0      VM1-129-Local1   Mon Feb 27 06:20:54 2..  detached                 N/A             200MB     19MB   thin         ext4             read-write  independent_persistent
Vol2Esx1Local2   local.1-0      VM1-129-Local2   Mon Feb 27 06:22:27 2..  detached                 N/A             200MB     19MB   thin         ext4             read-write  independent_persistent
Vol5Esx1Vsan11   vsanDatastore  VM1-129-Vsan1    Mon Feb 27 06:57:34 2..  detached                 [VSAN default]  200MB     96MB   thin         ext4             read-write  independent_persistent
Vol6Esx1Vsan22   vsanDatastore  VM1-129-Vsan2    Mon Feb 27 06:58:18 2..  detached                 [VSAN default]  200MB     96MB   thin         ext4             read-write  independent_persistent
Vol5Esx2Vsan11   vsanDatastore  VM2-65-Vsan1     Mon Feb 27 08:03:04 2..  detached                 [VSAN default]  400MB     144MB  thin         ext4             read-write  independent_persistent
Vol6Esx2Vsan22   vsanDatastore  VM2-65-Vsan2     Mon Feb 27 08:04:31 2..  detached                 [VSAN default]  400MB     144MB  thin         ext4             read-write  independent_persistent

[root@sc2-rdops-vm03-dhcp-107-129:~]

Trying to mount local container from ESX-1:

root@photon-xVmYMbyTn [ ~ ]# docker run -it --volume-driver=vmdk -v Vol1Esx1Local1@local.1-0:/vol1 --name ubVM2 ubuntu
Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu
8aec416115fd: Pull complete
695f074e24e3: Pull complete
946d6c48c2a7: Pull complete
bc7277e579f0: Pull complete
2508cbcde94b: Pull complete
Digest: sha256:71cd81252a3563a03ad8daee81047b62ab5d892ebbfbf71cf53415f29c130950
Status: Downloaded newer image for ubuntu:latest
docker: Error response from daemon: create Vol1Esx1Local1@local.1-0: VolumeDriver.Create: Invalid datastore 'local.1-0'.
Known datastores: sharedVmfs-0, sharedVmfs-1, local.0-0, local.0-1, vsanDatastore.
Default datastore: vsanDatastore.
See 'docker run --help'.

Mount vol5 from esx1 and vol5 from ESX2:

root@photon-xVmYMbyTn [ ~ ]# docker run -it --volume-driver=vmdk -v Vol5Esx1Vsan11@vsanDatastore:/vol5 -v Vol5Esx2Vsan11@vsanDatastore:/vol6 --name ubvm2 ubuntu
root@797c519983f8:/# ls
bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var  vol5  vol6
root@797c519983f8:/# cd vol5
root@797c519983f8:/vol5# ls
hi  lost+found
root@797c519983f8:/vol5# cat hi
I'm Vol5 from vm1
root@797c519983f8:/vol5# cd ..
root@797c519983f8:/# echo "Now from ESX2" > vol5/hi
root@797c519983f8:/# cat vol5/hi
Now from ESX2
root@797c519983f8:/# echo "Now from vol6 on ESX2" > vol6/hi
root@797c519983f8:/# exit
exit
root@photon-xVmYMbyTn [ ~ ]#

Removed all the volumes:

root@photon-xVmYMbyTn [ ~ ]# docker volume ls
DRIVER              VOLUME NAME
root@photon-xVmYMbyTn [ ~ ]#

@ashahi1
Copy link
Contributor

ashahi1 commented Feb 28, 2017

Scenario 2:

create tenant from E1 & E2:

[root@sc2-rdops-vm03-dhcp-107-129:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py tenant ls
Uuid                                  Name      Description               Default_datastore  VM_list
------------------------------------  --------  ------------------------  -----------------  -------
11111111-1111-1111-1111-111111111111  _DEFAULT  This is a default tenant
7daf7956-990f-4619-a8e8-f2fb5248eb72  tenant1

[root@sc2-rdops-vm03-dhcp-107-129:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py tenant vm add --name=tenant1 --vm-list=VM1-129-Vsan1
tenant vm add succeeded
[root@sc2-rdops-vm03-dhcp-107-129:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py tenant ls
Uuid                                  Name      Description               Default_datastore  VM_list
------------------------------------  --------  ------------------------  -----------------  -------------
11111111-1111-1111-1111-111111111111  _DEFAULT  This is a default tenant
7daf7956-990f-4619-a8e8-f2fb5248eb72  tenant1                                                VM1-129-Vsan1

[root@sc2-rdops-vm03-dhcp-107-129:~]

Creating tenant from ESX-2

[root@sc2-rdops-vm03-dhcp-117-65:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py tenant create --name=tenant1
tenant create succeeded
[root@sc2-rdops-vm03-dhcp-117-65:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py tenant ls
Uuid                                  Name      Description               Default_datastore  VM_list
------------------------------------  --------  ------------------------  -----------------  -------
11111111-1111-1111-1111-111111111111  _DEFAULT  This is a default tenant
c6df3559-eda2-4155-a0b8-15b6a688978c  tenant1

[root@sc2-rdops-vm03-dhcp-117-65:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py tenant vm add --name=tenant1 --vm-list=VM2-65-Vsan1
tenant vm add succeeded
[root@sc2-rdops-vm03-dhcp-117-65:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py tenant ls
Uuid                                  Name      Description               Default_datastore  VM_list
------------------------------------  --------  ------------------------  -----------------  ------------
11111111-1111-1111-1111-111111111111  _DEFAULT  This is a default tenant
c6df3559-eda2-4155-a0b8-15b6a688978c  tenant1                                                VM2-65-Vsan1

[root@sc2-rdops-vm03-dhcp-117-65:~]

Created volumes from other vms and verified that volumes are no longer visible from vms in tenant:

root@photon-xVmYMbyTn [ ~ ]# docker volume create --driver=vmdk --name=vol1Esx1Vsan1 -o size=200mb
vol1Esx1Vsan1
root@photon-xVmYMbyTn [ ~ ]# docker volume ls
DRIVER              VOLUME NAME
vmdk                vol1Esx1Vsan1@vsanDatastore
root@photon-xVmYMbyTn [ ~ ]# docker volume inspect vol1Esx1Vsan1
[
    {
        "Name": "vol1Esx1Vsan1",
        "Driver": "vmdk",
        "Mountpoint": "/mnt/vmdk/vol1Esx1Vsan1",
        "Status": {
            "access": "read-write",
            "attach-as": "independent_persistent",
            "capacity": {
                "allocated": "96MB",
                "size": "200MB"
            },
            "clone-from": "None",
            "created": "Mon Feb 27 09:15:01 2017",
            "created by VM": "VM1-129-Vsan1",
            "datastore": "vsanDatastore",
            "diskformat": "thin",
            "fstype": "ext4",
            "status": "detached"
        },
        "Labels": {},
        "Scope": "global"
    }
]

Docker volume ls from tenant vm:

root@photon-xVmYMbyTn [ ~ ]# docker volume ls
DRIVER              VOLUME NAME
root@photon-xVmYMbyTn [ ~ ]#

Newly created volumes are visble from other VMs:

root@photon-xVmYMbyTn [ ~ ]# docker volume ls
DRIVER              VOLUME NAME
vmdk                vol1Esx1Vsan1@vsanDatastore
vmdk                vol2Esx2Vsan2@vsanDatastore
root@photon-xVmYMbyTn [ ~ ]#

root@photon-xVmYMbyTn [ ~ ]# docker volume ls
DRIVER              VOLUME NAME
vmdk                vol1Esx1Vsan1@vsanDatastore
vmdk                vol2Esx2Vsan2@vsanDatastore
root@photon-xVmYMbyTn [ ~ ]#

Created volumes after vms joined tenants:

root@photon-xVmYMbyTn [ ~ ]# docker volume ls
DRIVER              VOLUME NAME
root@photon-xVmYMbyTn [ ~ ]# docker volume create --driver=vmdk --name=vol2Esx2Ten2 -o size=200mb
vol2Esx2Ten2
root@photon-xVmYMbyTn [ ~ ]# docker volume ls
DRIVER              VOLUME NAME
vmdk                vol2Esx2Ten2@vsanDatastore
root@photon-xVmYMbyTn [ ~ ]#


root@photon-xVmYMbyTn [ ~ ]# docker volume ls
DRIVER              VOLUME NAME
root@photon-xVmYMbyTn [ ~ ]# docker volume create --driver=vmdk --name=vol1Esx1Ten1 -o size=200mb
vol1Esx1Ten1
root@photon-xVmYMbyTn [ ~ ]# docker volume ls
DRIVER              VOLUME NAME
vmdk                vol1Esx1Ten1@vsanDatastore
root@photon-xVmYMbyTn [ ~ ]#

Set default datastore for tenant1:

[root@sc2-rdops-vm03-dhcp-107-129:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py tenant access add --name=tenant1 --datastore=vsanDatastore  --allow-create --default-datastore
tenant access add succeeded
[root@sc2-rdops-vm03-dhcp-107-129:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py tenant ls
Uuid                                  Name      Description               Default_datastore  VM_list
------------------------------------  --------  ------------------------  -----------------  -------------
11111111-1111-1111-1111-111111111111  _DEFAULT  This is a default tenant
7daf7956-990f-4619-a8e8-f2fb5248eb72  tenant1                             vsanDatastore      VM1-129-Vsan1

[root@sc2-rdops-vm03-dhcp-107-129:~]

[root@sc2-rdops-vm03-dhcp-117-65:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py tenant access add --name=tenant1 --datastore=sharedVmfs-0  --allow-create --default-datastore
tenant access add succeeded
[root@sc2-rdops-vm03-dhcp-117-65:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py tenant ls
Uuid                                  Name      Description               Default_datastore  VM_list
------------------------------------  --------  ------------------------  -----------------  ------------
11111111-1111-1111-1111-111111111111  _DEFAULT  This is a default tenant
c6df3559-eda2-4155-a0b8-15b6a688978c  tenant1                             sharedVmfs-0       VM2-65-Vsan1

[root@sc2-rdops-vm03-dhcp-117-65:~]

After setting default datastore:

root@photon-xVmYMbyTn [ ~ ]# docker volume create --driver=vmdk --name=vol11Esx1Ten1 -o size=200mb
vol11Esx1Ten1
root@photon-xVmYMbyTn [ ~ ]# docker volume ls
DRIVER              VOLUME NAME
vmdk                vol11Esx1Ten1@vsanDatastore
vmdk                vol1Esx1Ten1@vsanDatastore
root@photon-xVmYMbyTn [ ~ ]#


root@photon-xVmYMbyTn [ ~ ]# docker volume create --driver=vmdk --name=vol22Esx2Ten2 -o size=200mb
vol22Esx2Ten2
root@photon-xVmYMbyTn [ ~ ]# docker volume ls
DRIVER              VOLUME NAME
vmdk                vol22Esx2Ten2@sharedVmfs-0
vmdk                vol2Esx2Ten2@vsanDatastore
root@photon-xVmYMbyTn [ ~ ]#

Successfully removed the volumes:

root@photon-xVmYMbyTn [ ~ ]# docker volume rm vol11Esx1Ten1@vsanDatastore vol1Esx1Ten1@vsanDatastore
vol11Esx1Ten1@vsanDatastore
vol1Esx1Ten1@vsanDatastore
root@photon-xVmYMbyTn [ ~ ]#

root@photon-xVmYMbyTn [ ~ ]# docker volume rm vol22Esx2Ten2@sharedVmfs-0 vol2Esx2Ten2@vsanDatastore
vol22Esx2Ten2@sharedVmfs-0
vol2Esx2Ten2@vsanDatastore
root@photon-xVmYMbyTn [ ~ ]#

@ashahi1
Copy link
Contributor

ashahi1 commented Feb 28, 2017

Scenario 3 (vsan policy)

  1. create a valid vsan policy from E1 and E2
[root@sc-rdops-vm02-dhcp-48-54:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py policy create --name some-policy-E2 --content '(("proportionalCapacity" i50)''("hostFailuresToTolerate" i0))'
Successfully created policy: some-policy-E2
[root@sc-rdops-vm02-dhcp-48-54:~]

[root@sc-rdops-vm02-dhcp-32-249:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py policy create --name some-policy-E1 --content '(("proportionalCapacity" i50)''("hostFailuresToTolerate" i0))'
Successfully created policy: some-policy-E1
[root@sc-rdops-vm02-dhcp-32-249:~]

  1. create two volumes from vm on vsan on one with policy created on E1 and another with E2
root@photon-xVmYMbyTn [ ~ ]# docker volume create --driver=vmdk --name=Vol1VsanE1 -o size=500mb -o vsan-policy-name=some-policy-E1
Vol1VsanE1
root@photon-xVmYMbyTn [ ~ ]# docker volume create --driver=vmdk --name=Vol2VsanE1 -o size=500mb -o vsan-policy-name=some-policy-E2
Vol2VsanE1
root@photon-xVmYMbyTn [ ~ ]#

[root@sc-rdops-vm02-dhcp-32-249:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py policy ls
Policy Name     Policy Content                                               Active
--------------  -----------------------------------------------------------  -------------------
some-policy-E1  (("proportionalCapacity" i50)("hostFailuresToTolerate" i0))  In use by 1 volumes
some-policy-E2  (("proportionalCapacity" i50)("hostFailuresToTolerate" i0))  In use by 1 volumes

[root@sc-rdops-vm02-dhcp-32-249:~]


[root@sc-rdops-vm02-dhcp-32-249:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py ls
Volume      Datastore      Created By VM  Created                   Attached To VM (name/uuid)      Policy          Capacity  Used   Disk Format  Filesystem Type  Access      Attach As             
----------  -------------  -------------  ------------------------  ------------------------------  --------------  --------  -----  -----------  ---------------  ----------  ----------------------
Vol1VsanE1  vsanDatastore  VM1-249-Vsan1  Mon Feb 27 23:17:46 2017  detached                        some-policy-E1  500MB     256MB  thin         ext4             read-write  independent_persistent
Vol2VsanE1  vsanDatastore  VM1-249-Vsan1  Mon Feb 27 23:17:58 2017  detached                        some-policy-E2  500MB     256MB  thin         ext4             read-write  independent_persistent

[root@sc-rdops-vm02-dhcp-32-249:~]
  1. create two volumes from vm on vsan on one with policy created on E1 and another with E2
root@photon-xVmYMbyTn [ ~ ]# docker volume create --driver=vmdk --name=Vol1VsanE2 -o size=500mb -o vsan-policy-name=some-policy-E2
Vol1VsanE2
root@photon-xVmYMbyTn [ ~ ]# docker volume create --driver=vmdk --name=Vol2VsanE2 -o size=500mb -o vsan-policy-name=some-policy-E2
Vol2VsanE2
root@photon-xVmYMbyTn [ ~ ]#
[root@sc-rdops-vm02-dhcp-48-54:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py ls
Volume      Datastore      Created By VM  Created                   Attached To VM (name/uuid)  Policy          Capacity  Used   Disk Format  Filesystem Type  Access      Attach As
----------  -------------  -------------  ------------------------  --------------------------  --------------  --------  -----  -----------  ---------------  ----------  ----------------------
Vol1VsanE1  vsanDatastore  VM1-249-Vsan1  Mon Feb 27 23:17:46 2017  detached                    some-policy-E1  500MB     256MB  thin         ext4             read-write  independent_persistent
Vol2VsanE1  vsanDatastore  VM1-249-Vsan1  Mon Feb 27 23:17:58 2017  detached                    some-policy-E2  500MB     256MB  thin         ext4             read-write  independent_persistent
Vol1VsanE2  vsanDatastore  VM2-54-Vsan1   Mon Feb 27 23:21:07 2017  detached                    some-policy-E2  500MB     256MB  thin         ext4             read-write  independent_persistent
Vol2VsanE2  vsanDatastore  VM2-54-Vsan1   Mon Feb 27 23:21:20 2017  detached                    some-policy-E2  500MB     256MB  thin         ext4             read-write  independent_persistent

[root@sc-rdops-vm02-dhcp-48-54:~]
  1. perform attach/detach for all 4 volumes from both vms and perform data manipulation as well
root@photon-xVmYMbyTn [ ~ ]# docker run -it --volume-driver=vmdk -v Vol1VsanE1@vsanDatastore:/vol1 -v Vol1VsanE2@vsanDatastore:/vol2 -v Vol2VsanE1@vsanDatastore:/vol3 -v Vol2VsanE2@vsanDatastore:/vol4 --name ubE1 ubuntu
Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu

d54efb8db41d: Pull complete
f8b845f45a87: Pull complete
e8db7bf7c39f: Pull complete
9654c40e9079: Pull complete
6d9ef359eaaa: Pull complete
Digest: sha256:dd7808d8792c9841d0b460122f1acf0a2dd1f56404f8d1e56298048885e45535
Status: Downloaded newer image for ubuntu:latest
root@a2a09d754abc:/# ls
bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var  vol1  vol2  vol3  vol4
root@a2a09d754abc:/# echo " vol1 from vm1" > vol1/hellos
root@a2a09d754abc:/# echo " vol2 from vm1" > vol2/hellos
root@a2a09d754abc:/# exit
exit
root@photon-xVmYMbyTn [ ~ ]#
root@photon-xVmYMbyTn [ ~ ]# docker run -it --volume-driver=vmdk -v Vol1VsanE1@vsanDatastore:/vol1 -v Vol1VsanE2@vsanDatastore:/vol2 -v Vol2VsanE1@vsanDatastore:/vol3 -v Vol2VsanE2@vsanDatastore:/vol4 --name ubE2 ubuntu
root@9bfc575610dd:/# ls
bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var  vol1  vol2  vol3  vol4
root@9bfc575610dd:/# cat vol1/hellos
 vol1 from vm1
root@9bfc575610dd:/# cat vol2/hellos
 vol2 from vm1
root@9bfc575610dd:/# echo " vol3 from vm2" > vol3/hellos
root@9bfc575610dd:/# echo " vol4 from vm2" > vol4/hellos
root@9bfc575610dd:/# exit
exit
root@photon-xVmYMbyTn [ ~ ]#
  1. on vm3_vsan_E1: remove volumes created by vm4_vsan_E2
root@photon-xVmYMbyTn [ ~ ]# docker stop $(docker ps -a -q)
a2a09d754abc
root@photon-xVmYMbyTn [ ~ ]# docker rm $(docker ps -a -q)
a2a09d754abc
root@photon-xVmYMbyTn [ ~ ]# docker volume rm Vol1VsanE2@vsanDatastore Vol2VsanE2@vsanDatastore
Vol1VsanE2@vsanDatastore
Vol2VsanE2@vsanDatastore
root@photon-xVmYMbyTn [ ~ ]#
  1. On vm4_vsan_E2: remove volumes created by vm3_vsan_E1
root@photon-xVmYMbyTn [ ~ ]# docker stop $(docker ps -a -q)
9bfc575610dd
root@photon-xVmYMbyTn [ ~ ]# docker rm $(docker ps -a -q)
9bfc575610dd
root@photon-xVmYMbyTn [ ~ ]# docker volume rm Vol1VsanE1@vsanDatastore Vol2VsanE1@vsanDatastore
Vol1VsanE1@vsanDatastore
Vol2VsanE1@vsanDatastore
root@photon-xVmYMbyTn [ ~ ]#

@ashahi1
Copy link
Contributor

ashahi1 commented Feb 28, 2017

Scenario 4

  1. create a valid vsan policy from E1 and E2
[root@sc2-rdops-vm03-dhcp-107-129:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py policy create --name some-policy-E1 --content '(("proportionalCapacity" i50)''("hostFailuresToTolerate" i0))'  
Successfully created policy: some-policy-E1
[root@sc2-rdops-vm03-dhcp-107-129:~]

[root@sc2-rdops-vm03-dhcp-117-65:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py policy create --name some-policy-E2 --content '(("proportionalCapacity" i50)''("hostFailuresToTolerate" i0))'
Successfully created policy: some-policy-E2
[root@sc2-rdops-vm03-dhcp-117-65:~]
[root@sc2-rdops-vm03-dhcp-117-65:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py policy ls
Policy Name     Policy Content                                               Active
--------------  -----------------------------------------------------------  ------
some-policy-E2  (("proportionalCapacity" i50)("hostFailuresToTolerate" i0))  Unused
some-policy-E1  (("proportionalCapacity" i50)("hostFailuresToTolerate" i0))  Unused

[root@sc2-rdops-vm03-dhcp-117-65:~]


[root@sc2-rdops-vm03-dhcp-107-129:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py policy ls
Policy Name     Policy Content                                               Active
--------------  -----------------------------------------------------------  ------
some-policy-E2  (("proportionalCapacity" i50)("hostFailuresToTolerate" i0))  Unused
some-policy-E1  (("proportionalCapacity" i50)("hostFailuresToTolerate" i0))  Unused

[root@sc2-rdops-vm03-dhcp-107-129:~]
  1. create two volumes from vm3_vsan_E1 one with policy created on E1 and another with E2
docker volume create --driver=vmdk --name=MyVolume -o size=10gb -o vsan-policy-name=some-policy-E1
[root@sc2-rdops-vm03-dhcp-107-129:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py policy ls
Policy Name     Policy Content                                               Active
--------------  -----------------------------------------------------------  -------------------
some-policy-E1  (("proportionalCapacity" i50)("hostFailuresToTolerate" i0))  In use by 1 volumes
some-policy-E2  (("proportionalCapacity" i50)("hostFailuresToTolerate" i0))  In use by 1 volumes

[root@sc2-rdops-vm03-dhcp-107-129:~]
root@photon-xVmYMbyTn [ ~ ]# docker volume ls
DRIVER              VOLUME NAME
vmdk                MyVolume2E1@vsanDatastore
vmdk                MyVolume@vsanDatastore
vmdk                vol1Esx1Vsan1@vsanDatastore
vmdk                vol2Esx2Vsan2@vsanDatastore
root@photon-xVmYMbyTn [ ~ ]#
  1. create two volumes from vm4_vsan_E2 one with policy created on E1 and another with E2
root@photon-xVmYMbyTn [ ~ ]#  docker volume create --driver=vmdk --name=TestVolE2 -o size=400mb                                                                                                      

TestVolE2
root@photon-xVmYMbyTn [ ~ ]# docker volume create --driver=vmdk --name=MyVol1E1 -o size=10gb -o vsan-policy-name=some-policy-E1
MyVol1E1
root@photon-xVmYMbyTn [ ~ ]# docker volume create --driver=vmdk --name=MyVol2E1 -o size=10gb -o vsan-policy-name=some-policy-E2
MyVol2E1
root@photon-xVmYMbyTn [ ~ ]#
root@photon-xVmYMbyTn [ ~ ]# docker volume ls
DRIVER              VOLUME NAME
vmdk                MyVol1E1@vsanDatastore
vmdk                MyVol2E1@vsanDatastore
root@photon-xVmYMbyTn [ ~ ]#
[root@sc-rdops-vm02-dhcp-32-249:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py ls
Volume      Datastore      Created By VM  Created                   Attached To VM (name/uuid)      Policy          Capacity  Used   Disk Format  Filesystem Type  Access      Attach As             
----------  -------------  -------------  ------------------------  ------------------------------  --------------  --------  -----  -----------  ---------------  ----------  ----------------------
MyVol1E1    vsanDatastore  VM1-249-Vsan1  Mon Feb 27 21:35:41 2017  detached                        some-policy-E1  10GB      5GB    thin         ext4             read-write  independent_persistent
MyVol2E1    vsanDatastore  VM1-249-Vsan1  Mon Feb 27 21:36:08 2017  detached                        some-policy-E2  10GB      5GB    thin         ext4             read-write  independent_persistent
MyVol1E2    vsanDatastore  VM2-54-Vsan1   Mon Feb 27 21:38:51 2017  detached                        some-policy-E1  10GB      5GB    thin         ext4             read-write  independent_persistent
MyVol2E2    vsanDatastore  VM2-54-Vsan1   Mon Feb 27 21:39:01 2017  detached                        some-policy-E1  10GB      5GB    thin         ext4             read-write  independent_persistent
MyVol22E2   vsanDatastore  VM2-54-Vsan1   Mon Feb 27 21:43:54 2017  detached                        some-policy-E2  10GB      5GB    thin         ext4             read-write  independent_persistent

[root@sc-rdops-vm02-dhcp-32-249:~]
[root@sc-rdops-vm02-dhcp-32-249:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py policy ls
Policy Name     Policy Content                                               Active
--------------  -----------------------------------------------------------  -------------------
some-policy-E1  (("proportionalCapacity" i50)("hostFailuresToTolerate" i0))  In use by 3 volumes
some-policy-E2  (("proportionalCapacity" i50)("hostFailuresToTolerate" i0))  In use by 2 volumes

[root@sc-rdops-vm02-dhcp-32-249:~]
  1. perform attach/detach for all 4 volumes from both vms and perform data manipulation as well
root@photon-xVmYMbyTn [ ~ ]# docker run -it --volume-driver=vmdk -v MyVol1E1@vsanDatastore:/vol1 -v MyVol1E2@vsanDatastore:/vol2 -v MyVol22E2@vsanDatastore:/vol3 -v MyVol2E2@vsanDatastore:/vol4 --

name ub ubuntu
Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu

d54efb8db41d: Pull complete
f8b845f45a87: Pull complete
e8db7bf7c39f: Pull complete
9654c40e9079: Pull complete
6d9ef359eaaa: Pull complete
Digest: sha256:dd7808d8792c9841d0b460122f1acf0a2dd1f56404f8d1e56298048885e45535
Status: Downloaded newer image for ubuntu:latest
root@704b98db1b92:/# ls
bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var  vol1  vol2  vol3  vol4
root@704b98db1b92:/# echo "I'm MyVol1E1 from vm1" > vol1/hellos
root@704b98db1b92:/# echo "I'm MyVol1E1 from vm1" > vol2/hellos
root@704b98db1b92:/# exit
exit
root@photon-xVmYMbyTn [ ~ ]#
[root@sc-rdops-vm02-dhcp-48-54:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py ls
Volume      Datastore      Created By VM  Created                   Attached To VM (name/uuid)  Policy          Capacity  Used   Disk Format  Filesystem Type  Access      Attach As
----------  -------------  -------------  ------------------------  --------------------------  --------------  --------  -----  -----------  ---------------  ----------  ----------------------
MyVol1E1    vsanDatastore  VM1-249-Vsan1  Mon Feb 27 21:35:41 2017  VM2-54-Vsan1                some-policy-E1  10GB      5GB    thin         ext4             read-write  independent_persistent
MyVol2E1    vsanDatastore  VM1-249-Vsan1  Mon Feb 27 21:36:08 2017  detached                    some-policy-E2  10GB      5GB    thin         ext4             read-write  independent_persistent
MyVol1E2    vsanDatastore  VM2-54-Vsan1   Mon Feb 27 21:38:51 2017  VM2-54-Vsan1                some-policy-E1  10GB      5GB    thin         ext4             read-write  independent_persistent
MyVol2E2    vsanDatastore  VM2-54-Vsan1   Mon Feb 27 21:39:01 2017  VM2-54-Vsan1                some-policy-E1  10GB      5GB    thin         ext4             read-write  independent_persistent
MyVol22E2   vsanDatastore  VM2-54-Vsan1   Mon Feb 27 21:43:54 2017  VM2-54-Vsan1                some-policy-E2  10GB      5GB    thin         ext4             read-write  independent_persistent

[root@sc-rdops-vm02-dhcp-48-54:~]

Verify attach volumes and data manipulation:

root@photon-xVmYMbyTn [ ~ ]# docker run -it --volume-driver=vmdk -v MyVol1E1@vsanDatastore:/vol1 -v MyVol1E2@vsanDatastore:/vol2 -v MyVol22E2@vsanDatastore:/vol3 -v MyVol2E2@vsanDatastore:/vol4 --

name ubE2 ubuntu
Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu

d54efb8db41d: Pull complete
f8b845f45a87: Pull complete
e8db7bf7c39f: Pull complete
9654c40e9079: Pull complete
6d9ef359eaaa: Pull complete
Digest: sha256:dd7808d8792c9841d0b460122f1acf0a2dd1f56404f8d1e56298048885e45535
Status: Downloaded newer image for ubuntu:latest
root@26d3c18b11a8:/# ls
bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var  vol1  vol2  vol3  vol4
root@26d3c18b11a8:/# cd vol1
root@26d3c18b11a8:/vol1# ls
hellos  lost+found
root@26d3c18b11a8:/vol1# cat hellos
I'm MyVol1E1 from vm1
root@26d3c18b11a8:/vol1# cd ..
root@26d3c18b11a8:/# echo "I'm MyVol1E2 from vm2" > vol3/hellos
root@26d3c18b11a8:/# exit
exit
root@photon-xVmYMbyTn [ ~ ]#
  1. on vm3_vsan_E1: attach volume created by vm3_vsan_E1
root@photon-xVmYMbyTn [ ~ ]# docker run -it --volume-driver=vmdk -v MyVol1E1@vsanDatastore:/vol1 --name ubE1 ubuntu                                                                                  

root@fe56685bd690:/# ls
bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var  vol1
root@fe56685bd690:/# cd vol1
root@fe56685bd690:/vol1# ls
hellos  lost+found
root@fe56685bd690:/vol1# cat hellos
I'm MyVol1E1 from vm1
root@fe56685bd690:/vol1#
[root@sc-rdops-vm02-dhcp-32-249:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py ls
Volume      Datastore      Created By VM  Created                   Attached To VM (name/uuid)      Policy          Capacity  Used   Disk Format  Filesystem Type  Access      Attach As             
----------  -------------  -------------  ------------------------  ------------------------------  --------------  --------  -----  -----------  ---------------  ----------  ----------------------
MyVol1E1    vsanDatastore  VM1-249-Vsan1  Mon Feb 27 21:35:41 2017  VM1-249-Vsan1                   some-policy-E1  10GB      5GB    thin         ext4             read-write  independent_persistent
MyVol2E1    vsanDatastore  VM1-249-Vsan1  Mon Feb 27 21:36:08 2017  detached                        some-policy-E2  10GB      5GB    thin         ext4             read-write  independent_persistent
MyVol1E2    vsanDatastore  VM2-54-Vsan1   Mon Feb 27 21:38:51 2017  detached                        some-policy-E1  10GB      5GB    thin         ext4             read-write  independent_persistent
MyVol2E2    vsanDatastore  VM2-54-Vsan1   Mon Feb 27 21:39:01 2017  detached                        some-policy-E1  10GB      5GB    thin         ext4             read-write  independent_persistent
MyVol22E2   vsanDatastore  VM2-54-Vsan1   Mon Feb 27 21:43:54 2017  detached                        some-policy-E2  10GB      5GB    thin         ext4             read-write  independent_persistent

[root@sc-rdops-vm02-dhcp-32-249:~]
  1. on E1: kill the vm vm3_vsan_E1 to test fail over
root@photon-xVmYMbyTn [ ~ ]# docker run -it --volume-driver=vmdk -v MyVol1E1@vsanDatastore:/vol1 -v MyVol1E2@vsanDatastore:/vol2 -v MyVol22E2@vsanDatastore:/vol3 -v MyVol2E2@vsanDatastore:/vol4 --

name ubFailOvr ubuntu
root@62e7ebc579b6:/# ls
bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var  vol1  vol2  vol3  vol4
root@62e7ebc579b6:/# cd vol1
root@62e7ebc579b6:/vol1# ls
hellos  lost+found
root@62e7ebc579b6:/vol1# cat hellos
I'm MyVol1E1 from vm1
root@62e7ebc579b6:/vol1# cd vol4
bash: cd: vol4: No such file or directory
root@62e7ebc579b6:/vol1# cd ..
root@62e7ebc579b6:/# cd vol4
root@62e7ebc579b6:/vol4# ls
lost+found
root@62e7ebc579b6:/vol4# cd ..
root@62e7ebc579b6:/# cd vol3
root@62e7ebc579b6:/vol3# ls
hellos  lost+found
root@62e7ebc579b6:/vol3# cat hellos
I'm MyVol1E2 from vm2
root@62e7ebc579b6:/vol3#
root@sc-rdops-vm02-dhcp-48-54:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py ls
Volume      Datastore      Created By VM  Created                   Attached To VM (name/uuid)  Policy          Capacity  Used   Disk Format  Filesystem Type  Access      Attach As
----------  -------------  -------------  ------------------------  --------------------------  --------------  --------  -----  -----------  ---------------  ----------  ----------------------
MyVol1E1    vsanDatastore  VM1-249-Vsan1  Mon Feb 27 21:35:41 2017  VM2-54-Vsan1                some-policy-E1  10GB      5GB    thin         ext4             read-write  independent_persistent
MyVol2E1    vsanDatastore  VM1-249-Vsan1  Mon Feb 27 21:36:08 2017  detached                    some-policy-E2  10GB      5GB    thin         ext4             read-write  independent_persistent
MyVol1E2    vsanDatastore  VM2-54-Vsan1   Mon Feb 27 21:38:51 2017  VM2-54-Vsan1                some-policy-E1  10GB      5GB    thin         ext4             read-write  independent_persistent
MyVol2E2    vsanDatastore  VM2-54-Vsan1   Mon Feb 27 21:39:01 2017  VM2-54-Vsan1                some-policy-E1  10GB      5GB    thin         ext4             read-write  independent_persistent
MyVol22E2   vsanDatastore  VM2-54-Vsan1   Mon Feb 27 21:43:54 2017  VM2-54-Vsan1                some-policy-E2  10GB      5GB    thin         ext4             read-write  independent_persistent

[root@sc-rdops-vm02-dhcp-48-54:~]

Detach and remove the volumes:

root@photon-xVmYMbyTn [ ~ ]# docker stop $(docker ps -a -q)
62e7ebc579b6
704b98db1b92
root@photon-xVmYMbyTn [ ~ ]# docker rm $(docker ps -a -q)
62e7ebc579b6
704b98db1b92
root@photon-xVmYMbyTn [ ~ ]# docker volume rm MyVol1E1@vsanDatastore MyVol1E2@vsanDatastore MyVol22E2@vsanDatastore MyVol2E1@vsanDatastore MyVol2E2@vsanDatastore
MyVol1E1@vsanDatastore
MyVol1E2@vsanDatastore
MyVol22E2@vsanDatastore
Error response from daemon: get MyVol2E1@vsanDatastore: no such volume
MyVol2E2@vsanDatastore
root@photon-xVmYMbyTn [ ~ ]# docker volume ls
DRIVER              VOLUME NAME
vmdk                TestVol2@vsanDatastore
vmdk                TestVol@vsanDatastore
vmdk                TestVolE22@vsanDatastore
vmdk                TestVolE2@vsanDatastore
root@photon-xVmYMbyTn [ ~ ]#


[root@sc-rdops-vm02-dhcp-48-54:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py ls
Volume      Datastore      Created By VM  Created                   Attached To VM (name/uuid)  Policy          Capacity  Used   Disk Format  Filesystem Type  Access      Attach As
----------  -------------  -------------  ------------------------  --------------------------  --------------  --------  -----  -----------  ---------------  ----------  ----------------------

[root@sc-rdops-vm02-dhcp-48-54:~]

@ashahi1
Copy link
Contributor

ashahi1 commented Feb 28, 2017

Scenario 5 ( fail-over for docker host created on shared vmfs)

  1. create two volumes - two each on vms
root@photon-xVmYMbyTn [ ~ ]# docker volume create --driver=vmdk --name=Vol1E1SD -o size=300mb
Vol1E1SD
root@photon-xVmYMbyTn [ ~ ]# docker volume create --driver=vmdk --name=Vol2E1SD -o size=300mb
Vol2E1SD
root@photon-xVmYMbyTn [ ~ ]# docker volume ls
DRIVER              VOLUME NAME
vmdk                Vol1E1SD@sharedVmfs-0
vmdk                Vol1E2SD@sharedVmfs-0
vmdk                Vol2E1SD@sharedVmfs-0
vmdk                Vol2E2SD@sharedVmfs-0
root@photon-xVmYMbyTn [ ~ ]#
[root@sc-rdops-vm02-dhcp-32-249:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py ls
Volume      Datastore      Created By VM  Created                   Attached To VM (name/uuid)      Policy          Capacity  Used   Disk Format  Filesystem Type  Access      Attach As             
----------  -------------  -------------  ------------------------  ------------------------------  --------------  --------  -----  -----------  ---------------  ----------  ----------------------
Vol1E1SD    sharedVmfs-0   photon-VM0.2   Mon Feb 27 22:28:30 2017  detached                        N/A             300MB     21MB   thin         ext4             read-write  independent_persistent
Vol2E1SD    sharedVmfs-0   photon-VM0.2   Mon Feb 27 22:28:44 2017  detached                        N/A             300MB     21MB   thin         ext4             read-write  independent_persistent
Vol1E2SD    sharedVmfs-0   photon-VM0.0   Mon Feb 27 22:29:21 2017  detached                        N/A             300MB     21MB   thin         ext4             read-write  independent_persistent
Vol2E2SD    sharedVmfs-0   photon-VM0.0   Mon Feb 27 22:29:36 2017  detached                        N/A             300MB     21MB   thin         ext4             read-write  independent_persistent

[root@sc-rdops-vm02-dhcp-32-249:~]
  1. Verify attach-detach and data manipulation for all volumes
root@photon-xVmYMbyTn [ ~ ]# docker run -it --volume-driver=vmdk -v Vol1E1SD@sharedVmfs-0:/vol1 -v Vol1E2SD@sharedVmfs-0:/vol2 -v Vol2E1SD@sharedVmfs-0:/vol3 -v Vol2E2SD@sharedVmfs-0:/vol4 --name 

ubFlOvrSD1 ubuntu
Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu

d54efb8db41d: Pull complete
f8b845f45a87: Pull complete
e8db7bf7c39f: Pull complete
9654c40e9079: Pull complete
6d9ef359eaaa: Pull complete
Digest: sha256:dd7808d8792c9841d0b460122f1acf0a2dd1f56404f8d1e56298048885e45535
Status: Downloaded newer image for ubuntu:latest
root@c16337a86094:/# ls
bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var  vol1  vol2  vol3  vol4
root@c16337a86094:/# echo "I'm MyVol1E1 from vm1" > vol1/hellos
root@c16337a86094:/# echo "I'm MyVol2E1 from vm1" > vol2/hellos
root@c16337a86094:/# exit
exit
root@photon-xVmYMbyTn [ ~ ]#
[root@sc-rdops-vm02-dhcp-32-249:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py ls
Volume      Datastore      Created By VM  Created                   Attached To VM (name/uuid)      Policy          Capacity  Used   Disk Format  Filesystem Type  Access      Attach As             
----------  -------------  -------------  ------------------------  ------------------------------  --------------  --------  -----  -----------  ---------------  ----------  ----------------------
Vol1E1SD    sharedVmfs-0   photon-VM0.2   Mon Feb 27 22:28:30 2017  photon-VM0.2                    N/A             300MB     28MB   thin         ext4             read-write  independent_persistent
Vol2E1SD    sharedVmfs-0   photon-VM0.2   Mon Feb 27 22:28:44 2017  photon-VM0.2                    N/A             300MB     28MB   thin         ext4             read-write  independent_persistent
Vol1E2SD    sharedVmfs-0   photon-VM0.0   Mon Feb 27 22:29:21 2017  photon-VM0.2                    N/A             300MB     28MB   thin         ext4             read-write  independent_persistent
Vol2E2SD    sharedVmfs-0   photon-VM0.0   Mon Feb 27 22:29:36 2017  photon-VM0.2                    N/A             300MB     28MB   thin         ext4             read-write  independent_persistent
  1. Stopping and exiting containers and mounting volumes on another container on different esx and verifying the data
root@photon-xVmYMbyTn [ ~ ]# docker run -it --volume-driver=vmdk -v Vol1E1SD@sharedVmfs-0:/vol1 -v Vol1E2SD@sharedVmfs-0:/vol2 -v Vol2E1SD@sharedVmfs-0:/vol3 -v Vol2E2SD@sharedVmfs-0:/vol4 --name 

ubFlOvrSD2 ubuntu
Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu

d54efb8db41d: Pull complete
f8b845f45a87: Pull complete
e8db7bf7c39f: Pull complete
9654c40e9079: Pull complete
6d9ef359eaaa: Pull complete
Digest: sha256:dd7808d8792c9841d0b460122f1acf0a2dd1f56404f8d1e56298048885e45535
Status: Downloaded newer image for ubuntu:latest
root@7b43e187c076:/# ls
bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var  vol1  vol2  vol3  vol4
root@7b43e187c076:/# cat vol1/hellos
I'm MyVol1E1 from vm1
root@7b43e187c076:/# cat vol2/hellos
I'm MyVol2E1 from vm1
root@7b43e187c076:/# echo "I'm MyVol1E2 from vm2" > vol3/hellos
root@7b43e187c076:/# echo "I'm MyVol2E2 from vm2" > vol4/hellos
root@7b43e187c076:/# exit
exit
root@photon-xVmYMbyTn [ ~ ]#
[root@sc-rdops-vm02-dhcp-32-249:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py ls
Volume      Datastore      Created By VM  Created                   Attached To VM (name/uuid)      Policy          Capacity  Used   Disk Format  Filesystem Type  Access      Attach As             
----------  -------------  -------------  ------------------------  ------------------------------  --------------  --------  -----  -----------  ---------------  ----------  ----------------------
Vol1E1SD    sharedVmfs-0   photon-VM0.2   Mon Feb 27 22:28:30 2017  564d08e9-9a36-b2a2-8edd-11f1..  N/A             300MB     28MB   thin         ext4             read-write  independent_persistent
Vol2E1SD    sharedVmfs-0   photon-VM0.2   Mon Feb 27 22:28:44 2017  564d08e9-9a36-b2a2-8edd-11f1..  N/A             300MB     28MB   thin         ext4             read-write  independent_persistent
Vol1E2SD    sharedVmfs-0   photon-VM0.0   Mon Feb 27 22:29:21 2017  564d08e9-9a36-b2a2-8edd-11f1..  N/A             300MB     28MB   thin         ext4             read-write  independent_persistent
Vol2E2SD    sharedVmfs-0   photon-VM0.0   Mon Feb 27 22:29:36 2017  564d08e9-9a36-b2a2-8edd-11f1..  N/A             300MB     28MB   thin         ext4             read-write  independent_persistent

[root@sc-rdops-vm02-dhcp-32-249:~]
  1. Testing failover by attaching a volume to a container and then killing the vm.
[root@sc-rdops-vm02-dhcp-32-249:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py ls
Volume      Datastore      Created By VM  Created                   Attached To VM (name/uuid)      Policy          Capacity  Used   Disk Format  Filesystem Type  Access      Attach As             
----------  -------------  -------------  ------------------------  ------------------------------  --------------  --------  -----  -----------  ---------------  ----------  ----------------------
Vol1E1SD    sharedVmfs-0   photon-VM0.2   Mon Feb 27 22:28:30 2017  photon-VM0.2                    N/A             300MB     28MB   thin         ext4             read-write  independent_persistent
Vol2E1SD    sharedVmfs-0   photon-VM0.2   Mon Feb 27 22:28:44 2017  detached                        N/A             300MB     28MB   thin         ext4             read-write  independent_persistent
Vol1E2SD    sharedVmfs-0   photon-VM0.0   Mon Feb 27 22:29:21 2017  detached                        N/A             300MB     28MB   thin         ext4             read-write  independent_persistent
Vol2E2SD    sharedVmfs-0   photon-VM0.0   Mon Feb 27 22:29:36 2017  detached                        N/A             300MB     28MB   thin         ext4             read-write  independent_persistent
  1. Killed the vm and mounted the same volume to another container on separate esx
root@photon-xVmYMbyTn [ ~ ]# docker run -it --volume-driver=vmdk -v Vol1E1SD@sharedVmfs-0:/vol1 -v Vol1E2SD@sharedVmfs-0:/vol2 -v Vol2E1SD@sharedVmfs-0:/vol3 -v Vol2E2SD@sharedVmfs-0:/vol4 --name 

ubFlOvr2SD2 ubuntu
root@c7e38fccd2f4:/# ls
bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var  vol1  vol2  vol3  vol4
root@c7e38fccd2f4:/# cat vol1/hellos
I'm MyVol1E1 from vm1
root@c7e38fccd2f4:/# cat vol4/hellos
I'm MyVol2E2 from vm2
root@c7e38fccd2f4:/#
[root@sc-rdops-vm02-dhcp-48-54:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py ls
Volume      Datastore      Created By VM  Created                   Attached To VM (name/uuid)  Policy          Capacity  Used   Disk Format  Filesystem Type  Access      Attach As
----------  -------------  -------------  ------------------------  --------------------------  --------------  --------  -----  -----------  ---------------  ----------  ----------------------
Vol1E1SD    sharedVmfs-0   photon-VM0.2   Mon Feb 27 22:28:30 2017  photon-VM0.0                N/A             300MB     28MB   thin         ext4             read-write  independent_persistent
Vol2E1SD    sharedVmfs-0   photon-VM0.2   Mon Feb 27 22:28:44 2017  photon-VM0.0                N/A             300MB     28MB   thin         ext4             read-write  independent_persistent
Vol1E2SD    sharedVmfs-0   photon-VM0.0   Mon Feb 27 22:29:21 2017  photon-VM0.0                N/A             300MB     28MB   thin         ext4             read-write  independent_persistent
Vol2E2SD    sharedVmfs-0   photon-VM0.0   Mon Feb 27 22:29:36 2017  photon-VM0.0                N/A             300MB     28MB   thin         ext4             read-write  independent_persistent
  1. Exited all the containers:
[root@sc-rdops-vm02-dhcp-48-54:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py ls
Volume      Datastore      Created By VM  Created                   Attached To VM (name/uuid)  Policy          Capacity  Used   Disk Format  Filesystem Type  Access      Attach As
----------  -------------  -------------  ------------------------  --------------------------  --------------  --------  -----  -----------  ---------------  ----------  ----------------------
Vol1E1SD    sharedVmfs-0   photon-VM0.2   Mon Feb 27 22:28:30 2017  detached                    N/A             300MB     28MB   thin         ext4             read-write  independent_persistent
Vol2E1SD    sharedVmfs-0   photon-VM0.2   Mon Feb 27 22:28:44 2017  detached                    N/A             300MB     28MB   thin         ext4             read-write  independent_persistent
Vol1E2SD    sharedVmfs-0   photon-VM0.0   Mon Feb 27 22:29:21 2017  detached                    N/A             300MB     28MB   thin         ext4             read-write  independent_persistent
Vol2E2SD    sharedVmfs-0   photon-VM0.0   Mon Feb 27 22:29:36 2017  detached                    N/A             300MB     28MB   thin         ext4             read-write  independent_persistent
  1. Stopped and removed all the containers - deleted all the volumes
root@photon-xVmYMbyTn [ ~ ]# docker stop $(docker ps -a -q)
c7e38fccd2f4
7b43e187c076
root@photon-xVmYMbyTn [ ~ ]# docker rm $(docker ps -a -q)
c7e38fccd2f4
7b43e187c076
root@photon-xVmYMbyTn [ ~ ]# docker volume rm Vol1E1SD@sharedVmfs-0 Vol1E2SD@sharedVmfs-0 Vol2E1SD@sharedVmfs-0 Vol2E2SD@sharedVmfs-0
Vol1E1SD@sharedVmfs-0
Vol1E2SD@sharedVmfs-0
Vol2E1SD@sharedVmfs-0
Vol2E2SD@sharedVmfs-0
root@photon-xVmYMbyTn [ ~ ]# docker volume ls
DRIVER              VOLUME NAME

@shuklanirdesh82
Copy link
Contributor Author

Reopening to keep adding newly identified scenarios

@shuklanirdesh82
Copy link
Contributor Author

Identification is done and we have decided to query regression-test-needed while automating them.

# for free to subscribe to this conversation on GitHub. Already have an account? #.
Projects
None yet
Development

No branches or pull requests

3 participants