Skip to content
This repository has been archived by the owner on Nov 9, 2020. It is now read-only.

[Cross ESX] 'Attached to vm' field is not populated even though volume is attached to a container. #1053

Closed
ashahi1 opened this issue Mar 20, 2017 · 4 comments
Closed

[Cross ESX] 'Attached to vm' field is not populated even though volume is attached to a container. #1053

ashahi1 opened this issue Mar 20, 2017 · 4 comments
Assignees
@pshahzeb
Labels
kind/bug P1
Milestone
0.13

Comments

@ashahi1
Copy link
Contributor

ashahi1 commented Mar 20, 2017

Test Steps:

  1. Installed the managed plugin on VM-1 and VM-2
  2. Created two volumes - one from each vm
  3. Ran the container from vm-1 by mounting the volume created from vm-2.
  4. Did docker volume inspect from vm-1
  5. Did docker volume inspect from vm-2. "Attached-to-vm" field is not present.

Admin cli volume ls from both the ESXs does show the "Attached-to" field correctly.

Setup and command output detail is as follows:

Setup : 2 ESXs (E1 & E2 having 2 shared datastores (shared vmfs))

docker Version: 17.03.0-ce

ESX-1: 10.160.35.114
VM-1: 10.160.57.11
VM's Datastore: sharedVmfs-1


ESX-2: 10.160.50.83
VM-2: 10.160.63.76
VM's Datastore: sharedVmfs-0
  • Installed the plugin:
root@photon-xVmYMbyTn [ ~ ]# docker plugin install cnastorage/docker-volume-vsphere:1.0
Plugin "cnastorage/docker-volume-vsphere:1.0" is requesting the following privileges:
 - network: [none]
 - mount: [/dev]
 - mount: [/var/run]
 - mount: [/etc]
 - mount: [/var/log]
 - allow-all-devices: [true]
 - capabilities: [CAP_SYS_ADMIN]
Do you grant the above permissions? [y/N] y
1.0: Pulling from cnastorage/docker-volume-vsphere
2ba055944850: Download complete
Digest: sha256:aaa7ccee39b8ba5ae16465d60069469b0141057a839cc1d14b22ae0008f4902b
Status: Downloaded newer image for cnastorage/docker-volume-vsphere:1.0
Installed plugin cnastorage/docker-volume-vsphere:1.0
root@photon-xVmYMbyTn [ ~ ]# docker plugin ls
ID                  NAME                                   DESCRIPTION                           ENABLED
ced20f7c8668        cnastorage/docker-volume-vsphere:1.0   VMWare vSphere Docker Volume plugin   true
root@photon-xVmYMbyTn [ ~ ]#
  • Created a volume from VM-1: TestVol_VM1
root@photon-xVmYMbyTn [ ~ ]# docker volume ls
DRIVER              VOLUME NAME
root@photon-xVmYMbyTn [ ~ ]# docker volume create -d cnastorage/docker-volume-vsphere:1.0 --name=TestVol_VM1
TestVol_VM1
root@photon-xVmYMbyTn [ ~ ]# docker volume ls
DRIVER                                 VOLUME NAME
cnastorage/docker-volume-vsphere:1.0   TestVol_VM1@sharedVmfs-1
root@photon-xVmYMbyTn [ ~ ]#


root@photon-xVmYMbyTn [ ~ ]# docker volume inspect TestVol_VM1
[
    {
        "Driver": "cnastorage/docker-volume-vsphere:1.0",
        "Labels": {},
        "Mountpoint": "/mnt/vmdk/TestVol_VM1",
        "Name": "TestVol_VM1",
        "Options": {},
        "Scope": "global",
        "Status": {
            "access": "read-write",
            "attach-as": "independent_persistent",
            "capacity": {
                "allocated": "13MB",
                "size": "100MB"
            },
            "clone-from": "None",
            "created": "Mon Mar 20 22:55:12 2017",
            "created by VM": "photon-VM1.0",
            "datastore": "sharedVmfs-1",
            "diskformat": "thin",
            "fstype": "ext4",
            "status": "detached"
        }
    }
]
root@photon-xVmYMbyTn [ ~ ]#

Admin cli volume ls from ESX-1:

[root@sc-rdops-vm02-dhcp-35-114:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py ls
Volume        Datastore     VM-Group  Capacity  Used  Filesystem  Policy  Disk Format  Attached-to  Access      Attach-as               Created By    Created Date
------------  ------------  --------  --------  ----  ----------  ------  -----------  -----------  ----------  ----------------------  ------------  ------------------------
TestVol_VM2   sharedVmfs-0  _DEFAULT  100MB     13MB  ext4        N/A     thin         detached     read-write  independent_persistent  photon-VM1.3  Mon Mar 20 22:56:12 2017
TestVol_VM1   sharedVmfs-1  _DEFAULT  100MB     13MB  ext4        N/A     thin         detached     read-write  independent_persistent  photon-VM1.0  Mon Mar 20 22:55:12 2017

[root@sc-rdops-vm02-dhcp-35-114:~]
  • Created another volume from VM-2 - TestVol_VM22
root@photon-xVmYMbyTn [ ~ ]# docker volume create -d cnastorage/docker-volume-vsphere:1.0 --name=TestVol_VM22
TestVol_VM22
root@photon-xVmYMbyTn [ ~ ]# docker volume inspect TestVol_VM22
[
    {
        "Driver": "cnastorage/docker-volume-vsphere:1.0",
        "Labels": {},
        "Mountpoint": "/mnt/vmdk/TestVol_VM22",
        "Name": "TestVol_VM22",
        "Options": {},
        "Scope": "global",
        "Status": {
            "access": "read-write",
            "attach-as": "independent_persistent",
            "capacity": {
                "allocated": "13MB",
                "size": "100MB"
            },
            "clone-from": "None",
            "created": "Mon Mar 20 22:56:38 2017",
            "created by VM": "photon-VM1.3",
            "datastore": "sharedVmfs-0",
            "diskformat": "thin",
            "fstype": "ext4",
            "status": "detached"
        }
    }
]
root@photon-xVmYMbyTn [ ~ ]#

Admin cli volume ls from ESX-2:

[root@sc-rdops-vm02-dhcp-50-83:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py ls
Volume        Datastore     VM-Group  Capacity  Used  Filesystem  Policy  Disk Format  Attached-to  Access      Attach-as               Created By    Created Date
------------  ------------  --------  --------  ----  ----------  ------  -----------  -----------  ----------  ----------------------  ------------  ------------------------
TestVol_VM2   sharedVmfs-0  _DEFAULT  100MB     13MB  ext4        N/A     thin         detached     read-write  independent_persistent  photon-VM1.3  Mon Mar 20 22:56:12 2017
TestVol_VM22  sharedVmfs-0  _DEFAULT  100MB     13MB  ext4        N/A     thin         detached     read-write  independent_persistent  photon-VM1.3  Mon Mar 20 22:56:38 2017
TestVol_VM1   sharedVmfs-1  _DEFAULT  100MB     13MB  ext4        N/A     thin         detached     read-write  independent_persistent  photon-VM1.0  Mon Mar 20 22:55:12 2017

[root@sc-rdops-vm02-dhcp-50-83:~]
  • Ran the container from VM-1 and mounted the volume - TestVol_VM22
root@photon-xVmYMbyTn [ ~ ]# docker run -it --volume-driver=cnastorage/docker-volume-vsphere:1.0 -v TestVol_VM22@sharedVmfs-0:/vol1 --name ub ubuntu
Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu
d54efb8db41d: Pull complete
f8b845f45a87: Pull complete
e8db7bf7c39f: Pull complete
9654c40e9079: Pull complete
6d9ef359eaaa: Pull complete
Digest: sha256:dd7808d8792c9841d0b460122f1acf0a2dd1f56404f8d1e56298048885e45535
Status: Downloaded newer image for ubuntu:latest
root@cd1c97491eca:/#

Docker volume inspect from VM-2:

root@photon-xVmYMbyTn [ ~ ]# docker volume inspect TestVol_VM22
[
    {
        "Driver": "cnastorage/docker-volume-vsphere:1.0",
        "Labels": {},
        "Mountpoint": "/mnt/vmdk/TestVol_VM22",
        "Name": "TestVol_VM22",
        "Options": {},
        "Scope": "global",
        "Status": {
            "access": "read-write",
            "attach-as": "independent_persistent",
            "capacity": {
                "allocated": "15MB",
                "size": "100MB"
            },
            "clone-from": "None",
            "created": "Mon Mar 20 22:56:38 2017",
            "created by VM": "photon-VM1.3",
            "datastore": "sharedVmfs-0",
            "diskformat": "thin",
            "fstype": "ext4",
            "status": "attached" <<<<<<<<<<<  Shows "attached" but "attached to VM" field is not there.
        }
    }
]
root@photon-xVmYMbyTn [ ~ ]#

Docker volume inspect from the vm from where container was started - VM-1

root@photon-xVmYMbyTn [ ~ ]# docker volume inspect TestVol_VM22@sharedVmfs-0
[
    {
        "Driver": "cnastorage/docker-volume-vsphere:1.0",
        "Labels": null,
        "Mountpoint": "/mnt/vmdk/TestVol_VM22@sharedVmfs-0",
        "Name": "TestVol_VM22@sharedVmfs-0",
        "Options": {},
        "Scope": "global",
        "Status": {
            "access": "read-write",
            "attach-as": "independent_persistent",
            "attached to VM": "photon-VM1.0",      <<<<<<<<<<This field is not there in docker volume inspect from VM2
            "capacity": {
                "allocated": "15MB",
                "size": "100MB"
            },
            "clone-from": "None",
            "created": "Mon Mar 20 22:56:38 2017",
            "created by VM": "photon-VM1.3",
            "datastore": "sharedVmfs-0",
            "diskformat": "thin",
            "fstype": "ext4",
            "status": "attached"    
        }
    }
]
root@photon-xVmYMbyTn [ ~ ]#

Admin cli volume ls from ESX-1:

[root@sc-rdops-vm02-dhcp-35-114:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py ls
Volume        Datastore     VM-Group  Capacity  Used  Filesystem  Policy  Disk Format  Attached-to   Access      Attach-as               Created By    Created Date
------------  ------------  --------  --------  ----  ----------  ------  -----------  ------------  ----------  ----------------------  ------------  ------------------------
TestVol_VM2   sharedVmfs-0  _DEFAULT  100MB     13MB  ext4        N/A     thin         detached      read-write  independent_persistent  photon-VM1.3  Mon Mar 20 22:56:12 2017
TestVol_VM22  sharedVmfs-0  _DEFAULT  100MB     15MB  ext4        N/A     thin         photon-VM1.0  read-write  independent_persistent  photon-VM1.3  Mon Mar 20 22:56:38 2017
TestVol_VM1   sharedVmfs-1  _DEFAULT  100MB     13MB  ext4        N/A     thin         detached      read-write  independent_persistent  photon-VM1.0  Mon Mar 20 22:55:12 2017

[root@sc-rdops-vm02-dhcp-35-114:~]

Admin cli volume ls from ESX-2:

[root@sc-rdops-vm02-dhcp-50-83:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py ls
Volume        Datastore     VM-Group  Capacity  Used  Filesystem  Policy  Disk Format  Attached-to                         Access      Attach-as               Created By    Created Date            
------------  ------------  --------  --------  ----  ----------  ------  -----------  ----------------------------------  ----------  ----------------------  ------------  ------------------------
TestVol_VM2   sharedVmfs-0  _DEFAULT  100MB     13MB  ext4        N/A     thin         detached                            read-write  independent_persistent  photon-VM1.3  Mon Mar 20 22:56:12 2017
TestVol_VM22  sharedVmfs-0  _DEFAULT  100MB     15MB  ext4        N/A     thin         564ddb8f-36ab-f213-c1d3-7406dadd..  read-write  independent_persistent  photon-VM1.3  Mon Mar 20 22:56:38 2017
TestVol_VM1   sharedVmfs-1  _DEFAULT  100MB     13MB  ext4        N/A     thin         detached                            read-write  independent_persistent  photon-VM1.0  Mon Mar 20 22:55:12 2017

[root@sc-rdops-vm02-dhcp-50-83:~]
@ashahi1 ashahi1 added this to the 0.13 milestone Mar 20, 2017
@ashahi1
Copy link
Contributor Author

ashahi1 commented Mar 20, 2017

Logs:
docker-volume-vsphere-vm1.txt
docker-volume-vsphere-vm2.txt
vmdk_ops_esx1.txt
vmdk_ops_esx2.txt

@tusharnt tusharnt assigned pshahzeb and unassigned lipingxue Mar 21, 2017
@pshahzeb pshahzeb changed the title [Managed Plugin] 'Attached to vm' field is not populated even though volume is attached to a container. [Cross ESX] 'Attached to vm' field is not populated even though volume is attached to a container. Mar 21, 2017
@pshahzeb
Copy link
Contributor

pshahzeb commented Mar 21, 2017 •
edited
Loading

The reason attached to VM field is not populated is because the VM uuid is stored in volume metadata while a volume is mounted to a VM. If the docker volume inspect is done from a VM that resides on a different ESX as compared to the ESX on which the VM which mounted the volume resides on; we can't find the name of VM using uuid( because the later ESX isn't aware about the VM on other ESX).
A solution to this is to store the VM name in volume metadata along with uuid. If we cannot find the volume name from uuid, use the name from metadata. This name can be stale (VM got renamed) but we can document this.
CC @msterin

@pdhamdhere
Copy link
Contributor

I agree with proposed approach to store VM name in KV. In fact I recommend same for admin ls command to store vm name in addition to UUID in database. Optionally you can postfix VM Name with * indicating VM using volume is running on another host. VM getting renamed and inspect or admin ls returning is really really corner case. Also later with Hostd Notification, this problem will be solved anyway. In summary, go ahead with your fix. CC /@govint

@msterin
Copy link
Contributor

msterin commented Mar 22, 2017

Agreed.

pshahzeb pushed a commit that referenced this issue Mar 22, 2017
Persisting VM name in KV while attaching volume
6602e96 
1. Persisting vm name along with uuid in KV when a volume is attached
   to a VM
2. If vm name cannot be retrieved using uuid (VM might reside on another host)
   then use the vm name persisted in KV. Using this in both for admincli and plugin
   requests

Fixes #1053
@pshahzeb pshahzeb mentioned this issue Mar 22, 2017
Merged
pdhamdhere pushed a commit that referenced this issue Mar 24, 2017
@pdhamdhere
Persisting VM name in KV while attaching volume (#1064)
5c7614b 
* Persisting VM name in KV while attaching volume

1. Persisting vm name along with uuid in KV when a volume is attached
   to a VM
2. If vm name cannot be retrieved using uuid (VM might reside on another host)
   then use the vm name persisted in KV. Using this in both for admincli and plugin
   requests

Fixes #1053

* Handling VM names for in volume metadata for volumes created without persisting VM name in metadata

* Populating attached VM with uuid in case vm name cannot be retrieved on host and is not persisted in KV store
@ashahi1 ashahi1 mentioned this issue Mar 28, 2017
Closed
5 tasks
@shuklanirdesh82 shuklanirdesh82 mentioned this issue May 16, 2017
Closed
5 tasks
@ashahi1 ashahi1 mentioned this issue Jun 21, 2017
Closed
1 task
# for free to subscribe to this conversation on GitHub. Already have an account? #.
Assignees

@pshahzeb pshahzeb

Labels
kind/bug P1
Projects
None yet
Milestone
0.13
Development

No branches or pull requests
5 participants
@msterin @lipingxue @pdhamdhere @ashahi1 @pshahzeb