Skip to content
This repository has been archived by the owner on Nov 9, 2020. It is now read-only.

if datastore is renamed, all dockvols/<vmgroup> links are broken #1659

Closed
msterin opened this issue Jul 28, 2017 · 2 comments
Closed

if datastore is renamed, all dockvols/<vmgroup> links are broken #1659

msterin opened this issue Jul 28, 2017 · 2 comments
Assignees
@govint
Labels
component/server P0
Milestone
Sprint - Mufasa (...

Comments

@msterin
Copy link
Contributor

msterin commented Jul 28, 2017 •
edited
Loading

When ESX joins VC cluster (and I suspect on other occasions) a datastore can get renamed.
In my case, datastore1 got renamed to datastore7, and then _DEFAULT link in dockvols got broken since it point to non-existing DS:

[root@msterin-esx-137:/vmfs/volumes/58c376a8-17d63fce-11fe-000c29e5cb47/dockvols] ls -l _DEFAULT
lrwxrwxrwx    1 root     root            70 Apr  6 18:47 _DEFAULT -> /vmfs/volumes/datastore1/dockvols/11111111-1111-1111-1111-111111111111

the end result is somewhat unpredicatable behavior - many ops will succeed since they use vmgroup UUID, and some may fail with messages about non-exiting files or folders

We should use ./ instead of /fullpath/uuid when forming links
@lipingxue @tusharnt
@shuklanirdesh82
Copy link
Contributor

likely to be regression ... Please refer #818 where rename datastore related workflow was fixed and tested as well.

@govint govint self-assigned this Aug 21, 2017
@govint
Copy link
Contributor

govint commented Aug 23, 2017

This is what I'm seeing,

  1. No config on ESX
  2. Create multiple volumes
  3. Create a config
  4. _DEFAULT symlink is not created
  5. this is the dockvols dir
    [root@sc-rdops-vm16-dhcp-232-194:/vmfs/volumes/59474fd1-55672410-d21e-02001e5a2f6d/dockvols] ls
    11111111-1111-1111-1111-111111111111
  6. Rename the datastore to sharedVmfs-02
    [root@sc-rdops-vm16-dhcp-232-194:/vmfs/volumes/59474fd1-55672410-d21e-02001e5a2f6d/dockvols] python /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py volume ls
    Volume Datastore VMGroup Capacity Used Filesystem Policy Disk Format Attached-to Access Attach-as Created By Created Date

debugvol19 sharedVmfs-02 _DEFAULT 100MB 15MB ext4 N/A thin worker1-VM1.0 read-write independent_persistent worker1-VM1.0 Wed Jul 26 11:04:21 2017
ugvol2 sharedVmfs-02 _DEFAULT 100MB 15MB ext4 N/A thin master-VM0.0 read-write independent_persistent worker1-VM1.0 Wed Aug 16 06:07:36 2017
upvol1 sharedVmfs-02 _DEFAULT 100MB 15MB ext4 N/A thin worker1-VM1.0 read-write independent_persistent worker1-VM1.0 Thu Aug 17 04:11:35 2017
snapvol3 sharedVmfs-02 _DEFAULT 100MB 15MB ext4 N/A thin master-VM0.0 read-write persistent master-VM0.0 Tue Aug 8 11:00:16 2017
snapvol 4 sharedVmfs-02 _DEFAULT 100MB 15MB ext4 N/A thin detached read-write independent_persistent worker1-VM1.0 Thu Aug 10 06:48:44 2017
tvol1 sharedVmfs-02 _DEFAULT 100MB 15MB ext4 N/A thin worker1-VM1.0 read-write independent_persistent worker1-VM1.0 Fri Aug 18 08:46:45 2017
tvol2 sharedVmfs-02 _DEFAULT 100MB 15MB ext4 N/A thin worker1-VM1.0 read-write independent_persistent worker1-VM1.0 Fri Aug 18 08:53:28 2017
tvol3 sharedVmfs-02 _DEFAULT 100MB 15MB ext4 N/A thin worker1-VM1.0 read-write independent_persistent worker1-VM1.0 Fri Aug 18 09:32:33 2017
tvol4 sharedVmfs-02 _DEFAULT 100MB 15MB ext4 N/A thin detached read-write independent_persistent worker1-VM1.0 Fri Aug 18 10:03:33 2017
tvol5 sharedVmfs-02 _DEFAULT 100MB 15MB ext4 N/A thin detached read-write independent_persistent worker1-VM1.0 Fri Aug 18 10:12:36 2017
tvol6 sharedVmfs-02 _DEFAULT 100MB 15MB ext4 N/A thin detached read-write independent_persistent worker1-VM1.0 Mon Aug 21 08:46:58 2017
tvol7 sharedVmfs-02 _DEFAULT 100MB 15MB ext4 N/A thin detached read-write independent_persistent worker1-VM1.0 Mon Aug 21 10:07:39 2017
tvol8 sharedVmfs-02 _DEFAULT 100MB 15MB ext4 N/A thin detached read-write independent_persistent worker1-VM1.0 Mon Aug 21 10:28:00 2017
tvol9 sharedVmfs-02 _DEFAULT 100MB 15MB ext4 N/A thin detached read-write independent_persistent worker1-VM1.0 Tue Aug 22 05:52:46 2017
tvol10 sharedVmfs-02 _DEFAULT 100MB 13MB ext4 N/A thin detached read-write independent_persistent worker1-VM1.0 Wed Aug 23 05:20:48 2017
snapvol1 sharedVmfs-1 _DEFAULT 100MB 15MB ext4 N/A thin detached read-write independent_persistent master-VM0.0 Mon Jul 31 08:41:41 2017

  1. Rename the datastore to sharedVmfs-3
    [root@sc-rdops-vm16-dhcp-232-194:/vmfs/volumes/59474fd1-55672410-d21e-02001e5a2f6d/dockvols] python /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py volume ls
    Volume Datastore VMGroup Capacity Used Filesystem Policy Disk Format Attached-to Access Attach-as Created By Created Date

debugvol19 sharedVmfs-3 _DEFAULT 100MB 15MB ext4 N/A thin worker1-VM1.0 read-write independent_persistent worker1-VM1.0 Wed Jul 26 11:04:21 2017
ugvol2 sharedVmfs-3 _DEFAULT 100MB 15MB ext4 N/A thin master-VM0.0 read-write independent_persistent worker1-VM1.0 Wed Aug 16 06:07:36 2017
upvol1 sharedVmfs-3 _DEFAULT 100MB 15MB ext4 N/A thin worker1-VM1.0 read-write independent_persistent worker1-VM1.0 Thu Aug 17 04:11:35 2017
snapvol3 sharedVmfs-3 _DEFAULT 100MB 15MB ext4 N/A thin master-VM0.0 read-write persistent master-VM0.0 Tue Aug 8 11:00:16 2017
snapvol 4 sharedVmfs-3 _DEFAULT 100MB 15MB ext4 N/A thin detached read-write independent_persistent worker1-VM1.0 Thu Aug 10 06:48:44 2017
tvol1 sharedVmfs-3 _DEFAULT 100MB 15MB ext4 N/A thin worker1-VM1.0 read-write independent_persistent worker1-VM1.0 Fri Aug 18 08:46:45 2017
tvol2 sharedVmfs-3 _DEFAULT 100MB 15MB ext4 N/A thin worker1-VM1.0 read-write independent_persistent worker1-VM1.0 Fri Aug 18 08:53:28 2017
tvol3 sharedVmfs-3 _DEFAULT 100MB 15MB ext4 N/A thin worker1-VM1.0 read-write independent_persistent worker1-VM1.0 Fri Aug 18 09:32:33 2017
tvol4 sharedVmfs-3 _DEFAULT 100MB 15MB ext4 N/A thin detached read-write independent_persistent worker1-VM1.0 Fri Aug 18 10:03:33 2017
tvol5 sharedVmfs-3 _DEFAULT 100MB 15MB ext4 N/A thin detached read-write independent_persistent worker1-VM1.0 Fri Aug 18 10:12:36 2017
tvol6 sharedVmfs-3 _DEFAULT 100MB 15MB ext4 N/A thin detached read-write independent_persistent worker1-VM1.0 Mon Aug 21 08:46:58 2017
tvol7 sharedVmfs-3 _DEFAULT 100MB 15MB ext4 N/A thin detached read-write independent_persistent worker1-VM1.0 Mon Aug 21 10:07:39 2017
tvol8 sharedVmfs-3 _DEFAULT 100MB 15MB ext4 N/A thin detached read-write independent_persistent worker1-VM1.0 Mon Aug 21 10:28:00 2017
tvol9 sharedVmfs-3 _DEFAULT 100MB 15MB ext4 N/A thin detached read-write independent_persistent worker1-VM1.0 Tue Aug 22 05:52:46 2017
tvol10 sharedVmfs-3 _DEFAULT 100MB 13MB ext4 N/A thin detached read-write independent_persistent worker1-VM1.0 Wed Aug 23 05:20:48 2017
snapvol1 sharedVmfs-1 _DEFAULT 100MB 15MB ext4 N/A thin detached read-write independent_persistent master-VM0.0 Mon Jul 31 08:41:41 2017

  1. Create a new vmgroup and add the VM to it
    [root@sc-rdops-vm16-dhcp-232-194:/vmfs/volumes/59474fd1-55672410-d21e-02001e5a2f6d/dockvols] python /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py vmgroup ls
    Uuid Name Description Default_datastore VM_list

11111111-1111-1111-1111-111111111111 _DEFAULT This is a default vmgroup _VM_DS
5e32424c-9e1d-42c7-881d-e8cc2e99c18c ds-ren sharedVmfs-0 worker1-VM1.0

  1. This is dockvols dir,
    [root@sc-rdops-vm16-dhcp-232-194:/vmfs/volumes/59474fd1-55672410-d21e-02001e5a2f6d/dockvols] ls
    11111111-1111-1111-1111-111111111111 5e32424c-9e1d-42c7-881d-e8cc2e99c18c ds-ren

  2. Create volumes from VM in ds-ren
    [root@sc-rdops-vm16-dhcp-232-194:/vmfs/volumes/59474fd1-55672410-d21e-02001e5a2f6d/dockvols] python /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py volume ls -c volume,datastore,vmgroup
    Volume Datastore VMGroup

debugvol19 sharedVmfs-3 _DEFAULT
ugvol2 sharedVmfs-3 _DEFAULT
upvol1 sharedVmfs-3 _DEFAULT
snapvol3 sharedVmfs-3 _DEFAULT
snapvol 4 sharedVmfs-3 _DEFAULT
tvol1 sharedVmfs-3 _DEFAULT
tvol2 sharedVmfs-3 _DEFAULT
tvol3 sharedVmfs-3 _DEFAULT
tvol4 sharedVmfs-3 _DEFAULT
tvol5 sharedVmfs-3 _DEFAULT
tvol6 sharedVmfs-3 _DEFAULT
tvol7 sharedVmfs-3 _DEFAULT
tvol8 sharedVmfs-3 _DEFAULT
tvol9 sharedVmfs-3 _DEFAULT
tvol10 sharedVmfs-3 _DEFAULT
tvol11 sharedVmfs-3 ds-ren <------------
tvol12 sharedVmfs-3 ds-ren <------------
snapvol1 sharedVmfs-1 _DEFAULT

  1. Rename datastore to sharedVmfs-0
    [root@sc-rdops-vm16-dhcp-232-194:/vmfs/volumes/59474fd1-55672410-d21e-02001e5a2f6d/dockvols] python /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py volume ls -c volume,datastore,vmgroup
    Volume Datastore VMGroup

debugvol19 sharedVmfs-0 _DEFAULT
ugvol2 sharedVmfs-0 _DEFAULT
upvol1 sharedVmfs-0 _DEFAULT
snapvol3 sharedVmfs-0 _DEFAULT
snapvol 4 sharedVmfs-0 _DEFAULT
tvol1 sharedVmfs-0 _DEFAULT
tvol2 sharedVmfs-0 _DEFAULT
tvol3 sharedVmfs-0 _DEFAULT
tvol4 sharedVmfs-0 _DEFAULT
tvol5 sharedVmfs-0 _DEFAULT
tvol6 sharedVmfs-0 _DEFAULT
tvol7 sharedVmfs-0 _DEFAULT
tvol8 sharedVmfs-0 _DEFAULT
tvol9 sharedVmfs-0 _DEFAULT
tvol10 sharedVmfs-0 _DEFAULT
tvol11 sharedVmfs-0 ds-ren
tvol12 sharedVmfs-0 ds-ren
snapvol1 sharedVmfs-1 _DEFAULT

  1. Vmgroup ds-ren still points to the old DS name
    [root@sc-rdops-vm16-dhcp-232-194:/vmfs/volumes/59474fd1-55672410-d21e-02001e5a2f6d/dockvols] ls -l
    total 32
    drwxr-xr-x 1 root root 6580 Aug 23 05:20 11111111-1111-1111-1111-111111111111
    drwxr-xr-x 1 root root 1120 Aug 23 05:46 5e32424c-9e1d-42c7-881d-e8cc2e99c18c
    lrwxrwxrwx 1 root root 72 Aug 23 05:39 ds-ren ->
    /vmfs/volumes/sharedVmfs-3/dockvols/5e32424c-9e1d-42c7-881d-e8cc2e99c18c

  2. Attach the volume in ds-ren in the VM

root@photon-machine [ /vol/vpl-upg/docker-volume-vsphere/client_plugin ]# docker run -it --rm -v tvol11:/mnt busybox
/ #

  1. Attached as expected,
    root@photon-machine [ ~ ]# docker volume inspect tvol11
    [
    {
    "Driver": "vsphere:latest",
    "Labels": {},
    "Mountpoint": "/mnt/vmdk/tvol11",
    "Name": "tvol11",
    "Options": {},
    "Scope": "global",
    "Status": {
    "access": "read-write",
    "attach-as": "independent_persistent",
    "attached to VM": "worker1-VM1.0",
    "attachedVMDevice": {
    "ControllerPciSlotNumber": "160",
    "Unit": "0"
    },
    "capacity": {
    "allocated": "15MB",
    "size": "100MB"
    },
    "clone-from": "None",
    "created": "Wed Aug 23 05:39:52 2017",
    "created by VM": "worker1-VM1.0",
    "datastore": "sharedVmfs-0",
    "diskformat": "thin",
    "fstype": "ext4",
    "status": "attached"
    }
    }
    ]

This is working as expected and hence closing.

@govint govint closed this as completed Aug 23, 2017
# for free to subscribe to this conversation on GitHub. Already have an account? #.
Assignees

@govint govint

Labels
component/server P0
Projects
None yet
Milestone
Sprint - Mufasa (VMworld, US)
Development

No branches or pull requests
4 participants
@msterin @govint @pdhamdhere @shuklanirdesh82