[Multi-ESX] vmgroup is not removable when vms are added from both ESXs (configured as MultiNode)

[shuklanirdesh82]

Test bed setup:

1. 2 ESX (E1 & E2) accessing a shared datastore SD1
2. 2 VMs (VM1 & VM2):

• VM1 is created on SD1 (registered at E1)
• VM2 is created on SD1 (registered at E2)

Steps to reproduce:

1. configure MultiNode state on both E1 & E2
E1: vmdkops_admin.py config init --datastore vsanDatastore
E2: vmdkops_admin.py config init --datastore vsanDatastore

2. Create named vmgroup "T1" and add VM1 & VM2 to T1
E1: vmdkops_admin.py vmgroup create --name=T1 --default-datastore=vsanDatastore
E1: vmdkops_admin.py vmgroup vm add --name=T1 --vm-list=VM1
E2: vmdkops_admin.py vmgroup vm add --name=T1 --vm-list=VM2

3. E1/E2: invoke vmdkops_admin.py vmgroup ls to make sure VM & VM2 are added to T1 vmgroup
E1/E2:vmdkops_admin.py vmgroup ls

4. Invoke vmdkops_admin.py vmgroup rm --name=T1

Expectation: Admin CLI should allow to remove T1

Current behavior: Admin CLI is not allowing to remove and throwing following error ( vm UUID is not recognized by ESX where vm is not registered)

[root@sc2-rdops-vm05-dhcp-160-17:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py vmgroup rm --name=T1
Cannot complete vmgroup rm. Cannot find vm 564d4e04-f778-487a-5e1c-50e6215af0a5

While invoking vmgroup rm command from E1 (Admin CLI complains about the UUID of VM2 as it is registered on E2) and on E2 complains about UUID of VM1

On E1:

=== Authorization Config DB: 
DB_SharedLocation: /vmfs/volumes/vsanDatastore/dockvols/vmdkops_config.db
DB_Mode: MultiNode (local symlink pointing to shared DB)
DB_LocalPath: /etc/vmware/vmdkops/auth-db

on E2:

=== Authorization Config DB: 
DB_LocalPath: /etc/vmware/vmdkops/auth-db
DB_Mode: MultiNode (local symlink pointing to shared DB)
DB_SharedLocation: /vmfs/volumes/vsanDatastore/dockvols/vmdkops_config.db

/CC @lipingxue

[pdhamdhere]

check_volumes_mounted does findVmByUuid which is going to fail for VMs registered on some other host. This was introduced in #1060

[govint]

Is this a doc issue? Because, when adding the VM to the vmgroup the below is done,

E1: vmdkops_admin.py vmgroup vm add --name=T1 --vm-list=VM1
E2: vmdkops_admin.py vmgroup vm add --name=T1 --vm-list=VM2

The vmgroup add has to be done on each ESX host thats running the VM. The user (even with a script) needs to ssh a command to the respective ESX to add VMs on each host into a vmgroup.

The expectation to remove the vmgroup is incorrect in a multi-node case unless the actions during add are reversed. Which is by first removing the VMs from the vmgroup from the respective hosts and only then the vmgroup can be removed. The flexibility to remove the vmgroup with VMs in it seems impossible in a multi-node case.

Given we need to init config, add VMs from multiple hosts rm too should be done on each host in this case.

[pshahzeb]

@govint

The flexibility to remove the vmgroup with VMs in it seems impossible in a multi-node case.

Agree. In such cases should we return with error to force user to remove the VMs from the vmgroup first (from respective esx host)? This is only for shared mode; for local mode, vmgroup with VMs in it can be deleted.

This is requires user to do additional steps, but is graceful.

[govint]

Agree, given the VC based capability will remove the need to do commands per host and fact that even now it should be just SSH from a guest OS to add VMs on multiple ESX hosts, its not extra to expect the user to remove the VMs before removing the vmgroup. Config init and vmgroup add all need to be done per host anyway.

[pdhamdhere]

Why not skip check_volumes_mounted check, warn user and proceed with deleting vm_group?

[pshahzeb]

It breaks the detach logic, as explained in first paragraph of description of #1045

[pshahzeb]

@pdhamdhere @govint so the fix is: for vmgroup rm, return error to user if multinode mode and vmgroup has member VMs ? I will go ahead and generate a PR then.

[shaominchen]

@pshahzeb I agree it makes sense to disallow deleting vmgroup if there are related volumes attached. But why don't we disallow deleting a vmgroup with some VMs? The issue here is that "Cannot find vm 564d4e04-f778-487a-5e1c-50e6215af0a5". I think we have agreed that we should persist VM names in the DB. Adding VM names in the DB should fix this issue, right?

[pshahzeb]

@shaominchen
The error "Cannot find vm 564d4e04-f778-487a-5e1c-50e6215af0a5" is actually thrown by check_volumes_mounted() which we do for each VM inside a Vmgroup. This is to ensure that it doesn't have any volumes attached (to ensure fluent detach of volumes)
As part of check_volumes_mounted, we do findVmbyUuid. But as the VM reside(might reside) on another ESX, we aren't able to retrieve VM moref from SI. Hence the error.

[shaominchen]

@pshahzeb I see. That makes sense then, as we actually need the VM MoRef object instead of just VM name here.