vmdk-ops service unable to identify the newly mounted datastore on ESX host. But identifies after the service restart.

[BaluDontu]

Hi guys,

I have mounted a new NFS datastore with name "NFSStore" on ESX host.
When I execute following docker create volume command, it gives an error Invalid datastore.

docker volume create -d=vmdk --name=volume@NFSStore
Error response from daemon: create volume@NFSStore: VolumeDriver.Create: Invalid datastore 'NFSStore'.
Known datastores: datastore1, NFSDatastore, coke (1).
Default datastore: datastore1

I am able to see the volume create under /vmfs/volume/NFSStore on ESX host. However vmdk-ops service is unable to identify it.

[msterin]

So there is 'NFSDatastore' and also "NFSStore' ? just checking. Can you share the results of 'ls -l /vmfs/volumes` ?

[BaluDontu]

Also, it was able to identify only after the service vmdk-ops restart.

[root@xxxx:/vmfs/volumes] ls -l
total 2068
drwxrwxrwx    6 99964800 201           4096 Oct 17 06:48 3bdec0ec-219e0c95
drwxr-xr-x    1 root     root             8 Jan  1  1970 4df30da5-ad6ac294-f786-80c16e6cf13a
drwxr-xr-t    1 root     root          3920 Oct 25 23:04 4df30dac-644c9042-b81a-80c16e6cf13a
drwxr-xr-x    1 root     root             8 Jan  1  1970 4df30dac-829f533e-cff2-80c16e6cf13a
drwxr-xr-x    4 999900   201           4096 Jun 10 18:35 85e097de-d1f29be0
lrwxr-xr-x    1 root     root            17 Oct 28 19:54 NFSDatastore -> f3d216d7-219d7a2b
lrwxr-xr-x    1 root     root            17 Oct 28 19:54 NFSStore -> 85e097de-d1f29be0
lrwxr-xr-x    1 root     root            17 Oct 28 19:54 Sample -> 3bdec0ec-219e0c95
drwxr-xr-x    1 root     root             8 Jan  1  1970 bbe5f970-46eca9be-484d-dd18578281fe
lrwxr-xr-x    1 root     root            38 Oct 28 19:54 coke (1) -> vvol:064ac87e656a4505-97f9297bf3c4a0da
drwxr-xr-x    1 root     root             8 Jan  1  1970 d8916d7c-39123dc5-bc62-c7dcb54c918d
lrwxr-xr-x    1 root     root            35 Oct 28 19:54 datastore1 -> 4df30dac-644c9042-b81a-80c16e6cf13a
drwxr-xr-x   93 1000371  1010         12288 Oct 25 18:38 f3d216d7-219d7a2b
drwxr-xr-x    1 root     root           512 Oct 28 19:54 vvol:064ac87e656a4505-97f9297bf3c4a0da

[pdhamdhere]

def get_datastores():

    if datastores != None:  <<<< This is an issue.
        return datastores

Since datastores is not null, we do not rescan. We should. When you restart vmdksops, it rescans and re-populates datastores.

[pdhamdhere]

@BaluDontu Good catch. Do you want to do a PR?

[msterin]

We should not rescan, DS are cached for a reason - there are way too many cases when we fetch the list so there is a significant perf impact. In linux I'd use iNotify, in esx I hope we can get a notification when new DS is added ?

-- Mark
Sent from my phone. Excuse the brevity and the typos

On Fri, Oct 28, 2016 at 1:20 PM, Prashant Dhamdherenotifications@github.com wrote:
@BaluDontu Good catch. Do you want to do a PR?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[BaluDontu]

Yes. I would agree with Mark. We could use the inotify python package to watch a specific directory for create/remove operations (in our case /vmfs/volumes) and then only repopulate the datastores. If this is not a high priority PR, I can work on this.

[BaluDontu]

I would also like to share few more things on my analysis.

1. Using esxcli commands to create the datastore doesn't reproduce the issue.

For example,
esxcli storage nfs add --host=10.162.210.32 --share=/dbc/pa-dbc1132/baominw --volume-name=nfsstore-baominw --readonly

vmdk-ops service was able to identify the datastore "nfsstore-baominw" without any restart.

1. However using the VC to create a NFS datastore, I was able to reproduce this issue.

vmdk-ops service was not able to identify the datastore "nfsstore-baominw". I need to restart the service for vmdk-ops to identify this newly created datastore.

For both the above scenarios, I can see "nfsstore-baominw" volume created under /vmfs/volumes/.

I didn't understand why it behaves differently in these 2 cases. However a retry to get the latest datastores on failure would solve the problem. I want to check with you guys on why this could occur. I will also try it from my end.

[pdhamdhere]

Fixed with #714