Support a volume create -o"clone-from=<diskname>" option.

[govint]

This issue is to support a new volume create time option "clone-from" that allows a docker user to create a volume from an existing volume, either as a full copy (initially) or as a copy-on-write disk of the existing one.

Use cases:

1. User wants to take a backup of a disk - pause the container thats using the disk (to be cloned), create the clone and unpause the container. New clone is available as point-in-time snap of the disk. Backups like Veeam can also work this way than snapping a docker host VM that may have multiple unrelated containers and hence volumes. Rather than snap all those volumes, the user can create clones of specific disks to backup (for those containers needing backup) and then provide those disks to the backup app (say Veeam backup software in a container).
2. User wants to create multiple volume groups with specific attributes for each. User creates a volume with a specific set of attrs and then creates the rest as clones of the first one. Rather than the user specifying the attrs for each volume on creation. Yes, user can script this easily, but the first volume is like a template for the rest.

[pdhamdhere]

@govint Would you be okay if we use #60 to flush out overall design for snapshot, clone, backup? I would like to avoid bug sprawl as much as possible. Once we have design proposal, we can open separate tracking issues for individual features.

[govint]

For a future planned release. Sure, this issue isn't as much for doing backup as it allows users to clone disks via docker itself which isn't possible otherwise.

[brunotm]

Hello @govint @pdhamdhere ,

I have added basic cloning support at 44f91cc.
What do you think for a initial implementation?

From simple tests the only downside i noticed is the capacity info in docker volume inspect, the same doesn't happen in the admin cli. (i know they fetch information in different ways, but didn't investigate).

Also dependency tracking for deltas doesn't seems to be necessary, looks like that when deleting the parent it consolidates on the delta. Do you have available documentation about this behaviour?

Thanks!

[brunotm]

The output from the one basic test with diskformat=delta, which shows the consolidation on the clone volume after removal of master.

root@dev01:/# docker volume create -d vmdk --name master -o size=1gb
master

root@dev01:/# docker run -ti -v master:/mnt busybox
/ # for x in bin dev etc home root tmp usr var; do cp -a $x /mnt/; done
/ # dd if=/dev/zero of=/mnt/master.img bs=1M count=128
128+0 records in
128+0 records out
134217728 bytes (128.0MB) copied, 0.159301 seconds, 803.5MB/s
/ # df -hP|egrep "Filesystem|/mnt"
Filesystem                Size      Used Available Capacity Mounted on
/dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:1:0                               975.9M    130.4M    778.3M  14% /mnt


root@dev01:/# docker volume inspect master
[
    {
        "Name": "master",
        "Driver": "vmdk",
        "Mountpoint": "/mnt/vmdk/master",
        "Status": {
            "access": "read-write",
            "attach-as": "independent_persistent",
            "capacity": {
                "allocated": "183MB",
                "size": "1GB"
            },
            "clone-from": "None",
            "created": "Mon Oct 31 15:51:33 2016",
            "created by VM": "dev01",
            "datastore": "datastore2",
            "diskformat": "thin",
            "fstype": "ext4",
            "status": "detached"
        },
        "Labels": {},
        "Scope": "global"
    }
]
root@dev01:/# docker volume create -d vmdk --name clone1 -o clone-from=master -o diskformat=delta
clone1
root@dev01:/# docker volume inspect clone1
[
    {
        "Name": "clone1",
        "Driver": "vmdk",
        "Mountpoint": "/mnt/vmdk/clone1",
        "Status": {
            "access": "read-write",
            "attach-as": "independent_persistent",
            "capacity": {
                "allocated": "17MB",
                "size": "16MB"
            },
            "clone-from": "master",
            "created": "Mon Oct 31 15:56:51 2016",
            "created by VM": "dev01",
            "datastore": "datastore2",
            "diskformat": "delta",
            "fstype": "ext4",
            "status": "detached"
        },
        "Labels": {},
        "Scope": "global"
    }
]

Volume  Datastore   Created By VM  Created                   Attached To VM  Policy  Capacity  Used      Filesystem Type  Access      Attach As
------  ----------  -------------  ------------------------  --------------  ------  --------  --------  ---------------  ----------  ----------------------
master  datastore2  dev01          Mon Oct 31 15:51:33 2016  detached        N/A     1.00GB    183.00MB  ext4             read-write  independent_persistent
clone1  datastore2  dev01          Mon Oct 31 15:56:51 2016  detached        N/A     1.00GB    17.00MB   ext4             read-write  independent_persistent

root@dev01:/# docker run -ti -v clone1:/mnt busybox
/ # ls
bin         dev         etc         home        lost+found  master.img  root        tmp         usr         var
/ # dd if=/dev/zero of=/mnt/clone1.img bs=1M count=128
128+0 records in
128+0 records out
134217728 bytes (128.0MB) copied, 0.153344 seconds, 834.7MB/s
/ # ls
bin         clone1.img  dev         etc         home        lost+found  master.img  root        tmp         usr         var
/ # rm -rf /mnt/dev

root@dev01:/# docker volume inspect master
[
    {
        "Name": "master",
        "Driver": "vmdk",
        "Mountpoint": "/mnt/vmdk/master",
        "Status": {
            "access": "read-write",
            "attach-as": "independent_persistent",
            "capacity": {
                "allocated": "183MB",
                "size": "1GB"
            },
            "clone-from": "None",
            "created": "Mon Oct 31 15:51:33 2016",
            "created by VM": "dev01",
            "datastore": "datastore2",
            "diskformat": "thin",
            "fstype": "ext4",
            "status": "detached"
        },
        "Labels": {},
        "Scope": "global"
    }
]

root@dev01:/# docker volume inspect clone1
[
    {
        "Name": "clone1",
        "Driver": "vmdk",
        "Mountpoint": "/mnt/vmdk/clone1",
        "Status": {
            "access": "read-write",
            "attach-as": "independent_persistent",
            "capacity": {
                "allocated": "145MB",
                "size": "144MB"
            },
            "clone-from": "master",
            "created": "Mon Oct 31 15:56:51 2016",
            "created by VM": "dev01",
            "datastore": "datastore2",
            "diskformat": "delta",
            "fstype": "ext4",
            "status": "detached"
        },
        "Labels": {},
        "Scope": "global"
    }
]

[root@esx01:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py ls
Volume  Datastore   Created By VM  Created                   Attached To VM  Policy  Capacity  Used      Filesystem Type  Access      Attach As
------  ----------  -------------  ------------------------  --------------  ------  --------  --------  ---------------  ----------  ----------------------
master  datastore2  dev01          Mon Oct 31 15:51:33 2016  detached        N/A     1.00GB    183.00MB  ext4             read-write  independent_persistent
clone1  datastore2  dev01          Mon Oct 31 15:56:51 2016  dev01           N/A     1.00GB    81.00MB   ext4             read-write  independent_persistent



root@dev01:/# docker rm -vf $(docker ps -qa)
2141bb915e4e
3f605182dd16
root@dev01:/# docker volume rm master
master
root@dev01:/# docker volume inspect clone1
[
    {
        "Name": "clone1",
        "Driver": "vmdk",
        "Mountpoint": "/mnt/vmdk/clone1",
        "Status": {
            "access": "read-write",
            "attach-as": "independent_persistent",
            "capacity": {
                "allocated": "145MB",
                "size": "144MB"
            },
            "clone-from": "master",
            "created": "Mon Oct 31 15:56:51 2016",
            "created by VM": "dev01",
            "datastore": "datastore2",
            "diskformat": "delta",
            "fstype": "ext4",
            "status": "detached"
        },
        "Labels": {},
        "Scope": "global"
    }
]


[root@esx01:~] /usr/lib/vmware/vmdkops/bin/vmdkops_admin.py ls
Volume  Datastore   Created By VM  Created                   Attached To VM  Policy  Capacity  Used      Filesystem Type  Access      Attach As
------  ----------  -------------  ------------------------  --------------  ------  --------  --------  ---------------  ----------  ----------------------
clone1  datastore2  dev01          Mon Oct 31 15:56:51 2016  detached        N/A     1.00GB    145.00MB  ext4             read-write  independent_persistent


root@dev01:/# docker run -ti -v clone1:/mnt busybox
/ # cd /mnt/
/mnt # ls
bin         clone1.img  etc         home        lost+found  master.img  root        tmp         usr         var
/mnt # rm master.img
/mnt # mv tmp something
/mnt # ls
bin         clone1.img  etc         home        lost+found  root        something   usr         var


root@dev01:/# docker volume inspect clone1
[
    {
        "Name": "clone1",
        "Driver": "vmdk",
        "Mountpoint": "/mnt/vmdk/clone1",
        "Status": {
            "access": "read-write",
            "attach-as": "independent_persistent",
            "capacity": {
                "allocated": "145MB",
                "size": "144MB"
            },
            "clone-from": "master",
            "created": "Mon Oct 31 15:56:51 2016",
            "created by VM": "dev01",
            "datastore": "datastore2",
            "diskformat": "delta",
            "fstype": "ext4",
            "status": "detached"
        },
        "Labels": {},
        "Scope": "global"
    }
]

//CC @msterin

[govint]

I'm not sure we want to create a delta of the volume being cloned-from, or that should be a choice whether delta or a full clone of the volume is needed. Plus need a way to create deltas of just the disk alone vs. the VM as a whole.

The disk size issue I've raised a separate issue to handle that.

[brunotm]

I'm not sure we want to create a delta of the volume being cloned-from, or that should be a choice whether delta or a full clone of the volume is needed.

It's a choice, right now the default is the same for create (thin).

Plus need a way to create deltas of just the disk alone vs. the VM as a whole.

The code on brunotm.issue462 only clones the volume with virtualDiskManager.CopyVirtualDisk, there's no vm interaction.

The disk size issue I've raised a separate issue to handle that.
👍

[msterin]

Thanks @brunotm . I have a few emergencies to take care of and then will take a look.
The first feeling is - I am not sure backup vendors do back ups off disconnected VMDKs (I'd expect per-VM pivoting in backup UIs) . But the PR itself is great, as it would allow to provide multiple Docker engines READ-only access to a volume.

[brunotm]

@msterin
My main case for this PR is not backup(but it helps there), more of fast app environment and data cloning, and ability to rollback (fast). Also multiple delta rw clones :)

Luck with the emergencies.

[msterin]

I am back for now. Yes. I see the use case. checkpoint it nice. The code is good too. A few nits (the biggest one being the code from lower layer creeping into executeRequest() - I think misc. details eg. clone or DB handling, should be within *VMDK() functions, e.g. createVMDK, not in executeRequest. Are you planning to submit a PR ?

[brunotm]

@msterin
Yes, The plan was to do it after having the unit tests in place but the kV issue came into attention. I'll try to find time later today to write tests and see if the code need any changes.

[govint]

Supported via #726