Skip to content

Files

Latest commit

 

History

History
251 lines (198 loc) · 8.38 KB

device_assignment.md

File metadata and controls

251 lines (198 loc) · 8.38 KB

Getting started with device assignment

Device assignment allows a VM to have direct access to HW without host/hyp intervention. AVF uses vfio-platform for device assignment, and host kernel support is required.

This document explains how to setup and launch VM with device assignments.

VM device assignment DTBO (a.k.a. VM DTBO)

For device assignment, a VM device assignment DTBO (a.k.a. VM DTBO) is required. VM DTBO is a device tree overlay which describes all assignable devices information. Information includes physical reg, IOMMU, device properties, and dependencies.

VM DTBO allows to pass extra properties of assignable platform devices to the VM (which can't be discovered from the HW) while keeping the VMM device-agnostic.

When the host boots, the bootloader provides VM DTBO to both Android and pvmfw.

When a VM boots, the VMM selectively applies the DTBO based from provided labels, describing the assigned devices.

Prepare VM DTBO

VM DTBO should be included in the dtbo partition. It should be in its own entry, and not together with any host OS's. See DTB/DTBO Paritions for partition format.

Write VM DTS for VM DTBO

DTBO is compiled from device tree source (DTS) with dtc tool. DTBO syntax explains basic syntax of DTS.

Here are details and requirements:

Describe assignable devices

VM DTBO should describe assignable devices and their labels.

  • VM DTBO should have assignable devices in the &{/}, so it can be overlaid onto VM DT. Assignable devices should be backed by physical device.
    • We only support overlaying onto root node (i.e. &{/}) to prevent unexpected modification of VM DT.
  • VM DTBO should have labels for assignable devices, so AVF can recognize assignable device list. Labels should point to valid 'overlayable' nodes.
    • Overlayable node is a node that would be applied to the base device tree when DTBO is applied.

Describe physical devices and physical IOMMUs

VM DTBO should describe a /host node which describes physical devices and physical IOMMUs. The /host node only describes information for verification of assigned devices, and wouldn't be applied to VM DT. Here are details:

  • Physical IOMMU nodes
    • IOMMU nodes must have a phandle to be referenced by a physical device node.
    • IOMMU nodes must have <android,pvmfw,token> property. The property describes the IOMMU token. An IOMMU token is a hypervisor-specific <u64> which uniquely identifies a physical IOMMU. IOMMU token must be constant across the VM boot for provisioning by pvmfw remains valid. The token must be kept up-to-date across hypervisor updates.
    • IOMMU nodes should be multi-master IOMMUs. (i.e. #iommu-cells = <1>)
  • Physical device nodes
    • Physical device nodes must have a <android,pvmfw,target> property that references an overlayable node. The overlayable node contains the properties that would be included in VM DT.
    • Physical device nodes must have <reg> property to provide physical regions.
    • Physical device nodes can optionally contain <iommus> property. The property is a prop-encoded-array and contains a number of (iommu phandle, SID) pairs.
      • IOMMU can be shared among devices, but should use distinct SIDs. Sharing the same IOMMU-SID pair among multiple devices isn't supported for now.

Describe dependencies

VM DTBO may have dependencies via phandle references. When a device node is assigned, dependencies of the node are also applied to VM DT.

When dependencies are applied, siblings or children nodes of dependencies are ignored unless explicitly referenced.

VM DTBO example

Here's a simple example device tree source with four assignable devices nodes.

/dts-v1/;
/plugin/;

/ {
    // host node describes physical devices and IOMMUs, and wouldn't be applied to VM DT
    host {
        #address-cells = <0x2>;
        #size-cells = <0x1>;
        rng {
            reg = <0x0 0x12f00000 0x1000>;
            iommus = <&iommu0 0x3>;
            android,pvmfw,target = <&rng>;
        };
        light {
            reg = <0x0 0x00f00000 0x1000>, <0x0 0x00f10000 0x1000>;
            iommus = <&iommu1 0x4>, <&iommu2 0x5>;
            android,pvmfw,target = <&light>;
        };
        led {
            reg = <0x0 0x12000000 0x1000>;
            iommus = <&iommu1 0x3>;
            android,pvmfw,target = <&led>;
        };
        bus0 {
            #address-cells = <0x1>;
            #size-cells = <0x1>;
            backlight {
                reg = <0x300 0x100>;
                android,pvmfw,target = <&backlight>;
            };
        };
        iommu0: iommu0 {
            #iommu-cells = <0x1>;
            android,pvmfw,token = <0x0 0x12e40000>;
        };
        iommu1: iommu1 {
            #iommu-cells = <0x1>;
            android,pvmfw,token = <0x0 0x40000>;
        };
        iommu2: iommu2 {
            #iommu-cells = <0x1>;
            android,pvmfw,token = <0x0 0x50000>;
        };
    };
};

// Beginning of the assignable devices. Assigned devices would be applied to VM DT
&{/} {  // We only allows to overlay to root node
    rng: rng {
        compatible = "android,rng";
        android,rng,ignore-gctrl-reset;
    };
    light: light {
        compatible = "android,light";
        version = <0x1 0x2>;
    };
    led: led {
        compatible = "android,led";
        prop = <0x555>;
    };
    bus0 {
        backlight: backlight {
            compatible = "android,backlight";
            android,backlight,ignore-gctrl-reset;
        };
    };
};

If you compile the above with dtc -@, then you'll get __symbols__ for free. __symbol__ has label of nodes, and it's required for the next step.

    // generated __symbols__
    __symbols__ {
        iommu0 = "/host/iommu0";
        iommu1 = "/host/iommu1";
        iommu2 = "/host/iommu2";
        rng = "/fragment@rng/__overlay__/rng";
        light = "/fragment@sensor/__overlay__/light";
        led = "/fragment@led/__overlay__/led";
        backlight = "/fragment@backlight/__overlay__/bus0/backlight";
    };

Prepare AVF assignable devices XML

AVF requires assignable device information to unbind from the host device driver and bind to VFIO driver. The information should be provided in an XML file at /vendor/etc/avf/assignable_devices.xml.

Here's example.

<devices>
    <device>
        <kind>sensor</kind>
        <dtbo_label>light</dtbo_label>
        <sysfs_path>/sys/bus/platform/devices/16d00000.light</sysfs_path>
    </device>
</devices>
  • <kind>: Device kind. Currently only used for debugging purposes and not used for device assignment.
  • <dtbo_label>: Label in the VM DTBO (i.e. symbol in __symbols__). Must be non-empty and unique in the XML.
  • <sysfs_path>: Sysfs path of the device in host, used to bind to the VFIO driver. Must be non-empty and unique in the XML.

List support assignable devices

In order to query list of the devices that can be assigned to a pVM, run the following command:

adb shell /apex/com.android.virt/bin/vm info

All supported assignable devices will be located under the "Assignable devices:" section of the output.

Boot with VM DTBO

Bootloader should provide VM DTBO to both Android and pvmfw.

Provide VM DTBO index in dtbo.img

Bootloader should provide the VM DTBO index with sysprop ro.boot.hypervisor.vm_dtbo_idx.. DTBO index represents DTBO location in dtbo.img.

Provide VM DTBO in the pvmfw config

For protected VM, bootloader must provide VM DTBO to the pvmfw. pvmfw sanitizes incoming device tree with the VM DTBO.

For more detail about providing VM DTBO in pvmfw, see: pvmfw/README.md

Launch VM with device assignment

We don't support client API yet in Android V, but you can use CLI to test device assignment. Note that host kernel support is required.

Specify --devices ${sysfs_path} when booting VM. The parameter can be repeated multiple times for specifying multiple devices.

Here's an example:

adb shell /apex/com.android.virt/bin/vm run-microdroid --devices /sys/bus/platform/devices/16d00000.light