SmartOS Overlays

WORK IN PROGRESS

Overlays

Definition from overlay man page:

 Overlay devices are a GLDv3 device that allows users to create overlay
 networks that can be used to form the basis of network virtualization and
 software defined networking.  Overlay networks allow a single physical
 network, often called an underlay network, to provide the means for
 creating multiple logical, isolated, and discrete layer two and layer
 three networks on top of it.

To put it simply, overalys are used to create isolated network at the software level. You then assign your VMs with these virtual network devices. VMs don't know that NICs they are using are virtual, and they don't care :) The advantage is that even though all VMs are using the same hardware network device, they don't see each other's traffic.

Working with overlays

Unless stated otherwise everything here is done in the Global zone.

Creating overlays

Overlays can be created manually with dladm create-overlay command. On top of this overlay a vNIC is created, which is then used by VMs/zones. However, overlays will be created automagically, if you do some preparation before you use vmadm create -f vm_properties.json to create VM. As it happens, vmadm is smart little cookie and it can create overlays during VM creation, you just have to give it some incentive :)

First of all you need to define properties of the overlays to be created. This is done in /var/run/smartdc/networking/overlay_rules.json. The file has key-value form, where key is the name of the overlay and value is list of the parameters, that have same syntax and meaning as they have in dladm create-overlay command.

/var/run/smartdc/networking/overlay_rules.json example configuration:

{
"my_overlay1": "-e vxlan -p vxlan/listen_ip=192.168.100.100,vxlan/listen_port=4790 -s files -p files/config=/var/run/smartdc/networking/my_overlay1.json -p mtu=1400",
"my_overlay2": "-e vxlan -p vxlan/listen_ip=192.168.200.200,vxlan/listen_port=4791 -s files -p files/config=/var/run/smartdc/networking/my_overlay2.json -p mtu=1400",
"my_overlay3": "-e vxlan -p vxlan/listen_ip=0.0.0.0,vxlan/listen_port=4790 -s files -p files/config=/var/run/smartdc/networking/my_overlay3.json -p mtu=1400"
}

Here is the summary of the used parameters, so you don't have to go running to man dladm :)

• -e - encapsulation method (vxlan in this case)
• -s - search plugin used to determine where the packet should be sent
• -p - property specifying overlay
  • vxlan/listen_ip - the IP on which vxlan overlay traffic is sent (0.0.0.0 to listen on all interfaces)
  • vxlan/listen_port - the port on which vxlan overlay traffic is sent
  • files/config - specifying file where files search plugin should search for ARP information
  • mtu - describes MTU of the overlay and thus MTU on this network, VMs on this overlay may not exceed this value

The 0.0.0.0 interface

The fact that you can define overlay on all interfaces means that you can have VMs on internal network communicating with VMs on SmartOS hosts that are located elsewhere on the Internet. To keep communication private deploy IPSec. For tutorial how to do this on SmartOS see Enabling IPSec on SmartOS.

Search plugin

Search plugin is managed by the service called varpd i.e. virtual ARP daemon. This service is responsible for determining where to send which packet, based on information from files/config file. If the entry is added or deleted from files/config, the varpd service MUST be restarted, not just refreshed (refresh is not implemented in the service config anyhow, but svcadm refresh network/varpd will not fail).

---

WARNING: Both files (/var/run/smartdc/networking/overlay_rules.json and /var/run/smartdc/networking/my_overlay.json) must be valid JSON files, otherwise your varpd service will fail to start. The error message isn't going to tell you that JSON is invalid :) Check your files with json command on SmartOS before deploying any VMs.

---

---

WARNING: The /var/run/smartdc/networking/overlay_rules.json and /var/run/smartdc/networking/my_overlay.json are not persistent. To make your overlays persistent over reboots place this file in /opt/custom/networking/ and right after the system boots, but before you create any VMs, do one of the following:

1. Create symbolic link to this file
2. Copy file from permanent location to /var/run/smartdc/networking/overlay_rules.json
3. Set Variables OVERLAY_RULES, OVERLAY_RULES_FILE, and OVERLAY_RULES_DIR This is not an option, since some tools (e.g. nictagadm) have hard-coded path to the overlay_fules.json file. Thus they will not be able to list overlays defined in some other file location.

---

Connecting VM to an overlay

Once you have created your overlays you can start connecting VMs. This is done by setting interface in the nics section of the VM definition json. It's important to set:

• nic_tag - the name of your overlay
• ip - IP address (VMs on the same overlay must be on the same subnet)
• netmask - mask of your subnet
• mtu - MTU must be that of the overlay set in overlay_rules.json (if set to some other value it will not work)

{
 "brand": "joyent",
 "image_uuid": "643de2c0-672e-11e7-9a3f-ff62fd3708f8",
 "alias": "test02",
 "hostname": "test02",
 "max_physical_memory": 512,
 "quota": 20,
 "resolvers": ["8.8.8.8"],
 "nics": [
  {
    "interface": "net0",
    "nic_tag": "fileoverlay/2233",
    "ip": "10.88.88.53",
    "netmask": "255.255.255.0",
    "mtu": "1400"
  }
 ]
}

VM is created the usual way, with the command:

vmadm create -f vm_properties.json

Adding VM to file/config

When your VM is up and running, you need to tell other VMs how to reach it. This is done by creating/editing /var/run/smartdc/networking/fileoverlay.json, which was specified in the overlay_file.json definition file. It's important to know:

• the file MUST be valid JSON, otherwise varpd service will not read it properly.
• new VM entry must be added on every SmartOS host, which has the overlay defined (on the host with the VM you don't have to add this entry, however it doesn't hurt to do so :) )

The entries have following structure:

"XX:XX:XX:XX:XX:XX":{
     "arp": "10.88.88.88",
     "ip": "10.0.60.188",
     "port": 4789
     }

Values:

• "XX:XX:XX:XX:XX:XX" - MAC address of the new VM
• "arp" - IP address of the VM
• "ip" - IP address of the host on which overlay is listening i.e. vxlan/listen_ip from overlay_file.json
• "port" - vxlan/listen_port defined for the overlay from overlay_file.json

Deleting overlays

Overlays will continue to exist even after last VM connected to the overlay is deleted/stoped. This can be problem if you want to change the properties of the connection. First stop/delete all VMs connected to overlays, then do:

dladm delete-overlay my_overlay1