Xen 3.2, CentOS 5.1 and NAT HOWTO

The default configuration for networking with Xen DomUs is bridging the network interface to the Dom0. This will make a DomU network interface appear on the LAN just like any other machine.

This HOWTO explains how you can change the configuration to masquerade all the network interfaces of the DomUs behind the Dom0. This will turn the Dom0 into a NAT box, which may be useful if you want to build a cluster that doesn't expose itself to the network as much.

Virtual machines

A virtual machine description is a snippet of Python code that describes the components of the machine. Here is an example

name = "ce"
bootloader = "/usr/bin/pygrub"
vcpus=1
on_reboot = 'restart'
on_crash = 'destroy'
memory = "1024"
disk = [ 'phy:VolGroup00/ce,xvda,w' ]
vif = [ 'mac=00:16:3E:C2:AA:77' ]

name

each vm should have a unique name, for the sake of Xen's administration.

bootloader

an alternative to handing Xen a straight kernel/initrd to boot is to use a bootloader script that will do something smart. For instance, the pypxeboot bootloader mimics the pxeboot behaviour by asking the DHCP server which kernel to download, while the pygrub bootloader (included in Xen) behaves much like GRUB. This mimicry is necessary because the machines, which are paravirtualized, do not have a real BIOS system underneath.

vcpus

The number of CPU cores to assign to the machine.

on_reboot/crash

What to do in certain situations. Although 'destroy' sounds rather spectacular, it is only the internal machine description that is being dropped, including the memory contents; the disk contents persists and thus it is more like a powerdown.

memory

The amount of RAM (in MB) to assign to the machine.

disk

The assignment of block devices; in this case a mapping is made from a physical block device to an internal name, with write access (w). The alternative is to map (image) files, which has a little more overhead because of the file system.

vif

The virtual network devices. The number of options here is endless, but a 'real' machine only cares about its MAC address, and that is all we need. Some care goes in to keeping these unique. The first three bytes are fixed; this is the IEEE Organizationally Unique Identifier (OUI) handed out to Xensource, Inc.

NAT

The IP adresses behind the NAT box are assigned dynamically with DHCP from the private 10.x.x.x range. A lightweight DHCP/DNS server called dnsmasq is used to manage the assignments. Configuration files for dnsmasq can be placed in /etc/dnsmasq.d/ and must have the extension .conf.

This is the full text of /etc/dnsmasq.d/xencluster.conf.

dhcp-range=testbed,10.0.0.1,static,255.0.0.0,infinite
read-ethers
leasefile-ro
except-interface=eth0

Here's the configuration explained.

dhcp-range

every configuration must have at least one range of IP addresses to hand out, even if only static assignments are being done. The 'static' keyword replaces the end address of the range and means that only static addresses will be given. The lease time is infinite.

read-ethers

this directive tells dnsmasq to read the file /etc/ethers, to find out which IP addresses belong to which MAC addresses. This file will play an role later on.

leasefile-ro

don't bother with any leases; useful for just doing the static thing.

except-interface

tells dnsmasq to ignore DHCP request originating from the physical interface of the machine (so as not to conflict with other DHCP servers on the net).

With this configuration dnsmasq will respond to DHCP requests coming from the virtual interfaces, by handing out addresses as listed in /etc/ethers. Here's a snippet of what this file looks like.

00:16:3E:0D:B0:96 10.0.5.1
00:16:3E:96:71:4E 10.0.10.1
00:16:3E:CF:26:36 10.0.1.1
00:16:3E:0C:65:7F 10.0.11.1
00:16:3E:96:CD:76 10.0.3.1
00:16:3E:19:39:B9 10.0.12.1
00:16:3E:C2:AA:77 10.0.2.1
00:16:3E:48:88:75 10.0.6.1

It is up to the manager of the virtual machines to assign unique mac addresses. Looking at the example description above, you can see the the machine called 'ce' will get IP address 10.0.2.1.

Dnsmasq also acts as a caching DNS server, and in particular it looks at /etc/hosts to resolve hostnames on its local network. So a line in /etc/hosts like

10.0.2.1 ce.testbed ce

will make the 'ce' virtual machine find its own DNS name ce.testbed.

Automating hosts and ethers

Keeping the VM descriptions, /etc/ethers and /etc/hosts in sync is an arduous task. Thankfully this can be fully automated with some scripting.

Network interfaces in Xen are generated on the fly. The line

vif = [ 'mac=00:16:3E:C2:AA:77' ]

will make eth0 appear with said MAC address. In the Dom0, a virtual interface is created as its counterpart. The two are logically connected by an imaginary crossover cable.

On the DomU side, you get

eth0      Link encap:Ethernet  HWaddr 00:16:3E:C2:AA:77  
          inet addr:10.0.2.1  Bcast:10.255.255.255  Mask:255.0.0.0

And in the Dom0, you see

vif5.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF  
          inet addr:10.0.2.128  Bcast:0.0.0.0  Mask:255.255.255.255

Two scripts handle the management of these interfaces. One is called the network-script, which is just run once upon starting the xend daemon; the other is the vif-script, which is called each time a virtual interface needs to be created or torn down. The file names of the scripts to use are defined in the Xen daemon configuration file, /etc/xen/xend-config.xsp.

Install the new scripts network-nat-dns, vif-nat-dns and the replacement vif-common.sh. in /etc/xen/scripts/

Edit /etc/xen/xend-config.xsp and replace the lines

(network-script network-bridge)
(vif-script vif-bridge)

by

(network-script network-nat-dns)
(vif-script     vif-nat-dns)

and restart the xend daemon

/etc/init.d/xend restart