Hetzner Germany has very fast and not expensive rentals of Hardware servers available. In order to communicate internally via private network between Xen-DOMUs and DOM0, normally one would install Xen DOM0 network with bridge networking as follows:
DOM0:xenbr0(eth0) --- bridging==>> DOMUs:eth0
DOM0:xenbr1(dummy0) ---bridging==>> DOMUs:eth1

Because of the configuration of the network switches at Hetzner, one hardware server can have multiple IPs but only one MAC address (MAC of eth0 in DOM0). This means that Bridge networking for Internet connection (eth0) doesn’t work for multiple DOMUs, each one having its own IP AND MAC address.
The solution is to use routing for Internet access and bridging for private LAN as follows:
DOM0:eth0 --- routing===>> DOMUs:eth0
DOM0:xenbr1(dummy0) --- bridging==>> DOMUs:eth1

Note: The DISADVANTAGE of this solution is that DOM0 must use one IP from the subnet provided by hetzner to be used as a gateway for the running DOMUs to allow them to communicate with the Internet. In this case the following IP subnet of 8 IPs provided by by Hetzner could be for example:
CIDR Subnet:
Network addr: (unusable by DOMUs hosts)
Gateway addr: (used as gateway for DOMUs, unusable by DOMUs hosts)
DOMUs usable IPs: - (5 IPs)
Broadcast addr: (unusable for DOMUs hosts)

This means out of 8 IPs you got as a subnet from Hetzner you can only run 5 DOMUs in this Xen environment if each DOMU needs to have its own Internet reachable IP.


We will first install XEN in the main hardware server. This means installing the hypervisor, xen aware kernel and xen tools. This can be done by a installing the following packages and a few favorite tools 🙂
apt-get install xen-linux-system xen-tools bridge-utils mc ssh fail2ban ethtool
Debian Wheezy uses Grub 2 and as default boot manager. It lists normal kernels first, and then, if the xen kernel is installed, lists the Xen hypervisor and its kernels. You need to change this to cause Grub 2 to prefer to boot Xen as default kernel. It is done by changing the priority of Grub’s Xen configuration script (20_linux_xen) to be higher prority than the standard Linux config (10_linux). This is most easily done using dpkg-divert:
dpkg-divert --divert /etc/grub.d/08_linux_xen --rename /etc/grub.d/20_linux_xen
After any update to the Grub configuration you must apply the configuration by running:
Disable Xendomains save & restore
We disable the saving and restore feature of DOMUs mostly because my experience is that this feature doens’t always work well. I prefer to do the shutdown of each DOMU manually before rebooting DOM0, then after reboot of DOM0, restart each individual DOMU using a @reboot cron job for example:
# This will start 2 virtual machines 60 sec after reboot of DOM0
@reboot /bin/sleep 60; /usr/sbin/xl create /etc/xen/DOMU1.cfg; /usr/sbin/xl create /etc/xen/DOMU2.cfg
This way if power failure happens or anything that forces an unattended reboot of DOM0, all the DOMUs will automatically restart after reboot.

Now the disabling of the automatic Save/Restore of DOMUs:
Edit /etc/default/xendomains


Add the dummy network interface module
echo dummy >> /etc/modules
modprobe dummy

Network configuration
Edit file: /etc/network/interfaces
(Note: here you’ll need to adapt your own IPs etc. in this file)
# Loopback device:
auto lo
iface lo inet loopback
# device: eth0
auto eth0
iface eth0 inet static
iface eth0 inet6 static
address 2a01:4f7:192:4213::2
netmask 64
gateway fe80::1
# Used exclusively as Gateway for DOMUs for this subnet. Unfortunately losing one IP for Gateway purposes.
auto eth0:gw1
iface eth0:gw1 inet static
# Internal private network to DOMUs
iface dummy0 inet manual
auto xenbr1
iface xenbr1 inet static
bridge_ports dummy0
#other possibly useful options in a virtualized environment
bridge_stp off # disable Spanning Tree Protocol
bridge_waitport 0 # no delay before a port becomes available
bridge_fd 0 # no forwarding delay
post-up ethtool -K xenbr1 tx off
post-up ip link set xenbr1 promisc off

Switch to the XL Xen ToolStack
Edit /etc/default/xen
WARNING: The above entry is small ‘XL’ and not small ‘X1’ !!

Edit /etc/xen/xl.conf and make sure the entries are as follows:
# automatically balloon down dom0 when xen doesn't have enough free
# memory to create a domain
# full path of the lockfile used by xl during domain creation
# default vif script.

Note: Here we use a script which will set routing for eth0 and bridging for dummy0.
Create it.
touch /etc/xen/scripts/vif-route_eth0-bridge_dummy0
chmod 755 /etc/xen/scripts/vif-route_eth0-bridge_dummy0

Edit the file /etc/xen/scripts/vif-route_eth0-bridge_dummy0
# Custom vif script which allows to combine routing for Internet and bridging for internal LAN
dir=$(dirname "$0")
IFNUM=$(echo ${vif} | cut -d. -f2)
if [ "$IFNUM" = "0" ] ; then
"$dir/vif-route" "$@"
"$dir/vif-bridge" "$@"

Edit the file /etc/xen/xend-config.sxp
and make sure the already existing entries are disabled with ‘#’ and new lines entered as follows:
#(vif-script vif-bridge)
(network-script dummy)
#(vif-script vif-route)
(vif-script vif-route_eth0-bridge_dummy0)
# make sure DOM0 has enough memory
(dom0-min-mem 2048)

Setup the IP forwarding and ARP proxying in kernel:
Edit the file /etc/sysctl.conf
Either un-comment or add the following lines:
# Uncomment the next line to enable packet forwarding for IPv4
# ARP Proxying
net.ipv4.conf.eth0.proxy_arp = 1

To make this change take effect immediately run:
sysctl -p /etc/sysctl.conf
Finally, before we reboot the system we need to make sure we activate the proper toolstack and related features at boot time by running the following commands:
update-rc.d xendomains defaults
update-rc.d xen defaults
/etc/init.d/xen restart
/etc/init.d/xendomains restart

DOMUs Configuration

If your DOMUs configurations are set to use pygrub as boot loader,
then make sure the path to pygrub in the DOMU configuration file is correct as follows:
bootloader = '/usr/lib/xen-4.1/bin/pygrub'
In the same DOMU configuration file, make sure you are using a non duplicated MAC addresses with the network interfaces assignment for example:
vif = [ 'ip=,mac=00:16:34:D7:9C:F4' , 'ip=,mac=00:16:3E:D7:9C:F6',bridge=xenbr1]
Note: The first IP doesn’t need any bridge since it is routing controlled, the internal LAN is bridged with xenbr1 though.

NOTE:If you want to use the pyGrub as boot loader for each individual DOMUs which makes the DOMUs kernel independant from the DOM0, see the following article: