IT Tips and Tricks

The most common way of configuring the networking in Xen environment is by using bridges.
In the case of servers rented at Hetzner provider (Germany) this would not work because the infrastructure is allowing only one MAC address per server. It allows for multiple IPs but only one MAC address.
To circumvent this situation we are then obliged to use the ‘Routed’ networking for eth0 explained in:
http://tipstricks.itmatrix.eu/?p=462. This explained method uses also ‘routing’ method for internal VLAN.
This works for Xen 3.2.1 environment BUT!! if you installed Xen 4.0.x from Debian Squeeze the routing method for the internal LAN doesn’t work as with Xen 3.2.1. (At least I could not get it to work).
One solution is to do a mixed network environment:
ETH0 in ‘Routed’ (needed in Hetzner servers)
DUMMY0 in Bridged (Needed in Xen 4.0.x)
This will create a bridge in Xen DOM0 and make available an interface(eth1) in all the DOMx (in our case ‘vsystem1’).
(This howto was inspired from the link: http://wiki.xensource.com/xenwiki/XenNetRoutingWithPrivateNetwork)

NOTE:
If you want to do the same with a real interface eg. eth1 the just omit the step 1 below and replace the word ‘dummy0’ with ‘eth1’

Building a dummy interface and bridge in DOM0

1) Add the dummy interface driver to the auto-loaded moludes
echo dummy >> /etc/modules
2) Configure the network interface:
/etc/network/interfaces
auto dummy0
iface dummy0 inet static
address 192.168.100.1
netmask 255.255.255.0

3) Bring up the dummy0 interface:
ifup dummy0
4) Create a network settings wrapper:
/etc/xen/scripts/network-route-eth0_bridge-dummy0
#!/bin/sh
dir=$(dirname "$0")
"$dir/network-route" "$@" netdev=eth0
"$dir/network-bridge" "$@" netdev=dummy0
echo 1 >/proc/sys/net/ipv4/ip_forward

5) Set the running rights to the script
chmod 755 /etc/xen/scripts/network-route-eth0_bridge-dummy0
6) Instead of using the default network-script use the above new wrapper script:
/etc/xen/xend-config.sxp
(network-script network-route-eth0_bridge-dummy0)
7) Create manually the bridge for the dummy0 interface for now instead of booting.
(Because of the wrapper script it will be created automatically at boot-up)
Run the command:
/etc/xen/scripts/network-bridge start netdev=dummy0 antispoof=no
You should get the following message and then the normal shell prompt:
'Waiting for pdummy0 to negotiate link.'
8)Check if the new bridge is present:
ifconfig pdummy0
Good example of result:
pdummy0 Link encap:Ethernet HWaddr d2:1b:97:ac:b0:74
inet6 addr: fe80::d01b:97ff:feac:b074/64 Scope:Link
UP BROADCAST RUNNING NOARP PROMISC MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:12 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:888 (888.0 B)

Create the vif script which will set the vifxx.0 as ‘routed’ and vifxx.1 as bridged.(Used to connect the network DOMu networking)
9) Create a network vifxx settings wrapper:
/etc/xen/scripts/vif-route-eth0_bridge-dummy0
#!/bin/sh
# Custom vif script which allows to combine routing for Internet and bridging for internal LAN
dir=$(dirname "$0")
IFNUM=$(echo ${vif} | cut -d. -f2)
if [ "$IFNUM" = "0" ] ; then
"$dir/vif-route" "$@"
else
"$dir/vif-bridge" "$@"
fi

10) Set the running rights to the script
chmod 755 /etc/xen/scripts/vif-route-eth0_bridge-dummy0
11) Instead of using the default ‘vif-script’ use the above new wrapper script:
/etc/xen/xend-config.sxp
(vif-script vif-route-eth0_bridge-dummy0)
Modify the file /etc/xen/scripts/vif-common.sh
Around line 135 replace the following line
ip addr show "$1" | awk "/^.*inet.*$1\$/{print \$2}" | sed -n '1 s,/.*,,p'
with
ip -4 -o addr show primary dev $1 | awk '$3 == "inet" {print $4; exit}' | sed 's#/.*##'


DOMU network configuration

1) Configure your Xen Domx virtual machine for eth1:
(It should be written all in one line)
/etc/xen/vsystem1.cfg
vif = [ 'ip=91.184.57.186,mac=00:16:3E:D7:9C:F4,bridge=eth0' , 'ip=192.168.100.100,mac=00:16:3E:D7:9C:F6,bridge=dummy0']

2) Bring down your vsystem1 DOMx machine
xm shutdown vsystem1
3) Mount the virtual disk in loop (for configuring the eth1 interface in it)
(Here we are assuming the virtual disk is /Xen/domains/vsystem1/disk.img)
mkdir /mnt/vsystem1
mount -o loop,rw /Xen/domains/vsystem1/disk.img /mnt/vsystem1

4) Configure the eth1 in the virtual disk
vim /mnt/vsystem1/etc/network/interfaces
Add the following lines and save the file:
auto eth1
iface eth1 inet static
address 192.168.100.100
netmask 255.255.255.0
5) Unmount the virtual disk
umount /mnt/vsystem1
6) Start the virtual machine
xm create /etc/xen/vsystem1.cfg -c
7) Login as root and check that the eth1 is configured
ifconfig eth1

NOTE:
To configure more virtual machines to use eth1 repeat the above steps 9 to 15 for each virtual machine.

NOT TO FORGET:
– You’ll need to configure your firewall in the DOM0 to forward the packets from one machine to another
– Do change the MAC address for each virtual machine you configure this way
– Set the ip_forwarding in the kernel of DOM0
echo 1 >/proc/sys/net/ipv4/ip_forward