msgbartop
MAC OS X, Linux, Windows and other IT Tips and Tricks
msgbarbottom

15 Jun 11 Create a mixed (routed & bridged) private VLAN for Xen Virtual Machines

The most common way of configuring the networking in Xen environment is by using bridges.
In the case of servers rented at Hetzner provider (Germany) this would not work because the infrastructure is allowing only one MAC address per server. It allows for multiple IPs but only one MAC address.
To circumvent this situation we are then obliged to use the ‘Routed’ networking for eth0 explained in:
//tipstricks.itmatrix.eu/?p=462. This explained method uses also ‘routing’ method for internal VLAN.
This works for Xen 3.2.1 environment BUT!! if you installed Xen 4.0.x from Debian Squeeze the routing method for the internal LAN doesn’t work as with Xen 3.2.1. (At least I could not get it to work).
One solution is to do a mixed network environment:
ETH0 in ‘Routed’ (needed in Hetzner servers)
DUMMY0 in Bridged (Needed in Xen 4.0.x)
This will create a bridge in Xen DOM0 and make available an interface(eth1) in all the DOMx (in our case ‘vsystem1’).
(This howto was inspired from the link: //wiki.xensource.com/xenwiki/XenNetRoutingWithPrivateNetwork)

NOTE:
If you want to do the same with a real interface eg. eth1 the just omit the step 1 below and replace the word ‘dummy0’ with ‘eth1’

Building a dummy interface and bridge in DOM0

1) Add the dummy interface driver to the auto-loaded moludes
echo dummy >> /etc/modules
2) Configure the network interface:
/etc/network/interfaces
auto dummy0
iface dummy0 inet static
address 192.168.100.1
netmask 255.255.255.0

3) Bring up the dummy0 interface:
ifup dummy0
4) Create a network settings wrapper:
/etc/xen/scripts/network-route-eth0_bridge-dummy0
#!/bin/sh
dir=$(dirname "$0")
"$dir/network-route" "$@" netdev=eth0
"$dir/network-bridge" "$@" netdev=dummy0
echo 1 >/proc/sys/net/ipv4/ip_forward

5) Set the running rights to the script
chmod 755 /etc/xen/scripts/network-route-eth0_bridge-dummy0
6) Instead of using the default network-script use the above new wrapper script:
/etc/xen/xend-config.sxp
(network-script network-route-eth0_bridge-dummy0)
7) Create manually the bridge for the dummy0 interface for now instead of booting.
(Because of the wrapper script it will be created automatically at boot-up)
Run the command:
/etc/xen/scripts/network-bridge start netdev=dummy0 antispoof=no
You should get the following message and then the normal shell prompt:
'Waiting for pdummy0 to negotiate link.'
8)Check if the new bridge is present:
ifconfig pdummy0
Good example of result:
pdummy0 Link encap:Ethernet HWaddr d2:1b:97:ac:b0:74
inet6 addr: fe80::d01b:97ff:feac:b074/64 Scope:Link
UP BROADCAST RUNNING NOARP PROMISC MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:12 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:888 (888.0 B)

Create the vif script which will set the vifxx.0 as ‘routed’ and vifxx.1 as bridged.(Used to connect the network DOMu networking)
9) Create a network vifxx settings wrapper:
/etc/xen/scripts/vif-route-eth0_bridge-dummy0
#!/bin/sh
# Custom vif script which allows to combine routing for Internet and bridging for internal LAN
dir=$(dirname "$0")
IFNUM=$(echo ${vif} | cut -d. -f2)
if [ "$IFNUM" = "0" ] ; then
"$dir/vif-route" "$@"
else
"$dir/vif-bridge" "$@"
fi

10) Set the running rights to the script
chmod 755 /etc/xen/scripts/vif-route-eth0_bridge-dummy0
11) Instead of using the default ‘vif-script’ use the above new wrapper script:
/etc/xen/xend-config.sxp
(vif-script vif-route-eth0_bridge-dummy0)
Modify the file /etc/xen/scripts/vif-common.sh
Around line 135 replace the following line
ip addr show "$1" | awk "/^.*inet.*$1\$/{print \$2}" | sed -n '1 s,/.*,,p'
with
ip -4 -o addr show primary dev $1 | awk '$3 == "inet" {print $4; exit}' | sed 's#/.*##'

DOMU network configuration


1) Configure your Xen Domx virtual machine for eth1:
(It should be written all in one line)
/etc/xen/vsystem1.cfg
vif = [ 'ip=91.184.57.186,mac=00:16:3E:D7:9C:F4,bridge=eth0' , 'ip=192.168.100.100,mac=00:16:3E:D7:9C:F6,bridge=dummy0']

2) Bring down your vsystem1 DOMx machine
xm shutdown vsystem1
3) Mount the virtual disk in loop (for configuring the eth1 interface in it)
(Here we are assuming the virtual disk is /Xen/domains/vsystem1/disk.img)
mkdir /mnt/vsystem1
mount -o loop,rw /Xen/domains/vsystem1/disk.img /mnt/vsystem1

4) Configure the eth1 in the virtual disk
vim /mnt/vsystem1/etc/network/interfaces
Add the following lines and save the file:
auto eth1
iface eth1 inet static
address 192.168.100.100
netmask 255.255.255.0

5) Unmount the virtual disk
umount /mnt/vsystem1
6) Start the virtual machine
xm create /etc/xen/vsystem1.cfg -c
7) Login as root and check that the eth1 is configured
ifconfig eth1

NOTE:
To configure more virtual machines to use eth1 repeat the above steps 9 to 15 for each virtual machine.

NOT TO FORGET:
– You’ll need to configure your firewall in the DOM0 to forward the packets from one machine to another
– Do change the MAC address for each virtual machine you configure this way
– Set the ip_forwarding in the kernel of DOM0
echo 1 >/proc/sys/net/ipv4/ip_forward

Reader's Comments

  1.    

    Hi,Juste to your information, a private vlan is also a Cisco implementation of Vlan : //www.cisco.com/en/US/tech/tk389/tk814/tk840/tsd_technology_support_sub-protocol_home.html:-)

    Reply to this comment
  2.    

    The Xen4.0 Debian Squeeze route doesn’t work because there are some errors in vif route / network route scripts. Compare those files in Xen3.2.1 versus Xen 4. I don’t remember well, but I this I was downloding the original xen4 from xen site and compare the script.Anyway, is an error in network scripts in debian packages and I managed to work with VM’s in routed network.My problem now in another: how to use routed network, which is a must, because my isp gave me a subnet which I had to route myself, and in the mean time create and use fully virtualized vm’s (HVM)?I’m stuck with this for a week, but no solution until now.Thanks

    Reply to this comment
  3.    

    Hey, i have a hetzner server also, we needed this guide to create an internal networks for VM’s to communicate with each other.We set it up and got it working, but it’s intermittant, e.g it’ll work for a few tries, then stop or sometimes it’ll take forever to start connecting here is a sample…64 bytes from 10.0.0.5: icmp_seq=76 ttl=63 time=1292 ms64 bytes from 10.0.0.5: icmp_seq=77 ttl=63 time=292 ms64 bytes from 10.0.0.5: icmp_seq=78 ttl=63 time=0.192 mslook at the first ping.Any idea how we can fix this?Bradley

    Reply to this comment
    •    

      Hi Bradley, unfortunately for you and fortunately for me I didn’t get this problem or at least I didn’t notice it.
      Maybe I should check. In any case this is what I would do:
      I would first check the routing table and see if it’s correct.
      If OK then I would make sure you don’t have any iptables rules that interfere with this LAN communication.
      Also remember that there is a strange phenomenon happening with this type of routing, where packets being sent from the DOM0 to DOMu via this internal Virtual-LAN are seen(source address) as coming from the Internet IP of DOM0. This can make the DOMu quite confused and send the response back through the default gateway (via eth0) of DOMu, which in turns delivers the packets to DOM0 via Internet to the eth0 interface.
      A good firewall would most likely detect this as a spoofing packet and chuck it, because its source address would be an invalid Internet address.
      I guess some iptables or routing rules in DOM0 and possibly in DOMu could be cooked to correct that. I wish you good luck with it.

      If you find a way to solve it, please post it as comment to this article. This could help me and others if the problem ever occurred.

      Reply to this comment

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: