Introduction:
When I upgraded from Debian Lenny Xen 3.2.1 to Squeeze Xen 4.0, except for the DOMu hard disks names (from /dev/hdax to /dev/xvdax), I had very little changes to do and all went pretty well. Lately I wanted to upgrade from Squeeze Xen 4.0 to Wheezy Xen 4.1 and I had a few very unexpected and unwanted changes to do, here is what I ended up with which works.
Note: The difficulties may be due to some hardware incompatibility, I don’t know (eth0 & eth1: e1000e Intel(R) PRO/1000 Network). The following solution I finally got is dirty and I would appreciate if someone would help me clarify what is really going on and find a better and more elegant solution.
Constellation:
Here is what I needed:
eth0: connected to internet
eth1: connected to internal LAN (private LAN between hardware servers and the virtual machines)
In Squeeze the real interface was called ‘peth0’ and the bridge was called ‘eth0’
The same was the case for peth1 and eth1.
In Wheezy that worked for peth0 and eth0 but not for peth1 and eth1. The new Xen scripts(/etc/xen/scripts/network-bridge) is allowing to build only one bridge. I still don’t know why. If I commented that line, the script tried to build the second bridge for eth1 but it didn’t work.
Here is the code that only allows one bridge(eth0) to be created:
/etc/xen/scripts/network-bridge
Line 219:
if [ `brctl show | wc -l` != 1 ]; then
return
fi
Here is what I had used in Squeeze:
/etc/xen/xend-config.sxp
# Using the (self-made) eth0-eth1 network script wrapper
(network-script network-bridge-eth0-eth1)
/etc/xen/scripts/network-bridge-eth0-eth1
#!/bin/sh
# needs to get used in the xen-tools to creat 2 bridges: eth0 and eth1
/etc/xen/scripts/network-bridge "$@" netdev=eth0
sleep 4
/etc/xen/scripts/network-bridge "$@" netdev=eth1
Configuration in Wheezy
Extra problem:
When I started a virtual machine using only one bridge (eth0), it didn’t start and the following error showed up: ‘File already exists. Bringing up eth0 failed’ or something like that.
Someone suggested to add the following line(which works) to:
/etc/network/interfaces
pre-up ip addr del 98.184.49.14/24 dev eth0 2> /dev/null || true
From this point I had a properly working ‘peth0’ as physical interface and eth0 as bridge, but could not configure the eth1 the same way at all. I really tried many different configurations including creating the bridge by hand with brctl commands then adding the bridge in the /etc/network/interfaces etc. to no avail. Here is the only combination which worked so far, which is as I wrote previously, very dirty and not elegant and would hope to get to know a better way.
The solution:
The result is the following:
Physical Interface Bridge
================== ======
peth0 eth0
eth1 xenbr1
Here I will only show the content of the files which are concerned:
/etc/network/interfaces
# The loopback network interface
auto lo
iface lo inet loopback
#
# The primary network interface
auto eth0
iface eth0 inet static
address 98.184.49.14
netmask 255.255.255.0
network 98.184.49.0
broadcast 98.184.49.255
gateway 98.184.49.254
#
pre-up ip addr del 98.184.49.14/24 dev eth0 2> /dev/null || true
#
iface eth1 inet manual
#
iface xenbr1 inet manual
address 192.168.0.5
netmask 255.255.255.0
network 192.168.0.0
broadcast 192.168.0.255
bridge_ports eth1
bridge_maxwait 0
Note: in the above file the Internet IP is just an example and not the real IP of my server.
/root/bin/start_Xen_env.sh
#!/bin/bash
# Constants
eth1_IP=$(grep -i flash2.srv /etc/hosts | awk '{print $1}')
/etc/xen/scripts/network-bridge start netdev=eth0
/sbin/ifup eth1
/sbin/ifup xenbr1
/sbin/ifconfig eth1 $eth1_IP
/sbin/ifconfig xenbr1 up
route del -net 192.168.0.0 netmask 255.255.255.0 dev eth1
route add -net 192.168.0.0 netmask 255.255.255.0 dev xenbr1
Example of network configuration of a DOMu Xen configuration:
.....
vif = [ 'ip=98.184.53.164,mac=00:16:3E:78:1C:64,bridge=eth0' , 'ip=192.168.0.164,mac=00:16:3E:D7:9C:64,bridge=xenbr1' ]
.....
Process:
Here is the sequence of event that needs to happen after reboot before I can start any DOMUs.
– After reboot the eth0 interface is configured as normal interface connected to Internet
– eth1 and xenbr1 interfaces are configured but not activated.
– I login as root to the server through eth0 and run the script /root/bin/start_Xen_env.sh manually:
These steps prepare the Xen networking environment which I need to be able to start DOMUs.
Note: for some strange reason which I would like to know why, if I ran this above script from root crontab as follows, it failed to create the peth0.
@reboot /bin/sleep 10 ; /root/bin/start_Xen_env.sh
If I run the same script by hand after a reboot then all goes well.
After I ran the script, here is the result of the output of the brctl and ifconfig commands:
flash2 ~ # brctl show
bridge name bridge id STP enabled interfaces
eth0 8000.00259033cc14 no peth0
xenbr1 8000.00259033cc15 no eth1
flash2 ~ # ifconfig
eth0 Link encap:Ethernet HWaddr 00:25:90:33:cc:14
inet addr:98.184.49.14 Bcast:98.184.49.255 Mask:255.255.255.0
inet6 addr: fe80::225:90ff:fe33:cc14/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:209 errors:0 dropped:4 overruns:0 frame:0
TX packets:44 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:11687 (11.4 KiB) TX bytes:5364 (5.2 KiB)
.
eth1 Link encap:Ethernet HWaddr 00:25:90:33:cc:15
inet addr:192.168.0.5 Bcast:192.168.0.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:5 errors:0 dropped:0 overruns:0 frame:0
TX packets:4 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:320 (320.0 B) TX bytes:314 (314.0 B)
Interrupt:17 Memory:fb6e0000-fb700000
.
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:1544 errors:0 dropped:0 overruns:0 frame:0
TX packets:1544 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:102145 (99.7 KiB) TX bytes:102145 (99.7 KiB)
.
peth0 Link encap:Ethernet HWaddr 00:25:90:33:cc:14
inet6 addr: fe80::225:90ff:fe33:cc14/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:436462 errors:0 dropped:11264 overruns:0 frame:0
TX packets:4791 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:29190641 (27.8 MiB) TX bytes:1020853 (996.9 KiB)
Interrupt:16 Memory:fb5e0000-fb600000
.
xenbr1 Link encap:Ethernet HWaddr 00:25:90:33:cc:15
inet6 addr: fe80::225:90ff:fe33:cc15/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:5 errors:0 dropped:0 overruns:0 frame:0
TX packets:3 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:230 (230.0 B) TX bytes:238 (238.0 B)
Any improvement or suggestions regarding this issue is very welcome.