msgbartop
MAC OS X, Linux, Windows and other IT Tips and Tricks
msgbarbottom

07 Aug 14 Install Xen 4.1 on Debian Wheezy in a Hetzner Dedicated server

Hetzner Germany has very fast and not expensive rentals of Hardware servers available. In order to communicate internally via private network between Xen-DOMUs and DOM0, normally one would install Xen DOM0 network with bridge networking as follows:
DOM0:xenbr0(eth0) --- bridging==>> DOMUs:eth0
DOM0:xenbr1(dummy0) ---bridging==>> DOMUs:eth1

BUT!!!!
PROBLEM:
Because of the configuration of the network switches at Hetzner, one hardware server can have multiple IPs but only one MAC address (MAC of eth0 in DOM0). This means that Bridge networking for Internet connection (eth0) doesn’t work for multiple DOMUs, each one having its own IP AND MAC address.
SOLUTION:
The solution is to use routing for Internet access and bridging for private LAN as follows:
DOM0:eth0 --- routing===>> DOMUs:eth0
DOM0:xenbr1(dummy0) --- bridging==>> DOMUs:eth1

Note: The DISADVANTAGE of this solution is that DOM0 must use one IP from the subnet provided by hetzner to be used as a gateway for the running DOMUs to allow them to communicate with the Internet. In this case the following IP subnet of 8 IPs provided by by Hetzner could be for example:
CIDR Subnet: 140.231.213.168/28
Network addr: 140.258.213.168 (unusable by DOMUs hosts)
Gateway addr: 140.231.213.169 (used as gateway for DOMUs, unusable by DOMUs hosts)
DOMUs usable IPs: 140.231.213.170 - 140.231.213.174 (5 IPs)
Broadcast addr: 140.231.213.175 (unusable for DOMUs hosts)

This means out of 8 IPs you got as a subnet from Hetzner you can only run 5 DOMUs in this Xen environment if each DOMU needs to have its own Internet reachable IP.

XEN INSTALLATION


We will first install XEN in the main hardware server. This means installing the hypervisor, xen aware kernel and xen tools. This can be done by a installing the following packages and a few favorite tools đŸ™‚
apt-get install xen-linux-system xen-tools bridge-utils mc ssh fail2ban ethtool
Debian Wheezy uses Grub 2 and as default boot manager. It lists normal kernels first, and then, if the xen kernel is installed, lists the Xen hypervisor and its kernels. You need to change this to cause Grub 2 to prefer to boot Xen as default kernel. It is done by changing the priority of Grub’s Xen configuration script (20_linux_xen) to be higher prority than the standard Linux config (10_linux). This is most easily done using dpkg-divert:
dpkg-divert --divert /etc/grub.d/08_linux_xen --rename /etc/grub.d/20_linux_xen
After any update to the Grub configuration you must apply the configuration by running:
update-grub
Disable Xendomains save & restore
We disable the saving and restore feature of DOMUs mostly because my experience is that this feature doens’t always work well. I prefer to do the shutdown of each DOMU manually before rebooting DOM0, then after reboot of DOM0, restart each individual DOMU using a @reboot cron job for example:
# This will start 2 virtual machines 60 sec after reboot of DOM0
@reboot /bin/sleep 60; /usr/sbin/xl create /etc/xen/DOMU1.cfg; /usr/sbin/xl create /etc/xen/DOMU2.cfg
This way if power failure happens or anything that forces an unattended reboot of DOM0, all the DOMUs will automatically restart after reboot.

Now the disabling of the automatic Save/Restore of DOMUs:
Edit /etc/default/xendomains
Content:
#XENDOMAINS_SAVE=/var/lib/xen/save
XENDOMAINS_SAVE=
#
#XENDOMAINS_RESTORE=true
XENDOMAINS_RESTORE=false

NETWORKING:


Add the dummy network interface module
echo dummy >> /etc/modules
modprobe dummy

Network configuration
Edit file: /etc/network/interfaces
(Note: here you’ll need to adapt your own IPs etc. in this file)
Content:
# Loopback device:
auto lo
iface lo inet loopback
#
# device: eth0
auto eth0
iface eth0 inet static
address 123.45.67.89
broadcast 123.45.67.255
netmask 255.255.255.0
gateway 123.45.67.1
#
iface eth0 inet6 static
address 2a01:4f7:192:4213::2
netmask 64
gateway fe80::1
#
# Used exclusively as Gateway for DOMUs for this subnet. Unfortunately losing one IP for Gateway purposes.
auto eth0:gw1
iface eth0:gw1 inet static
address 140.231.213.169
netmask 255.255.255.248
network 140.231.213.168
broadcast 140.231.213.175
#
# Internal private network to DOMUs
iface dummy0 inet manual
#
auto xenbr1
iface xenbr1 inet static
address 192.168.100.1
netmask 255.255.255.0
network 192.168.100.0
broadcast 192.168.100.255
bridge_ports dummy0
#
#other possibly useful options in a virtualized environment
bridge_stp off # disable Spanning Tree Protocol
bridge_waitport 0 # no delay before a port becomes available
bridge_fd 0 # no forwarding delay
post-up ethtool -K xenbr1 tx off
post-up ip link set xenbr1 promisc off

Switch to the XL Xen ToolStack
Edit /etc/default/xen
TOOLSTACK=xl
WARNING: The above entry is small ‘XL’ and not small ‘X1’ !!

Edit /etc/xen/xl.conf and make sure the entries are as follows:
# automatically balloon down dom0 when xen doesn't have enough free
# memory to create a domain
autoballoon=1
#
# full path of the lockfile used by xl during domain creation
lockfile="/var/lock/xl"
#
# default vif script.
#vifscript="vif-bridge"
vifscript="/etc/xen/scripts/vif-route_eth0-bridge_dummy0"

Note: Here we use a script which will set routing for eth0 and bridging for dummy0.
Create it.
touch /etc/xen/scripts/vif-route_eth0-bridge_dummy0
chmod 755 /etc/xen/scripts/vif-route_eth0-bridge_dummy0

Edit the file /etc/xen/scripts/vif-route_eth0-bridge_dummy0
Content:
#!/bin/sh
# Custom vif script which allows to combine routing for Internet and bridging for internal LAN
dir=$(dirname "$0")
IFNUM=$(echo ${vif} | cut -d. -f2)
if [ "$IFNUM" = "0" ] ; then
"$dir/vif-route" "$@"
else
"$dir/vif-bridge" "$@"
fi

Edit the file /etc/xen/xend-config.sxp
and make sure the already existing entries are disabled with ‘#’ and new lines entered as follows:
#.......
#(vif-script vif-bridge)
(network-script dummy)
#
#(vif-script vif-route)
(vif-script vif-route_eth0-bridge_dummy0)
#
# make sure DOM0 has enough memory
(dom0-min-mem 2048)
#.......

Setup the IP forwarding and ARP proxying in kernel:
Edit the file /etc/sysctl.conf
Either un-comment or add the following lines:
# Uncomment the next line to enable packet forwarding for IPv4
net.ipv4.ip_forward=1
# ARP Proxying
net.ipv4.conf.eth0.proxy_arp = 1

To make this change take effect immediately run:
sysctl -p /etc/sysctl.conf
Finally, before we reboot the system we need to make sure we activate the proper toolstack and related features at boot time by running the following commands:
update-rc.d xendomains defaults
update-rc.d xen defaults
/etc/init.d/xen restart
/etc/init.d/xendomains restart

DOMUs Configuration


PyGRUB
If your DOMUs configurations are set to use pygrub as boot loader,
then make sure the path to pygrub in the DOMU configuration file is correct as follows:
bootloader = '/usr/lib/xen-4.1/bin/pygrub'
In the same DOMU configuration file, make sure you are using a non duplicated MAC addresses with the network interfaces assignment for example:
vif = [ 'ip=140.258.213.170,mac=00:16:34:D7:9C:F4' , 'ip=192.168.0.18,mac=00:16:3E:D7:9C:F6',bridge=xenbr1]
Note: The first IP doesn’t need any bridge since it is routing controlled, the internal LAN is bridged with xenbr1 though.

NOTE:If you want to use the pyGrub as boot loader for each individual DOMUs which makes the DOMUs kernel independant from the DOM0, see the following article:
http://tipstricks.itmatrix.eu/?s=pygrub&x=0&y=0

Reader's Comments

  1.    

    With Hetzner you can add mac addresses to ip’s for free, at least when you apply for the 3 extra ip’s. I assume with subnets you should be able to do the same?

    Reply to this comment
    •    

      Yes I noticed that you can ask for a MAC address per IP up to 3 if you apply for them. But I could not find anywhere the same possibility for Subnets. In the IP page the subnet is simply listed as a CIDR subnet with no individual IP listed. I guess one muct then ask Hetzner if that is also possible for subnets.

      Reply to this comment
  2.    

    I am using Xen4Centos on hetzner dedicated server. Created a bridge network and VM.

    Single IP on VM is working fine but adding second IP in same VM is not working.

    Is there a way to add second IP in same subnet in one single VM at hetzner?

    I have tried multiple option with no luck.

    If any one knows then please let me know.

    Reply to this comment
    •    

      Well the following solution has not been tested but I do believe it could work:
      Note: here I assume that the second IP is within the same subnet as the first IP and that the Gateway of your IPs are the same.
      – Makes sure you have generated a MAC address in the Hetzner robot site of your server for your second IP.
      – Set a second network interface in the xen configuration for the specific DOMU as follows:
      vif = [ ‘ip=140.158.213.170,mac=00:16:34:D7:9C:F4’ , ‘ip=140.158.213.172,mac=00:16:34:47:6C:A3’]
      – Create a second interface(eth1) configuration in DOMU for example as follows:
      Note: Sorry for the difficulties of text formatting where all the lines are completely left aligned, please adjust yours as they should be in the /etc/network/interfaces file.

      auto eth0
      iface eth0 inet static
      address 140.158.213.170
      netmask 255.255.255.248
      network 140.158.213.168
      gateway 140.258.213.174
      .
      iface eth1 inet static
      address 140.158.213.172
      netmask 255.255.255.248
      network 140.158.213.168
      gateway 140.258.213.174

      With a similar configuration as above you should be able to use 2 reachable IPs in the same DOMU.

      Reply to this comment
      •    

        @admin,

        Thank You so much for looking into this issue and replying to my comment.

        The solution which you have provided that is already tried but second IP was not reachable.

        Finally, I have figure out the way to listen the second IP from public network.

        I have created a routing table for eth1 and added default gateway to eth1. [ ip route add default via XXX.XX.XXX.112 dev eth1 table eth1].

        Doing this way second IP in VM was reachable from public network.

        Reply to this comment

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: