msgbartop
MAC OS X, Linux, Windows and other IT Tips and Tricks
msgbarbottom

03 Feb 16 Installing Xen 4.4 on Ubuntu Server 14.04 LTS (Trusty)

Introduction:

This HowTo assumes that the Internet access from VMs via DOM0 and the private LAN are done using the Bridge method. In the previous versions of Xen installation the bridges were dynamically built via the Xen scripts, in this version the bridges are built permanently as the DOM0 boots up.
DOM0:xenbr0(eth0) ---bridging==>> DOMUs:eth0
DOM0:pdummy0(dummy0) ---bridging==>> DOMUs:eth1

IMPORTANT: If you are installing Xen in a Hetzner(Germany) dedicated server and use only the available(max 3) IPs for the DOMUs, then you need to make sure you are generating a MAC address for each DOMU IP in the Hetzner robot site of your server, then use this MAC address in your DOMU Xen configuration. If you are using a subnet of 8 IP or more in Hetzner server for DOMUs, this bridging method would not work. Follow the instructions shown here instead: https://wp.me/pKZRY-F9

Install Xen Hypervisor and some useful tools
apt-get install xen-hypervisor-4.4-amd64 xen-utils-4.4 bridge-utils ethtool iptables

Some extra preparations

Since every virtual disk needs to be mounted using a loop device, we need to make sure there are enough of them available in the system.
Edit the file /etc/modules and add:
loop max_loop=64
dummy

We also need to turn on the IPv4 forwarding in the kernel.
Edit the file /etc/sysctl.conf (around line 44) activate the line by removing the ‘#’ as follows:
net.ipv4.ip_forward=1
The run the following command to activate it:
sysctl -p /etc/sysctl.conf

CONFIGURING THE NETWORK in DOM0

Based on the IP assumptions above, here is the content of the file /etc/network/interfaces.
# Internet Access nterface
auto xenbr0
iface xenbr0 inet static
address 85.114.145.5
netmask 255.255.255.0
network 85.114.145.0
broadcast 85.114.145.255
gateway 85.114.145.1
bridge_ports eth0
#
auto eth0
iface eth0 inet manual
#
# Internal LAN between VMs and DOM0
auto pdummy0
iface pdummy0 inet static
address 192.168.100.1
netmask 255.255.255.0
bridge_ports dummy0
#
auto dummy0
iface dummy0 inet manual

In order to make sure Xen scripts don’t create the normal bridges when a DOMu is started, we need to hinder this process by:
editing the file /etc/xen/xend-config.sxp and change the line:(around line 176)
FROM:
(network-script network-bridge)
TO:
(network-script none)
reboot

Configuring the DOMUs

DOMUs Configuration

PyGRUB
If your DOMUs configurations are set to use pygrub as boot loader,
then make sure the path to pygrub in the DOMU configuration file is correct as follows:
bootloader = '/usr/lib/xen-4.4/bin/pygrub'
In the same DOMU configuration file, make sure you are using a non duplicated MAC addresses with the network interfaces assignment as well as define the bridge that will be used by this DOMu, for example:
vif = [ 'ip=46.7.178.112,mac=00:16:34:D7:9C:12,bridge=xenbr0', 'ip=192.168.100.112,mac=00:16:3E:D7:1C:12,bridge=pdummy0' ]
NOTE:If you are not using the PyGRUb and want to use it as boot loader for each individual DOMUs, which makes the DOMUs kernel independent from the DOM0, see the following article. Please notice that in Ubuntu 14.04 the path to pygrub is different than in the article. Each new version of Xen has a different path to PyGRUB th rest of the article is fully accurate for Ubuntu as well.
http://tipstricks.itmatrix.eu/?s=pygrub&x=0&y=0

DOMus Network Configuration

Each DOMu will get an interface lo and eth0 with the following configuration:
I’m using the first IP of our subnet for this DOMU and will therefore be configured as follows:
Note: This configuration not really standard as it uses each IP with the netmask /32 (255.255.255.255).
This setting allows each IP of the subnet to be usable by each DOMu.
File: /etc/network/interfaces
Content:
# The loopback network interface
auto lo
iface lo inet loopback
#
# The primary network interface
auto eth0
iface eth0 inet static
address 46.7.178.112
netmask 255.255.255.255
gateway 46.7.178.1
#
# The internale LAN interface(will be connected to pdummy0 on DOM0)
auto eth1
iface eth1 inet static
address 192.168.100.112
netmask 255.255.255.0

Reader's Comments

  1.    

    Thanks for ur effort. However I cant connect to internet anymore… Why can it be?

    Reply to this comment
    •    

      Hi, I’m sorry that this is the result of using the set-up I described in this post. Unfortunately there could be a few or many causes for this to happen and without doing research with some Linux tools I cannot answer your question. I would suggest you look at your firewall where the Internet NIC interface is no more eth0 but xenbr0. This can make the firewall block outgoing connections. I would suggest to disable temporarily your firewall and try again connection to Internet. Good luck.

      Reply to this comment
  2.    

    Hi, thanks for the tutorial. Are the IP address used in your example above standard or specific to your machine?

    Reply to this comment
    •    

      The IPs shown here are only examples that might be own by someone else.
      The local LAN IPs are simply chosen from the B private network(192.168.xxx.0/24).
      You therefore you need to use the Internet IPs that got given to you for your Xen server,
      but you can use the same IPs of Internal LAN that appears in this tutorial if you want.

      Reply to this comment
      •    

        Thank you for the quick reply. Kindly allow me to take more liberties with your time because I’m really in a fix. I have a server from Hetzner with the following IP (not the real IP)

        Main IP: 4.5.6.111

        I then bought 4 extra IPs

        3.5.6.111
        3.5.6.112
        3.5.6.113
        3.5.6.114

        I’m trying to set up Xen using bridged networking. Through the Hetzner robot, I am able to create MAC addresses for each of these IPs.

        I have install Xen but I can’t get the network to work. Please any leads you can give me using the specific IP examples of mine?

        Sorry for the trouble and thanks again.

        Reply to this comment
        •    

          Hi, I can understand your confusion. I didn’t get it the first time either. So I will try to give you the configuration of the Main Xen server as well as of the DOMu using the IPs you gave me.

          CONFIGURING THE NETWORK in DOM0
          # Internet Access Interface
          auto xenbr0
          iface xenbr0 inet static
          address 4.5.6.111
          netmask 255.255.255.0
          network 4.5.6.0
          broadcast 4.5.6.255
          gateway 4.5.6.1
          bridge_ports eth0
          #
          auto eth0
          iface eth0 inet manual
          #
          # Internal LAN between VMs and DOM0
          auto pdummy0
          iface pdummy0 inet static
          address 192.168.100.1
          netmask 255.255.255.0
          bridge_ports dummy0
          #
          auto dummy0
          iface dummy0 inet manual
          ————————————————-

          CONFIGURING THE NETWORK in DOMU(VMs)

          # The loopback network interface
          auto lo
          iface lo inet loopback

          #
          # The primary network interface
          auto eth0
          iface eth0 inet static
          address 3.5.6.111
          netmask 255.255.255.255
          gateway 4.5.6.1
          #
          # The internale LAN interface(will be connected to pdummy0 on DOM0)

          auto eth1
          iface eth1 inet static
          address 192.168.100.111
          netmask 255.255.255.0

          ———————————————-

          Make sure again that you also did all the other steps shown in the article.
          Good luck and let me know if you still have trouble.

          Reply to this comment
          •    

            One more thing: make sure you don’t have a firewall in your Xen server. The reason is that it’s very difficult to configure a firewall on the Xen server that doesn’t interfere with the DOMU’s network. For protection I usually make sure that no other service than SSH and possibly Postfix(if needed) are running in the Main Xen server. This way you don’t have to protect ports that are not opened. For SSH I always protect it with the package ‘fail2ban’ and it’s doing a great job.

  3.    

    Wow, wow, wow!!! This worked! Thank you very much. This has literarily cost me sleepless nights on so many days. Being the sort of person I am, once I am hooked on a problem I don’t give up until I find a solution, except that this time there seemed to be no solution and I was as ignorant as they come.

    Just yesterday I finally conceded defeat and reached out to a freelancer who would have taken about $200 to fix this for me. Thankfully, I had no immediate cash to pay him 🙂 And so I decided to reach out to you and you helped. Thank you!

    Your solution may also have gained me an extra IP address. In a previous xen installation I was forced to use one of the 4 IPs as gateway, but not this time.

    I am hugely indebted to you sir!

    Reply to this comment
    •    

      Great! I’m really glad that you finally got it to work. It’s very encouraging to do so much work to document a HowTO and to see that others can benefit from it. This site is up since at least 5 years and I’m getting around 300 visitors per day since about 3 years and only once in a while do I get a ‘thank you’ like yours. Good luck in continuing your work with the Xen servers. Have a good day, Michel

      Reply to this comment
  4.    

    Thank you Michel.

    I found your site through “Google”, so I guess I should update this with more details in case there is someone else out there pulling out his hair.

    I have now completed the Xen installation and I have 4 brand new virtual machines using the 4 IPs from Hetzner.

    For information, I am using Xen 4.6.0 installed on Ubuntu 16.04 (Xenial).

    I’m going to paste Michel’s scripts above, with comments about what I changed (not much).

    1.

    vif = [ ‘ip=46.7.178.112,mac=00:16:34:D7:9C:12,bridge=xenbr0’, ‘ip=192.168.100.112,mac=00:16:3E:D7:1C:12,bridge=pdummy0’ ]

    The first “ip” belongs to one of the four VM IPs I bought from Hetzner. I got the MAC address from Hetzner robot and changed the one above.

    I changed the second ip above to 192.168.100.111 so that it matches with the Internal LAN interface configuration of DomU as Michel recommended above.

    2.

    CONFIGURING THE NETWORK in DOM0
    # Internet Access Interface
    auto xenbr0
    iface xenbr0 inet static
    address 4.5.6.111 (Main IP)
    net mask 255.255.255.0 (Get yours from Hetzner robot for the main IP)
    network 4.5.6.0 (Get yours from Hetzner robot)
    broadcast 4.5.6.255 (Get yours from Hetzner robot)
    gateway 4.5.6.1 (Get yours from Hetzner robot)
    bridge_ports eth0
    #
    auto eth0
    iface eth0 inet manual
    #
    # Internal LAN between VMs and DOM0
    auto pdummy0
    iface pdummy0 inet static
    address 192.168.100.1 (I used this)
    net mask 255.255.255.0 (I used this)
    bridge_ports dummy0
    #
    auto dummy0
    iface dummy0 inet manual
    ————————————————-

    CONFIGURING THE NETWORK in DOMU(VMs)

    # The loopback network interface
    auto lo
    iface lo inet loopback

    #
    # The primary network interface
    auto eth0
    iface eth0 inet static
    address 3.5.6.111 (Change to your VM IP)
    net mask 255.255.255.255 (I used this)
    gateway 4.5.6.1 (This is the gateway of the Main IP, DOMO)
    #
    # The internale LAN interface(will be connected to pdummy0 on DOM0)

    auto eth1
    iface eth1 inet static
    address 192.168.100.111 (I used this)
    net mask 255.255.255.0 (I used this)

    Thanks again Michel!

    Reply to this comment
  5.    

    Hi there!, for the new Xen Version 4.4+, the file /etc/xen/xend-config.sxp doesn’t exist. in which file can i config to avoid that xen script’s create the normal bridges.

    Reply to this comment
    •    

      Well, for the Xen 4.4.+ the replacement file is /etc/xen/xl.conf. It doesn’t have the same configuration syntax as the xend-config.sxp but it holds the most important config keywords regarding the set-up of the bridges. In fact fro that version on the interface bridges have to be pre-created by definitions in the file /etc/network/interfaces unless you are using ‘routing’ method for routing the packet to each individual DOMUs. Regarding the VIFs a script is still needed and it is declared in the xl.conf file. Here is any example of my xl.conf under Debian Jessie which runs Xen 4.4.

      ## Global XL config file ##
      # Control whether dom0 is ballooned down when xen doesn't have enough
      # free memory to create a domain. "auto" means only balloon if dom0
      # starts with all the host's memory.
      #autoballoon="auto"
      # full path of the lockfile used by xl during domain creation
      lockfile="/var/lock/xl"
      # default output format used by "xl list -l"
      #output_format="json"
      # first block device to be used for temporary VM disk mounts
      #blkdev_start="xvda"
      # default option to run hotplug scripts from xl
      # if disabled the old behaviour will be used, and hotplug scripts will be
      # launched by udev.
      #run_hotplug_scripts=1
      # default backend domain to connect guest vifs to. This can be any
      # valid domain identifier.
      #vif.default.backend="0"
      # default gateway device to use with vif-route hotplug script
      #vif.default.gatewaydev="eth0"
      # default vif script to use if none is specified in the guest config
      #vif.default.script="vif-bridge"
      vif.default.script="vif-route_eth0-bridge_xenbr1"
      # default bridge device to use with vif-bridge hotplug scripts
      vif.default.bridge="xenbr1"
      # Reserve a claim of memory when launching a guest. This guarantees immediate
      # feedback whether the guest can be launched due to memory exhaustion
      # (which can take a long time to find out if launching huge guests).
      # see xl.conf(5) for details.
      #claim_mode=1

      The content of the script /etc/xen/scripts/vif-route_eth0-bridge_xenbr1
      #!/bin/sh
      # needs to get used in the xen-tools to create routing with eth0 and bridging with eth1
      /etc/xen/scripts/network-route "$@" netdev=eth0
      sleep 4
      /etc/xen/scripts/network-bridge "$@" netdev=eth1
      echo 1 >/proc/sys/net/ipv4/ip_forward

      Content of /etc/network/interfaces
      auto lo
      iface lo inet loopback
      #
      auto eth0
      # The primary network interface
      iface eth0 inet static
      address 85.17.18.106
      netmask 255.255.255.255
      gateway 85.17.18.254
      pointopoint 85.17.18.254
      #
      # eth1 and xenbr1 Bridge
      auto xenbr1
      iface xenbr1 inet static
      bridge_ports eth1
      address 192.168.0.106
      netmask 255.255.255.0
      network 192.168.0.0
      broadcast 192.168.0.255
      bridge_stp off
      post-up ethtool -K xenbr1 tx off
      post-up ip link set xenbr1 promisc off
      #
      auto eth1
      iface eth1 inet manual

      This above configuration is using routing(via eth0) for internet connections and a bridge(xenbr1) for internal LAN between DOMUs.
      To make the routing work for DOMUs that are using a different subnet as for the main Xen server I use an extra IP as gateway for them.
      Here is any example of such configuration in /etc/network/interfaces:
      #------------ DOMUs Gateway ------------
      # Gateway for network 92.184.55.0/24
      auto eth0:gw1
      iface eth0:gw1 inet static
      address 92.184.55.3
      netmask 255.255.255.0
      network 92.184.55.0
      broadcast 92.184.55.255

      I hope that is of some help.

      Reply to this comment

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: