IT Tips and Tricks

Hetzner Germany has very fast and not expensive rentals of Hardware servers available. In order to communicate internally via private network between Xen-DOMUs and DOM0, normally one would install Xen DOM0 network with bridge networking as follows:
DOM0:xenbr0(eth0) ===bridging===>> DOMUs:eth0

BUT!!!!
PROBLEM:
Because of the configuration of the network switches at Hetzner, one hardware server can have multiple IPs but only one MAC address (MAC of eth0 in DOM0). This means that Bridge networking for Internet connection (eth0) doesn’t work for multiple DOMUs, each one having its own IP AND MAC address. The situation is quite different if you order 1 to 3(max) extra IPs that can be added to each hardware server. Those IPs can be configured in the Xen DOMUs and use bridge as the method. On the Hetzner Robot site you can generate a MAC address per IP which you can use in your Xen DOMUs configuration and Hetzner switch will route it properly.(See this site for these instructions: https://wp.me/pKZRY-OE) BUT, this is not (yet?) the case with requested IP subnets from Hetzner. Therefore the following solution is the best found so far.

SOLUTION:
The solution is to use routing for Internet access. DOM0 does the routing of the traffic from internet to each DOMU. It also does routing the traffic between DOMUs, making this a private connection since this communication never leaves DOM0.
Note: This solution was presented in Hetzner documentation at http://wiki.hetzner.de/index.php/KVM_mit_Nutzung_aller_IPs_aus_Subnetz/en for KVM installation, which offers the possibility of using ALL of the subnet IPs, as opposed to the traditional way of using routing, which prevents the use of the first and last IP of the subnet as DOMU IP and also needs an extra IP as subnet Gateway for DOMUs. For example:
Traditional way of routing:
CIDR Subnet: 46.5.178.112/29
Network addr: 46.5.178.112 (unusable by DOMUs hosts)
Gateway addr: 46.5.178.113 (used as gateway for DOMUs, unusable by DOMUs hosts)
DOMUs usable IPs: 46.5.178.114 - 46.5.178.118 (5 IPs)
Broadcast addr: 46.5.178.119 (unusable for DOMUs hosts)
This means out of 8 IPs subnet (/29) you can only run 5 DOMUs in this Xen environment if each DOMU needs to have its own Internet reachable IP.

This specific routing method:
Every IP (8 IPs) of the subnet can be used for DOMUs: 46.5.178.112 – 46.5.178.119
No IP is lost for being the network or broadcast IP or as subnet gateway.
Internet ===>>(DOM0:eth0) --- routing ===>>(DOMu Bridge)===>>(DOMu VIF === DOMu:eth0)
Short explanation:
Each DOMu gets a bridge which contains a private network address(172.30.xx.1) used to link DOM0 to the DOMu Internet address.
Example:
DOM0:eth0 ===routing===>> (Bridge[172.30.112.1]) ===>> (vif1.0 === DOMu:eth0 [46.5.178.112])
DOM0:eth0 ===routing===>> (Bridge[172.30.113.1]) ===>> (vif2.0 === DOMu:eth0 [46.5.178.113])
DOM0:eth0 ===routing===>> (Bridge[172.30.114.1]) ===>> (vif3.0 === DOMu:eth0 [46.5.178.114])
DOM0:eth0 ===routing===>> (Bridge[172.30.115.1]) ===>> (vif4.0 === DOMu:eth0 [46.5.178.115])
DOM0:eth0 ===routing===>> (Bridge[172.30.116.1]) ===>> (vif5.0 === DOMu:eth0 [46.5.178.116])
DOM0:eth0 ===routing===>> (Bridge[172.30.117.1]) ===>> (vif6.0 === DOMu:eth0 [46.5.178.117])
DOM0:eth0 ===routing===>> (Bridge[172.30.118.1]) ===>> (vif7.0 === DOMu:eth0 [46.5.178.118])
DOM0:eth0 ===routing===>> (Bridge[172.30.119.1]) ===>> (vif8.0 === DOMu:eth0 [46.5.178.119])
DOM0:dummy0 ===routing===>> (Bridge:pdummy0) ===>> (vifx.0 === DOMu:eth1 [192.168.100.x])

Please notice the 3rd number in the bridge IP corresponds to the last number of the subnet IP of its respective DOMu. This is used just to identify the different subnets created in each bridge. They simply need to differ between each other.
The vifx.0 Virtual Interface is created automatically by the Xen scripts at the start of a DOMu. It is the internal link between the DOMu eth0 interface and its associated bridge located in DOM0.
The netmask of the private subnet of each bridge being 255.2555.255.0 is only a practical way of limiting the range of each subnet so they don’t overlap each other.

Note: In this HowTo I also use the virtual interface dummy0 to connect the DOMUs between each other in a private virtual LAN based on the network: 192.168.100.0/24. To realize this I set-up a dummy0 virtual interface and its attached bridge pdummy0.

Steps to create the virtual private LAN:

Edit /etc/modules and add the following line:
dummy
This will load the module dummy to the kernel automatically at boot time.
Now to avoid having to reboot we load it manually by issuing the command:
modprobe dummy
Interface configuration:
Edit /etc/network/interfaces and add the following lines:
(Replace IPs to your preferred IP Network)
auto dummy0
iface dummy0 inet manual
#
auto pdummy0
iface pdummy0 inet static
address 192.168.100.1
netmask 255.255.255.0
network 192.168.100.0
broadcast 192.168.100.255
bridge_ports dummy0
bridge_stp off
bridge_fd 0
bridge_maxwait 0
Now we bring the dummy0 and bridge pdummy0 interfaces up:
ifup dummy0
ifup pdummy0
Note: at this point no worry about the error message. We can ignore it for now.
Check the configuration:
ifconfig dummy0
ifconfig pdummy0
You should get something like this:
dummy0 Link encap:Ethernet HWaddr 76:99:e1:48:64:f5
UP BROADCAST RUNNING NOARP MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:15 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:1230 (1.2 KB)
#
pdummy0 Link encap:Ethernet HWaddr 76:99:e1:48:64:f5
inet addr:192.168.100.1 Bcast:192.168.100.255 Mask:255.255.255.0
inet6 addr: fe80::7499:e1ff:fe48:64f5/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:407 errors:0 dropped:0 overruns:0 frame:0
TX packets:530 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:82394 (82.3 KB) TX bytes:57166 (57.1 KB)

The routing table looks then like this:
# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 178.61.78.129 0.0.0.0 UG 0 0 0 eth0
46.5.178.112 0.0.0.0 255.255.255.255 UH 0 0 0 br112
46.5.178.113 0.0.0.0 255.255.255.255 UH 0 0 0 br113
46.5.178.114 0.0.0.0 255.255.255.255 UH 0 0 0 br114
46.5.178.115 0.0.0.0 255.255.255.255 UH 0 0 0 br115
46.5.178.116 0.0.0.0 255.255.255.255 UH 0 0 0 br116
46.5.178.117 0.0.0.0 255.255.255.255 UH 0 0 0 br117
46.5.178.118 0.0.0.0 255.255.255.255 UH 0 0 0 br118
46.5.178.119 0.0.0.0 255.255.255.255 UH 0 0 0 br119
172.30.112.0 0.0.0.0 255.255.255.0 U 0 0 0 br112
172.30.113.0 0.0.0.0 255.255.255.0 U 0 0 0 br113
172.30.114.0 0.0.0.0 255.255.255.0 U 0 0 0 br114
172.30.115.0 0.0.0.0 255.255.255.0 U 0 0 0 br115
172.30.116.0 0.0.0.0 255.255.255.0 U 0 0 0 br116
172.30.117.0 0.0.0.0 255.255.255.0 U 0 0 0 br117
172.30.118.0 0.0.0.0 255.255.255.0 U 0 0 0 br118
172.30.119.0 0.0.0.0 255.255.255.0 U 0 0 0 br119
178.61.78.129 0.0.0.0 255.255.255.255 UH 0 0 0 eth0
192.168.100.0 0.0.0.0 255.255.255.0 U 0 0 0 pdummy0


ASSUMPTIONS in these examples

Current network settings:
DOM0 IP: 178.61.78.140
Default Gateway: 178.61.78.129
IP Net netmask: 255.255.255.240

Extra IPs subnet:
Subnet: 46.5.178.112/29 (46.4.178.112 – 46.4.178.119)
Netmask: 255.255.255.248
Broadcast: 46.5.178.119

Local virtual LAN
Subnet: 192.168.100.0/24
Netmask: 255.255.255.0
Broadcast: 192.168.100.255

XEN INSTALLATION

We will first install XEN in the main hardware server. This means installing the hypervisor, xen aware kernel and xen tools. This can be done by a installing the following packages and a few favorite tools 🙂
apt-get install xen-hypervisor-4.4-amd64 xen-utils-4.4 bridge-utils ethtool iptables mc ssh fail2ban

Some extra preparations

Since every virtual disk needs to be mounted using a loop device, we need to make sure there are enough of them available in the system.
Edit the file /etc/modules and add:
loop max_loop=64

We also need to turn on the IPv4 forwarding in the kernel.
Edit the file /etc/sysctl.conf (around line 44) activate the line by removing the ‘#’ as follows:
net.ipv4.ip_forward=1
The run the following command to activate it:
sysctl -p /etc/sysctl.conf

CONFIGURING THE NETWORK in DOM0

Based on the IP assumptions above, here is the content of the file /etc/network/interfaces:
Note: The configuration of the eth0 below is not standard. Please see the explanation of it at:
http://wiki.hetzner.de/index.php/KVM_mit_Nutzung_aller_IPs_aus_Subnetz/en
# Loopback device:
auto lo
iface lo inet loopback
#
## device: eth0 for normal operation
# The primary network interface for KVM operation
auto eth0
iface eth0 inet static
address 178.61.78.140
netmask 255.255.255.255
gateway 178.61.78.129
pointopoint 178.61.78.129
#
iface eth0 inet6 static
address 2a01:4f8:121:30ea::2
netmask 64
gateway fe80::1
#
auto dummy0
iface dummy0 inet manual
#
auto pdummy0
iface pdummy0 inet static
address 192.168.100.1
netmask 255.255.255.0
network 192.168.100.0
broadcast 192.168.100.255
gateway 192.168.0.1
bridge_ports dummy0
bridge_stp off
bridge_fd 0
bridge_maxwait 0
#
################# Individual bridges for each extra VM #################
auto br112
iface br112 inet static
address 172.30.112.1
netmask 255.255.255.0
pre-up brctl addbr $IFACE
post-up route add -host 46.5.178.112 $IFACE
post-down brctl delbr $IFACE
#
auto br113
iface br113 inet static
address 172.30.113.1
netmask 255.255.255.0
pre-up brctl addbr $IFACE
post-up route add -host 46.5.178.113 $IFACE
post-down brctl delbr $IFACE
#
auto br114
iface br114 inet static
address 172.30.114.1
netmask 255.255.255.0
pre-up brctl addbr $IFACE
post-up route add -host 46.4.178.114 $IFACE
post-down brctl delbr $IFACE
#
auto br115
iface br115 inet static
address 172.30.115.1
netmask 255.255.255.0
pre-up brctl addbr $IFACE
post-up route add -host 46.5.178.115 $IFACE
post-down brctl delbr $IFACE
#
auto br116
iface br116 inet static
address 172.30.116.1
netmask 255.255.255.0
pre-up brctl addbr $IFACE
post-up route add -host 46.5.178.116 $IFACE
post-down brctl delbr $IFACE
#
auto br117
iface br117 inet static
address 172.30.117.1
netmask 255.255.255.0
pre-up brctl addbr $IFACE
post-up route add -host 46.5.178.117 $IFACE
post-down brctl delbr $IFACE
#
auto br118
iface br118 inet static
address 172.30.118.1
netmask 255.255.255.0
pre-up brctl addbr $IFACE
post-up route add -host 46.5.178.118 $IFACE
post-down brctl delbr $IFACE
#
auto br119
iface br119 inet static
address 172.30.119.1
netmask 255.255.255.0
pre-up brctl addbr $IFACE
post-up route add -host 46.5.178.119 $IFACE
post-down brctl delbr $IFACE


In order to make sure Xen scripts don’t create the normal bridges when a DOMu is started, we need to hinder this process by:
editing the file /etc/xen/xend-config.sxp and change the line:(around line 176)
FROM:
(network-script network-bridge)
TO:
(network-script none)
Reboot for the new network configuration to take effect:
reboot

DOMUs Configuration

PyGRUB
If your DOMUs configurations are set to use pygrub as boot loader,
then make sure the path to pygrub in the DOMU configuration file is correct as follows:
bootloader = '/usr/lib/xen-4.4/bin/pygrub'
In the same DOMU configuration file, make sure you are using a non duplicated MAC addresses with the network interfaces assignment as well as define the bridge that will be used by this DOMu, for example:
vif = [ 'ip=46.5.178.112,mac=00:16:34:D7:9C:F8,bridge=br112', 'ip=192.168.100.112,mac=00:16:3E:D7:1C:13,bridge=pdummy0' ]

NOTE:If you are not using the PyGRUb and want to use it as boot loader for each individual DOMUs, which makes the DOMUs kernel independent from the DOM0, see the following article. Please notice that in Ubuntu 14.04 the path to pygrub is different than in the article. Each new version of Xen has a different path to PyGRUB th rest of the article is fully accurate for Ubuntu as well.
http://tipstricks.itmatrix.eu/?s=pygrub&x=0&y=0

DOMus Network Configuration

Each DOMu will get an interface lo, eth0 and eth1 with the following configuration:
I’m using the first IP of our subnet for this DOMU and will therefore be configured as follows:
Note: This configuration is not really standard as it uses each IP with the netmask /32 (255.255.255.255). This setting allows each IP of the subnet to be usable by each DOMu. The configuration pointopoint allows it to reach the gateway.

File: /etc/network/interfaces
Content:
# The loopback network interface
auto lo
iface lo inet loopback
#
# The primary network interface
auto eth0
iface eth0 inet static
address 46.5.178.112
netmask 255.255.255.255
gateway 178.61.78.140
pointopoint 178.61.78.140
#
auto eth1
iface eth1 inet static
address 192.168.100.112
netmask 255.255.255.0