msgbartop
MAC OS X, Linux, Windows and other IT Tips and Tricks
msgbarbottom

07 May 15 TCP Load balancing email/web servers with NginX

I’ve got 2 synchronized email servers running and, in order to make sure I don’t have to change the servername settings of my mail client in case one server goes down, I was looking for a straight TCP layer load balancer. There are a few software packages on the market that can do that , eg. Lvs-kiss etc. I found the solution of using NginX quite interesting, since the load balancing seems to be better built especially regarding the return route which gave me some headaches with LVS-KISS. Here is an example of using NginX and TCP load balancing to 2 IMAPs/SMTPs/Webmail servers.

Note: Unfortunately my IMAPs/SMTPs servers are using a legitimate certificates but because the email clients do use the address of an extra server for Loadbalancing, the certificate is declared invalid by the email clients. This TCP load balancing operates at a lower layer(TCP) than application layer where SSL certificates can be used to authenticate. It is therefore not possible to add a certificate to this load balancer. For this reason it is recommended to use a wildcard certificate in both back-end Mail servers for production use of Mail services load-balancing. For HTTP and HTTPS there is no need for a wildcard certificate. Normal certificates installed in both web servers will do.

NginX shortcomings


NginX did have the TCP load-balancing feature compiled only in the Pro version with all the features of backend health check etc.
Since the version 1.9 they introduced a limited version of the TCP load-balancing feature into the community version.
Unfortunately if you need other pro features like backend health-checks you are out of luck.
Fortunately some 3rd parties have created a patch for earlier versions which implement the necessary features of a good low level TCP load balancer. In this tutorial I will show how to compile and patch an earlier version of Nginx to achieve the TCP Load-balancer.

Steps


Login as root and run the following commands:
apt-get remove nginx
apt-get install libssl-dev build-essential git
mkdir ~/build ; cd ~/build
wget -O - http://nginx.org/download/nginx-1.6.2.tar.gz | tar xfvz -
git clone git://github.com/yaoweibin/nginx_tcp_proxy_module
cd nginx-1.6.2/
patch -p1 < ../nginx_tcp_proxy_module/tcp.patch ./configure --add-module=../nginx_tcp_proxy_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --without-http_rewrite_module --without-http_charset_module --without-http_gzip_module --without-http_ssi_module --without-http_userid_module --without-http_access_module --without-http_auth_basic_module --without-http_autoindex_module --without-http_geo_module --without-http_map_module --without-http_split_clients_module --without-http_referer_module --without-http_proxy_module --without-http_fastcgi_module --without-http_uwsgi_module --without-http_scgi_module --without-http_memcached_module --without-http_limit_conn_module --without-http_limit_req_module --without-http_empty_gif_module --without-http_browser_module --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --http-client-body-temp-path=/var/lib/nginx/body make && make install mkdir -p /usr/share/nginx/logs mkdir /var/log/nginx mkdir -p /var/lib/nginx/body touch /var/log/nginx/error.log /var/log/nginx/access.log touch /usr/share/nginx/logs/tcp_access.log chown -R www-data: /var/{lib,log}/nginx /usr/share/nginx/logs

Check it's version:
/usr/share/nginx/sbin/nginx -V
Create an init start/stop script
touch /etc/init.d/nginx
chmod 755 /etc/init.d/nginx
mcedit /etc/init.d/nginx

Content:
#!/bin/sh
### BEGIN INIT INFO
# Provides: nginx
# Required-Start: $local_fs $remote_fs $network $syslog
# Required-Stop: $local_fs $remote_fs $network $syslog
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: starts the nginx web server
# Description: starts nginx using start-stop-daemon
### END INIT INFO
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DAEMON=/usr/share/nginx/sbin/nginx
NAME=nginx
DESC=nginx
# Include nginx defaults if available
if [ -f /etc/default/nginx ]; then
. /etc/default/nginx
fi
test -x $DAEMON || exit 0
set -e
. /lib/lsb/init-functions
test_nginx_config() {
if $DAEMON -t $DAEMON_OPTS >/dev/null 2>&1; then
return 0
else
$DAEMON -t $DAEMON_OPTS
return $?
fi
}
case "$1" in
start)
echo -n "Starting $DESC: "
test_nginx_config
# Check if the ULIMIT is set in /etc/default/nginx
if [ -n "$ULIMIT" ]; then
# Set the ulimits
ulimit $ULIMIT
fi
start-stop-daemon --start --quiet --pidfile /var/run/$NAME.pid \
--exec $DAEMON -- $DAEMON_OPTS || true
echo "$NAME."
;;
stop)
echo -n "Stopping $DESC: "
start-stop-daemon --stop --quiet --pidfile /var/run/$NAME.pid \
--exec $DAEMON || true
echo "$NAME."
;;
restart|force-reload)
echo -n "Restarting $DESC: "
start-stop-daemon --stop --quiet --pidfile \
/var/run/$NAME.pid --exec $DAEMON || true
sleep 1
test_nginx_config
# Check if the ULIMIT is set in /etc/default/nginx
if [ -n "$ULIMIT" ]; then
# Set the ulimits
ulimit $ULIMIT
fi
start-stop-daemon --start --quiet --pidfile \
/var/run/$NAME.pid --exec $DAEMON -- $DAEMON_OPTS || true
echo "$NAME."
;;
reload)
echo -n "Reloading $DESC configuration: "
test_nginx_config
start-stop-daemon --stop --signal HUP --quiet --pidfile /var/run/$NAME.pid \
--exec $DAEMON || true
echo "$NAME."
;;
configtest|testconfig)
echo -n "Testing $DESC configuration: "
if test_nginx_config; then
echo "$NAME."
else
exit $?
fi
;;
status)
status_of_proc -p /var/run/$NAME.pid "$DAEMON" nginx && exit 0 || exit $?
;;
*)
echo "Usage: $NAME {start|stop|restart|reload|force-reload|status|configtest}" >&2
exit 1
;;
esac
exit 0

The NGinX configuration here TCP-load-balances the following ports: 143, 993, 587, 465, 80 & 443.

Configuration:
To use only the TCP load balancing feature of NginX we configure the strict minimum:
Rename the created configuration file
mv /etc/nginx/nginx.conf /etc/nginx/nginx.conf.orig
Create the new configuration file with the following content:
mcedit /etc/nginx/nginx.conf
Content:
user www-data;
worker_processes 1;
events {
worker_connections 1024;
}
# ---------- TCP Load balancer for IMAPs, IMAP and SMTPS -----------------
tcp {
# ----------------- IMAPs ------------------
upstream cluster_imaps {
server mail2.itmatrix.eu:993;
server mail3.itmatrix.eu:993;
check interval=5000 rise=2 fall=5 timeout=2000 type=tcp;
ip_hash;
}
server {
listen 993;
proxy_pass cluster_imaps;
}
# ----------------- IMAP -------------------
upstream cluster_imap {
server mail2.itmatrix.eu:143;
server mail3.itmatrix.eu:143;
check interval=5000 rise=2 fall=5 timeout=2000 type=imap;
ip_hash;
}
server {
listen 143;
proxy_pass cluster_imap;
}
# ----------------- SMTP -------------------
upstream cluster_smtp {
server mail2.itmatrix.eu:587;
server mail3.itmatrix.eu:587;
check interval=5000 rise=2 fall=5 timeout=2000 type=smtp;
ip_hash;
}
server {
listen 587;
proxy_pass cluster_smtp;
}
# ----------------- SMTP -------------------
upstream cluster_smtps {
server mail2.itmatrix.eu:465;
server mail3.itmatrix.eu:465;
check interval=5000 rise=2 fall=5 timeout=2000 type=smtp;
ip_hash;
}
server {
listen 465;
proxy_pass cluster_smtps;
}
# ----------------- HTTP -------------------
upstream cluster_http {
server mail2.itmatrix.eu:80;
server mail3.itmatrix.eu:80;
check interval=5000 rise=2 fall=5 timeout=2000 type=tcp;
ip_hash;
}
server {
listen 80;
proxy_pass cluster_http;
}
# ----------------- HTTPS -------------------
upstream cluster_https {
server mail2.itmatrix.eu:443;
server mail3.itmatrix.eu:443;
check interval=5000 rise=2 fall=5 timeout=2000 type=tcp;
ip_hash;
}
server {
listen 443;
proxy_pass cluster_https;
}
#--------------- End of TCP Block --------------
}

Start NginX server:
service nginx start

Some explanations of this NginX configuration


UPSTREAM OPTION: check
========================
interval=2000 # Alive-Check interval for the back-end mail servers, 2s
rise=2 # How many Alive-Checks must be successful in order to consider the server as ON-line
fall=5 # How many Alive-Checks must fail in order to consider the server as OFF-line
timeout=1000 # Timeout for responses of Alive-Checks Here: 1 sec.
type=imap; # Type of Alive-Check

STICKY Sessions
ip_hash; # Based on client IP

An overview of types from NGinX Git repositories:
1. tcp is a simple tcp socket connect and peek one byte.
2. ssl_hello sends a client ssl hello packet and receives the server ssl hello packet.
3. http sends a http request packet, receives and parses the http response to diagnose if the upstream server is alive.
4. smtp sends a smtp request packet, receives and parses the smtp response to diagnose if the upstream server is alive. The response begins with ‘2’ should be an OK response.
5. mysql connects to the mysql server, receives the greeting response to diagnose if the upstream server is alive.
6. pop3 receives and parses the pop3 response to diagnose if the upstream server is alive. The response begins with ‘+’ should be an OK response.
7. imap connects to the imap server, receives the greeting response to diagnose if the upstream server is alive.

Reader's Comments

  1.    

    Thanks for this interesting post. It is very informative indeed. Can you also suggest anyway to handle the Real IP Address problem as after following your post if I loadbalance, my mail server is getting the loadbalancer’s IP as incoming connection, whereas I want the client’s actual IP to be forwarded to the mail server.

    Reply to this comment
    •    

      Well regarding retaining the source(Client) IP address in the TCP requests being load-balanced by nGinX, I havent found a solution yet. If such feature is absolutely needed then I suggest using HAProxy and look at the following article at the very end: https://wp.me/pKZRY-Ni See the last paragraph.

      Reply to this comment

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: