IT Tips and Tricks

Since Icinga2 is not part of the stable version of Debian Wheezy we need to install it from the wheezy-backports. Therefore the following steps are suggested:

Add the backports to the Debian repositories list:
mcedit /etc/apt/sources.list
Add the following lines:
deb http://ftp.debian.org/debian/ wheezy-backports main contrib non-free
deb-src http://ftp.debian.org/debian/ wheezy-backports main contrib non-free
Update the repo db.
apt-get update
Install icinga2 and Graphite from backports
apt-get install -t wheezy-backports icinga2 icinga2-classicui libapache2-mod-wsgi
During the installation the password of Icinga Admin user icingaadmin will be asked to be created.
Give this password 2 times.

PNP4NAGIOS

(Ref: Based on: https://wiki.icinga.org/display/howtos/Setting+up+PNP+with+Icinga2+on+Debian)
apt-get update && apt-get install --no-install-recommends pnp4nagios rrdcached
update-rc.d rrdcached defaults
We will install PNP with “Bulk mode with NPCD”
icinga2-enable-feature perfdata
Edit the file /etc/pnp4nagios/apache.conf and change the line:
AuthUserFile /etc/nagios3/htpasswd.users
TO:
AuthuserFile /etc/icinga2/classicui/htpasswd.users
Edit the file /etc/pnp4nagios/npcd.cfg and make the changes as follows:
#perfdata_spool_dir = /var/spool/pnp4nagios/npcd/
perfdata_spool_dir = /var/spool/icinga2/perfdata
Enable the npcd daemon
Edit /etc/default/npcd and change the Run line to:
Run="yes"
update-rc.d npcd defaults
mkdir -p /var/cache/rrdcached
chown nagios: /var/cache/rrdcached
Edit /etc/default/rrdcached and add the following line:
OPTS="-w 1800 -z 1800 -j /var/cache/rrdcached -s nagios -m 0660 -l unix:/var/run/rrdcached.sock"
Put the Apache user www-data into the nagios group
usermod -G nagios www-data
Edit the file /etc/pnp4nagios/config.php and modify these 2 variables:
#$conf['nagios_base'] = "/cgi-bin/nagios3";
$conf['nagios_base'] = "/cgi-bin/icinga";
...
#$conf['RRD_DAEMON_OPTS'] = '';
$conf['RRD_DAEMON_OPTS'] = 'unix:/var/run/rrdcached.sock';

Edit the file /etc/pnp4nagios/process_perfdata.cfg and change the RRD_DAEMON_OPTS to:
RRD_DAEMON_OPTS = unix:/var/run/rrdcached.sock

Edit the file /etc/icinga2/conf.d/templates.conf and
– Add the following templates at the very beginning of the file:
template Host "pnp-hst" {
action_url = "/pnp4nagios/graph?host=$HOSTNAME$' class='tips' rel='/pnp4nagios/popup?host=$HOSTNAME$&srv=_HOST_"
}
template Service "pnp-svc" {
action_url = "/pnp4nagios/graph?host=$HOSTNAME$&srv=$SERVICEDESC$' class='tips' rel='/pnp4nagios/popup?host=$HOSTNAME$&srv=$SERVICEDESC$"
}

– And add the import lines:
In the Host template:
template Host "generic-host" {
max_check_attempts = 5
check_interval = 1m
retry_interval = 30s
check_command = "hostalive"
import "pnp-hst"
}
In the Services template
template Service "generic-service" {
max_check_attempts = 3
check_interval = 1m
retry_interval = 30s
import "pnp-svc"
}

Restart All the Daemons
service rrdcached restart
service npcd restart
service icinga2 restart

GRAPHITE

Install Graphite-Carbon
apt-get install -t wheezy-backports graphite-carbon
Enable the Graphite Module
icinga2-enable-feature graphite
Enable the auto-start of Icinga2
update-rc.d icinga2 defaults
Enable the auto-start of Graphite-Carbon
Edit /etc/default/graphite-carbon
Change the line: CARBON_CACHE_ENABLED=false
TO CARBON_CACHE_ENABLED=true
Then run: update-rc.d carbon-cache defaults

Graphite WEB Interface

Although Graphite-carbon is part of Wheezy, Graphite-Web is not.
So we need to install it using pip
(Instructions based on: https://gist.github.com/tristanbes/4046457)
Install Graphite-Web
apt-get install -y python2.6 python-pip python-cairo python-django python-django-tagging
apt-get install -y libapache2-mod-wsgi python-twisted python-memcache python-pysqlite2 python-simplejson build-essential python-dev
pip install graphite-web
Add the Apache config symlink:
cp /opt/graphite/examples/example-graphite-vhost.conf /opt/graphite/examples/graphite.conf
ln -s /opt/graphite/examples/graphite.conf /etc/apache2/sites-enabled/
Adapt the following Apache configuration to your environment
/opt/graphite/examples/graphite.conf
For Debian you also need to change the following Entry:
FROM: WSGISocketPrefix run/wsgi
TO: WSGISocketPrefix /var/run/apache2/wsgi
Configure Graphite
cd /opt/graphite/conf/
cp dashboard.conf.example dashboard.conf
cp graphTemplates.conf.example graphTemplates.conf
cp graphite.wsgi.example graphite.wsgi
Symlink the Debian Graphite-Carbon data directory to this Graphite-Web installation:
rmdir /opt/graphite/storage/whisper
ln -s /var/lib/graphite/whisper /opt/graphite/storage/whisper
Configure the local settings of Graphite:
cp /opt/graphite/webapp/graphite/local_settings.py.example /opt/graphite/webapp/graphite/local_settings.py
Edit the file /opt/graphite/webapp/graphite/local_settings.py
and change the following settings:
#SECRET_KEY = 'UNSAFE_DEFAULT'
TO: SECRET_KEY = 'YOUR_OWN_KEY_WITHOUT_SPACES'
Change the following lines FROM:
#LOG_RENDERING_PERFORMANCE = True
#LOG_CACHE_PERFORMANCE = True
#LOG_METRIC_ACCESS = True
TO:
LOG_RENDERING_PERFORMANCE = False
LOG_CACHE_PERFORMANCE = False
LOG_METRIC_ACCESS = False
Note:Activate and adapt other values as you see appropriate. Like:
TIME_ZONE = 'Europe/Berlin'

Finish the Graphite-Web installation process
cd /opt/graphite/webapp/graphite
python manage.py syncdb
You’ll be presented with the following question:
You just installed Django's auth system, which means you don't have any superusers defined.
Would you like to create one now? (yes/no):
Answer no.
Now you might get some error message but for now they should be ignored.
The administrator account will be created later.

Make the data area writable by Apache user (www-data)
chown -R www-data:www-data /opt/graphite/storage/

Configure Graphite-Carbon to receive Icinga data properly
Edit the file: /etc/carbon/storage-schemas.conf
Add the following lines:
[icinga_internals]
pattern = ^icinga\..*\.(max_check_attempts|reachable|current_attempt|execution_time|latency|state|state_type)
retentions = 5m:7d
#
[icinga_default]
# intervals like PNP4Nagios uses them per default
pattern = ^icinga\.
retentions = 1m:2d,5m:10d,30m:90d,360m:4y

Make a fix of graphite carbon:
Otherwise the start of carbon gives the following error:
‘ImportError: cannot import name daemonize’
pip install daemonize
vim /usr/lib/python2.7/dist-packages/carbon/util.py
And change the line:
from twisted.scripts._twistd_unix import daemonize
to
import daemonize

Start Graphite-Carbon daemon
service carbon-cache start
Reload Apache2 configurations
service apache2 reload

Access the Icinga admin web interface via:
http://my.monitoring.com/icinga2-classicui/
Login:
Name: icingaadmin
Password: (admin password you created during installation)

Access the Graphite graphs via:
http://graphite.my.monitoring.com/
graphite.yourserver.com Is the Name of ‘ServerName’ config in /opt/graphite/examples/graphite.conf

Configure a SuperAdmin for enabling saving Graphite graphs.
cd /opt/graphite/webapp/graphite
export LC_ALL=en_GB.UTF-8
export LANG=en_GB.UTF-8
python manage.py createsuperuser
And give the new administrator credentials.

Saving graphs in graphite:
In order to be able to save graphs in Graphite-Web you need to:
– login as administrator on the graphite-web interface with the URL:
http://your.graphite.com/admin/
– Create new user(s) (give access rights in thier profile)
– Login in the graphite-web as one of the users created.
– A new Disk button will appear on top left of the Graph composer window.

Using the Nagios/Icinga agent NRPE

Note: Icinga is happy to use the regular method of calling nagios scripts, in the remote monitored server, by using the SSH command. BUT in case you blocked this port, in whatever manner, I suggest using the NRPE agent which listens on port 5666 and is dedicated for monitoring only.
INSTALLATION:
This installation is to be repeated in each server you wish to monitor.
apt-get install nagios-nrpe-server nagios-plugins sudo
The following package is recommended to make tests locally of the configured NRPE daemon:
apt-get install nagios-nrpe-plugin
To allow some commands that require being root, you allow the user nagios running the scripts contained in in the nagios /usr/lib/nagios/plugins/
WARNING: In some systems this might pose a security problem. So be careful when using this method.
nagios ALL=(ALL) NOPASSWD: /usr/lib/nagios/plugins/
Edit the NRPE configuration file /etc/nagios/nrpe_local.cfg and add the following entry:
command_prefix=/usr/bin/sudo
Note: If you have a firewall make sure the port 5666 is open, otherwise you can change the port in this same file as desired.
server_port=5666
If your monitored server has more than one network interface, it might be wise to limit the connection to only one interface through which the connection from Icinga2 server is made,. Otherwise do not use this entry and nrpe daemon will bind to all interfaces.
Example:
server_address=12.23.34.56
This is the address of the interface NRPE should bind to.
Tip: For testing locally the nrpe commands you can:
– set the above server_address=127.0.0.1
– restart the nrpe daemon(service nagios-nrpe-server restart)
– and use the following command format to test any nrpe command:
/usr/lib/nagios/plugins/check_nrpe -H 127.0.0.1 ........
And the most important setting is to limit the NRPE connections to the Icinga server by configuring this entry.
If you have 2 or more servers that should be allowed to talk to NRPE, delimit them with a comma.
Example:
allowed_hosts=34.45.23.67,34.45.23.68,127.0.0.1

Calling NRPE

It is recommended to always add the localhost(127.0.0.1) in order to be able to test the NRPE commands locally for debug purposes.
Each command you want to call from Icinga2 has to be entered here. The command arguments can be fixed like the following:
command[check_users]=/usr/lib/nagios/plugins/check_users -w 5 -c 10
command[check_load]=/usr/lib/nagios/plugins/check_load -w 15,10,5 -c 30,25,20
command[check_hda1]=/usr/lib/nagios/plugins/check_disk -w 20% -c 10% -p /dev/hda1
command[check_zombie_procs]=/usr/lib/nagios/plugins/check_procs -w 5 -c 10 -s Z
command[check_total_procs]=/usr/lib/nagios/plugins/check_procs -w 150 -c 200
If you want to control the WARNING, CRITICAL levels and other arguments from Icinga when calling the nrpe command, then use the following method:
In this case remember to add the following entry in the configuration file:
dont_blame_nrpe=1
Examples:
command[check_users]=/usr/lib/nagios/plugins/check_users -w $ARG1$ -c $ARG2$
command[check_load]=/usr/lib/nagios/plugins/check_load -w $ARG1$ -c $ARG2$
command[check_disk]=/usr/lib/nagios/plugins/check_disk -w $ARG1$ -c $ARG2$ -p $ARG3$
command[check_procs]=/usr/lib/nagios/plugins/check_procs -w $ARG1$ -c $ARG2$ -s $ARG3$
Note: Take a look at the commands provided by the installed packages in the directory /usr/lib/nagios/plugins/ and enter the ones you will need for monitoring this server.
So resume, a sample of my NRPE server configuration(/etc/nagios/nrpe_local.cfg):
######################################
# Do any local nrpe configuration here
######################################
server_port=5666
server_address=192.168.100.5
allowed_hosts=192.168.100.3,127.0.0.1
command_prefix=/usr/bin/sudo
dont_blame_nrpe=1
command[check_users]=/usr/lib/nagios/plugins/check_users -w $ARG1$ -c $ARG2$
command[check_load]=/usr/lib/nagios/plugins/check_load -w $ARG1$ -c $ARG2$
command[check_disk]=/usr/lib/nagios/plugins/check_disk -w $ARG1$ -c $ARG2$ -p $ARG3$
command[check_procs]=/usr/lib/nagios/plugins/check_procs -w $ARG1$ -c $ARG2$ -s $ARG3$
........ (a long list of commands are listed here)
In the above configuration I’m using the internal LAN to connect from the Icinga2 server (192.168.100.5) to all of my servers to be monitored. To make temporary tests locally, the server_address needs to be changed to 127.0.0.1, restart the nrpe daemon(service nagios-nrpe-server restart) and use the /usr/lib/nagios/plugins/check_nrpe command with arguments as above to test the local nrpe daemon. When finished, revert to the proper binding address of server_address.

When finished restart NRPE:
service nagios-nrpe-server restart

Compiling NRPE from sources

If you find the distribution package of NRPE too old and want to compile your own do the following:
– Install the Debian nagios-nrpe-server and nagios-nrpe-plugin as shown above
– Compile and overwrite the Debian nrpe binaries with the following commands.
mkdir -p /usr/src/NRPE
cd /usr/src/NRPE
wget http://downloads.sourceforge.net/project/nagios/nrpe-2.x/nrpe-2.15/nrpe-2.15.tar.gz
tar fvxz nrpe-2.15.tar.gz
cd nrpe-2.15
./configure --with-ssl-lib=$(dpkg -L libssl1.0.0 | grep libcrypto | cut -d/ -f1,2,3,4)
make
Installing the daemon and the check plugin:
cp src/nrpe /usr/sbin/
cp src/check_nrpe /usr/lib/nagios/plugins/
Testing the local NRPE daemon locally with an example command:
/usr/lib/nagios/plugins/check_nrpe -H 127.0.0.1 -c check_disk -a 15 17 /
Example of proper Result:
DISK OK - free space: / 17576 MB (91% inode=94%);| /=1557MB;20143;20141;0;20158

Icinga server configuration for issuing remote NRPE commands

To be continued ……

Note: In order to make sure you are installing only Icinga2 etc. from backports and leave all other packages to stable version in future installations, after installation of Icinga2 just comment out the above added backport lines in /etc/apt/sources.list and update the repo. db again (apt-get update).

Happy Monitoring!!