msgbartop
MAC OS X, Linux, Windows and other IT Tips and Tricks
msgbarbottom

22 Aug 14 Installing Icinga2, pnp4nagios, NRPE Agent and Graphite in Debian Wheezy

Since Icinga2 is not part of the stable version of Debian Wheezy we need to install it from the wheezy-backports. Therefore the following steps are suggested:

Add the backports to the Debian repositories list:
mcedit /etc/apt/sources.list
Add the following lines:
deb //ftp.debian.org/debian/ wheezy-backports main contrib non-free
deb-src //ftp.debian.org/debian/ wheezy-backports main contrib non-free

Update the repo db.
apt-get update
Install icinga2 and Graphite from backports
apt-get install -t wheezy-backports icinga2 icinga2-classicui libapache2-mod-wsgi
During the installation the password of Icinga Admin user icingaadmin will be asked to be created.
Give this password 2 times.

PNP4NAGIOS


(Ref: Based on: //wiki.icinga.org/display/howtos/Setting+up+PNP+with+Icinga2+on+Debian)
apt-get update && apt-get install --no-install-recommends pnp4nagios rrdcached
update-rc.d rrdcached defaults

We will install PNP with “Bulk mode with NPCD”
icinga2-enable-feature perfdata
Edit the file /etc/pnp4nagios/apache.conf and change the line:
AuthUserFile /etc/nagios3/htpasswd.users
TO:
AuthuserFile /etc/icinga2/classicui/htpasswd.users
Edit the file /etc/pnp4nagios/npcd.cfg and make the changes as follows:
#perfdata_spool_dir = /var/spool/pnp4nagios/npcd/
perfdata_spool_dir = /var/spool/icinga2/perfdata

Enable the npcd daemon
Edit /etc/default/npcd and change the Run line to:
Run="yes"
update-rc.d npcd defaults
mkdir -p /var/cache/rrdcached
chown nagios: /var/cache/rrdcached

Edit /etc/default/rrdcached and add the following line:
OPTS="-w 1800 -z 1800 -j /var/cache/rrdcached -s nagios -m 0660 -l unix:/var/run/rrdcached.sock"
Put the Apache user www-data into the nagios group
usermod -G nagios www-data
Edit the file /etc/pnp4nagios/config.php and modify these 2 variables:
#$conf['nagios_base'] = "/cgi-bin/nagios3";
$conf['nagios_base'] = "/cgi-bin/icinga";
...
#$conf['RRD_DAEMON_OPTS'] = '';
$conf['RRD_DAEMON_OPTS'] = 'unix:/var/run/rrdcached.sock';

Edit the file /etc/pnp4nagios/process_perfdata.cfg and change the RRD_DAEMON_OPTS to:
RRD_DAEMON_OPTS = unix:/var/run/rrdcached.sock

Edit the file /etc/icinga2/conf.d/templates.conf and
– Add the following templates at the very beginning of the file:
template Host "pnp-hst" {
action_url = "/pnp4nagios/graph?host=$HOSTNAME$' class='tips' rel='/pnp4nagios/popup?host=$HOSTNAME$&srv=_HOST_"
}
template Service "pnp-svc" {
action_url = "/pnp4nagios/graph?host=$HOSTNAME$&srv=$SERVICEDESC$' class='tips' rel='/pnp4nagios/popup?host=$HOSTNAME$&srv=$SERVICEDESC$"
}

– And add the import lines:
In the Host template:
template Host "generic-host" {
max_check_attempts = 5
check_interval = 1m
retry_interval = 30s
check_command = "hostalive"
import "pnp-hst"
}

In the Services template
template Service "generic-service" {
max_check_attempts = 3
check_interval = 1m
retry_interval = 30s
import "pnp-svc"
}

Restart All the Daemons
service rrdcached restart
service npcd restart
service icinga2 restart

GRAPHITE


Install Graphite-Carbon
apt-get install -t wheezy-backports graphite-carbon
Enable the Graphite Module
icinga2-enable-feature graphite
Enable the auto-start of Icinga2
update-rc.d icinga2 defaults
Enable the auto-start of Graphite-Carbon
Edit /etc/default/graphite-carbon
Change the line: CARBON_CACHE_ENABLED=false
TO CARBON_CACHE_ENABLED=true
Then run: update-rc.d carbon-cache defaults

Graphite WEB Interface


Although Graphite-carbon is part of Wheezy, Graphite-Web is not.
So we need to install it using pip
(Instructions based on: //gist.github.com/tristanbes/4046457)
Install Graphite-Web
apt-get install -y python2.6 python-pip python-cairo python-django python-django-tagging
apt-get install -y libapache2-mod-wsgi python-twisted python-memcache python-pysqlite2 python-simplejson build-essential python-dev
pip install graphite-web

Add the Apache config symlink:
cp /opt/graphite/examples/example-graphite-vhost.conf /opt/graphite/examples/graphite.conf
ln -s /opt/graphite/examples/graphite.conf /etc/apache2/sites-enabled/

Adapt the following Apache configuration to your environment
/opt/graphite/examples/graphite.conf
For Debian you also need to change the following Entry:
FROM: WSGISocketPrefix run/wsgi
TO: WSGISocketPrefix /var/run/apache2/wsgi
Configure Graphite
cd /opt/graphite/conf/
cp dashboard.conf.example dashboard.conf
cp graphTemplates.conf.example graphTemplates.conf
cp graphite.wsgi.example graphite.wsgi

Symlink the Debian Graphite-Carbon data directory to this Graphite-Web installation:
rmdir /opt/graphite/storage/whisper
ln -s /var/lib/graphite/whisper /opt/graphite/storage/whisper

Configure the local settings of Graphite:
cp /opt/graphite/webapp/graphite/local_settings.py.example /opt/graphite/webapp/graphite/local_settings.py
Edit the file /opt/graphite/webapp/graphite/local_settings.py
and change the following settings:
#SECRET_KEY = 'UNSAFE_DEFAULT'
TO: SECRET_KEY = 'YOUR_OWN_KEY_WITHOUT_SPACES'
Change the following lines FROM:
#LOG_RENDERING_PERFORMANCE = True
#LOG_CACHE_PERFORMANCE = True
#LOG_METRIC_ACCESS = True

TO:
LOG_RENDERING_PERFORMANCE = False
LOG_CACHE_PERFORMANCE = False
LOG_METRIC_ACCESS = False

Note:Activate and adapt other values as you see appropriate. Like:
TIME_ZONE = 'Europe/Berlin'

Finish the Graphite-Web installation process
cd /opt/graphite/webapp/graphite
python manage.py syncdb

You’ll be presented with the following question:
You just installed Django's auth system, which means you don't have any superusers defined.
Would you like to create one now? (yes/no):

Answer no.
Now you might get some error message but for now they should be ignored.
The administrator account will be created later.

Make the data area writable by Apache user (www-data)
chown -R www-data:www-data /opt/graphite/storage/

Configure Graphite-Carbon to receive Icinga data properly
Edit the file: /etc/carbon/storage-schemas.conf
Add the following lines:
[icinga_internals]
pattern = ^icinga\..*\.(max_check_attempts|reachable|current_attempt|execution_time|latency|state|state_type)
retentions = 5m:7d
#
[icinga_default]
# intervals like PNP4Nagios uses them per default
pattern = ^icinga\.
retentions = 1m:2d,5m:10d,30m:90d,360m:4y

Make a fix of graphite carbon:
Otherwise the start of carbon gives the following error:
‘ImportError: cannot import name daemonize’
pip install daemonize
vim /usr/lib/python2.7/dist-packages/carbon/util.py

And change the line:
from twisted.scripts._twistd_unix import daemonize
to
import daemonize

Start Graphite-Carbon daemon
service carbon-cache start
Reload Apache2 configurations
service apache2 reload

Access the Icinga admin web interface via:
//my.monitoring.com/icinga2-classicui/
Login:
Name: icingaadmin
Password: (admin password you created during installation)

Access the Graphite graphs via:
//graphite.my.monitoring.com/
graphite.yourserver.com Is the Name of ‘ServerName’ config in /opt/graphite/examples/graphite.conf

Configure a SuperAdmin for enabling saving Graphite graphs.
cd /opt/graphite/webapp/graphite
export LC_ALL=en_GB.UTF-8
export LANG=en_GB.UTF-8
python manage.py createsuperuser

And give the new administrator credentials.

Saving graphs in graphite:
In order to be able to save graphs in Graphite-Web you need to:
– login as administrator on the graphite-web interface with the URL:
//your.graphite.com/admin/
– Create new user(s) (give access rights in thier profile)
– Login in the graphite-web as one of the users created.
– A new Disk button will appear on top left of the Graph composer window.

Using the Nagios/Icinga agent NRPE


Note: Icinga is happy to use the regular method of calling nagios scripts, in the remote monitored server, by using the SSH command. BUT in case you blocked this port, in whatever manner, I suggest using the NRPE agent which listens on port 5666 and is dedicated for monitoring only.
INSTALLATION:
This installation is to be repeated in each server you wish to monitor.
apt-get install nagios-nrpe-server nagios-plugins sudo
The following package is recommended to make tests locally of the configured NRPE daemon:
apt-get install nagios-nrpe-plugin
To allow some commands that require being root, you allow the user nagios running the scripts contained in in the nagios /usr/lib/nagios/plugins/
WARNING: In some systems this might pose a security problem. So be careful when using this method.
nagios ALL=(ALL) NOPASSWD: /usr/lib/nagios/plugins/
Edit the NRPE configuration file /etc/nagios/nrpe_local.cfg and add the following entry:
command_prefix=/usr/bin/sudo
Note: If you have a firewall make sure the port 5666 is open, otherwise you can change the port in this same file as desired.
server_port=5666
If your monitored server has more than one network interface, it might be wise to limit the connection to only one interface through which the connection from Icinga2 server is made,. Otherwise do not use this entry and nrpe daemon will bind to all interfaces.
Example:
server_address=12.23.34.56
This is the address of the interface NRPE should bind to.
Tip: For testing locally the nrpe commands you can:
– set the above server_address=127.0.0.1
– restart the nrpe daemon(service nagios-nrpe-server restart)
– and use the following command format to test any nrpe command:
/usr/lib/nagios/plugins/check_nrpe -H 127.0.0.1 ........
And the most important setting is to limit the NRPE connections to the Icinga server by configuring this entry.
If you have 2 or more servers that should be allowed to talk to NRPE, delimit them with a comma.
Example:
allowed_hosts=34.45.23.67,34.45.23.68,127.0.0.1

Calling NRPE


It is recommended to always add the localhost(127.0.0.1) in order to be able to test the NRPE commands locally for debug purposes.
Each command you want to call from Icinga2 has to be entered here. The command arguments can be fixed like the following:
command[check_users]=/usr/lib/nagios/plugins/check_users -w 5 -c 10
command[check_load]=/usr/lib/nagios/plugins/check_load -w 15,10,5 -c 30,25,20
command[check_hda1]=/usr/lib/nagios/plugins/check_disk -w 20% -c 10% -p /dev/hda1
command[check_zombie_procs]=/usr/lib/nagios/plugins/check_procs -w 5 -c 10 -s Z
command[check_total_procs]=/usr/lib/nagios/plugins/check_procs -w 150 -c 200

If you want to control the WARNING, CRITICAL levels and other arguments from Icinga when calling the nrpe command, then use the following method:
In this case remember to add the following entry in the configuration file:
dont_blame_nrpe=1
Examples:
command[check_users]=/usr/lib/nagios/plugins/check_users -w $ARG1$ -c $ARG2$
command[check_load]=/usr/lib/nagios/plugins/check_load -w $ARG1$ -c $ARG2$
command[check_disk]=/usr/lib/nagios/plugins/check_disk -w $ARG1$ -c $ARG2$ -p $ARG3$
command[check_procs]=/usr/lib/nagios/plugins/check_procs -w $ARG1$ -c $ARG2$ -s $ARG3$

Note: Take a look at the commands provided by the installed packages in the directory /usr/lib/nagios/plugins/ and enter the ones you will need for monitoring this server.
So resume, a sample of my NRPE server configuration(/etc/nagios/nrpe_local.cfg):
######################################
# Do any local nrpe configuration here
######################################
server_port=5666
server_address=192.168.100.5
allowed_hosts=192.168.100.3,127.0.0.1
command_prefix=/usr/bin/sudo
dont_blame_nrpe=1
command[check_users]=/usr/lib/nagios/plugins/check_users -w $ARG1$ -c $ARG2$
command[check_load]=/usr/lib/nagios/plugins/check_load -w $ARG1$ -c $ARG2$
command[check_disk]=/usr/lib/nagios/plugins/check_disk -w $ARG1$ -c $ARG2$ -p $ARG3$
command[check_procs]=/usr/lib/nagios/plugins/check_procs -w $ARG1$ -c $ARG2$ -s $ARG3$
........ (a long list of commands are listed here)

In the above configuration I’m using the internal LAN to connect from the Icinga2 server (192.168.100.5) to all of my servers to be monitored. To make temporary tests locally, the server_address needs to be changed to 127.0.0.1, restart the nrpe daemon(service nagios-nrpe-server restart) and use the /usr/lib/nagios/plugins/check_nrpe command with arguments as above to test the local nrpe daemon. When finished, revert to the proper binding address of server_address.

When finished restart NRPE:
service nagios-nrpe-server restart

Compiling NRPE from sources

If you find the distribution package of NRPE too old and want to compile your own do the following:
– Install the Debian nagios-nrpe-server and nagios-nrpe-plugin as shown above
– Compile and overwrite the Debian nrpe binaries with the following commands.
mkdir -p /usr/src/NRPE
cd /usr/src/NRPE
wget //downloads.sourceforge.net/project/nagios/nrpe-2.x/nrpe-2.15/nrpe-2.15.tar.gz
tar fvxz nrpe-2.15.tar.gz
cd nrpe-2.15
./configure --with-ssl-lib=$(dpkg -L libssl1.0.0 | grep libcrypto | cut -d/ -f1,2,3,4)
make

Installing the daemon and the check plugin:
cp src/nrpe /usr/sbin/
cp src/check_nrpe /usr/lib/nagios/plugins/

Testing the local NRPE daemon locally with an example command:
/usr/lib/nagios/plugins/check_nrpe -H 127.0.0.1 -c check_disk -a 15 17 /
Example of proper Result:
DISK OK - free space: / 17576 MB (91% inode=94%);| /=1557MB;20143;20141;0;20158

Icinga server configuration for issuing remote NRPE commands


To be continued ……

Note: In order to make sure you are installing only Icinga2 etc. from backports and leave all other packages to stable version in future installations, after installation of Icinga2 just comment out the above added backport lines in /etc/apt/sources.list and update the repo. db again (apt-get update).

Happy Monitoring!!

Reader's Comments

  1.    

    Thanks so much for this!

    Reply to this comment
  2.    

    followed your tips.
    couldnt get the performance graphs to show inside the classicui interface.
    any ideas?

    Reply to this comment
    •    

      I guess you mean the graphs of pnp4nagios. If so then the question is:
      Do you see a small gear like icon next to the localhost item in icinga GUI?
      If yes then by clicking in it you get to have a set of graphs of this performance data inside the HTML frame of the icinga GUI.
      If you don’t see any gear icon then maybe some steps got forgotten. Just verify again that all the steps got done.
      It could also be that some services involved need to be restarted like the npcd, apache2, rrdcached or icinga2 services.
      Take a look as well at some of the logs related to those services to see if anything went wrong.
      For the moment I cannot see what else to propose.
      Good luck. Let me know if you find anything strange in the logs.

      Reply to this comment
  3.    

    I get gear icon in classicui but when clicked it i get “Not found -> URL /pnp4nagios/graph not found on this server”

    Reply to this comment
    •    

      When I created this tutorial I had just learned it myself. That is the main purpose of those tips-and-tricks blog articles. I write them every now and then as a reminder for things I just learned and most probably will soon forget it. Unfortunately right after I created this article I had to leave this subject and go on to other tasks not related to it at all. I really wanted to got to it soon but it simply didn’t happen. So sorry but that means I can’t help you with this problem at the moment. I wish you good luck searching for a solution in the net.

      Reply to this comment
  4.    

    root@icinga2:~# sudo apt-get install pnp4nagios
    Reading package lists… Done
    Building dependency tree
    Reading state information… Done
    E: Unable to locate package pnp4nagios

    Reply to this comment
    •    

      Hi, unfortunately I also got the same message when I tried to install it in a newer version of Debian.
      So far since I have lots of other things to do and the need for this is very low I have to put the search for a solution on ice for now.
      If you find a solution I would appreciate and many others as well that you share it with all of us as a comment in the post.
      Thanks.

      Reply to this comment

Leave a Reply to Damr Cancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: