IT Tips and Tricks

In the area of monitoring systems using GPL software Nagios is a big player but the organization of
displaying the history graphs is not so ideal.
Zabbix is wonderful as well, it produces also good graphs but it creates them dynamically, resulting
in a long wait time when we need to display a large amount of them at once. To achieve this Cacti
is one of the best GPL History-Graphs-Only monitoring program out there.
Note:
I have put together here a step-by-step HOWTO to help setting-up some of the most important items to be monitored in a short time. Although the theme says to use ‘Lenny’ the same should apply to Squeeze with very minor modification if any.

The difficulty with the version of Cacti provided with Debian Lenny/Squeeze is that it’s outdated.
I tried to install the latest version of Cacti and somehow got into some difficulties which
I felt I didn’t want to bother solving. I knew there was a faster solution to this and didn’t have the time
to do troubleshooting.

The fast solution is to install the regular (and outdated) Debian version of Cacti
and then replace its components with the latest version(0.8.7.g).
For later versions, simply replace the version number where applied. So here is how I did it:

Install the Debian Lenny/Squeeze version

apt-get install cacti
Note: Please make sure you complete the whole installation especially concerning the creation of the Mysql Database.

Download the current Cacti version

(at the time of this article the version 0.8.7g was the latest one.)
wget http://www.cacti.net/downloads/cacti-0.8.7g.tar.gz

Put the Debian Cacti files aside to leave space to the new one.

mv /usr/share/cacti /usr/share/cacti_Debian

Prepare space for the new version

mkdir -p /usr/share/cacti-0.8.7g

Install the new version with the same directory structure as with the Debian version

tar fvxz cacti-0.8.7g.tar.gz -C /usr/share/cacti-0.8.7g/
mv /usr/share/cacti-0.8.7g/cacti-0.8.7g /usr/share/cacti-0.8.7g/site
ln -s /usr/share/cacti-0.8.7g /usr/share/cacti
mv /usr/share/cacti-0.8.7g/site/rra /usr/share/cacti-0.8.7g/site/rra_orig
ln -s /var/lib/cacti/rra /usr/share/cacti-0.8.7g/site/rra
ln -s /usr/share/cacti-0.8.7g/site/resource /usr/share/cacti-0.8.7g/resource


Get a copy of the config.php from Debian version

This php file refers to the configuration file /etc/cacti/debian.php as installed by the original Debian version.
mv /usr/share/cacti/site/include/config.php /usr/share/cacti/site/include/config.php.orig
cp /usr/share/cacti_Debian/site/include/config.php /usr/share/cacti/site/include/config.php


Restart your web server

/etc/init.d/apache2 restart

Upgrade to the new cacti

http://myserver.com/cacti
and simply follow the on screen instructions to upgrade to the new version.

NOTES:
1) Debian Lenny/Squeeze uses the SNMP protocol Ver. 2.0 therefore make sure you do
configure the correct version in the cati web interface.

2) For better compatibility, in the /etc/snmp/snmpd.conf I used only the following configuration:
com2sec readonly default public
group MyROSystem v1 paranoid
group MyROSystem v2c paranoid
group MyROSystem usm paranoid
group MyROGroup v1 readonly
group MyROGroup v2c readonly
group MyROGroup usm readonly
group MyRWGroup v1 readwrite
group MyRWGroup v2c readwrite
group MyRWGroup usm readwrite
view all included .1 80
view system included .iso.org.dod.internet.mgmt.mib-2.system
access MyROSystem "" any noauth exact system none none
access MyROGroup "" any noauth exact all none none
access MyRWGroup "" any noauth exact all all none
syslocation Unknown (configure /etc/snmp/snmpd.local.conf)
syscontact Root (configure /etc/snmp/snmpd.local.conf)
Restart SNMPD Daemon
/etc/init.d/snmpd restart
Testing the SNMP configuration on the monitored system
snmpwalk -Os -c public -v 1 localhost system
3) I prefer to use the UDP Ping instead of the SNMP Ping for finding out if the server is available. This way if the SNMP service dies in the monitored host I get the proper information regarding if the monitored host died or not.

4) In our case, since we are going to get all sorts in information from the system where the user snmp is not allowed, we need to run the snmpd daemon as ‘root’ instead as ‘snmp’. To do this do the following changes:
vim /etc/default/snmpd
snmpd options (use syslog, close stdin/out/err).
SNMPDOPTS='-Lsd -Lf /var/log/snmpd.log -u root -I -smux -p /var/run/snmpd.pid 192.168.0.3,127.0.0.1'
Note: You also need to include the IP of the interface from where the monitoring server will be making its queries. In the above SNMPDOPTS settings, both interfaces having the IPs 192.168.0.3 and 127.0.0.1 will accept snmp queries. If any of the connected to the internet, make sure the /etc/hosts.allow/ & /etc/hosts.deny are correctly configured as follows:

Allowing the monitoring server acccess to snmpd daemon
In /etc/hosts.allow
snmpd: 90.78.145.72
Making sure all other hosts are denied access
In /etc/hosts.deny
snmpd: ALL

Installing some cacti templates

Apache statistics

In the monitored hosts
Make sure that in the client host you configure Apache for extended server-status and if needed configure the access control to accept the monitoring server where cacti runs.

In the monitoring server
– Download the apache template locally in desktop from
– Import the template in Cacti interface
wget http://public.itmatrix.eu/Cacti_templates/Apache/cacti_host_template_webserver_apache.xml
– Login as root:
cd /usr/share/cacti/site/scripts
wget http://public.itmatrix.eu/Cacti_templates/Apache/ws_apachestats.pl


Postfix queues

In the monitored host:
– download the postfix queues script in to /usr/local/bin/
cd /usr/local/bin/
wget http://public.itmatrix.eu/Cacti_templates/Postfix_queues/getmailq.sh
chmod 755 /usr/local/bin/getmailq.sh
Add the necessary SNMP configuration and restart
echo 'extend .1.3.6.1.4.1.2021.53.101.0 mailq /usr/local/bin/getmailq.sh' >> /etc/snmp/snmpd.conf
/etc/init.d/snmpd restart
Testing the script in the monitored host:
/usr/bin/snmpwalk -Oav -v2c -c public 127.0.0.1 .1.3.6.1.4.1.2021.53.101.0.4 | cut -d'"' -f2
You should get an output similar to this:
incoming:0 active:0 deferred:6 hold:0
In the monitoring server:
Log into the monitoring server and get the data query script
cd /usr/share/cacti/site/scripts
wget http://public.itmatrix.eu/Cacti_templates/Postfix_queues/postfixqueues.sh
chmod 755 /usr/share/cacti/site/scripts/postfixqueues.sh
Download and Import the template in cacti interface
http://public.itmatrix.eu/Cacti_templates/Postfix_queues/cacti_graph_template_postfix_queues.xml

Postfix statistics

In the monitored host
Download the postfix stats script in to /usr/local/bin/
cd /usr/local/bin/
wget http://public.itmatrix.eu/Cacti_templates/Postfix_stats/fetch_mail_statistics.pl
chmod 755 /usr/local/bin/fetch_mail_statistics.pl
Add the necessary SNMP configuration and restart
echo 'pass .1.3.6.1.4.1.2021.252 /usr/local/bin/fetch_mail_statistics.pl /var/log/mail.log /var/log/mailstats.db .1.3.6.1.4.1.2021.252' >> /etc/snmp/snmpd.conf
/etc/init.d/snmpd restart
Testing the script on the monitored host:
/usr/bin/snmpwalk -Oav -v2c -c public 127.0.0.1 .1.3.6.1.4.1.2021.252
You should get an output similar to this:
Output Description
============= ====================================
INTEGER: 25 Received
INTEGER: 1 Sent
INTEGER: 58 Deferred (Saved and to be resent later)
INTEGER: 0 Bounced (Bounced back)
INTEGER: 293 Rejected (Total rejected mails)
INTEGER: 15 Clean
INTEGER: 188 Pspam (Possible SPAM GreyListed)
INTEGER: 287 Spam (RBL and spammassassin Blocked)
INTEGER: 0 Infected (Blocked Virus)
INTEGER: 1 Bad Header
INTEGER: 1 Banned (Banned content)
In the Monitoring server (Via cacti web interface)
Download and Import the host template in cacti interface
http://public.itmatrix.eu/Cacti_templates/Postfix_stats/cacti_host_template_postfix_mailserver.xml

DISK IO statistics

In the monitored host:
Download the DISK IO stats script into /usr/local/bin/
cd /usr/local/bin/
wget http://public.itmatrix.eu/Cacti_templates/Disk_IO/snmpdiskio
chmod 755 /usr/local/bin/snmpdiskio

Add entries to /etc/snmp/snmpd.conf
echo 'extend .1.3.6.1.4.1.2021.54 hdNum /usr/local/bin/snmpdiskio hdNum' >> /etc/snmp/snmpd.conf
echo 'extend .1.3.6.1.4.1.2021.55 hdIndex /usr/local/bin/snmpdiskio hdIndex' >> /etc/snmp/snmpd.conf
echo 'extend .1.3.6.1.4.1.2021.56 hdDescr /usr/local/bin/snmpdiskio hdDescr' >> /etc/snmp/snmpd.conf
echo 'extend .1.3.6.1.4.1.2021.57 hdInBlocks /usr/local/bin/snmpdiskio hdInBlocks' >> /etc/snmp/snmpd.conf
echo 'extend .1.3.6.1.4.1.2021.58 hdOutBlocks /usr/local/bin/snmpdiskio hdOutBlocks' >> /etc/snmp/snmpd.conf
Restart snmpd service
/etc/init.d/snmpd restart
Testing the data query on the monitored host:
/usr/bin/snmpwalk -Oav -v2c -c public 127.0.0.1 .1.3.6.1.4.1.2021.54.4
/usr/bin/snmpwalk -Oav -v2c -c public 127.0.0.1 .1.3.6.1.4.1.2021.55.4
/usr/bin/snmpwalk -Oav -v2c -c public 127.0.0.1 .1.3.6.1.4.1.2021.56.4
/usr/bin/snmpwalk -Oav -v2c -c public 127.0.0.1 .1.3.6.1.4.1.2021.57.4
/usr/bin/snmpwalk -Oav -v2c -c public 127.0.0.1 .1.3.6.1.4.1.2021.58.4
In the Monitoring server:
Install the needed xml file
mkdir -p /usr/share/cacti/extras
cd /usr/share/cacti/site/resource/
wget http://public.itmatrix.eu/Cacti_templates/Disk_IO/partition.xml
ln -s /usr/share/cacti/site/resource/snmp_queries/partition.xml /usr/share/cacti/extras/partition.xml

Download on local desktop and import the host template using the Cacti web interface
http://public.itmatrix.eu/Cacti_templates/Disk_IO/cacti_data_query_snmp_disk_statistics.xml
http://public.itmatrix.eu/Cacti_templates/Disk_IO/cacti_graph_template_disk_io_bytessec.xml

Advanced MySQL statistics

In the monitored host:
– Create a special user for monitoring in MySQL server with the same username and password for all monitored hosts
– This user must have the SUPER_Privilege only
– Use ‘PASSWORD’ type and NOT MD5 or something else

Make sure the mysql server binds to ALL network interfaces:
Edit /etc/mysql/my.cnf
[mysqld]
........
#bind-address = 127.0.0.1
bind-address = 0.0.0.0
Protect the mysql server from Internet access using a firewall or the /etc/hosts.allow & /etc/hosts.deny as follows:
————————————-
Allowing the monitoring server acccess to snmpd daemon
In /etc/hosts.allow
mysqld: 127.0.0.1 192.168.100.0/255.255.255.0
Making sure all other hosts are denied access
In /etc/hosts.deny
mysqld: ALL
In the Monitoring server:
Download and install the php script
cd /usr/share/cacti/site/scripts/
wget http://public.itmatrix.eu/Cacti_templates/Improved_Mysql/ss_get_mysql_stats.php
ln -s /usr/share/cacti/site/scripts/ss_get_mysql_stats.php /usr/share/cacti/extras/

Edit the name and password in /usr/share/cacti/site/scripts/ss_get_mysql_stats.php
eg.
$mysql_user = 'monitor';
$mysql_pass = 'monitor';

Download on desktop and Import the host template using the cacti web interface
http://public.itmatrix.eu/Cacti_templates/Improved_Mysql/cacti_host_template_x_db_server_ht_0.8.6j.xml

TeMySQL statistics

In the monitored host:
– Create a special user for monitoring in MySQL server with the same username and password for all monitored hosts
– This user must have the SUPER_Privilege only
– Use ‘PASSWORD’ type and nothing else

Make sure the mysql server binds to ALL network interfaces:
Edit /etc/mysql/my.cnf
[mysqld]
........
#bind-address = 127.0.0.1
bind-address = 0.0.0.0

Protect the mysql server from Internet access using a firewall or the /etc/hosts.allow & /etc/hosts.deny as follows:

Allowing the monitoring server acccess to snmpd daemon
In /etc/hosts.allow
mysqld: 127.0.0.1 192.168.100.0/255.255.255.0
Making sure all other hosts are denied access
In /etc/hosts.deny
mysqld: ALL
In the Monitoring server:
Download and install the php scripts
cd /usr/share/cacti/site/scripts/
wget http://public.itmatrix.eu/Cacti_templates/teMySQLcacti/mysql_stats.php
ln -s /usr/share/cacti/site/scripts/mysql_stats.php /usr/share/cacti/extras/

Download on desktop and Import the host template in cacti interface
http://public.itmatrix.eu/Cacti_templates/teMySQLcacti/cacti_host_template_temysql_host-step300-heartbeat600.xml

During the creation of teMysql graphs in each device, enter the name and password of the mysql user configured in the monitored host for this purpose.
eg.
Name: monitor
PW: monitor


Network statistics

In the monitoring server:
Log into the monitoring server and get the data query script
cd /usr/share/cacti/site/scripts
wget http://public.itmatrix.eu/Cacti_templates/Netstat/lvm_netstat_tcp.pl
chmod 755 /usr/share/cacti/site/scripts/lvm_netstat_tcp.pl
ln -s /usr/share/cacti/site/scripts/lvm_netstat_tcp.pl /usr/share/cacti/extras/

Download and Import the template using the cacti web interface
http://public.itmatrix.eu/Cacti_templates/Netstat/cacti_graph_template_snmp_get_tcp_connection_status.xml

Advanced Ping

In the monitoring server:
Log into the monitoring server and get the data query script:
cd /usr/share/cacti/site/scripts
wget http://public.itmatrix.eu/Cacti_templates/Advanced_ping/ss_fping.php
ln -s /usr/share/cacti/site/scripts/ss_fping.php /usr/share/cacti/extras/

Download and Import the template in cacti interface
http://public.itmatrix.eu/Cacti_templates/Netstat/cacti_graph_template_ping_advanced_ping_v1_3.xml

SNMP Disk Usage

In the monitoring server:
Log into the monitoring server and get the query xml file
cd /usr/share/cacti/site/resource/snmp_queries/
wget http://public.itmatrix.eu/Cacti_templates/hrStorageTable/hrStorageTable.xml
ln -s /usr/share/cacti/site/resource/snmp_queries/hrStorageTable.xml /usr/share/cacti/extras/

Download and Import the template in cacti interface
http://public.itmatrix.eu/Cacti_templates/hrStorageTable/cacti087d_data_query_snmp_-_hrstoragetable.xml

HTTP Response Time

In the monitoring server:
Log into the monitoring server and get the script file
cd /usr/share/cacti/site/scripts
wget http://public.itmatrix.eu/Cacti_templates/HTTP_response_time/http_response_time.pl
ln -s /usr/share/cacti/site/scripts/http_response_time.pl /usr/share/cacti/extras/

Download and Import the template using cacti web interface
http://public.itmatrix.eu/Cacti_templates/HTTP_response_time/cacti_data_template_http_reponse_time.xml
http://public.itmatrix.eu/Cacti_templates/HTTP_response_time/cacti_graph_template_http_response_time.xml

NOTE: If by any chance you get no graphs created after a good 10 minutes then look at this solution:
Click on ‘Graph Management’, Click on one of the ‘HTTP Response Time ‘ graphs links and if you get no graph and the ‘RRDTool Says:’
ERROR: the RRD does not contain an RRA matching the chosen CF
Then the following might be the reason:
You are using (the Graph Template includes) the LAST consolidation function.
But your RRA definitions do not include this CF.
Edit RRAs when clicking onto “Data Sources”, then select “RRAs”
and edit all of them to add “LAST” CF
Then, you’ll have to re-create the failing rrd file
By deleting the graph from Graph management and under each device create the graph again.

PostGres Statistics

PostGres Data Base Cacti templates:
Ref: http://forums.cacti.net/viewtopic.php?f=12&t=23300
Installation instruction:
1. Make sure you have enabled Statistic Collector in Postgres (postgresql.conf)

Till postgres 8.2
stats_start_collector = true
stats_command_string = true
stats_block_level = true
stats_row_level = true
stats_reset_on_server_start = false

From postgres 8.3
track_activities = on
track_counts = on
update_process_title = on

Also in this configuration file:
Make sure PostGresql server listens to the interfaces that will be used for connections:
Here you give the IP of the server’s local interfaces.
eg.
listen_addresses = '127.0.0.1,192.168.100.117'
2. Edit the access rights file /etc/postgresql/8.3/main/pg_hba.conf
Then make sure the clients will be allowed in via the same interface Postgres is listening on above:
Add the access rights line:
host all all 192.168.100.0/24 md5

3. Create new User in PostgreSQL ,for example ‘monitor’
(you don’t have to grant any special roles to this users AFAIK, any user has access to Statistic Tables).
su - postgres
createuser --pwprompt --encrypted --no-adduser --no-createdb monitor

– Enter password for new role: monitor
– Enter it again: monitor
– Shall the new role be allowed to create more new roles? (y/n) n

4. Try to connect remotely under this user to postgres Database:
psql -h YOUR_POSTGRES_HOST -U monitor -W postgres
and try execute following SQL:
select * from pg_stat_all_tables;
If everything work proceed with installation otherwise fix your problem till you get results.

5.In monitoring server:
Commands:
cd /usr/share/cacti/site/scripts
wget http://public.itmatrix.eu/Cacti_templates/PostGres/postgresql_database_stats/pgsql_stats.php
cd /usr/share/cacti/site/resource/script_queries
wget http://public.itmatrix.eu/Cacti_templates/PostGres/postgresql_database_stats/postgres_dbstat.xml


Description of templates.
pgsql_stats.php - the Script, put this in /scripts/
postgres_dbstat.xml - the Definition, put this in /resource/script_queries
cacti_data_query_postgresql_database_stats.xml - Data Query + Graphs - import into Cacti

6. Install the debian package ‘php-pgsql’
apt-get install php5-pgsql
/etc/init.d/apache2 restart
7. Download and Import the template using cacti web interface
http://public.itmatrix.eu/Cacti_templates/PostGres/postgresql_database_stats/cacti_data_query_postgresql_database_stats.xml

8. Edit pgsql_stats.php line 173 and 174 and put username and password you have created at point 1.
(I don’t know how to pass username and password parameters to Data Queries. If you know how,
let me know so I can remote username and password from script)

9. Edit your Host Template or Device and add PostgreSQL Database Stats in Data Queries.

10. Create your Graphs 🙂

My Processes

Note: this self made cacti monitoring template does the monitoring of the number of processes per specific services as follows:
OUTPUT example:
allprocs:226 apache2:12 mysqld:19 mysqld2:0 bash:1 gpg:0 pop3:3
Its called : MyProcessesMGT2 in the monioring list

In the monitoring server:
– get the script file
cd /usr/share/cacti/site/scripts
wget http://public.itmatrix.eu/Cacti_templates/My_Processes/my_processes.sh
chmod 755 /usr/share/cacti/site/scripts/my_processes.sh

Make sure the user www-data on the monitoring server can run via ssh without password the script /usr/share/cacti/site/scripts/my_processes.sh
Commands in monitoring server:
mkdir /home/www-data
usermod -d /home/www-data -s /bin/bash www-data
chown www-data. /home/www-data
su - www-data
ssh-keygen -t rsa
(just press Enter key at every question)


In each monitored hosts:
-Modify the user www-data in each monitored hosts, create a home directory for it
-Commands in monitored hosts:
mkdir /home/www-data
usermod -d /home/www-data -s /bin/bash
chown www-data. /home/www-data
passwd www-data
(Give 2 times the password ‘www-data’ (no worry it will be deleted right after we are finished)

When all monitored hosts are modified as above then go back to the monitoring server
Commands in monitoring server:
su - www-data (only if not already logged-in as www-data)
ssh-copy-id clientHost_1_Name (Give the 'www-data' password)
ssh-copy-id clientHost_2_Name "" "" ""
ssh-copy-id clientHost_3_Name "" "" ""
........


Try the connection
IMPORTANT: use the same hostname exactly as it is given as hostname in cacti for every monitored host here as clientHost_1_Name. If you get a question like:
Are you sure you want to continue connecting (yes/no)?
Answer ‘yes’.
ssh clientHost_1_Name (if all ok then CTRL-D to exit the session on this remote host)
ssh clientHost_2_Name (if all ok then CTRL-D to exit the session on this remote host)
ssh clientHost_3_Name (if all ok then CTRL-D to exit the session on this remote host)
.......


Disable all password of www-data in all monitored hosts
Make sure you are back loggged in as ‘root’
Commands in all monitored hosts:
ssh clientHost_1_Name sed -i 's/www-data:/www-data:*/' /etc/shadow
ssh clientHost_2_Name sed -i 's/www-data:/www-data:*/' /etc/shadow
ssh clientHost_3_Name sed -i 's/www-data:/www-data:*/' /etc/shadow
.........


Download and Import the template in cacti interface:
http://public.itmatrix.eu/Cacti_templates/My_Processes/cacti_data_template_myprocessesmdt2.xml
http://public.itmatrix.eu/Cacti_templates/My_Processes/cacti_graph_template_myprocessesmgt2.xml