msgbartop
MAC OS X, Linux, Windows and other IT Tips and Tricks
msgbarbottom

13 Dec 16 Changing Linux system password in bash script

Introduction:
I created a script that creates a full email account for a new user including setting the password.
I was looking for a way to change/set the password programmatically in the script using clear text passwords provided in variables.
I took this information for the man page of chpasswd

Some solution examples:
Changing/setting the password using the existing encryption provided by PAM system:
echo 'username:cleartext_password' | chpasswd
Changing/setting the password using a different encryption method(Not recommended)
echo 'username:cleartext_password' | chpasswd -c METHOD
METHODS:
Use the specified method to encrypt the passwords.
The available methods are DES, MD5, NONE, and SHA256 or SHA512 if your libc support these methods.

Changing/setting the password using already encrypted password:
echo 'username:encrypted_password' | chpasswd -e
Changing/setting the password for list of users:passowrds
chpasswd < users:passwords_list_file
(one line per user:password in the file)

02 Dec 16 Discover live hosts on a network under Linux

There are lost of tools available to scan a network and list the IPs of the hosts that are live.
Here is one that I find quite cool using nmap with the following command(for example for the LAN network 192.168.100.0/24):
Reference: http://security.stackexchange.com/questions/36198/how-to-find-live-hosts-on-my-network
nmap 192.168.100.0/24 -n -sP | grep report | awk '{print $5}'
the result could be for example:
192.168.1.1
192.168.1.11
192.168.1.12
192.168.1.13
192.168.1.14
192.168.1.15
192.168.1.118
192.168.1.122
192.168.1.123
192.168.1.126
192.168.1.129
192.168.1.133
192.168.1.134
192.168.1.156
192.168.1.159
192.168.1.168
192.168.1.170

22 Nov 16 Forcing pam users to use only FTPS and block SFTP/SSH

Introduction:
I needed to force certain PAM users(configured in /etc/passwd) to use FTPS and block them from using SSH or SFTP.
Here is a solution I found in:
http://askubuntu.com/questions/93411/simple-easy-way-to-jail-users

Solution:
Add to /etc/shells a new shell:
vim /etc/shells
Add one line:
/bin/false
Save. For each user you want to deny ssh/sftp, change the user’s shell:
usermod -s /bin/false userx
Now userx cannot log in via ssh/sftp.
Install vsftpd:
apt-get install vsftpd
Edit the config file:
vim /etc/vsftpd.conf
And some changes….
anonymous_enable=NO
local_enable=YES
chroot_local_user=YES

Save. Restart vsftpd:
/etc/init.d/vsftpd restart

30 Oct 16 Resolving Mysql error: Too many open files

Introduction:
As I upgraded from Mysql 5.5 to 5.6 suddenly some sites were showing the following error:
...... Too many open files
The issue has to do with the present limitations given to the system and PAM system to open max 1024 files. After doing some research I found this site below here which is in German in which the following is explained:
https://www.filewalker.de/ulimit-mysql-open_files_limit/

STEPS:
Check the files-open limits or running mysql server:
mysql -p -u root
mysql> SHOW VARIABLES LIKE 'open%';

The very possible output:
+------------------+-------+
| Variable_name | Value |
+------------------+-------+
| open_files_limit | 1024 |
+------------------+-------+
1 row in set (0.00 sec)

That means that Mysql server gets to open maximum 1024 which seems too little for Mysql 5.6.

Raising this limit
Edit the file /etc/security/limits.conf and add the following lines:
mysql hard nofile 65535
mysql soft nofile 65535

This will raise the limit of open files to 65535 for the user mysql only.
If you want to rise this limit for all users then replace the word mysql for *
eg.
* hard nofile 65535
* soft nofile 65535

And according to this site edit the file /etc/pam.d/common-session and add this line at the end:
session required pam_limits.so
Note: I’m not sure this step is really needed though. Some people did try without it and it also worked. For me, in Debian Wheezy, I had to do this otherwise I was still getting the error.

For systems that run systemd instead of InitV do the following:
Edit file /usr/lib/systemd/system/mysqld.service
Add these 2 lines at the end:
LimitNOFILE=65535
LimitNPROC=65535

Restart Mysql server and test it again
service mysql restart
mysql -p -u root
mysql>> SHOW VARIABLES LIKE 'open%';

The hopeful output:
+------------------+-------+
| Variable_name | Value |
+------------------+-------+
| open_files_limit | 65535 |
+------------------+-------+
1 row in set (0.00 sec)

This error should no more appear.

26 Sep 16 Blocking all traffic from individual countries using IPSet and IPTables

Introduction:
As I looked at the syslog I found too often that SSH login attacks are coming either from China or from Russia. Having enough of that, and anyway not expecting any traffic from any of these countries, although I use fail2ban still I decided to block any traffic coming from these 2 countries. Fail2ban is a great tool for blocking unsuccessful attempts to login. BUT it seems that many attemps(thousands) per day are still going through. I can only assume that the attacker is using parallel sending of attempts and till Fail2ban(based on auth logs) reacts many have gone through.
To do a full blocking of IPs with preloaded lists of IP ranges, I started by using iptables with one rule per IP range, BUT the loading of the rules was taking too long and more importantly, loading all IP ranges, as pure iptables rules, made my server unstable, meaning … it crashed!!! It is said that more than max. 25,000 rules in the iptables, especially larger than 27,000, rules may put the kernel in an unstable state.
To remedy to that I use ipset with iptables. ipset is especially conceived to deal with large IP Range lists(fast access hash tables) which can contain up to 65536 entries.

NOTE: The script below is only valid for Debian based Linux distributions. For other distributions you will need to adapt the script accordingly.

Principle of this Countries Blocking protection:
The IP ranges in CIDR format are picked up from the web site http://www.ipdeny.com/ipblocks/data/countries/ and entered in the IPSet’s individual lists which are named by country codes and then referenced by iptables to define the TARGET (what to do with the IPs that match:DROP)
NOTE: In this example the script sets iptables rules target to DROP instead of REJECT to avoid high traffic of TCP/IP stack rejection responses. DROP doesn’t respond anything.

IMPORTANT NOTE: This script below should be run AFTER you loaded your regular firewall rules. It INSERTS the new iptables rules in such a way that any incoming packets from those defined countries will be blocked BEFORE any further processing in your firewall.

Steps:
#!/bin/bash
# Description: Uses IPSET and IPTABLES to block full countries from accessing the server for all ports and protocols
# Syntax: countries_firewall.sh countrycode [countrycode] ......
# Use the standard locale country codes to get the proper IP list. eg.
# countries_firewall.sh cn ru ro
# Will create tables that block all requests from China, Russia and Romania
# Changes: 13.11.2016 Initial creation of script
# Note: To get a sorted list of the inserted IPSet IPs for example China list(cn) run the command:
# ipset list cn | sort -n -t . -k 1,1 -k 2,2 -k 3,3 -k 4,4
# #############################################################################
# Defining some defaults
iptables="/sbin/iptables"
tempdir="/tmp"
sourceURL="http://www.ipdeny.com/ipblocks/data/countries/"
#
# Verifying that the program 'ipset' is installed
if ! (dpkg -l | grep '^ii ipset' &>/dev/null) ; then
echo "ERROR: 'ipset' package is not installed and required."
echo "Please install it with the command 'apt-get install ipset' and start this script again"
exit 1
fi
[ -e /sbin/ipset ] && ipset="/sbin/ipset" || ipset="/usr/sbin/ipset"
#
# Verifying the number of arguments
if [ $# -lt 1 ]; then
echo "ERROR: wrong number of arguments. Must be at least one."
echo "countries_firewall.sh countrycode [countrycode] ......"
echo "Use the standard locale country codes to get the proper IP list. eg."
echo "countries_firewall.sh cn ru ro"
exit 2
fi
#
# Now load the rules for blocking each given countries and insert them into IPSet tables
for country ; do
# Read each line of the list and create the IPSet rules
# Making sure only the valid country codes and lists are loaded
if wget -q -P $tempdir ${sourceURL}${country}.zone ; then
# Destroy the IPSet list if it exists
$ipset flush $country &>/dev/null
# Create the IPSet list name
echo "Creating and filling the IPSet country list: $country"
$ipset create $country hash:net &>/dev/null
(for IP in $(cat $tempdir/${country}.zone) ; do
# Create the IPSet rule from each IP in the list
echo -n "$ipset add $country $IP --exist - "
$ipset add $country $IP -exist && echo "OK" || echo "FAILED"
done) > $tempdir/IPSet-rules.${country}.txt
# Destroy the already existing rule if it exists and insert the new one
$iptables -D INPUT -p tcp -m set --match-set $country src -j DROP &>/dev/null
$iptables -I INPUT -p tcp -m set --match-set $country src -j DROP
# Delete the temporary downloaded counties IP lists
rm $tempdir/${country}.zone
else
echo "Argument $country is invalid or not available as country IP list. Skipping"
fi
done
# Display the result of the iptables rules in INPUT chain
echo "======================================"
echo "IPSet lists registered in iptables:"
$iptables -L INPUT -n -v | grep 'match-set'
# Dispaly the number of IP ranges entered in the IPset lists
echo "--------------------------------------"
for country ; do
echo "Number of ip ranges entered in IPset list '$country' : $($ipset list $country | wc -l)"
done
echo "======================================"
#
#eof

Loggin of the script operation:

As you can see in the script, the adding of IP ranges in the IPSet tables is logged in: $tempdir/IPSet-rules.${country}.txt which is overwritten every time the script is run for the same country.

Starting the script

In order to make sure this firewall add-on starts properly on each boot, but after the user defined firewall, I start it via cron @reboot job after a delay of about 40 sec.(which can be adjusted to you needs) to let other services start, including the firewall service. I’m aware that this method of starting the script is not very elegant but it fits pretty much all variations of Linux distributions whether Sysinit-V or Systemd based. Example of root cron job:
@reboot /bin/sleep 40 ; /bin/bash -c ". /root/.bashrc ; /root/bin/countries_firewall.sh cn ru"

Sending traffic reports per email

In order to send traffic reports per email every day I collect every day the traffic data from iptables, format it and send it by email using the following bash script. This script will then be run every day by cron (put in /etc/cron.daily/) and the traffic counter will be reset.
#!/bin/bash
# Purpose: Sends the blocked traffic report per email and resets the counter
# Syntax: iptables_report
# Dependencies: Systems tools: iptables, awk, column, whois, sendmail
# Changes: 13.11.2016 First implementation of script
#----------------------------------------------------------
HOST=$(cat /etc/hostname | tr 'a-z' 'A-Z')
email="me@myserver.com"
reportsender="cron@$HOST"
subject="BLOCKED Packets report on $(hostname | tr 'a-z' 'A-Z')"
tempdir="/tmp"
file1="iptables_report1.txt"
file2="iptables_report2.txt"
#
#------------ Build the header of the mail to send ------------
echo "From: $reportsender" > $tempdir/$file1
echo "To: $email" >> $tempdir/$file1
echo "Subject: $subject" >> $tempdir/$file1
echo "MIME-Version: 1.0" >> $tempdir/$file1
echo 'Content-Type: text/html; charset="ISO-8859-15"' >> $tempdir/$file1
echo "" >> $tempdir/$file1
echo "<br />" >> $tempdir/$file1
echo -e "<font size=3 FACE='Courier'><pre>" >> $tempdir/$file1
# Formatted message starts here
# Add the country at the end of each line
# Load the header and data to the temporary file 2
echo -e "Packets Bytes Source \n======= ========= ======" >$tempdir/$file2
/sbin/iptables -L -n -v | /bin/grep -v '^ 0' | /bin/grep 'match-set' | /usr/bin/awk '{print $1" "$2" "$11}' >> $tempdir/$file2
#
# Format temp file2 into temp file1
cat $tempdir/$file2 | column -t >> $tempdir/$file1
#
#
# Add the last HTML preformatting End
echo -e "</pre>" >> $tempdir/$file1
echo "" >> $tempdir/$file1
#
#----------------- Send the prepared email ---------------------------
# now format the report and send it by email
cat $tempdir/$file1 | /usr/sbin/sendmail -t
rm $tempdir/$file1 $tempdir/$file2
#
# Reset the iptables counters
/sbin/iptables -Z
#
# eof

Creating the cron job for regular daily reports
Save that file in eg. /etc/cron.daily/iptables_report
Make it runable:
chmod 755 /etc/cron.daily/iptables_report
Example of report email:
Packets Bytes Source
======= ========= ======
188 7852 ru
19295 1150K cn

Manual status:
To get manual status of the iptables rules for the INPUT chain run the command:
iptables -L INPUT -v -n
To get a sorted listing of an IPSet IP list run the command(eg. for Russia):
ipset list ru | sort -n -t . -k 1,1 -k 2,2 -k 3,3 -k 4,4

26 Apr 16 Activating SPDY in Apache 2.4 (Ubuntu 14.04)

SPDY is a new protocol created by Google and given to the Apache Foundation which allows faster Web traffic under SSL. Apache 2.4 is SPDY capable but its module is not included in Ubuntu 14.04 LTS Server. Here are some instructions that allow to get, install and enable SPDY feature foe Apache 2.4 under Ubuntu 14.04.

Get the third party Apache 2.4 module, extract and install it:
cd /usr/local/
https://www.rivy.org/static/mod_spdy.tar.gz
tar zxf mod_spdy.tar.gz
cd /usr/lib/apache2/modules
mv /usr/local/mod-spdy/mod-spdy/src/mod_ssl.so .
mv /usr/local/mod-spdy/mod-spdy/src/out/Release/libmod_spdy.so mod_spdy.so
echo "LoadModule spdy_module /usr/lib/apache2/modules/mod_spdy.so" | sudo tee /etc/apache2/mods-available/spdy.load
echo "SpdyEnabled on" | sudo tee /etc/apache2/mods-available/spdy.conf
a2enmod spdy
service apache2 restart

In case you get the error message in log file eg:
...... [core:notice] [pid 7842:tid 140724740487067] AH00052: child pid 8025 exit signal Segmentation fault (11)
Then that means that you are using a version of the MPM that is not compatible with SPDY module. You might need to change to the thread-stable mpm_prefork as follows:
a2dismod mpm_event
a2enmod mpm_prefork
service apache2 restart

Test your sites for SPDY protocol:
https://spdycheck.org/

15 Apr 16 Installing Webmin in Debian 8(Jessie)

These instructions are a ‘Plagiat’ of the site:
http://www.christophe-casalegno.com/2015/07/14/how-to-install-webmin-on-debian-8/

To install webmin on Debian 8 just follow this instructions :
cd /root
wget http://www.webmin.com/jcameron-key.asc
apt-key add jcameron-key.asc
echo "deb http://download.webmin.com/download/repository sarge contrib" >> /etc/apt/sources.list
echo "deb http://webmin.mirror.somersettechsolutions.co.uk/repository sarge contrib" >> /etc/apt/sources.list
apt-get update
apt-get -y install webmin

If it’s too long for you, you can also just do this :
wget http://www.christophe-casalegno.com/tools/install_webmin.sh
chmod +x install_webmin.sh
./install_webmin.sh

31 Mar 16 Fixing Spamassassin in Debian Jessie(8)

Introduction:
For a long time under Debian Wheezy Spamassassin was running quite well until I upgraded the system to Jessie. That is when Spamassassin(spamd) started to crash every now and then without giving much reasons why.

Cause of error message:
Looking in the system logs(/var/log/syslog) I found the following error:
spamd[7490]: util: refusing to untaint suspicious path: "/${SAHOME}"
I’m not sure if this is the cause of the crashes but it certainly doesn’t help. So I figured I should first try to solve this error first. According to this site since the Spamassassin is now started via ‘systemd’ the variables set in the init config file (/etc/default/spamassassin) are not expanded and they are passed on ‘as-is’ on the command line for starting spamd process. eg.
SAHOME="/var/lib/spamassassin/"
OPTIONS="--create-prefs --max-children 5 --username spamd --helper-home-dir ${SAHOME}"

Solution:
Since this file will not be overwritten during updates the suggestion was to write the value of this variable directly in the OPTIONS line in (/etc/default/spamassassin) as follows:
OPTIONS="--create-prefs --max-children 5 --username spamd --helper-home-dir /var/lib/spamassassin/"
Now at least this error doesn’t occur any more and time will tell if the crashes of spamd are still happening.

24 Mar 16 Using HTTPS as proxy backend in Apache 2.4

Introduction:
In Apache 2.4 in a Vhost in order to be able to proxy to a backend with HTTPS using either a self-signed or expired certificate on the backend we need to include the following directives:
SSLProxyEngine on
SSLProxyVerify none
SSLProxyCheckPeerCN off
SSLProxyCheckPeerName off
SSLProxyCheckPeerExpire off

You also need to enable the required Apache2 modules as follows:
a2enmod proxy
a2enmod proxy_http
a2enmod proxy_connect
service apache2 restart

Example when using Apache 2.4 to proxy to Webmin port 10000:
RewriteEngine On
SSLProxyEngine on
SSLProxyVerify none
SSLProxyCheckPeerCN off
SSLProxyCheckPeerName off
SSLProxyCheckPeerExpire off
RewriteRule ^/(.*) https://127.0.0.1:10000/$1 [P]
ProxyPassReverse / https://127.0.0.1:10000

21 Mar 16 Redirecting HTTP to HTTPS in NginX

Here is a working method of redirecting any requested HTTP URL to HTTPS in NginX VirtualHosts that handles both HTTP and HTTPS.
For example, to have a single vhost support both HTTP and HTTPS you have normally the following directives:
# Support for HTTP and HTTPS
listen 80;
listen 443 ssl;
ssl_certificate /etc/letsencrypt/live/www.myserver.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/www.myserver.com/privkey.pem;

Then to redirect all HTTP requests to HTTPS within this vhost without creating infinite loops you add the following redirect:
if ($scheme != "https") {rewrite ^ https://$host/$request_uri permanent;}
Other methods can be seen here:
http://serverfault.com/questions/67316/in-nginx-how-can-i-rewrite-all-http-requests-to-https-while-maintaining-sub-dom