msgbartop
MAC OS X, Linux, Windows and other IT Tips and Tricks
msgbarbottom

24 Dec 16 Force reboot a remote Linux server

Introduction:
After having tried to do a reboot of a remote Linux server via the command reboot which had no effect, I tried to find a command that would force the server to reboot immediately. I found the commands that do exactly that at:
https://major.io/2009/01/29/linux-emergency-reboot-or-shutdown-with-magic-commands/

Commands:
echo 1 > /proc/sys/kernel/sysrq
echo b > /proc/sysrq-trigger

This is pretty brutal and pretty much the same as pressing the reset button on the server (if equipped). No daemons will be shut down gracefully, no filesystem sync will occur, and you may get the wrath of a fsck (or worse, a non-booting server) upon reboot. BUT when all else fails, this is the next best thing to do.

19 Dec 16 Display MySQL databases types in bash

Based on the site: http://stackoverflow.com/questions/213543/how-can-i-check-mysql-engine-type-for-a-specific-table

Here is a bash command that will display the databases types(innoDB or MyISAM) of all the MySQL databases except for the defaults ones(mysql, etc)
for i in $(mysql -u root -p -e "show databases;" | egrep -v 'Database|information_schema|mysql|performance_schema'); do echo "--------------------$i--------------------"; mysql -u root -p -e "use $i; show table status;"; done | awk '{print $1" "$2}' | column -t

You will need to type as many times as there are databases the MySQL root password.

Alternative:
Create a mysql user and use it’s password on the command line as follows:
for i in $(mysql -u <user> -p --password='secret' -e "show databases;" | egrep -v 'Database|mysql|performance_schema'); do echo "--------------------$i--------------------"; mysql -u <user> -p --password='secret' -e "use $i; show table status;"; done | awk '{print $1" "$2}' | column -t

22 Nov 16 Forcing pam users to use only FTPS and block SFTP/SSH

Introduction:
I needed to force certain PAM users(configured in /etc/passwd) to use FTPS and block them from using SSH or SFTP.
Here is a solution I found in:
http://askubuntu.com/questions/93411/simple-easy-way-to-jail-users

Solution:
Add to /etc/shells a new shell:
vim /etc/shells
Add one line:
/bin/false
Save. For each user you want to deny ssh/sftp, change the user’s shell:
usermod -s /bin/false userx
Now userx cannot log in via ssh/sftp.
Install vsftpd:
apt-get install vsftpd
Edit the config file:
vim /etc/vsftpd.conf
And some changes….
anonymous_enable=NO
local_enable=YES
chroot_local_user=YES

Save. Restart vsftpd:
/etc/init.d/vsftpd restart

05 Nov 16 Displaying the list of all ‘at’ jobs and their respective commands

Introduction:
The command atq gives me the list of at jobs waiting to be executed and their execution times. Each line starting with the job number.
The command at -c JobNumber gives me the content of the job including the environment variables.
What I wanted is a command that would give me the list of the jobs (just like the command atq) but expanded so that each job line is followed by the commands that will be executed, excluding their environment variables. eg.
34 Tue Jan 17 10:22:00 2017 a root
Commands
35 Tue Jan 24 17:50:00 2017 a root
Commands
28 Tue Dec 13 23:00:00 2016 a root
Commands
24 Mon Nov 14 20:27:00 2016 a root
Commands
31 Sun Jan 15 00:21:00 2017 a root
Commands

Humm… could not find such command to display this. So I created this bash script which does the job:

#!/bin/bash
# Description: Displays all 'at' jobs and their respective commands
# Syntax: atlist.sh
# Changes: 05.11.2016 First implementation
########################################################################
# Get the short jobs list and expand from there
atq | while read line ; do
jobnr=$(echo $line | awk '{print $1}')
echo $line
# Pickup all the command lines after first line matching '}'.
# This excludes all the environment variables and the at exit line.
# Exclude the matching '}' line and empty lines
# Add an offset of 8 chars to each command line.
at -c $jobnr | sed -e '1,/^\}/d' -e '/^$/d' -e 's/^/ /'
done

05 Apr 16 Useful ps options

Here are some (growing) tricks in order to get the maximum of the PS command:
I set the command’s options in an alias to simplify the running of it.
alias psa='ps --headers axf -o pid,ppid,pri,state,user:15,group:15,nlwp,%cpu,%mem,rss,vsz,maj_flt,time,start,comm'
#

One advantage of this above command is that it displays the username in its full length even if it’s long. Normally the ps command displays long usernames as their PIDs only.

13 Feb 16 Verifying the validity of an NFS mount

Introduction:
Every now and then if an NFS mount is no more connected to the server or something goes wrong with the NFS connection, running the command ‘ls mountpoint’ hangs the terminal till I press CTRL-C. So I tried to figure out a script that will be run as cron job and will tell me when an NFS mount has gone wrong. I had to revert to unorthodox tricks since doing a simple command ‘stat mountpoint &’ within the script would also hang the script. So I use the command ‘at now’ which runs the command independently for the script that initiated it. Here is an example of such script.

#!/bin/bash
# Name: MOUNT_CHECK.sh
# Purpose: Checks the health of the NFS mountpoint given by argument
# it kills the at/stat process and exits with an exit code 2 if the timeout has expired.
#-------------------------------------------------------------------
startdelay=3
timeout=10
# processes to be excluded in the 'ps | grep' test
excludes="openvpn|istatd|rpc.statd"
if [ $# -ne 1 ]; then
echo "ERROR: Needs mountpoint as argument"
echo "Usage: MOUNT_CHECK.sh MountPoint"
exit 2
fi
#
echo "/usr/bin/stat $1" | /usr/bin/at now
sleep $startdelay
while (ps ax | egrep -v "grep|$excludes" | grep -q stat); do
let count=${count}+1
sleep 1
if [ $count -ge $timeout ]; then
kill $(pidof stat)
#echo "Mountpoint $1 : FAILED to connect before timeout of $timeout sec."
exit 2
fi
done

29 Jan 16 Recursively delete files securely: shredding

Description:
Every administrator should know that when we delete a file on the hard disk, almost nothing is really deleted. The space used by the files is simply put back on the list of ‘free to use‘ space in the filesystem and will no more appear in the directory listing. This means forensic tools can be used to retrieve these ‘deleted‘ files if the space they were using hasn’t been overwritten by newly written files.

In order to prevent this the best method is to overwrite the files to be deleted many times (10 times should do)before they get deleted. That is the job of the tool ‘shred‘ which is used in this following bash script: rec_shred.sh

Script content: rec_shred.sh
#!/bin/bash
# Purpose: SHREDS files recursively
# Use: rec_shred.sh /dir/to/shred/
if [ $# -ne 1 ]; then
echo "ERROR: The script needs one and only one argument"
exit 1
fi
# Verify if the given path is valid
if [ -d $1 ]; then
cd $1/
# Shred the files
find . -type f | while read file ; do
shred --remove --zero -n 10 $file
done
else
echo "ERROR: The given directory $1 does not exist. Exitting."
exit 2
fi

NOTE:
This script will shred the files only and leave all sub-directories empty for you to verify and delete with the command:
rm -rf /directory/

Happy shredding 🙂

14 Jan 16 Using SS tool for network troubleshooting

Introduction:

The following article is been copied completely 1 to 1 (full plagiat!!)from the following site inn order to be able to refer to it here in case the article disappears from Internet access or moves location.
http://www.linux-magazine.com/Issues/2015/181/Querying-Sockets-with-ss

Linux Magazine. Article from Issue 181/2015
Author(s): Chris Binnie

The unassuming ss utility is easy to understand and easy to type, but it adds some powerful options to your admin toolkit.

The names for a few Linux utilities are so small that you find yourself unexpectedly launching them by entering a typo on the command line. Why bother typing lengthy words when a perfectly suitable abbreviation will suffice? One minuscule command (both in name and its pocket-sized footprint on your hard disk) is a little utility called ss.

Ss punches several levels above its flyweight class. If you’re familiar with any of the popular tools used by sys admins for checking network links, I’m sure you’ll be glad to hear that its functionality won’t be too tricky to get your head around.

For the curious among us, the “ss” abbreviation is apparently for the words “socket statistics.” Ss is bundled with the iproute2 package. If, for some highly unusual reason you don’t find ss on your Debian-like system, you can always install it by running:

sudo apt-get install iproute2

A socket, is a port and an IP address. You can think of a socket as identifying a service listening on a specific port number of a specific computer. A socket pair, in this case, consists of a client IP address, a client port number, a server IP address, and a server port number. Querying information by socket therefore lets you zero in quickly on a specific service running at a specific IP address.

I would be remiss in not mentioning Unix Domain Sockets. Unix Domain Sockets, which facilitate communication between processes running on a local machine, serve a number of useful purposes, such as enabling the permissions needed to access resources between processes that would otherwise be non-privileged.
Getting Started with ss

In most scenarios, ss will run from an ordinary user account. On my system, ss resides in the /usr/sbin/ss directory. I’ll start off with some basic uses for ss. The following command shows the output for IPv4 networks:
# ss -4
The abbreviated output (Listing 1) shows client and server communication. Bear in mind that a client can also be a server, and vice versa, depending on the direction of the information flow.

Listing 1
Basic ss Output
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 0 192.168.0.2:37564 192.168.0.100:www
ESTAB 0 0 192.168.0.2:47592 192.168.0.156:smtp
ESTAB 0 0 192.168.0.2:ssh 192.168.0.49:64009

Replace the -4 option with -6 to output information on IPv6 connections.

As you can see in Listing 1, 192.168.0.2 is the local machine IP address, and the useful /etc/services utility has converted some port numbers into names (such as ssh, www, and smtp).

The excellent ss also offers information about TCP, UDP, local Unix Domain Sockets, and remote sockets. As I’ll describe later, what makes ss exceptionally powerful is its ability to deal with the state of connections.

I often use ss to query which ports are opened by daemons installed on the computer. Use the l option to check for listening ports (Listing 2):

Listing 2

Viewing the Listening Ports
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 *:ssh *:*
LISTEN 0 100 127.0.0.1:smtp *:*
LISTEN 0 128 *:sunrpc *:*
LISTEN 0 128 127.0.0.1:http *:*

# ss -l
You might want to check for the listening ports during your daily sys admin routine to ensure that unexpected and potentially insecure services haven’t been left enabled. Or, you might want to check for less sophisticated rootkits that might not hide their open ports effectively.

Either of the commands
lsof -i
netstat -tulpn

will help you figure out which processes (PIDs) are opening up your ports, so you can kill them off if necessary.
Simply running the ss command without any options provides a list of current “connections” (note these are actually “sockets”), as shown in Listing 3.

Listing 3
ss Unadorned

# ss
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 0 192.168.0.2:37564 192.168.0.49:tcpmux
ESTAB 0 64 192.168.0.2:ssh 192.168.0.143:64009
ESTAB 0 0 192.168.0.2:47609 192.168.0.88:gopher

PIDs

If you can’t get access to lsof, and you don’t like netstat (I’m not a massive fan), the super-duper ss utility can also report on PIDs pertinent to your open ports. To see which processes are using sockets directly, just throw a -p into the mix:
# ss -p
One caveat is I need to log in as root to get the extended information. Listing 4 shows an example of what to expect.

Listing 4
ss -p
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 0 10.10.10.20:ssh 10.10.10.10:52918 users:(("sshd",31195,3),("sshd",31204,3))

Working backward from the application name, which is sshd in this case, you can see the PIDs 31195 and 31204; 31204 belongs to a non-privileged user (my login user), and 31195 appears to result from the clever privilege separation employed by sshd.

SSH is designed to minimize the chances for programming bugs causing very serious systemic issues by limiting access and restricting the service to a chrooted jail for the network-facing SSH process. To achieve this goal, SSH uses two processes. The root-owned process monitors the progress of the non-privileged process, which has a previously unused UID and GID.
Looking Deeper

You can use ss -s to retrieve screeds of statistics relating to how many sockets are open and which protocols they are using (Listing 5).

Listing 5

ss -s
Total: 201 (kernel 218)
TCP: 6 (estab 2, closed 0,
orphaned 0, synrecv 0,
timewait 0/0), ports 5
Transport Total IP IPv6
* 218 - -
RAW 0 0 0
UDP 5 5 0
TCP 6 6 0
INET 11 1 0
FRAG 0 0 0

It’s not the best of comparisons, but you could get a vaguely similar result using the following netstat command:
# netstat -tan | grep -v "Proto" | grep -v "Active" | awk '{print $6}' | uniq -c
It’s definitely worth mentioning that the super ss utility has a habit of outperforming other network tools in benchmark tests. Compared with the widely-used netstat, for example, ss delivers its results very rapidly indeed. Keep in mind, however, that ss is not designed to answer every possible question but, rather, reflects the Unix philosophy that each system component should “do one thing well.” Experienced users often combine ss with tools like netstat then prune the output using tools such as grep, awk, and sed.

For instance, if you would like to see which application is tying up a specific port, you could use ss with grep, as follows:
# ss | grep 58620

Getting Specific

You can look at only TCP sockets with:
# ss -t
Alternatively, you could write out the –tcp option. Ramp up the level of detail with the -a (for “all”) switch:
# ss -t -a
Altering that command ever-so slightly generates output to include UDP, Raw, and Unix sockets. For a verbose view of all UDP sockets, enter
# ss -u -a
and to view all Raw sockets, enter:
# ss -w -a
If you really must indulge yourself with several screen’s worth of mind-boggling text, you can choose to view all Unix Domain sockets with:
# ss -x -a

Also included in the seemingly bottomless toolbox provided by the ss utility is the ability to watch out for DCCP sockets; DCCP is a less common network protocol that has the connection-oriented, error-checking traits of TCP with the broadcast-type features of UDP. The DCCP protocol is often used for media streaming. Check for DCCP traffic with this nifty little command:
# ss -d -a
You can also monitor the status of connections to your computer by extending the level of detail using the -e option (Listing 6).

Listing 6
# ss -e
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 0 192.168.0.2:ssh 192.168.0.88:58302 timer:(keepalive,40min,0) ino:2184870 sk:ffff880138d26700
ESTAB 0 0 192.168.0.2:48246 192.1680.56:ntp timer:(keepalive,105min,0) ino:2187726 sk:ffff880138d2748

Listing 6 shows timer options that offer an insight into the current status of any keepalives on a connection. This feature can be useful for services such as HTTP or SSH, which tend to employ keepalives (see the box titled “Keepalive Notes”). Listing 7 shows similar output using the netstat -to command.

Keepalive Notes

A keepalive packet generally has a TCP ACK configured with a sequence number set to one less than the current number used on the connection. Any machine receiving a nudge from such a packet will simply respond with the correct sequence number and, Frankenstein jokes aside, announce that it is indeed alive.

These types of packets are sometimes empty and generally have three associated parameters. The retry parameter will declare how many times to send a packet before coming to the conclusion that the other end of the connection has given up the ghost. The time setting is configured as the frequency of the checks, and finally the interval dictates the length of time between two sent packets if no response is received.

Inside the kernel, you can alter these settings by editing these values in the eye-watering pseudo-filesystem known and loved as /proc. One file, in this case, is /proc/sys/net/ipv4/tcp_keepalive_time, and you can edit it as follows:
# echo 75 > /proc/sys/net/ipv4/tcp_keepalive_intvl
# echo 9 > /proc/sys/net/ipv4/tcp_keepalive_probes

Listing 7
# netstat -to
Proto Recv-Q Send-Q Local Address Foreign Address State Timer
tcp 0 0 host-one:48340 host-four:45358 ESTABLISHED keepalive (6830.00/0/0)
tcp 0 64 host-two:ssh host-three:58302 ESTABLISHED on (0.45/0/0)

Netstat also lets you prepend the versatile watch command and see real-time updates:
# watch netstat -to
It is a close call, but I have to admit that, in this instance, netstat keeps its output nice and succinct and looks as good as the ss utility’s output.
Source and Destination
The following command lets you find information on any sockets with a destination (dst) aimed at a specific IP address:
# ss dst 192.168.0.1
Conversely, it’s not going to take a massive leap to consider that using the src option reveals information on the source side of the socket:
# ss src 192.168.0.2
Handy, huh? This syntax is easy to remember if you’re in a rush, saving the day when a testy boss is breathing down your neck during an outage.
You can even use CIDR network notations in the address callout:
# ss dst 192.168.0.1/24
Add a colon at the end, and you can check for a very specific port and a very wide IP address range at the same time:
# 192.168.0.1/24:53
I really appreciate this functionality in situations with lots of traffic and numerous open ports. For instance, this option lets you monitor all DNS-related activity on an entire /24 subnet with one simple ss command.
Regular Expressions
In addition to DNS names and IP addresses, you can also use regular expression (regex) operators in your ss syntax. Have a look at this little nugget:
# ss dport != :53
This command excludes the destination dport on the DNS port 53.
If you need to retain some sanity and avoid just looking at numbers, you can also translate ports into the /etc/services format:
# ss 192.168.0.1:http
If you’re ready to be impressed, you can also include greater-than, lesser-than, less-than-or-equal-to, and so forth:
# ss dport > :53
Other special characters are possible also, although sometimes special characters need escaping. Alphabetic equivalents, such as eq (equal), ne (not equal), gt (greater than), and le (less than) also work. Your mileage might vary with different versions of the ss utility.

Connection State

You can use the –query or -A query addition to dump a sockets table. The magic word autobound checks for ephemeral ports that sockets have attached themselves to. Prepare yourself for screeds of output, even on a quiet system. An abbreviated version of the output is shown in Listing 8.

Listing 8

# ss -a -A all autobound
01 u_str ESTAB 0 0 * 11984 * 0
02 u_str ESTAB 0 0 * 11996 * 0
03 u_str ESTAB 0 0 * 12003 * 0
04 u_str ESTAB 0 0 * 12005 * 0
05 u_str ESTAB 0 0 * 12010 * 0

You can also filter by TCP states; for instance, the following command filters for sFTP traffic:
# ss state connected dport = :sftp
Anything treated as “current” in relation to the sFTP port is displayed promptly.
You can complicate the command a little more with a boolean operator:
# ss ( sport = :ftp or dport = :http )
You can even use ss to find connections that are in a specific TCP state, including the established, syn-sent, syn-recv, fin-wait-1, fin-wait-2, time-wait, closed, close-wait, last-ack, listen, and closing.

The TCP state parameters let you do some very powerful querying. For example, checking for FIN–WAIT–1 states lets you identify whether your application has closed its side of a connection, but a remote host has not closed its side, thus tying up your machine’s precious ports:

ss -o state fin-wait-1 '( sport = \
:ftp or sport = :http )' \
dst 10.10.3.3/24:22

Sso It Ends

The ss utility is a powerful tool that will help you query your network in significant detail. Ss is extremely high performance for both manual and automated queries, and it requires very few keystrokes to execute common commands.

This tiny but heroic tool helps flex the muscles of any sys admin. If you want to increase the power of your admin toolkit, try practicing some of the more complex commands in your day-to-day work.

14 Jan 16 Preventing a bash script from running concurrently

Introduction: In order to prevent a bash script instance from running more than once concurrently, here is a small tip on how to write the script.

Script template:
#!/bin/bash
# Prevents that an instance of the script starts while another instance of it is still running
scriptname=$(basename $0)
lockfile="/tmp/${scriptname}.lock"
if [ -e $lockfile ]; then exit 1 ; fi
touch $lockfile.lock
# Delete lock file if CTRL-C typed from the keyboard
trap 'rm $lockfile ; exit' SIGINT SIGQUIT
#----------------------------------
# ############ Put your script code here #####################
#----------------------------------
# delete the lock file
rm $lockfile
# .eof

15 Aug 15 Finding the absolute path of a running script

In bash scripts we often need to know in which directory the running script is found especially when the script is in the $PATH and may be occurring in multiple places:
Here is a reliable way to find it out:
Based on this site:
http://stackoverflow.com/questions/4774054/reliable-way-for-a-bash-script-to-get-the-full-path-to-itself
# Absolute path to this script. /home/user/bin/foo.sh
SCRIPT=$(readlink -f $0)
# Absolute path this script is in. /home/user/bin
SCRIPTPATH=$(dirname $SCRIPT)