msgbartop
MAC OS X, Linux, Windows and other IT Tips and Tricks
msgbarbottom

14 Jan 16 Using SS tool for network troubleshooting

Introduction:

The following article is been copied completely 1 to 1 (full plagiat!!)from the following site inn order to be able to refer to it here in case the article disappears from Internet access or moves location.
http://www.linux-magazine.com/Issues/2015/181/Querying-Sockets-with-ss

Linux Magazine. Article from Issue 181/2015
Author(s): Chris Binnie

The unassuming ss utility is easy to understand and easy to type, but it adds some powerful options to your admin toolkit.

The names for a few Linux utilities are so small that you find yourself unexpectedly launching them by entering a typo on the command line. Why bother typing lengthy words when a perfectly suitable abbreviation will suffice? One minuscule command (both in name and its pocket-sized footprint on your hard disk) is a little utility called ss.

Ss punches several levels above its flyweight class. If you’re familiar with any of the popular tools used by sys admins for checking network links, I’m sure you’ll be glad to hear that its functionality won’t be too tricky to get your head around.

For the curious among us, the “ss” abbreviation is apparently for the words “socket statistics.” Ss is bundled with the iproute2 package. If, for some highly unusual reason you don’t find ss on your Debian-like system, you can always install it by running:

sudo apt-get install iproute2

A socket, is a port and an IP address. You can think of a socket as identifying a service listening on a specific port number of a specific computer. A socket pair, in this case, consists of a client IP address, a client port number, a server IP address, and a server port number. Querying information by socket therefore lets you zero in quickly on a specific service running at a specific IP address.

I would be remiss in not mentioning Unix Domain Sockets. Unix Domain Sockets, which facilitate communication between processes running on a local machine, serve a number of useful purposes, such as enabling the permissions needed to access resources between processes that would otherwise be non-privileged.
Getting Started with ss

In most scenarios, ss will run from an ordinary user account. On my system, ss resides in the /usr/sbin/ss directory. I’ll start off with some basic uses for ss. The following command shows the output for IPv4 networks:
# ss -4
The abbreviated output (Listing 1) shows client and server communication. Bear in mind that a client can also be a server, and vice versa, depending on the direction of the information flow.

Listing 1
Basic ss Output
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 0 192.168.0.2:37564 192.168.0.100:www
ESTAB 0 0 192.168.0.2:47592 192.168.0.156:smtp
ESTAB 0 0 192.168.0.2:ssh 192.168.0.49:64009

Replace the -4 option with -6 to output information on IPv6 connections.

As you can see in Listing 1, 192.168.0.2 is the local machine IP address, and the useful /etc/services utility has converted some port numbers into names (such as ssh, www, and smtp).

The excellent ss also offers information about TCP, UDP, local Unix Domain Sockets, and remote sockets. As I’ll describe later, what makes ss exceptionally powerful is its ability to deal with the state of connections.

I often use ss to query which ports are opened by daemons installed on the computer. Use the l option to check for listening ports (Listing 2):

Listing 2

Viewing the Listening Ports
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 *:ssh *:*
LISTEN 0 100 127.0.0.1:smtp *:*
LISTEN 0 128 *:sunrpc *:*
LISTEN 0 128 127.0.0.1:http *:*

# ss -l
You might want to check for the listening ports during your daily sys admin routine to ensure that unexpected and potentially insecure services haven’t been left enabled. Or, you might want to check for less sophisticated rootkits that might not hide their open ports effectively.

Either of the commands
lsof -i
netstat -tulpn

will help you figure out which processes (PIDs) are opening up your ports, so you can kill them off if necessary.
Simply running the ss command without any options provides a list of current “connections” (note these are actually “sockets”), as shown in Listing 3.

Listing 3
ss Unadorned

# ss
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 0 192.168.0.2:37564 192.168.0.49:tcpmux
ESTAB 0 64 192.168.0.2:ssh 192.168.0.143:64009
ESTAB 0 0 192.168.0.2:47609 192.168.0.88:gopher

PIDs

If you can’t get access to lsof, and you don’t like netstat (I’m not a massive fan), the super-duper ss utility can also report on PIDs pertinent to your open ports. To see which processes are using sockets directly, just throw a -p into the mix:
# ss -p
One caveat is I need to log in as root to get the extended information. Listing 4 shows an example of what to expect.

Listing 4
ss -p
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 0 10.10.10.20:ssh 10.10.10.10:52918 users:(("sshd",31195,3),("sshd",31204,3))

Working backward from the application name, which is sshd in this case, you can see the PIDs 31195 and 31204; 31204 belongs to a non-privileged user (my login user), and 31195 appears to result from the clever privilege separation employed by sshd.

SSH is designed to minimize the chances for programming bugs causing very serious systemic issues by limiting access and restricting the service to a chrooted jail for the network-facing SSH process. To achieve this goal, SSH uses two processes. The root-owned process monitors the progress of the non-privileged process, which has a previously unused UID and GID.
Looking Deeper

You can use ss -s to retrieve screeds of statistics relating to how many sockets are open and which protocols they are using (Listing 5).

Listing 5

ss -s
Total: 201 (kernel 218)
TCP: 6 (estab 2, closed 0,
orphaned 0, synrecv 0,
timewait 0/0), ports 5
Transport Total IP IPv6
* 218 - -
RAW 0 0 0
UDP 5 5 0
TCP 6 6 0
INET 11 1 0
FRAG 0 0 0

It’s not the best of comparisons, but you could get a vaguely similar result using the following netstat command:
# netstat -tan | grep -v "Proto" | grep -v "Active" | awk '{print $6}' | uniq -c
It’s definitely worth mentioning that the super ss utility has a habit of outperforming other network tools in benchmark tests. Compared with the widely-used netstat, for example, ss delivers its results very rapidly indeed. Keep in mind, however, that ss is not designed to answer every possible question but, rather, reflects the Unix philosophy that each system component should “do one thing well.” Experienced users often combine ss with tools like netstat then prune the output using tools such as grep, awk, and sed.

For instance, if you would like to see which application is tying up a specific port, you could use ss with grep, as follows:
# ss | grep 58620

Getting Specific

You can look at only TCP sockets with:
# ss -t
Alternatively, you could write out the –tcp option. Ramp up the level of detail with the -a (for “all”) switch:
# ss -t -a
Altering that command ever-so slightly generates output to include UDP, Raw, and Unix sockets. For a verbose view of all UDP sockets, enter
# ss -u -a
and to view all Raw sockets, enter:
# ss -w -a
If you really must indulge yourself with several screen’s worth of mind-boggling text, you can choose to view all Unix Domain sockets with:
# ss -x -a

Also included in the seemingly bottomless toolbox provided by the ss utility is the ability to watch out for DCCP sockets; DCCP is a less common network protocol that has the connection-oriented, error-checking traits of TCP with the broadcast-type features of UDP. The DCCP protocol is often used for media streaming. Check for DCCP traffic with this nifty little command:
# ss -d -a
You can also monitor the status of connections to your computer by extending the level of detail using the -e option (Listing 6).

Listing 6
# ss -e
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 0 192.168.0.2:ssh 192.168.0.88:58302 timer:(keepalive,40min,0) ino:2184870 sk:ffff880138d26700
ESTAB 0 0 192.168.0.2:48246 192.1680.56:ntp timer:(keepalive,105min,0) ino:2187726 sk:ffff880138d2748

Listing 6 shows timer options that offer an insight into the current status of any keepalives on a connection. This feature can be useful for services such as HTTP or SSH, which tend to employ keepalives (see the box titled “Keepalive Notes”). Listing 7 shows similar output using the netstat -to command.

Keepalive Notes

A keepalive packet generally has a TCP ACK configured with a sequence number set to one less than the current number used on the connection. Any machine receiving a nudge from such a packet will simply respond with the correct sequence number and, Frankenstein jokes aside, announce that it is indeed alive.

These types of packets are sometimes empty and generally have three associated parameters. The retry parameter will declare how many times to send a packet before coming to the conclusion that the other end of the connection has given up the ghost. The time setting is configured as the frequency of the checks, and finally the interval dictates the length of time between two sent packets if no response is received.

Inside the kernel, you can alter these settings by editing these values in the eye-watering pseudo-filesystem known and loved as /proc. One file, in this case, is /proc/sys/net/ipv4/tcp_keepalive_time, and you can edit it as follows:
# echo 75 > /proc/sys/net/ipv4/tcp_keepalive_intvl
# echo 9 > /proc/sys/net/ipv4/tcp_keepalive_probes

Listing 7
# netstat -to
Proto Recv-Q Send-Q Local Address Foreign Address State Timer
tcp 0 0 host-one:48340 host-four:45358 ESTABLISHED keepalive (6830.00/0/0)
tcp 0 64 host-two:ssh host-three:58302 ESTABLISHED on (0.45/0/0)

Netstat also lets you prepend the versatile watch command and see real-time updates:
# watch netstat -to
It is a close call, but I have to admit that, in this instance, netstat keeps its output nice and succinct and looks as good as the ss utility’s output.
Source and Destination
The following command lets you find information on any sockets with a destination (dst) aimed at a specific IP address:
# ss dst 192.168.0.1
Conversely, it’s not going to take a massive leap to consider that using the src option reveals information on the source side of the socket:
# ss src 192.168.0.2
Handy, huh? This syntax is easy to remember if you’re in a rush, saving the day when a testy boss is breathing down your neck during an outage.
You can even use CIDR network notations in the address callout:
# ss dst 192.168.0.1/24
Add a colon at the end, and you can check for a very specific port and a very wide IP address range at the same time:
# 192.168.0.1/24:53
I really appreciate this functionality in situations with lots of traffic and numerous open ports. For instance, this option lets you monitor all DNS-related activity on an entire /24 subnet with one simple ss command.
Regular Expressions
In addition to DNS names and IP addresses, you can also use regular expression (regex) operators in your ss syntax. Have a look at this little nugget:
# ss dport != :53
This command excludes the destination dport on the DNS port 53.
If you need to retain some sanity and avoid just looking at numbers, you can also translate ports into the /etc/services format:
# ss 192.168.0.1:http
If you’re ready to be impressed, you can also include greater-than, lesser-than, less-than-or-equal-to, and so forth:
# ss dport > :53
Other special characters are possible also, although sometimes special characters need escaping. Alphabetic equivalents, such as eq (equal), ne (not equal), gt (greater than), and le (less than) also work. Your mileage might vary with different versions of the ss utility.

Connection State

You can use the –query or -A query addition to dump a sockets table. The magic word autobound checks for ephemeral ports that sockets have attached themselves to. Prepare yourself for screeds of output, even on a quiet system. An abbreviated version of the output is shown in Listing 8.

Listing 8

# ss -a -A all autobound
01 u_str ESTAB 0 0 * 11984 * 0
02 u_str ESTAB 0 0 * 11996 * 0
03 u_str ESTAB 0 0 * 12003 * 0
04 u_str ESTAB 0 0 * 12005 * 0
05 u_str ESTAB 0 0 * 12010 * 0

You can also filter by TCP states; for instance, the following command filters for sFTP traffic:
# ss state connected dport = :sftp
Anything treated as “current” in relation to the sFTP port is displayed promptly.
You can complicate the command a little more with a boolean operator:
# ss ( sport = :ftp or dport = :http )
You can even use ss to find connections that are in a specific TCP state, including the established, syn-sent, syn-recv, fin-wait-1, fin-wait-2, time-wait, closed, close-wait, last-ack, listen, and closing.

The TCP state parameters let you do some very powerful querying. For example, checking for FIN–WAIT–1 states lets you identify whether your application has closed its side of a connection, but a remote host has not closed its side, thus tying up your machine’s precious ports:

ss -o state fin-wait-1 '( sport = \
:ftp or sport = :http )' \
dst 10.10.3.3/24:22

Sso It Ends

The ss utility is a powerful tool that will help you query your network in significant detail. Ss is extremely high performance for both manual and automated queries, and it requires very few keystrokes to execute common commands.

This tiny but heroic tool helps flex the muscles of any sys admin. If you want to increase the power of your admin toolkit, try practicing some of the more complex commands in your day-to-day work.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: