Unix netstat -an Output Explained

$ netstat -an

TCP: IPv4

Local Address Remote Address Swind Send-Q Rwind Recv-Q State

-------------------- -------------------- ----- ------ ----- ------ -----------

*.* *.* 0 0 49152 0 IDLE

10.128.27.50.16043 192.168.170.31.54871 49640 0 49640 0 ESTABLISHED

10.128.27.50.14353 10.128.14.211.44398 24820 0 49640 0 ESTABLISHED

127.0.0.1.57958 127.0.0.1.57957 49152 0 49152 0 TIME_WAIT

10.128.27.50.57959 10.128.27.108.11919 49640 0 49640 0 TIME_WAIT

10.128.27.50.16041 10.128.17.34.62013 49640 0 49640 0 TIME_WAIT

10.128.27.50.62393 10.128.27.51.16001 49640 0 49640 0 CLOSE_WAIT

10.128.27.50.36198 10.128.27.39.11035 49640 0 49640 0 ESTABLISHED

10.128.27.50.16035 *.* 0 0 49152 0 LISTEN

Useful Command : netstat -na | awk '{print $7}' | sort | uniq -c | sort -n

Swind/Rwind Are the window size for sending/receiving packets

Send-Q/Recv-Q Are the send/receive queue, meaning how many packets are waiting to be sent/received

LISTEN- Waiting to receive a connection

ESTABLISHED- A connection is active

CLOSE_WAIT:

This is a state where socket is waiting for the application to execute close()

CLOSE_WAIT is not something that can be configured where as TIME_WAIT can be set through tcp_time_wait_interval (The attribute tcp_close_wait_interval has nothing to do with close_wait state and this was renamed to tcp_time_wait_interval starting from Solaris 7)

A socket can be in CLOSE_WAIT state indefinitely until the application closes it.

Faulty scenarios would be like filedescriptor leak, server not being execute close() on socket leading to pile up of close_wait sockets. (At java level, this manifests as "Too many open files" error)

TIME_WAIT :

When the TCP socket closes, the side starting the close puts the socket into the TIME_WAIT state.this should last only a minute or two, then change back to LISTEN. The socket pair cannot be re-used as long the TIME_WAIT state persists.This is just a time based wait on socket before closing down the connection permanently.

Under most circumstances, sockets in TIME_WAIT is nothing to worry about.

A netstat will short the sockets in the TIME_WAIT state. The following shows an example of the TIME_WAIT sockets generated while benchmarking. Each client connection has a unique ephemeral port and the server always uses its public port:

Typical Benchmarking Netstat

unix> netstat

...

tcp 0 0 localhost:25033 localhost:8080 TIME_WAIT

tcp 0 0 localhost:25032 localhost:8080 TIME_WAIT

tcp 0 0 localhost:25031 localhost:8080 TIME_WAIT

tcp 0 0 localhost:25030 localhost:8080 TIME_WAIT

tcp 0 0 localhost:25029 localhost:8080 TIME_WAIT

tcp 0 0 localhost:25028 localhost:8080 TIME_WAIT

The socket will remain in the TIME_WAIT state for a system-dependent time, generally 120 seconds, but usually configurable. Since there are less than 32k ephemeral socket available to the client, the client will eventually run out and start seeing connection failures. On some operating systems, including RedHat Linux, the default limit is only 4k sockets. The full 32k sockets with a 120 second timeout limits the number of connections to about 250 connections per second.

If mod_caucho or isapi_srun are misconfigured, they can use too many connections and run into the TIME_WAIT limits. Using keepalives effectively avoids this problem. Since keepalive connections are reused, they won't go into the TIME_WAIT state until they're finally closed. A site can maximize the keepalives by setting thread-keepalive large and setting live-time and request-timeout to large values. thread-keepalive limits the maximum number of keepalive connections. live-time and request-timeout will configure how long the connection will be reused.

TCP Parameters on Different Platforms:

# ndd /dev/icmp \? - Solaris 10.

# /proc/sys/net/   All TCP/IP tuning parameters are located under this directory