Who's on first? Identifying port utilization in Solaris

Setting up Apache2 on Solaris 10 is normally about as challenging as brushing your teeth. But in this case, I was humbled by an unexpected troubleshooting adventure. I needed to transfer a TWiki site from an Apache server running on Solaris 9 to an Apache2 server running on Solaris 10. Sounds pretty straight forward, but I abandoned discipline at one point in the game and that detour came back to bite me.

I started carelessly thinking that to make things simple I would just use the legacy apache. This would save any initial headaches with module incompatibilities (if any existed). So, I started out copying the config file in place and trying to start the daemons. It didn't work, and after a few minutes of fiddling with the new httpd.conf I changed course. My reasoning went something like this, "If I'm going to spend much time fiddling, I might as well fiddle with Apache2 and have something better than I started with." And so it began.

I stopped the legacy Apache daemons and followed a similar process with Apache2, ending with the same result: No daemons. I did some fiddling and located a minor typo I'd made in the configuration which is not of consequence to this story. I issued a "svcadm restart apache2" command. Yeeha! Now I had five httpd processes just chomping at the bit for a chance to serve those Wiki pages.

Or did I? It turned out that no matter what I did with my web browser remotely or locally I couldn't get a response. So, I tried a quick telnet to port 80 to see what there was to see... And of course I received a response, so all must be well. Somewhere in my troubleshooting process I made two mistakes:

First, I didn't remove the httpd.conf file from /etc/apache, which means the legacy Apache starts up and conflicts with Apache2 on a reboot. I've already written an article that goes into some detail about why the current legacy Apache's integration isn't ideal, so I won't expand on my frustration in this one. This problem was quickly solved, and could have been avoided if I had adhered to my Jedi training.

Second, I assumed that when I directed a Telnet session to port 80 it was reaching the Apache2 server. In fact, it was not. I shut down the Apache2 server and again issues the Telnet command to port 80. Surprise! The same greeting appeared. So, some process on the system had claimed port 80 before Apache could do so. Now... To find it!

Linux distributions typically ship with the lsof utility. This provides a quick and convenient way to identify what process is using what TCP port. Solaris doesn't have lsof in the integrated Open Source software (/usr/sfw) or the companion CD (/opt/sfw). It's not hard to obtain and compile, but it's just inconvenient enough that I'm inclined not to do it. My next logical question became, "what is the Solaris way to accomplish my goal?".

Solaris has no way to natively solve this issue without a shell script. There are a number of similar scripts available on-line through a quick Google search. None are particularly complex, but complex enough that you wouldn't want to write them every time you need it. Here's what I ended up with:

#!/bin/sh

if [ `/usr/xpg4/bin/id -u` -ne 0 ]; then
echo "ERROR: This script must run as root to access pfiles command."
exit 1
fi

if [ $# -eq 1 ]; then
port=$1
else
printf "which port?> "
read port
echo "Searching for processes using port $port...";
echo
fi

for pid in `ps -ef -o pid | tail +2`
do
foundport=`/usr/proc/bin/pfiles $pid 2>&1 | grep "sockname:" | egrep "port: $port$"`
if [ "$foundport" != "" ];
then
echo "proc: $pid, $foundport"
fi
done

exit 0


When executed, it will produce output similar to the following. Note that it requires root permissions to traverse the proc directories...

cgh@testbox{tmp}$ sudo ./portpid 80
proc: 902, sockname: AF_INET 0.0.0.0 port: 80
sockname: AF_INET 192.168.1.4 port: 80
sockname: AF_INET 127.0.0.1 port: 80

A quick "ps -ef " command told be that our Citrix server was to blame for the port conflict...


cgh@testbox{tmp}$ ps -ef | nawk '$2 ~ /^902$/ {print $0}'
ctxsrvr 902 1 0 May 18 ? 7:00 /opt/CTXSmf/slib/ctxxmld


Ah ha! Problem solved. I'd like to see the Solaris engineering team add a "p" command, or an option to an existing command to make this functionality a standard part of Solaris. Another option would be to integrate the Linux syntax for the fuser command to make this possible.

The demise of Linux

I read a quick editorial which suggests that Ubuntu Linux is going to be the downfall of Red Hat. The premise being that as Red Hat grew more commercial, and abandoned its community in favor of its stockholders, the sys-admins who used Red Hat as their desktop OS drifted away from the Red Hat camp. As Ubuntu came into being, and did so in a very strong way, those SA-types who were driven away from Red Hat will now want to put what they are more familiar with (Ubuntu) on their servers when they have the choice.

This whole discussion brought me right back down memory lane. I started using Linux before Red Hat existed with a few early Slackware distributions. I remember writing all those 3" diskettes - somewhere around 80 of them by the time I burned the X-windows distribution as well. Shortly after came Red Hat, and at that time I was helping to set up the first Linux environment at SUNY Plattsburgh. We switched over from Slackware to Red Hat, and loved it.

I ended up sticking with Red Hat for the next 10 years. It was the OS of choice when I lead a project to build servers for our local Boy Scouts of America council, and remained there until Red Hat went totally commercial, burning the bridges out from under us. Make no mistake, I was extremely disappointed with their decision. We switched over to Fedora Core, and for the most part it has been a smooth transition. Despite its big red "DEVELOPMENT" stamp, Fedora has been very good to our availability. In fact, at the moment we've got an impressive uptime on an FC2 system:

# uptime
21:41:15 up 412 days, 2:37, 2 users, load average: 0.00, 0.00, 0.00


But thinking back on the experience I have an entirely different memory of what direction I was forced in, and where I wanted to be. The places I've worked would not consider using Linux for their production environments. Linux has made some great in roads t the corporate world, but there aren't a lot of Fortune 500 companies running their SAP central instance on Linux. I'm sorry, it's just not happening. HPUX, AIX, and Solaris are king in the land of mission-critical highly scalable UNIX servers.

Although Solaris never made a particularly compelling desktop, it's what I've wanted to use on every server I've ever built. It's rock-solid, well documented, and very cohesive. When you use Solaris and Sun products, you rarely get the impression that 10,000 individual developers all tried to do it "their way" when the final build was cast in stone. What always stopped me was cost. Solaris x86 had pathetic support and commitment in the past. It's so incredibly painful to migrate between operating environments that I never wanted to risk Solaris x86 being yanked - which it was.

The second big barrier was cost. If you went with Sparc, you had to have money. Lots of money. Oodles of money! What I had was a basement full of x86 architecture hardware, and the not-for-profits I volunteer at had the same. There was simply no funding for shiny Sun hardware no matter how badly we wanted it.

And then the sleeping giant awoke. After being pummelled by the dot-com crash, Sun figured out what went wrong, and executed one of the m most amazing feats of corporate intertia changes I've ever seen. In a very short time frame, support for Solaris x86 was restored at a full commitment level. And it was made free. Then they continued to make their Java Enterprise System free to download and use as well.

So, the making of a fantastic Linux in Ubuntu may hurt Red Hat, but it's not what will deliver the killing blow. Red Hat has an opportunity right now to try to pull off a corporate inertia swing of Sun's magnitude. They need to restore faith in the community restore the religion they destroyed and find some kind of innovation to draw people back in. Solaris has done all of this and created an affordable support model that doesn't intimidate the small businesses who were once driven to Linux.

The first blood has been drawn by Solaris, but the second wound is far deeper. This second wound is bleeding internally and missing a lot of coverage. Mac OS-X is the killer desktop. If you have a reason to be using UNIX on a desktop, then using anything other than Mac OS-X is a tough sell in my book. Hardware is a bit more expensive, sure. But it's the best of every world, and solid as a rock. It doesn't hurt that it looks great either.

A recent seminar I attended talked about business models and knowing when to have the guts to drop a design. The idea was that you need to look at things you're developing and ask whether or not they give you a long-term sustainable advantage. I have to use that same litmus test to examine Linux. In the server world I can get free and open Solaris which is out-innovating Linux in my observation. And on the desktop, while Linux continues to improve, it's not even close to Mac OS-X.

In the end, these observations mean little to the tech-hobbyist who loves Linux for its religion. But in the business world, religion doesn't make IT choices. Competitive advantage does.