Who's on first? Identifying port utilization in Solaris

Setting up Apache2 on Solaris 10 is normally about as challenging as brushing your teeth. But in this case, I was humbled by an unexpected troubleshooting adventure. I needed to transfer a TWiki site from an Apache server running on Solaris 9 to an Apache2 server running on Solaris 10. Sounds pretty straight forward, but I abandoned discipline at one point in the game and that detour came back to bite me.

I started carelessly thinking that to make things simple I would just use the legacy apache. This would save any initial headaches with module incompatibilities (if any existed). So, I started out copying the config file in place and trying to start the daemons. It didn't work, and after a few minutes of fiddling with the new httpd.conf I changed course. My reasoning went something like this, "If I'm going to spend much time fiddling, I might as well fiddle with Apache2 and have something better than I started with." And so it began.

I stopped the legacy Apache daemons and followed a similar process with Apache2, ending with the same result: No daemons. I did some fiddling and located a minor typo I'd made in the configuration which is not of consequence to this story. I issued a "svcadm restart apache2" command. Yeeha! Now I had five httpd processes just chomping at the bit for a chance to serve those Wiki pages.

Or did I? It turned out that no matter what I did with my web browser remotely or locally I couldn't get a response. So, I tried a quick telnet to port 80 to see what there was to see... And of course I received a response, so all must be well. Somewhere in my troubleshooting process I made two mistakes:

First, I didn't remove the httpd.conf file from /etc/apache, which means the legacy Apache starts up and conflicts with Apache2 on a reboot. I've already written an article that goes into some detail about why the current legacy Apache's integration isn't ideal, so I won't expand on my frustration in this one. This problem was quickly solved, and could have been avoided if I had adhered to my Jedi training.

Second, I assumed that when I directed a Telnet session to port 80 it was reaching the Apache2 server. In fact, it was not. I shut down the Apache2 server and again issues the Telnet command to port 80. Surprise! The same greeting appeared. So, some process on the system had claimed port 80 before Apache could do so. Now... To find it!

Linux distributions typically ship with the lsof utility. This provides a quick and convenient way to identify what process is using what TCP port. Solaris doesn't have lsof in the integrated Open Source software (/usr/sfw) or the companion CD (/opt/sfw). It's not hard to obtain and compile, but it's just inconvenient enough that I'm inclined not to do it. My next logical question became, "what is the Solaris way to accomplish my goal?".

Solaris has no way to natively solve this issue without a shell script. There are a number of similar scripts available on-line through a quick Google search. None are particularly complex, but complex enough that you wouldn't want to write them every time you need it. Here's what I ended up with:

#!/bin/sh

if [ `/usr/xpg4/bin/id -u` -ne 0 ]; then
echo "ERROR: This script must run as root to access pfiles command."
exit 1
fi

if [ $# -eq 1 ]; then
port=$1
else
printf "which port?> "
read port
echo "Searching for processes using port $port...";
echo
fi

for pid in `ps -ef -o pid | tail +2`
do
foundport=`/usr/proc/bin/pfiles $pid 2>&1 | grep "sockname:" | egrep "port: $port$"`
if [ "$foundport" != "" ];
then
echo "proc: $pid, $foundport"
fi
done

exit 0


When executed, it will produce output similar to the following. Note that it requires root permissions to traverse the proc directories...

cgh@testbox{tmp}$ sudo ./portpid 80
proc: 902, sockname: AF_INET 0.0.0.0 port: 80
sockname: AF_INET 192.168.1.4 port: 80
sockname: AF_INET 127.0.0.1 port: 80

A quick "ps -ef " command told be that our Citrix server was to blame for the port conflict...


cgh@testbox{tmp}$ ps -ef | nawk '$2 ~ /^902$/ {print $0}'
ctxsrvr 902 1 0 May 18 ? 7:00 /opt/CTXSmf/slib/ctxxmld


Ah ha! Problem solved. I'd like to see the Solaris engineering team add a "p" command, or an option to an existing command to make this functionality a standard part of Solaris. Another option would be to integrate the Linux syntax for the fuser command to make this possible.

Apache in Solaris 10: 3 Simple Things I Would Change

The Apache legacy run control script in Solaris 10 (/etc/init.d/apache) provides an excellent example of a few practices to avoid when writing init scripts.

Take a look at the code snippet below:

if [ ! -f ${CONF_FILE} ]; then
exit 0
fi


Are you kidding me? Of course this is easy to debug, but let's look at what it does anyway: If the configuration file is missing, when you ask to start Apache, and it will exit with a code of zero when it doesn't find the /etc/apache/httpd.conf file. In case you didn't catch the first four words of this paragraph I'll repeat them. Are you kidding me?

Here's a simple improvement...

if [ ! -f ${CONF_FILE} ]; then
echo "ERROR: ${CONF_FILE} not found. Exiting."
exit 1
fi


The first change was to exit with a non-zero status. Zero is the UNIX standard exit code representing successful completion. If the configuration file is missing and you request a startup, it should NOT exit with a zero status.

The second change is to provide a concise error message indicating why the exit code is going to be zero. There is no benefit to bolstering the cryptic nature of UNIX. In my mind the best systems are designed such that a tired SA at 4AM has a reasonable chance of accurate debug and corrective action.

Having said all this, the reason the code is necessarily convoluted because the not-yet-configured service has an active set of init scripts in the run control directories.

cgh@testbox{etc}$ ls -i /etc/init.d/apache 21813 /etc/init.d/apache*
cgh@testbox{etc}$ find /etc/rc?.d -inum 2813
/etc/rc0.d/K16apache
/etc/rc1.d/K16apache
/etc/rc2.d/K16apache
/etc/rc3.d/S50apache
/etc/rcS.d/K16apache


So the root cause of our problem is that someone decided to make it easy for someone who doesn't understand the Solaris Run Control facility to start Apache by simply creating the httpd.conf file. Is that really a good idea? I would argue that for many reasons it's a bad practice. If a service is not configured to run, it should not be active in any run level.

The third detail I would change is Solaris' default behavior of installing active sym-links in the legacy rc directories, and instead use an SMF manifest that adheres to standards.

None of this impacts the otherwise excellent web server that Sun has integrated into their OS, and I'm grateful that Sun has provided it in their standard OS rather than leaving it to the semi-integrated Companion CD. I woudl, however, like to see that integration brought up to Jedi standards.

5/21/07 Postscript: I probably should have made it clear that the Apache2 server is implemented nicely using SMF, and is probably what you ought be to using on Solaris 10 if you've decided to forego the JES Web Server. I don't think that excuses the older Apache server from maintaining Jedi discipline, but it does move the issue a bit toward the background.

Turn off the LAMP and Reuse Acronyms

I've never been a fan of the LAMP acronym because it's too restrictive. It gives the impression that to be socially responsible in the Linux community one needs to be a LAMP developer.

In this month's Linux Journal magazine I found an article explaining one perspective on why PostgreSQL is a more desirable database than MySQL. I've had the exact same thought process for years now. Truth be known, I also prefer Perl development to PHP, and I prefer running the stack on Solaris over Linux. I guess that SAPP doesn't have the same sexy ring as LAMP. There's probably an odd trademark thing with an ERP company as well.

Now before you get too bent out of shape, I am aware that the acronym has some poetic license with it, and people often swap Perl and PHP, and in theory any other letter can be swapped out. Why invent a new acronym that doesn't convey the real idea when a perfectly good acronym already exists?

There is nothing wrong with simply stating that an application is built on an Open Source Stack. The acronym OSS (Open Source Software) is well known and conveys a lot more than LAMP. It stands for a methodology rather than a point solution, and embraces the foundation that made "LAMP" so successful. Why limit yourself to MySQL and PHP? Wouldn't you be more valuable as an architect capable of leveraging the most appropriate components Open Source has to offer?

Another Round with the Laptop

Having recently switched a bunch of older Fedora Core servers to CentOS 4, I became very excited to see the announcement that CentOS 5 has been released. I am very pleased with the polish and stability of CentOS 4 and thought 5 would be a perfect update for my (then) Ubuntu laptop. My laptop is a fairly old IBM Thinkpad T23 - certainly not bleeding edge hardware.

The display was maxed out at 800x600 rather than the native 1400x1050 I'm accustomed to and my NetGear wifi card was recognized, but not functional. It comes as no surprise that none of my Thinkpad buttons worked. I spent the past 10 years of my hacking career tweaking Linux boxes, and I was using X windows in the days when you actually risked toasting your monitor with a bad config. So, yes, I could make X work. And I have gone through the Windows wifi card firmware dance, so yes, I could make the Wifi card work as well. I could also recompile the kernel and add Thinkpad buttons and ACPI events. Yuck. My rebuild just turned into a lot of research and work.

On a whim I thought I'd try Fedora Core 6 just in case my problems stemmed from the more conservative approach CentOS takes. Same problems, although I love the eye candy in Fedora - their art is great.

I didn't bother to try Solaris x86 because I know it won't detect any of my special hardware. Someday I'd love to be able to use Solaris outside of work, but that day hasn't come yet on the desktop.

So here I am, back with Ubuntu. It detects everything and works right out of the box. Within an hour I had a fully functional laptop, and despite my 1 GHz CPU in a 5GHz world it performs great. I suppose a year from now I'll try it again since I prefer a Red Hat based distribution to a Debian base. But what matters most comes down to two words: "It Works."

Thanks, Ubuntu.

Does Your Virualization Strategy Have a Blind Spot?

I just finished reading a very interesting article about virtualization. It described the test results of running two sample web applications under virtual and physical environments. The idea was simply to check how the virtualization affected the application's performance. The results are interesting.

Before I go further, let's clarify that this article is about Windows IIS and Win2k3 server, so it is not about Solaris, or any other flavor UNIX beyond the fact that the underlying OS for VMWare in this case was CentOS. None the less I believe the story's moral is highly applicable to any operating environment, definitely including UNIX.

There is a lot of detail in the article, but if we can assume their methodology to be relevant, we can reduce it down to a short extract from its conclusion: "a virtualized server running a typical web application may experience a 43% loss of total capacity when compared to a native server running on equivalent hardware."

Wow.

Of course, I have no reason to think that this function carries over directly to Zones, Xen, or even VMWare running Linux. To believe this without research would be to abandon one's Jedi training. It would be a very valuable experiment to try such an excercise though, because I have no reason (at this moment) to think it would not have some relevance.

The big point here I want to make is that I would bet there are a great many sites out there who dilligently track low hanging fruit metrics like CPU utilization, and use that metric both in planning and asessing their virtualization projects. Server "A" runs an application and on average sits at 10% utilization. Lets call it 100Mhz and find it a home on the consolidation farm. Not so fast.

The problem is that if you completely ignore business metrics, you will fail to identify this article's identified flaw in virtualization. Hey, we moved from 8 servers down to one, and its at 80% all the time. Success! But what if that new server is running at 80% and only processing 50% of the volume it used to? I would suggest that the article I referenced might be a great reason to reflect on your site's metric stratgey and see if you have the ability to accurately asess a consolidaton stratgy. If not, you could be making an expensive transition.

On Proper Use of English

While my mind was entering a virtual screen-saver (yes, oddly enough my mind seems to run a GUI over its CLI interface) I overheard a conversation in the next cubicle. A project manager said, "...and then we give it to the engineers for solutioning". I started to think about how often I'd heard that expression and quickly realized the count was quite high.

Shortly after that exciting revelation (much more exciting than the status bar I watched during an in-progress installation) I recalled another expression I'm finding has become prevalent on our site: The act of "dialoguing". I'm not completely sure of the spelling of this colloquialism, but I'm hearing it all the time.

If I'm understanding them correctly, my job as an engineer entails dialoguing with customers to gain requirements which I can use in the solutioning process. Ain't that it?

From the Merriam Webster Dictionary, we learn that "solution" is actually a noun. Just to remove any ambiguity from my position on this matter, let me also state that a noun in this scenario cannot moonlight as a verb and still retain its dignity.

Continuing on our lexical journey, the MW Dictionary also has an entry for dialogue. It's simply amazing to find such an artifact considering how few people have been able to study this elusive part of speech. But, in the interest of open sharing of knowledge I'd like to share what it contains. Dialog is also a noun. Amazing!

Having solved this perplexing grammar mystery I can now return to designing a solution to the problem of how to virtualize our Directory Service. I hope you've enjoyed this dialogue.

JET: Controlling custom_files with a custom extension

Any site running Sun hardware with more than one system should be looking at JumpStart to ensure that systems can be rebuilt consistently. the corollary to this is that any site running JumpStart environment should be using Sun's Jumpstart Enterprise Toolkit (JET). JET provides a consistent framemwork for accomplishing most common tasks, and a consistent framework to write extensions within. Standards and discipline are good.

One of the modules which comes with JET is called simly enough, custom. The custom module allows you specify either packages or files which should be added to a server during any of N predetermined reboots. This allows you to ensure that a change which requires a reboot can be made prior to a dependent process being started. Sounds good so far.

Following a recent Solaris 9 server build I was perusing the system for problems by auditing log files. In the messages file I discovered some lines indicating that a Kerberos problem was rearing its ugly head:
Kerberos mechanism library initialization error: No profile file open.
Our site does not use Kerberos, so it had to be a recent configuration change - not surprising considering we had just updated the patch set. After some research I arrived at BugID 5020096. This bug indicates that the issue can be resolved by removing some offending lines from /etc/krb5/krb5.conf.

This should be easy enough to fix in future builds. Just add the modified krb5.conf to the JET template's custom_files variable, and we'll be in good shape. Ahh, not so fast. How will we know what the file originally contained? A true Solaris Jedi will always manage an audit trail of his activities. If I were making the change manually I would copy the file to file.orig, or file.datestamp. Automation is not an excuse for abandoning discipline.

The trouble with JET is that its custom module's functionality for installing files is limited to two operations: overwrite or append. Overwrite simply clobbers any file which may exist. For example, to install the /etc/motd file I would palce my custom file in the configured JET file location, then add a line like this to the JET template:
custom_files_1="motd:o:/etc/motd"

motd is a fairly harmless little file, but knowing little about Kerberos, I dind't want to blindly whack the original file. The right solution to this problem lies in creating a simple extension to the JET toolkit. I began by examining the code from the custom module. Two modules specifically are relevant to this project: install, and postinstall. Within them is a simple case statement which handles the "o" or "a" functionality:

case ${mode} in
a) case ${fn2} in
/etc/hosts) JS_merge_hosts ${filefound};;
*) JS_cat ${filefound} ${ROOTDIR}${fn2}
;;
esac;;
o) JS_cp ${filefound} ${ROOTDIR}${fn2};;
esac


So, when I use an "o" in my custom_files module, it called JS_cp. I now needed to find the library which contains these core functions. Eventually, a colleague and I traced it back to /opt/SUNWjet/utils/lib. Looking at the JS_cp function revealed exactly what I expected: a simple copy routine wrapped in some voodoo.

Feeling a bt optimistic, I copied JS_cp to JS_cp_preserve and modified the code a bit so it would first check to see if the destination file exists, and if so, backup the file with a datestamp. Once the backup was in place, the original copy operation was performed. This was very trivial shell scripting. Here's what I ended up with:

if [ "$#" != "2" ]; then
JS_error "`basename $0`: Illegal Arguments. Usage: "
fi

JS_FROM=$1
JS_TO=$2

JS_display "Copying file `echo ${JS_FROM} | sed -e \"s?^${SI_CONFIG_DIR}/??\"` to ${JS_TO}"

if [ -f ${JS_TO} ] ; then
datestamp="`/usr/bin/date +%Y%m%d`"
/bin/cp -p ${JS_TO} ${JS_TO}.jet.${datestamp}
case $? in
0) # Success
JS_display "Successfully preserved ${JS_TO}.jet.${datestamp}"
;;
1) # Failure
JS_display "WARNING: Failed to preserve original file ${JS_TO}"
;;
esac
fi

/bin/cp -p ${JS_FROM} ${JS_TO}

if [ "$?" != "0" ]; then
JS_error "JS_cp:\t\tError occured while copying ${JS_FROM} to ${JS_TO}"
fi


Next, I returned to the install and postinstall code, and modified the case statements to accept a "b" operation (b for backup). I then executed a test Jump and was very pleased to see my JET extension had worked! I can now have custom_files install the workaround krb5.conf, and maintain a backup of the original. Here's the modified code:

case ${mode} in
a) case ${fn2} in
/etc/hosts) JS_merge_hosts ${filefound};;
*) JS_cat ${filefound} ${ROOTDIR}${fn2};;
esac;;
o) JS_cp ${filefound} ${ROOTDIR}${fn2}
b) JS_cp_preserve ${filefound} ${ROOTDIR}${fn2};;
esac


Note that you need to make this modification in both /opt/SUNWjet/Products/custom/isntall and postinstall.

Now, all I need to do it specify something in the custom_files module like this:
custom_files_N="krb5.workaround:b:/etc/krb5/krb5.conf"

And I will get a clean backup of the original file. Such a simple tweak - I hope the Sun folks who maintain JET will add something similar. While some limitations of JET can be frustrating, its intuitive layout and ease of extension make it something I grow more fond of each time I use it.

Frustration with Solaris Packages

I have a love / hate relationship with Solaris packaging. When you need to crank out a simple package I find it much easier to deal with than RPM. I also like the simple file system or streams based stucture vs. the binary mode of RPM. All things considered, it gets the job done, and has been a tremendous help in standardizing our provisioning system. There are, however, a few things that I'm not crazy about.

In Sun's model, the package is used to release functionality while the patch is a vehicle to fix existing functionality. If I want to add feature X to my software, I need to release a new package version. In contrast, if feature X is broken in a package then I need to release a patch. Seems simple on the surface.

One place this model gets sketchy is if I have the following situation: Package FOO needs to be updated to a new revision, but package BAR depends on it, and is required for system operations. In this case I need to first remove package BAR, then update package FOO, and finally, reinstall package BAR. In my mind this causes an unjustified level of system disruption. An RPM or dpkg based system would use an update option to perform this in-place. I'm told that there's an "in place upgrade" capability in the Solaris packaging system, but I haven't yet discovered it or found it documented. I will be looking though.

I have also noticed documentation gaps in the use of patches. Sun does provide instructions on how to produce a patch-package, but they omit naming conventions. Clearly, it would be a bad thing to produce package 123456-01 and then have Sun release the same one. This conflict could be very disruptive to a patch process. It seems that by selecting an upper range (ie 90001-01) you can have safety similar to selecting a 10.0.0.0 network address. I'd feel quite a bit better if Sun woudl explicitly define this range so we'd know it was safe. In the interim, I've been fixing bugs by creating minor revisions of packages rather than using patches.

The last point I wanted to touch on in this article is the use of package prototypes. In packaging nomenclature, a prototype file is the list of files included in the package, and their ownership and permission attributes. Here's an example of a prototype I'm durrently working for a custom sendmail solution:

d none etc 0755 root sys
d none etc/mail 0755 root mail
f none etc/mail/foo-client-v10sun.cf 0644 root bin
f none etc/mail/foo-server-v10sun.cf 0644 root bin
d none usr 0755 root sys
d none usr/lib 0755 root bin
d none usr/lib/mail 0755 root mail
d none usr/lib/mail/cf 0755 root mail
f none usr/lib/mail/cf/proto.m4 0444 root mail
f none usr/lib/mail/cf/foo.m4 0644 root mail
f none usr/lib/mail/cf/foo-client-v10sun.mc 0644 root mail
f none usr/lib/mail/cf/foo-server-v10sun.mc 0644 root mail


Pay particular attention to what I call placeholder lines. Those are lines in the prototype referring to directories which this package depends on, but are really part of another package by virtue of already being registered. Of course, a directory like /usr is chocked full of nested package dependencies:

# grep ^/usr /var/sadm/install/contents head
/usr d none 0755 root sys FJSVvplu SUNWctlu SUNWcsr TSBWvplu SUNWocfd SUNWncft SUNWGlib SUNWgcmn SUNWGtku SUNWctpls SUNWxwdv SUNWpl5u SUNWcpp FJSVcpc SUNWopl5p FJSVcpcx FJSVmdb FJSVmdbx IPLTadman SUNWowbcp SUNWpamsc SUNWpamsx SUNWpcmcu IPLTdsman SUNWadmj SUNWmcdev SUNWjsnmp SUNWtftp SUNWbsu SUNWpd SUNWsckmu SUNWpdx SUNWpiclh SUNWuxflu SUNWuxfl1 SUNWeurf SUNW1251f SUNWuxfl2 SUNWuxfl4 SUNWuxfle SUNWmgapp SUNWrmui SUNWpiclx SUNWpl5p SUNWTcl SUNWjpg SUNWTiff SUNWTk SUNWaccu SUNWaclg SUNWadmap SUNWpng SUNWpool SUNWpoolx SUNWant SUNWrcmdc SUNWpppd SUNWpppdu SUNWpppdt SUNWpppdx SUNWpppg SUNWfns SUNWsadml SUNWapct SUNWascmn SUNWasac SUNWqosu SUNWjaf SUNWjmail SUNWxsrt SUNWxrgrt SUNWxrpcrt SUNWiqfs SUNWiqjx SUNWiqu SUNWiquc SUNWiqum SUNWjaxp SUNWasu SUNWasdem SUNWrmodu SUNWrmwbx SUNWrpm SUNWrsg SUNWfnsx SUNWrsgx SUNWdfbh SUNWsadmi SUNWi15cs SUNWsadmx SUNWi1cs ... (lines omitted)


That was a tiny fraction of the list...

I don't think there is anything wrong with declaring a package as being dependent on a pre-existing directory, but I have a problem with how easy it is for a new package to overwrite the intende dattributes of that directory. Note that in my custom package's prototype I need to declare the attributes for /usr. This typocally means that I need to look at a clean operating system on the platform I intend to deploy on (ie - consistent Solaris revision) and pick the attributes from there.

I'd like to see the packaging facility accept a prorotype entry that has no attributes, and instead inherit the attributes from the package which initially registered the directory. This would minimize the chances of stray patches and packages conflicting with intended system permissions.

Having spent all this time complaining, let me end on a positive note by reinforcing how much efficiency we have gained by moving from tarballs and custom scripts to version controlled packages. I'd do it again in a heartbeat. I'm hoping Jedi discipline will eventually reverse the chaos inherent to the current packaging architecture.

Perl: Beware when counting array elements

Sometimes it's the simple things that put a halt to my productivity. In fact, it's rarely the big things. Master Yoda gave us an excellent phrase to consider when addressing a programming problem: "Judge me by my size do you? And well you should NOT!".

When I need to check how big an array is in Perl, it's quite intuitive to access that value using the $#array_name special variable. It works like a charm. But what happens when the array you need to count elements in is a referenced array?

A typical de-reference operation would be @$array_reference. This gives us the underlying array. Forgetting my Jedi training, I allowed the Dark Side to i$nfluence me as I flailed through syntactic permutations.


$$#array_reference
@$#array_reference
$#{$array_reference}


Much to my dismay, these futile expressions continued to plague me. I then focused my energies, and put the blast shield down.

...From WikiQuote

Ben: Remember a Jedi can feel the Force flowing through him.
Luke: You mean it controls your actions?
Ben: Partially, but it also obeys your commands.
Han: Hokey religions and ancient weapons are no match for a good blaster at your side, kid.
Luke: [to Han] You don't believe in the Force, do you?
Han: Kid, I've flown from one side of this galaxy to the other. I've seen a lot of strange stuff, but I've never seen anything to make me believe there's one all-powerful Force controlling everything. There's no mystical energy field controls my destiny! It's all a lot of simple tricks and nonsense.
Ben: I suggest you try it again Luke. This time, let go your conscious self and act on instinct.
[Ben places a helmet on Luke's head with the blast-shield down to blind him]
Luke: But with the blast shield down I can't see a thing!
Ben: Your eyes can deceive you. Don't trust them. Stretch out with your feelings.
[Luke calmly evades and deflects three pulses from the remote, successfully using the Force]
Han: I call it luck.
Ben: In my experience, there's no such thing as luck.
Han: Look. Good against remotes is one thing. Good against the living…that's something else.
Ben: [to Luke] You've taken your first step into a larger world.


I wrote a quick code fragment to isolate the problem and experiment with it. In doing so, I was reminded of the true nature of dereferencing and array contexts.


#!/usr/bin/perl -w
use strict;

my @a = qw(one two three four five six);
my $aref = \@a;

print "Number of elements in the array: $#a\n";
print "Number of elements in the array (ref): " . @$aref . "\n";


Looking at this code, ask yourself what values will be produced in each print statement. With carelessness likened to use of a blaster, one may assume they will produce the same integer. In fact they will not; A fact whose presence will be known through the Force. Let's take a closer look at these expressions.

@a is a literal array. The $# special variable is a direct vector to the last element used by that array. Knowing that Perl indexes its arrays beginning at zero, we then know this line will print the integer 5.

@$aref is a de-referencing operation. It represents the literal array, rather than the scalar that points to it. In this case, we are printing the array in scalar context, which will evaluate to the number of elements in the array: 6.

A simple problem with a not-so-simple explanation. What if your code had thousands of elements in it... Would your test plan have covered this condition when you decided to use referenced arrays rather than literals in one of your code branches? Remember that the Jedi Code tells us ...there is no ignorance; there is knowledge. Code, and learn.

Packaging delinquency in 3rd party software

We're working hard at the moment on migrating unmanaged scripts and solutions into Solaris pkgadd format, a.k.a. "packages". This has a number of benefits. First, it avoids the need to have complicated manual installation routines; By including preinstall / postinstall scriptsinstallation can be automated. Second, it is easy to know what revision of a software environment is installed right down to the installation process. Third, it ensures that not only can I ensure software is installed with the ight attributes and in the right places, but also that I can validate at a later time things are still as intended using pkgchk. I refer to this as managed files vs. unmanaged files.

The easy part was packaging our custom site scripts. We standardized on a hierarchy under /opt which contained bin, man, etc, lib, and sbin subdirectories. We then inventoried the unmanaged scripts and determined which were still valid and which could be discarded. The "keepers" were then incorperated into packages which were fairly simple overall.

Phase two has been an asessment of what unmanaged files for third party applications are being deployed. We found a lot of opportunity here and decided to start with the utility software like monitoring, security, and other non-revenue generating software. Here is where we have been uncovering nothing less than a mess.

For some reason, third party software providers in the UNIX space seem determined to make it impossible to manage their files. We've seen many interesting perversons of best practices that I thought woudl be interesting to collect in one place.

One product choose to adopt a package management solution called LSM, which I believe stands for Linux Software Manager. Note that this solution is for Solaris which has a perfectly good vendor provided and supported standard for software management. It turns out to be quite a technical feat to reversen engineer the format of LSM and directly convert to packaged.

Another product did not use any software management, but went so far as to encrypt their pre-installation bundles so as to make it impossible to install via a standards-based management system. This really blew my mind. What could be of such critical intellectual property in an installation routine that it justified encryption? And wouldn't the real IP be available once the software was installed anyway?

We routinely encounter software that has highly interactive installation processes that become cumbersome to integrate into packages because on each new release the routines would need to be re-ported to pre/post install scripts. The idea of managing software is to reduce work - not increase maintenance.

The biggest thorn in our side is Oracle. It's deployed everywhere in our environment and is intalled manually each time because we're told that's just how its done, and from observation it seems to be in the PITA bucket as far as automation goes. Contrast this to PostgreSQL which as of Solaris 10 (6/06) is integrated into the Operating Environment in clean packages.

So here's a message to all you third party software developers who provide Solaris solutions: Sun publishes an excellent guide to software packaging that any reasonably technical person could use to master the process in a few hours. Let me summarize a few key points in advance:

(1) You don't need to have a conversation to install software. Just copy the files, then configure it later.

(2) Sometimes I don't want to have a conversation. Let me put the answers in a file and feed that instead.

(3) Some of your customers have too many systems to install manually.

(4) Don't use a non-standard solution when the OS vendor provides a perfectly usable solution.

To err is human

I am writing this post as a catharsis and purification. A centering of my spiritual engineering energy that may otherwise be out of balance. Three days ago I made a typo which eliminated the /etc directory on a fairly important server. It was amazing how long that server continued to plug away after being lobatomized. Let me take you through the story as I relive the moment, and ensure that I learn from it.

Like many of the tasks I juggle, this was to be a short time-slice effort. I needed a distraction from a longer term project, and wanted to bite off a small piece of something that didn't require significant thought. Part of our Jumpstart environment deploys a tar archive to the client which is later unpacked and massaged by a custom script. My task was to eliminate the usr/local/etc directory from that archive and than recreate it. As my fingers systematically hit the keys, one extraneous finger made its imprint on the keyboard.

"r" "m" "-" "r" "." "/" "e" "t" "c".

The world slowed down as my finger hit enter, and I felt my heart stop beating. I believe I actually flat-lined that morning. Could it be? Had I really deleted /etc? Yes. I had. The command I entered was: "rm -r . /etc". I removed the current working directory and the server's /etc directory.

Why was I using elevated privileges for mundane work? The tarball had root-owned files in it. This is a downfall of our approach at the moment. When using pkgadd format, anyone can own the files which are given attributes at installation time. This makes day to day maintenance much safer. Ironically, I was editing the archive because I had just created a package to replace the files I was deleting. It was almost as if the prior bad practice were vomiting on me as I excercised it from the server.

Fortunately we had an excellent SA on hand to boot from CD and restore the missing file system, and it was back in business a relatively short while later. Eningeering nad operations are segregated in duties at my current site, so I was unable to clean up my own mess. A very humbling experience indeed, and this is what it taught me:

(1) Mirrored operating system disks are a good thing, but they don't protect you from human error propogating mistakes across both disks. While I've been a bit critical of maintaining a third contingency disk, there are other similar solutions which I have a heightened respect for.

(2) Whenever executing commands using RBAC, sudo, or the root account, count to three before hitting enter. No matter how much longer it takes to get your work done, no matter how good you are with UNIX, and no moatter how long it has been since you made a mistake, counting to three will always be quicker than restoring a file system from tape.

Sun loves Oracle, Sun loves PostgreSQL

Sun and Oracle have announced they will work together for another ten years. Not only that, but there's a new bundle in town that includes Oracle Enterprise with Sun servers. I haven't exactly figured out what it means to have software included for free that will require a support contract; Is that still free? But there were words indicating that processor count may not be relevant and that's probably where the savings lie. Maybe it's just saving you download time?

I don't really care about the pricing because big companies don't seem to hesitate to throw down dollars for Oracle licensing. What made this interesting to me was that Solaris, as of the 6/06 update now includes PostgreSQL natively - and there's no catches there. If you perform a "full distribution" install you already have an RDBMS. What's more, if you want to take that database into the critical waters of the production pool Sun will offer their world-class software support which means the company that knows their own operating system better than onyone else will also know the RDBMS sitting on top of it. Tres chique, n'est pas?

I'd imagine with Oracle's market share Sun has to play nice in the short term, but I give them a lot of credit for including PostgreSQL and picking a side. Right or wrong, in the age of mediocrity they made a decision. PostgreSQL is a phenominal database that competes aggressively with Oracle in many venues.

I'm fascinated with what the future will hold for relational databases on Solaris now that Sun has picked a side. This isn't just another open source database running on Linux farms - this battle will take place in the big data centers that Linux is just starting to scratch the surface of. I love Linux as much as the next guy, but how many sites do you know of running systems as large as an Enterprise 25K with Linux under the hood? Not too many - it's not in the heritage of Linux kernel - at least not yet.

So, where will this take Postgres? Methinks Oracle had better keep close tabs on Postgres over the next five years or so.

Adventures on eBay

I'm finally creating a Sparc based lab outside my place of employment. I want to make sure I can work on projects without any hint of conflict of interest of proprietary intellectual capital nonsense. My work is for hire from 8-5, but my ideas remain my own, and I want to be able to implement them outside of work with freedom.

I've been using an old AMD based system with Solaris x86 for a while now, but since my real interest is in Enterprise Engineering I really wanted to have some Sparc based equipment and at least one disk array. This will open the doors to many projects that would only be feasible on expensive modern x86 hardware. Funny how those same features have been on Sparc hardware since the early days...

For budget reasons I've gone with a v240 as my workhorse server. It doesn't have the sweet LOM that many Sparc system enjoy, but it does have four CPUs and four GB RAM. That will give me plenty of horsepower to run a few Oracle databases and work on some Zone projects. A v440 would have been ideal for my goals, but the price point on those systems is WAY too high for the investment returns I'll be looking at in the short-term.

I have the v240 server in my office right now, and have been playing with it a bit before it goes to the basement rack. After attaching a console cable and booting I was met with a hostname from the test.aol.com domain. I was a bit surprised to find that AOL doesn't wipe their disks before sending servers to auction, but after a quick look around I discovered that there isn't really anything on the disks anyway; just a Solaris 8 OS image. I almost skipped checking it out... I have zero interest in cracking, least of all a Solaris 8 image. I did have a small interest in their best practices, but I'm far more interested in my own projects, so the box will be getting a fresh load of Solaris 10 (6/06) this weekend.

My storage array of choice is the D1000. I'm not going to do much that requires massive expansion or throughput. I just need a bunch of disks I can put into ZFS, share for a cluster, and run databases on. The D1000 has a 12 disk Ultra-SCSI backplane that provides plenty of throughput for my needs. And hey, there's not much worry about driver obsolescence. It's such a simple device that it's just going to work as long as Solaris continues to support SCSI. The D200 is very cool, but I just couldn't find a value-add for my project list, so I went with budget. I won the bid for this unit this morning, so I'll have it in a week or so. Then I need to fill it with disks.

I upgraded my measly 16 port 10mb SuperStack II hub to a 24 port Superstack II Switch. This will ensure the servers have plenty of bandwidth to talk amongst themselves. I'm particulary proud of this purchase - only $1 plu s shipping. Works for me.

A few white papers from now I'm planning to add a second v240 for work on SunCluster and alternative clustering solutions. I'll also be adding a pair of Netra T1 AC200 servers for a directory services project.

And that should just about round out the new data center. I'm amazed at how much capability old servers have. CPU technology has progressed so much faster than the remaining system bottlenecks that many of today's systems simply will never show significant CPU utilization. For me, this means that if I'm willing to run servers at a level that makes them sweat, I can accomplish the same work for a fraction of the cost. This is the premise for a white paper I'll be working on that explores the use of old enterprise class hardware in not-for profit or small business shops. Stay tuned!

Shells and the path of standards

It seems that shells evoke almost as much controversy as religion. Everyone has a favorite, and for each favorite, there's another that would sooner grind their knuckles on the back of a circuit board than follow.

As long as I've been using Solaris systems I've never found common ground with the C-shell. Its programming capabilities are more limited compared to other options and its syntax is just too far from standards to be worth learning to me. But C is a standard UNIX language, you say. How can I call C-shell non-standard?

I'm a systems engineer, not an application developer (and now's not the time to debate that grey area!). If I were to write a script to traverse the entire Operating Environment, and calculate a distribution for the various shell I think we'd find 97% written in Bourne shell, 2% written in ksh, and 1% written in Bash. Please remember that I'm talking about Solaris here, and not Linux. When in Rome, do as the Romans do.

Because I create a LOT of shell scripts, I'm also painfully aware of Bourne shell's limitations. Not the least of which is its anemic data structure capability. For that reason, I started to use the Korn shell, or ksh. I have used ksh on Solaris for about 10 years both as my interactive shell and as my programming shell. It's still painful for data structures when compared to Perl, but it's MUCH nicer to work with than the Bourne Subset.

In addition to my Solaris career, I have a life in a parallel universe as a Linux engineer, volunteering at a local not-for-profit organization. There we use Linux for servers because Solaris was too expensive when we chose the initial environment. In the Linux world, the Bourne Again Shell, or Bash, is the de-facto standard for good reason. It's a very capable shell with great programming constructs, and a pleasant demeanor for interactive shell use. I dig it.

In fact, I dug it so much that I recently changed my Solaris interactive shell to it. Bash has been included in Solaris for quite a while now, and I consider it a standard-enough component that it's safe to learn. I love the file completion and history browsing with tab and arrow keys. It's so much more intuitive than the equivalent in ksh. I've been happily bashing around for a few months on Solaris, but recently did an about face in my thinking while reading up on Role Based Access Control, or RBAC.

My current site has a sudo to RBAC conversion on the roadmap, which should be a great project. In the RBAC world, you assume a role to complete an activity that is outside your default privileges. Those roles will have one of three shells: pfcsh, pfsh, or pfksh. Did you notice that Bash is not amongst them? Not for the foreseeable future.

So, in a Solaris-standard world that leverages the powerful RBAC facility, you will not have the option of working in a bash compatible shell if you want to perform an activity outside your normal rights. That's enough reason for me to drop the bells and whistles of Bash and go back to ksh. I'll probably miss it for a few weeks before I forget completely.

This experience proved to be yet another reminder for me that staying on the path of standards is full of temptations. While Bash is stable and common, it is not a native Solaris standard. That's fine for an end-user or software developer that operates outside the OS internals, but as a systems engineer my job is to live in the internals, and at this point in time Korn shell seems to give me the best functionality, programming, and standards compliance.

Revision control bites... Unless you train it!

I've recently begun the task of placing my world under revision control. Keeping track of edits, versions, builds, etc. had become too much of a chore considering that Solaris has a facility to do it for me. The Source Code Control System, or SCCS, is the Solaris standard for revision control in just about every release I'm familiar with. There are many other revision control system out there, each with pros and cons. Popular alternatives include RCS, CVS, and Subversion. In my case, I want to choose a system that I know will be available in all reelases of Solaris on all sites, even in galaxies far, far away. SCCS is the only one that meets my key criteria.

Having chosen my platform for version control, I begain moving projects into its protective custody. All went well for the first few weeks, and I began to develop the the comfort level that usually preceeds a problem. The most recent project I put under SCCS control is a package with a simple preinstall script. The script is responsible for checking for the existance of a file prior to package installation, and making a backup before overwriting it. The backup is named FILENAME.PACKAGENAME.DATESTAMP. To implement this datestamp I set a variable in a backup subroutine as follows:

backup_file () {
DATESTAMP=`date +%Y%m%d%H%M`
test -f "${1}" && /usr/bin/cp ${1} ${1}.CGHfoopkg.${DATESTAMP}
return ${?}
} #end backup_file


After testing successfully on my development machine, I checked the file back in (sccs delget) and rebuilt the package. When I transferred the package to my staging server, the preinstall failed miserably with strange syntax originating at lines two and three of the above segment.

The first thing I questioned was whether there is some difference in versions of the time command between my development server and staging server, but it only took a second to prove that theory null and void.

After some mucking around, I discovered that the problem only exists when the source code was checked in to SCCS. This quickly lead me to realize that SCCS was expanding keywords in my DATESTAMP variable.

SCCS keywords allow you to have SCCS dynamically insert a revision, filename, check-in time, and other metadata when a file is checked in. By placing this into my checked out code:
#
# SCCS Revision Control:
# %M% %I% %H% %T%
#


I end up with this when the code is checked-in:
#
# SCCS Revision Control:
# preinstall 1.5 09/21/06 09:01:46
#


The problem is, SCCS by default does not stop at the header. It goes all the way through the code. In this case, it was hitting my DATESTAMP variable and changing its value, thus breaking my backup_file function.

It took some digging to find the solution, but I finally discovered the sccs admin command. Using this command I was able to specify that only the first ten lines of the file should be considered for keyword expansion using the following command:
# sccs admin -fs10 preinstal


After making this change, and checking the file back in, my problems were gone. The source stayed clean, and I was once again blissfully coding. My newly trained SCCS repository has been behaving wonderfully ever since.

To install links, or not to install links... That is the Question!

I'm working on a package at the moment that integrates Oracle 9i / 10g startups with Solaris Resource Manager (SRM). One of its functions is providing init scripts to start and stop the database and listeners during reboots. Seems simple at first, but I ran into an interesting dilemma.

At my current site, Oracle is not an automated installation, and not part of our Jumpstart framework. The challenge this presents is that we do not want to install active run control links for software that is not yet installed on the system, but we want to be able to install these packages during the jumpstart.

Ideally, I'd like to see the rc scripts registered via pkgadd so they can be easily identified, and cleanly removed if the package is de-installed. In the end, I had to compromise, but I think it turned out pretty safe because I followed standards in use of those rc / init scripts and used a hard link.

Although it may sound odd at first glance, I chose not to include the rc links in my package prototype. The links are installed in a disabled mode (pre-pended underscore character) by a postinstall script. I chose this route because it creates a placeholder for the startup order I selected rather than hoping whoever does the installation gets it right. This eliminates inconsistency and human error.

Next, I created a preremove script. The preremove works by issuing a find command on the /etc/rc?.d scripts that looks for any files with an inode matching the init script's inode. If proper convention for using hard-links was followed, this method will find the associated rc scripts even if their startup order or link name are changed over the life of the system.

LINKS=`find /etc/rc?.d -inum ${INITSCRIPTINODE} -print`

The final touch was to create a simple script that can activate all the sym-links at once. It leverages the preremove's inode hunting strategy and finds entries that have an initial underscore, and renames them using a simple Sed expression:

NEWLINKNAME=`echo $LINKNAME | sed -e 's/^_//'`

I'll be adding a reverse-function for the enabler script that can disable as well, and this will live in our NFS-accessible admin scripts repository for convenience. No need to distribute software that isn't critical to operation when the network is down.

The only downfall to my solution is not being able to search for the rc links in the Solaris software registry (/var/sadm/install/contents). This would have allowed someone unfamiliar with the solution to identify its components. The problem with registering RC links is that they change a lot over time and it's very difficult to keep the registry current. A link may change its name, start order, may be active, or inactive.

The effort required to manage that kind of a dynamic file would have been almost as large as the project I was packaging, so I decided to keep my scope tight and assume that as an acceptable risk. When we move from Solaris 9 to Solaris 10 this problem will be eliminated by integration with SMF, so no point in getting wrapped around the axles.

Auditing challenged with SSH

I recently had a real dilemma thrown at me by the security team at a site I work on. The site's policy dictates that no authentication should take place without a password. Any exceptions require both a business case justification on file, and an expiration. This presents a real challenge with SSH. SSH allows public key authentication as an alternative to passwords, and the private key can be created without a passphrase. In addition, there is no way to enforce a key expiration at a server level (at least, not that I've been able to find).

SSH supports two primary means of authentication. Password authentication is essentially the same process that occurs through Telnet, except we use a secure tunnel instead. Public Key authentication bypasses the system's password channel completely. At first glance, it's easy to say, "so disable public key authentication and be done with it."

There are two benefits to using public key authentication with SSH. First, you have the ability to confiigure your private key (residing on the client) without a passphrase. In doing so, you increase the risk of private key compromise, but enable passwordless authentication. This makes batch jobs much easier because it elminiates an interactive session.

The second benefit is that public key authentication is a two-factored authentication. It combines something you have, a private key, and something you know: a passphrase. As such, it's much more secure than traditional password based authentication which onyl uses single factor authentication. Even if someone compromises your private key file, it can't be used unless they know the passphrase, which SHOULD be a complex phrase.

The problem is that there is no way to control who on a server is authorized to use public key encryption, no way to enforce passphrase complexity, and no way to expire a public key. I could create a public / private key pair on a machine under my desk that isn't subject to production data center security scans and audits. Let's next say that I decided not to use a passphrase, or that the passphrase was weak and easily guessed. Next, assume I'm the DBA for the company's Oracle Fincancials encironment. It would require very little effort for someone to compromise my under-desk system and gain passwordless access to the company's critical systems. This is a low-risk, high payoff scenario that people who know what they're doing would be likely to attempt.

Another issue with the public key encryption capabilities of SSH is that you can enter any passphrase (or no passphrase). With my system password, I can install a crack module into the Pluggable Authentication Module (PAM) stack and enforce very complex passwords. Unfortunately, because key pairs are generated on clients, there is no way to enforce sanity at the server level.

After some research I determined that the version of Sun SSH which ships with Solaris 9 is far less capable than the OpenSSH releases one could obtain and build. Using OpenSSH it is possible to move the default $HOME/.ssh to another base directory (like /var). In doing so, it is much easier to create a root controlled environment where someone can not use the authorized_keys file unless authroized by a superuser. Unfortunately, the maintenance issues created by doing this are not justified, and our policy is to stick with vedor provided and supported software in all but the most extenuating circumstances. Under Solaris 9, there's just no safe and auditable way to allow publick key authentication. In an ideal world, I should be able to not just configure public key authentication at a server leve, but specify which keys were respected by the server's SSH daemon.

There is no solution I can see to the issue of enforcing passphrase complexity, or auditing use of non-interactive key pairs because that part of the process is handled entirely by the client. It's very difficult to convnice a security officer that key generated on an uncontrolled device can be trusted for authentication against Sarbox servers.

Our decision, much to my dismay, was to disable public key authentication site-wide. I feel like we're throwing the baby out with the bath water, but at the same time I understand the need to audit system access, and be able to enforce policy. I'm anxious for our Solaris 9 fleet to turn over to Solaris 10 so we can begin using the more capable version of SSH it includes.

AppleCare not up to Sun Service

One of the big reasons I was so excited to move my desktop environment to Mac OS X was its underlying UNIX operating environment. Being a UNIX guy, I'm well aware of how well instrumented it is, and how surgically it can (often) be debugged. This, of course is in contrast to the Windows world where it has become not only common, but almost accepted that troubleshooting step #1 is to reboot and sacrifice a chicken.

Over time my G5 was starting to crash with increasing frequency. At first it was once in a rare while, although I was still surprised that it happened. Recently it accelerated to the point where it crashed almost once per day. Given how much I paid to have rock solid hardware, and AppleCare behind it, this was not acceptable to me. So, I finally grew tired of my Mac's grey text-box of death and called my friendly AppleCare representative.

I began by telling my story, and adding a detail I felt was critical. Each time the system crashes I generated a crash dump report, and the stack trace always pointed back to the USB driver. I grabbed the text below from another site as an example, but it's very similar to what I was seeing.


1 Unresolved kernel trap(cpu 0): 0x300 - Data access DAR=0xdeadbeef PC=0x0e692550
2 Latest crash info for cpu 0:
3 Exception state (sv=0x0EB5DA00)
4 PC=0x0E692550; MSR=0x00009030; DAR=0xDEADBEEF; DSISR=0x42000000; LR=0x0E692530;
R1=0x081DBC20; XCP=0x0000000C (0x300 - Data access)
5 Backtrace:
6 0x0E6924A8 0x00213A88 0x00213884 0x002141D4 0x00214830
0x00204CB0 0x00204C74
7 Kernel loadable modules in backtrace (with dependencies):
8 com.apple.dts.driver.PanicDriver(1.0)@0xe691000
9 dependency: com.apple.iokit.IOUSBFamily(1.9.2)@0xed9c000
10 Proceeding back via exception chain:
11 Exception state (sv=0x0EB5DA00)
12 previously dumped as "Latest" state. skipping...
13 Exception state (sv=0x0EB64A00)
14 PC=0x00000000; MSR=0x0000D030; DAR=0x00000000;
DSISR=0x00000000; LR=0x00000000; R1=0x00000000; XCP=0x00000000 (Unknown)


Note that USB line? I was seeing it in EVERY crash. This tends to be something worth investigating. In my case I have a USB card reader attached to the USB ports in the bottom of my CinemaDisplay, and a Palm Pilot USB cable plugged into the front of my case. Another observation I made was that each time my system crashed it was essentially idle. It usually happened at night, or while I was at work. I would return to the sound of a jet engine coming from that silver box.

My first suggestion was that we look at the panic logs and try to identify the faulty components, but this didn't get too much traction. AppleCare is set up so that if the basic rubber stamp checks (slightly better than a reboot, but not by much) fail, they redirect you to a local store. In my case this wasn't appealing. It's a 40 minute drive from here, and the issue is intermittant. I could end up being without my Mac for more than a week if things went well.

So, we went through and erased all the caches and preferences, then reset the NVRAM. My system was brought back to factory specs, although I really did almost nothing abnormal to it. I don't use funky extensions or other hacks; I use mainstream well supported stuff.

Much to my surprise, the system has been stable since the activities. I'll be the first to eat my words, but I'm not used to voodoo troubleshooting. This was like chemo-therapy where we just bombard the system in hopes of getting all the cancerous code. I'm used to working in a surgical environment where we see CPU 0 corrupting data on an interval that indicates it needs replacing.

As much as I'm a hopeless fan of Solaris, I have to say that I don't think it's a huge quality difference between Sun and Apple that gives me ths uneasy feeling about this experience; I think it's the quality of Sun Service. They are used to dealing with mission critical servers more than art-critical desktops. No offense to the Mac world - I'm one of you... But it's a very different world.

Initology 101: A lesson in proper use of Solaris run control scripts

Starting and stopping applications through init scripts ought to be a simple thing that doesn't cause much debate, but in fact its just the opposite. I routinely see servers with functional but non-standard artifacts nested in the rc directories. I also hear many justifications for these configurations; some reasonable, others somewhat less so. But in the end, I believe that a systems engineering approach to using init scripts will filter the options, and this article intends to do just that.

There are three specific conventions that I want to address:

1. Which run levels should be used for starting and stopping typical applications.
2. Should a symbolic link (sym-link) or hard-link be used?
3. How should a link be disabled

Let's begin with identifying the correct run levels to start and stop a common application. By common application I mean something that is not a core part of the operating system, but rather in the application layer that depends on the operating environment's core features. Oracle and web servers are common examples of what I consider common applications. Knowing that the Solaris Operating Environment has well defined run level states, the first step is to consult the docs.sun.com web site for your particular Solaris version and refer to those definitions. Let's take the case of Solaris 9 (9/05) which is that last release in the Solaris 9 series. I am not going to address Solaris 10 in this context because it uses the new Service Management Facility as part of the new Predictive Self Healing feature to replace init scripts.

According to the Solaris 9 (9/04) System Administration Guide: Basic Administration Section 8: Run Levels and Boot Files We have the following run levels and explanations:

Run Level	Description
0	Shut down all processes and power down to ok> prompt (sparc).
S	Run as a single user with some file systems mounted and accessible.
1	Administrative state with access to all file systems, but no user logins permitted
2	Multi-user state. For normal operations. Multiple users can access the system and all file system. All daemons are running except for the NFS server daemons.
3	Multi-user state: For normal operations with NFS resources shared. This is the default run level for the Solaris environment.
4	Alternative MU state. This is not used by Solaris, but is available for site customization if needed. I recommend NOT using it.
5	Power down after shutting down all processes.
6	Reboot the system.

In theory, we need to consider that a system may transition from any run level to any other run level. This means that when the system enters run level S, if our application is running, we need to ensure it is stopped. The same thing goes for 0, 1, and 2. Run level three is the conventional system state associated with end user applications being loaded. Putting this into practice, we will need to install the following links to fully integrate with Solaris' run levels:

/etc/rc0.d/K##fooapp
/etc/rc1.d/K##fooapp
/etc/rc2.d/K##fooapp
/etc/rc3.d/S##fooapp
/etc/rcS.d/K##fooapp


These will ensure that our application is started in run level 3, and stopped in any other run level. This contrasts with what I see in most data centers where rc scripts are installed to run level 2 or 3 for start up, and 0 for shut down. While this approach can work for reboots it has a down fall. How many times have you been told that before patching you need to reboot a server into single user mode? This is because kill scripts are not installed for all applications for all run level transitions. I still advocate rebooting into single user mode to be safe, but in a perfect world this would not be necessary.

Having selected the run control directories, you are now ready to put the links in place. But wait! You have another decision to make. Should you use a symbolic link or a hard link? There are all kinds of reasons for and against either method if you approach the question from an emotional standpoint. However, as a Solaris Jedi, you do not allow your emotions to control you. You look for standards.

Referring again to the docs.sun.com web site, we return to the Solaris 9 (9/04) System Administration Guide: Basic Administration. This time, to Section 8, How to Add a Run Control Script. The examples on the page clearly show how to use the ln command to create a hard link. This is where the discussion should end. You didn't write Solaris, and you didn't do the integration testing. You are disciplined, and you follow standards; This is the way of the Jedi.

I have heard numerous arguments for using sym-links in place of hard links, and I believe each of them stems from not fully understanding how UNIX file system inodes work, and how Solaris commands can be used to understand them. Using the "ls -i" command you can prove that the files reference the same inode, and are thus the same.

cgh@soleil{/etc/rc0.d}# ls -li /etc/rc3.d/S90samba
9731 -rwxr--r-- 6 root sys 324 Jan 14 2006 /etc/rc3.d/S90samba*
cgh@soleil{/etc/rc0.d}# ls -li /etc/init.d/samba
9731 -rwxr--r-- 6 root sys 324 Jan 14 2006 /etc/init.d/samba*
cgh@soleil{/etc/rc0.d}#


Notice the first field in each record shows the integer, 9731? That is the inode number. The next field to attend to is the third. In this case, a "6" for each record. This refers to the link count, or number of links that point to the same piece of data.

Another approach to observing all rc links associated with an init script is to use the find command to search a branch of the file system for the inode number matching the init script. Let's look at the standard Samba service included with Solaris 10. We know from the prior example that inode #9731 references the samba script. The following command will seek out all of the hard links:

cgh@soleil{/etc/rc0.d}# find /etc/rc?.d -inum 9731
/etc/init.d/samba
/etc/rc0.d/K03samba
/etc/rc1.d/K03samba
/etc/rc2.d/K03samba
/etc/rc3.d/S90samba
/etc/rcS.d/K03samba
cgh@soleil{/etc/rc0.d}#


If these link were symbolic the task would not be as simple, and we would not have the benefit of a link counter to ensure the integrity of our boots.

The last facet of initology I want to discuss is proper convention for disabling an init script on a Solaris server. As with the above examples, the correct process comes right out of the Basic Administration Guide, Section 8. The init scripts only process files that begin with an "S" or a "K". I most often see the upper-case letter replaced with lower case. The number two method I've observed is to remove the links altogether, leaving (hopefully) the init script in place.

The correct process for disabling an init script is almost always to prepend an underscore. The underscore stands out clearly in the list while lower cases characters tend to have less contrast next to the upper case entries. It sounds trivial, but how goood is your eye sight at 3am after your pager goes off? Another benefit is the grouping of all disabled scripts in the directory listing so you can tell at a glance what is turned off. Finally, by not removing it altogether we can preserve the ordering of the scripts, which is some cases is critical. Take a look at the example below, and hopefully my suggestions will be apparent:

cgh@soleil{/etc/rc3.d}# ls -l
total 44
-rw-r--r-- 1 root sys 1285 Jan 21 2005 README
-rwxr--r-- 6 root sys 474 Jan 21 2005 S16boot.server*
-rwxr--r-- 6 root sys 1649 Jan 8 2005 S50apache*
-rwxr--r-- 6 root sys 5840 Jan 29 2004 S52imq*
-rwxr-xr-x 1 root sys 491 Apr 10 12:49 S75seaport*
-rwxr--r-- 6 root sys 685 Jan 21 2005 S76snmpdx*
-rwxr--r-- 6 root sys 1125 Jan 21 2005 S77dmi*
-rwxr--r-- 6 root sys 344 Jan 21 2005 S80mipagent*
-rwxr--r-- 6 root sys 513 May 15 19:21 S81volmgt*
-rwxr-xr-x 5 root sys 2225 Apr 10 12:49 S82initsma*
-rwxr--r-- 5 root sys 824 May 26 2004 S84appserv*
-rwxr--r-- 6 root sys 324 Jan 14 2006 S90samba*
-rw-r--r-- 1 root root 0 Aug 31 21:31 _S92foodb
-rw-r--r-- 1 root root 0 Aug 31 21:31 _S95fooapp


Henceforth, you will properly integrate your scripts with the entire run level facility using hard links. When those magical links need to be disabled you will prepend underscores to them. You are now a master of the Solaris init scripts, and ready to carry this knowledge to others. You are also ready to explore the Solaris 10 SMF and enjoy all that it has to offer.

Spotlight on Richard McDougall

If you haven't yet visited Richard McDougall's Blog you should fire up your browser and head over. I had the pleasure of meeting Richard at a SunUP Network Conference in Singapore where we were both giving presentations. We met up again at later conferences in Sydney Australia and Boston in the same scenario, and he hit it out of the park every time he got in front of customers. It's very rare to meet someone as brilliant as Richard who is also so down to Earth and generous with his knowledge; he is a true Jedi Master in the land of Solaris.

One characteristic you observe early in Richard's presentations is his enthusiasm for Solaris and its potential. This article on Chip Multi-Threading is a classic example of his style, and was what inspired me to write this entry. I remember him speaking about some of Sun Volume Manager's (SVM) new (at the time) features which were specifically designed to address the reasons customers had chosen Veritas Volume Manager. Rather than attacking the message with the technical nuts and bolts, he hit on a few topics and delivered the message that Sun was listening. I am certain that more people re-examined SVM after his delivery than any speeds and feeds preso would have motivated. A true Jedi master delivers important messages without patronizing through understanding the intended recipients.

The other item I want to draw your attention to is his new set of books: Solaris Internals and Solaris(TM) Performance and Tools: DTrace and MDB Techniques for Solaris 10 and OpenSolaris (Hardcover) which were just delivered to me from Amazon. First of all, I hate poorly bound books. I buy books to use as reference manuals - tools of my trade. These books feel like professional tools that you will appreciate returning to. Remember buying that Calculus book in college? The one that weighs 25 lbs? This is that book. I love it! They are expensive, but good books aren't cheap, and the investment the author made in sharing his skills isn't cheap either. The first edition of Solaris Internals is well known as the authoritative reference on Solaris plumbing, and with all of the exciting changes Solaris 10 brings, this book is timely. I'm anxious to dig into it and post a review, but in the mean time please check the books out. This page has more information, and sample content.

I'd like to start paying tribute to some of the Jedi Masters I've benefited from, and this post serves as the first. Please take a moment to read Richard's Blog. Check back frequently - if he posts it, you should know about it. And if you need a diversion from Solaris, he's also a great photographer.