Initology 101: A lesson in proper use of Solaris run control scripts

Starting and stopping applications through init scripts ought to be a simple thing that doesn't cause much debate, but in fact its just the opposite. I routinely see servers with functional but non-standard artifacts nested in the rc directories. I also hear many justifications for these configurations; some reasonable, others somewhat less so. But in the end, I believe that a systems engineering approach to using init scripts will filter the options, and this article intends to do just that.

There are three specific conventions that I want to address:

1. Which run levels should be used for starting and stopping typical applications.
2. Should a symbolic link (sym-link) or hard-link be used?
3. How should a link be disabled

Let's begin with identifying the correct run levels to start and stop a common application. By common application I mean something that is not a core part of the operating system, but rather in the application layer that depends on the operating environment's core features. Oracle and web servers are common examples of what I consider common applications. Knowing that the Solaris Operating Environment has well defined run level states, the first step is to consult the docs.sun.com web site for your particular Solaris version and refer to those definitions. Let's take the case of Solaris 9 (9/05) which is that last release in the Solaris 9 series. I am not going to address Solaris 10 in this context because it uses the new Service Management Facility as part of the new Predictive Self Healing feature to replace init scripts.

According to the Solaris 9 (9/04) System Administration Guide: Basic Administration Section 8: Run Levels and Boot Files We have the following run levels and explanations:

Run Level	Description
0	Shut down all processes and power down to ok> prompt (sparc).
S	Run as a single user with some file systems mounted and accessible.
1	Administrative state with access to all file systems, but no user logins permitted
2	Multi-user state. For normal operations. Multiple users can access the system and all file system. All daemons are running except for the NFS server daemons.
3	Multi-user state: For normal operations with NFS resources shared. This is the default run level for the Solaris environment.
4	Alternative MU state. This is not used by Solaris, but is available for site customization if needed. I recommend NOT using it.
5	Power down after shutting down all processes.
6	Reboot the system.

In theory, we need to consider that a system may transition from any run level to any other run level. This means that when the system enters run level S, if our application is running, we need to ensure it is stopped. The same thing goes for 0, 1, and 2. Run level three is the conventional system state associated with end user applications being loaded. Putting this into practice, we will need to install the following links to fully integrate with Solaris' run levels:

/etc/rc0.d/K##fooapp
/etc/rc1.d/K##fooapp
/etc/rc2.d/K##fooapp
/etc/rc3.d/S##fooapp
/etc/rcS.d/K##fooapp


These will ensure that our application is started in run level 3, and stopped in any other run level. This contrasts with what I see in most data centers where rc scripts are installed to run level 2 or 3 for start up, and 0 for shut down. While this approach can work for reboots it has a down fall. How many times have you been told that before patching you need to reboot a server into single user mode? This is because kill scripts are not installed for all applications for all run level transitions. I still advocate rebooting into single user mode to be safe, but in a perfect world this would not be necessary.

Having selected the run control directories, you are now ready to put the links in place. But wait! You have another decision to make. Should you use a symbolic link or a hard link? There are all kinds of reasons for and against either method if you approach the question from an emotional standpoint. However, as a Solaris Jedi, you do not allow your emotions to control you. You look for standards.

Referring again to the docs.sun.com web site, we return to the Solaris 9 (9/04) System Administration Guide: Basic Administration. This time, to Section 8, How to Add a Run Control Script. The examples on the page clearly show how to use the ln command to create a hard link. This is where the discussion should end. You didn't write Solaris, and you didn't do the integration testing. You are disciplined, and you follow standards; This is the way of the Jedi.

I have heard numerous arguments for using sym-links in place of hard links, and I believe each of them stems from not fully understanding how UNIX file system inodes work, and how Solaris commands can be used to understand them. Using the "ls -i" command you can prove that the files reference the same inode, and are thus the same.

cgh@soleil{/etc/rc0.d}# ls -li /etc/rc3.d/S90samba
9731 -rwxr--r-- 6 root sys 324 Jan 14 2006 /etc/rc3.d/S90samba*
cgh@soleil{/etc/rc0.d}# ls -li /etc/init.d/samba
9731 -rwxr--r-- 6 root sys 324 Jan 14 2006 /etc/init.d/samba*
cgh@soleil{/etc/rc0.d}#


Notice the first field in each record shows the integer, 9731? That is the inode number. The next field to attend to is the third. In this case, a "6" for each record. This refers to the link count, or number of links that point to the same piece of data.

Another approach to observing all rc links associated with an init script is to use the find command to search a branch of the file system for the inode number matching the init script. Let's look at the standard Samba service included with Solaris 10. We know from the prior example that inode #9731 references the samba script. The following command will seek out all of the hard links:

cgh@soleil{/etc/rc0.d}# find /etc/rc?.d -inum 9731
/etc/init.d/samba
/etc/rc0.d/K03samba
/etc/rc1.d/K03samba
/etc/rc2.d/K03samba
/etc/rc3.d/S90samba
/etc/rcS.d/K03samba
cgh@soleil{/etc/rc0.d}#


If these link were symbolic the task would not be as simple, and we would not have the benefit of a link counter to ensure the integrity of our boots.

The last facet of initology I want to discuss is proper convention for disabling an init script on a Solaris server. As with the above examples, the correct process comes right out of the Basic Administration Guide, Section 8. The init scripts only process files that begin with an "S" or a "K". I most often see the upper-case letter replaced with lower case. The number two method I've observed is to remove the links altogether, leaving (hopefully) the init script in place.

The correct process for disabling an init script is almost always to prepend an underscore. The underscore stands out clearly in the list while lower cases characters tend to have less contrast next to the upper case entries. It sounds trivial, but how goood is your eye sight at 3am after your pager goes off? Another benefit is the grouping of all disabled scripts in the directory listing so you can tell at a glance what is turned off. Finally, by not removing it altogether we can preserve the ordering of the scripts, which is some cases is critical. Take a look at the example below, and hopefully my suggestions will be apparent:

cgh@soleil{/etc/rc3.d}# ls -l
total 44
-rw-r--r-- 1 root sys 1285 Jan 21 2005 README
-rwxr--r-- 6 root sys 474 Jan 21 2005 S16boot.server*
-rwxr--r-- 6 root sys 1649 Jan 8 2005 S50apache*
-rwxr--r-- 6 root sys 5840 Jan 29 2004 S52imq*
-rwxr-xr-x 1 root sys 491 Apr 10 12:49 S75seaport*
-rwxr--r-- 6 root sys 685 Jan 21 2005 S76snmpdx*
-rwxr--r-- 6 root sys 1125 Jan 21 2005 S77dmi*
-rwxr--r-- 6 root sys 344 Jan 21 2005 S80mipagent*
-rwxr--r-- 6 root sys 513 May 15 19:21 S81volmgt*
-rwxr-xr-x 5 root sys 2225 Apr 10 12:49 S82initsma*
-rwxr--r-- 5 root sys 824 May 26 2004 S84appserv*
-rwxr--r-- 6 root sys 324 Jan 14 2006 S90samba*
-rw-r--r-- 1 root root 0 Aug 31 21:31 _S92foodb
-rw-r--r-- 1 root root 0 Aug 31 21:31 _S95fooapp


Henceforth, you will properly integrate your scripts with the entire run level facility using hard links. When those magical links need to be disabled you will prepend underscores to them. You are now a master of the Solaris init scripts, and ready to carry this knowledge to others. You are also ready to explore the Solaris 10 SMF and enjoy all that it has to offer.

Spotlight on Richard McDougall

If you haven't yet visited Richard McDougall's Blog you should fire up your browser and head over. I had the pleasure of meeting Richard at a SunUP Network Conference in Singapore where we were both giving presentations. We met up again at later conferences in Sydney Australia and Boston in the same scenario, and he hit it out of the park every time he got in front of customers. It's very rare to meet someone as brilliant as Richard who is also so down to Earth and generous with his knowledge; he is a true Jedi Master in the land of Solaris.

One characteristic you observe early in Richard's presentations is his enthusiasm for Solaris and its potential. This article on Chip Multi-Threading is a classic example of his style, and was what inspired me to write this entry. I remember him speaking about some of Sun Volume Manager's (SVM) new (at the time) features which were specifically designed to address the reasons customers had chosen Veritas Volume Manager. Rather than attacking the message with the technical nuts and bolts, he hit on a few topics and delivered the message that Sun was listening. I am certain that more people re-examined SVM after his delivery than any speeds and feeds preso would have motivated. A true Jedi master delivers important messages without patronizing through understanding the intended recipients.

The other item I want to draw your attention to is his new set of books: Solaris Internals and Solaris(TM) Performance and Tools: DTrace and MDB Techniques for Solaris 10 and OpenSolaris (Hardcover) which were just delivered to me from Amazon. First of all, I hate poorly bound books. I buy books to use as reference manuals - tools of my trade. These books feel like professional tools that you will appreciate returning to. Remember buying that Calculus book in college? The one that weighs 25 lbs? This is that book. I love it! They are expensive, but good books aren't cheap, and the investment the author made in sharing his skills isn't cheap either. The first edition of Solaris Internals is well known as the authoritative reference on Solaris plumbing, and with all of the exciting changes Solaris 10 brings, this book is timely. I'm anxious to dig into it and post a review, but in the mean time please check the books out. This page has more information, and sample content.

I'd like to start paying tribute to some of the Jedi Masters I've benefited from, and this post serves as the first. Please take a moment to read Richard's Blog. Check back frequently - if he posts it, you should know about it. And if you need a diversion from Solaris, he's also a great photographer.

The verdict: Ubuntu Linux is a keeper

I'm writing now from the keyboard of my reborn laptop. Having just completed the installation and configuration of Ubuntu Linux on it, and happily retired its installation of Windows XP. Since this blog is really about Sun Solaris and Systems Engineering I don't want to spend too much time talking about this from a technical standpoint. It does have relevance to the theme as we all need a portable means of working on Solaris systems. If you use UNIX as your primary operating environment, you know how awkward it is to depend on Windows as your interface to the systems you support.

So far, Unbuntu "just works" with no headaches at all; It auto-detects and configures my Netgear WG511 "G" Network Card, and can successfully enter and exit both hibernate and suspend modes. These were the two big headaches for me under Fedora Core. I am really impressed that the special volume and mute keys worked as well. Those used to require installing a separate Thinkpad buttons package called tpb. The boot screens look slick, the theme is very clean and coherant, and the desktop is clean and EMPTY. I love that! I give it two thumbs up. It looks like I'm finally going to learn the Debian flavor Linux after years of being a die-hard Red-Hat camper.

Now let me clarify this position; I'm not changing my opinion about Mac OS being the ultimate desktop. But, I can obtain an old IBM Thinkpad T20 for a fraction of the cost of a PowerBook or MacBook. I wouldn't want to process my photographs on it, but for a tool that lets me perform systems work, and use typical Office Software, I'm very happy.

Microsoft's Genuine Advantage

I have an older IBM Thinkpad T23 laptop which I purchased after sending most of my scrapyard through eBay. I bought it with the intention of running 90% Linux, and occasionally using Windows when I need some odd utility, or have to connect to something that only speaks Windows. The T23 is a rock solid machine, and being far from the bleeding edge, also has pretty decent hardware compatibility. With 1GB of RAM and a 1GHz CPU, the machine is plenty fast for it's intended role as a terminal web browser, email client, and occasional OpenOffice platform. I bet the most used application on it was gnome-terminal if I really analyzed the accounting records; Nothing stressful.

When I first loaded Fedora Linux it was a simple process to get the machine useable. Useable and optimial turned out to be divided by a full-strength, adult size, bang-a-roo of a headache. The little things like getting it to play MP3s didn't phase me too much. In fact, taken one by one the entire list isn't anything that can't be handled. The problem is that I'm tired of having to handle things. I just want my computers to let me do what I want without HAVING to hack. I'd rather hack by choice than for base survival.

I was able to get Wireless ethernet working after some digging, but what really sent me over the edge was ACPI. My power consumption was awful, and getting it to be even close to Windows proved as complex as tuning 100 Oracle instances fighting for the resources of a SparcStation 5. Not fun at all. Eventually, I decided that despite my inability to mentally mesh with Window's gears I would dump Linux and stick to the main stream.

I bought a copy of Windows XP Pro off eBay, complete with hologram media, funky sticker, and all of those gimmicky little things they do to make ou think you're getting something official and important. I downloaded all the updates, I filled out the registration, I did all the things someone would do when they are an IT professional who wants to be legitimate. After using it for about a year with no issues, including the "Windows Genuine Advantage" thingy which used to think I has a legitimate copy.

Today, after a long hiatus, my laptop was booted and it informed me that Windows Genuine Advantage had changed its mind. Warning boxes were popping up left and right, and graciously giving me the opportunity to "purchase genuine Windows". You know what? I already did. It was shrink wrapped, and had so many gimmicky little security things that it was gaudy. And now you want to give me an opportunity to do it again? No thanks. From a quick Google search it looks like I'm not the first person to be annoyed.

I did notice that my system clock was really goofed up, and I've heard that the validation process involves hardware checks, so maybe something in my configuration triggered it. I don't know, but frankly I don't care. I don't want to know why it happened. I'm going back to a world that doesn't include helpful paper clips and other rediculous instantiations of a help system.

Since my battery is nearly dead, I've decided not to worry about ACPI. Windows is being scrapped tonight and I'm going to either run Ubuntu or Fedora linux. I'm not crazy about Solaris on the desktop because it has less standard productivity software, and the updates for non-Solaris software are not convenient. I'm the #1 fan for servers, but on the desktop I'm a Linux guy until I can afford a PowerBook or MacBook.

I'll post more about the final choice I make, but I felt it necessary to document this eve of liberation for all to see. And now I must end this post as I have a date with fdisk to catch...

IM Ruining Grammar?

This one is a bit off theme, but I couldn't resist. Apparently it has been found that the prolonged use of IM does not truly impair one's grammatical ability. Thank the University of Toronto for this piece of knowledge...

Are you serious? It's not grammar that gets hurt when IM is abused. It's one's social skills. The article I mention above is really talking about kids who get carried away, but living in a cube farm, I've seen the adult version as well. At some point, we've all been guilty of IM'ing someone close enough to hit with a paper airplane.

Between email, voicemail, wikis, and IM, not to mention remote work, just about everything is driving a wedge between developing the personal relationships that foster good working environments. When I've met someone in person I immediately feel more at ease trusting them for the role they will play in a project.

Body language plays a HUGE role in our ability to communicate effectively, and to dismiss it for the "efficiency" of electronic communication is naive at best. The next time you need to talk to someone, walk to the other side of the building, or schedule the time to drive to their site. You'll be glad you did, and they probably will too.

The demise of Linux

I read a quick editorial which suggests that Ubuntu Linux is going to be the downfall of Red Hat. The premise being that as Red Hat grew more commercial, and abandoned its community in favor of its stockholders, the sys-admins who used Red Hat as their desktop OS drifted away from the Red Hat camp. As Ubuntu came into being, and did so in a very strong way, those SA-types who were driven away from Red Hat will now want to put what they are more familiar with (Ubuntu) on their servers when they have the choice.

This whole discussion brought me right back down memory lane. I started using Linux before Red Hat existed with a few early Slackware distributions. I remember writing all those 3" diskettes - somewhere around 80 of them by the time I burned the X-windows distribution as well. Shortly after came Red Hat, and at that time I was helping to set up the first Linux environment at SUNY Plattsburgh. We switched over from Slackware to Red Hat, and loved it.

I ended up sticking with Red Hat for the next 10 years. It was the OS of choice when I lead a project to build servers for our local Boy Scouts of America council, and remained there until Red Hat went totally commercial, burning the bridges out from under us. Make no mistake, I was extremely disappointed with their decision. We switched over to Fedora Core, and for the most part it has been a smooth transition. Despite its big red "DEVELOPMENT" stamp, Fedora has been very good to our availability. In fact, at the moment we've got an impressive uptime on an FC2 system:

# uptime
21:41:15 up 412 days, 2:37, 2 users, load average: 0.00, 0.00, 0.00


But thinking back on the experience I have an entirely different memory of what direction I was forced in, and where I wanted to be. The places I've worked would not consider using Linux for their production environments. Linux has made some great in roads t the corporate world, but there aren't a lot of Fortune 500 companies running their SAP central instance on Linux. I'm sorry, it's just not happening. HPUX, AIX, and Solaris are king in the land of mission-critical highly scalable UNIX servers.

Although Solaris never made a particularly compelling desktop, it's what I've wanted to use on every server I've ever built. It's rock-solid, well documented, and very cohesive. When you use Solaris and Sun products, you rarely get the impression that 10,000 individual developers all tried to do it "their way" when the final build was cast in stone. What always stopped me was cost. Solaris x86 had pathetic support and commitment in the past. It's so incredibly painful to migrate between operating environments that I never wanted to risk Solaris x86 being yanked - which it was.

The second big barrier was cost. If you went with Sparc, you had to have money. Lots of money. Oodles of money! What I had was a basement full of x86 architecture hardware, and the not-for-profits I volunteer at had the same. There was simply no funding for shiny Sun hardware no matter how badly we wanted it.

And then the sleeping giant awoke. After being pummelled by the dot-com crash, Sun figured out what went wrong, and executed one of the m most amazing feats of corporate intertia changes I've ever seen. In a very short time frame, support for Solaris x86 was restored at a full commitment level. And it was made free. Then they continued to make their Java Enterprise System free to download and use as well.

So, the making of a fantastic Linux in Ubuntu may hurt Red Hat, but it's not what will deliver the killing blow. Red Hat has an opportunity right now to try to pull off a corporate inertia swing of Sun's magnitude. They need to restore faith in the community restore the religion they destroyed and find some kind of innovation to draw people back in. Solaris has done all of this and created an affordable support model that doesn't intimidate the small businesses who were once driven to Linux.

The first blood has been drawn by Solaris, but the second wound is far deeper. This second wound is bleeding internally and missing a lot of coverage. Mac OS-X is the killer desktop. If you have a reason to be using UNIX on a desktop, then using anything other than Mac OS-X is a tough sell in my book. Hardware is a bit more expensive, sure. But it's the best of every world, and solid as a rock. It doesn't hurt that it looks great either.

A recent seminar I attended talked about business models and knowing when to have the guts to drop a design. The idea was that you need to look at things you're developing and ask whether or not they give you a long-term sustainable advantage. I have to use that same litmus test to examine Linux. In the server world I can get free and open Solaris which is out-innovating Linux in my observation. And on the desktop, while Linux continues to improve, it's not even close to Mac OS-X.

In the end, these observations mean little to the tech-hobbyist who loves Linux for its religion. But in the business world, religion doesn't make IT choices. Competitive advantage does.

Solaris - the Wine

soalriswine.jpg
Originally uploaded by cghubbell.

I believe this may be one of the best system tools for evenings when a long day of UNIX has left the Force unbalanced in your mind. I stumbled on this wine while perusing a liquor store in Horseheads, NY. I hardly drink at all these days, but definitely enjoy a nice glass of wine when I unwind after a long day. If you're into this kind of thing and enjoy seeing Solaris outside the data center, check out the Solaris Winery!

Did you hear what I sed?

When battling the dark side of UNIX, it is critical that you not let your eyes betray your instincts. Windows teaches you to trust what you see, which is in itself a good reason to be wary. Today's lesson will involve our old friend awk, and a not so well known friend, od (octal dump).

I was working on a section of code, which decided whether or not arguments were passed by checking in a case statement for an empty string, or anything other than an empty string. It looks like this:

case "$SID_LIST" in
"" ) # No arguments passed, go with default.
echo "Stopping all configured Oracle databases."
su oracle -c "$ORACLE_HOME/bin/dbshut"
;;
* ) # SID list paased - pass it on to dbshut
echo "Stopping specified Oracle database(s)."
su oracle -c "$ORACLE_HOME/bin/dbshut $SID_LIST"
;;
esac


This structure comes from the Oracle 10g dbshut script which I'm applying some customizations to. As a result, I'm trying not to completely restructure the script. If I were to write it myself, I'd be more tempted to put this in an if statement, and test for a null string (test -z). But, since I'm working with someone else's code, I'm trying to stick to minimizing my impact.

If you call this particular script with arguments (an argument is a token that follows the command, like do_something RED BLUE) I detect the extra arguments from the command line, and put them into a variable called SID_LIST as follows:

ACTION=$1 # Assign first argument to action
shift; # shift arg pointer past $1 (action)
SID_LIST="$*" # Assign any remaining args to the argument list


So, when I call the script with a command like, "dbshut TESTDBA TESTDBB" I expect to see SID_LIST end up with the values "TESTDBA TESTDBB". Good enough! But what if someone repeats an argument? We don't want to iterate through arguments we have already processed, so I decided to add my own personal garnish of ensuring the list is unique. And this little detour is where the fun began...

The modification I made looked like this:

SID_LIST=`echo $SID_LIST | tr " " "\n" | sort | uniq | tr "\n" " "`


Let's break this down into logical steps:
First, translate any spaces into newlines because the next commands in the pipeline will expect to see things in multi-line form. This turns "one two one three" into:

one
two
one
three

Next, sort the output alphabetically to ensure similar items are immediately next to each other, which is necessary for the following piece of the command. Now we send the sorted list to a program called uniq which removes duplicates. The output now looks something like this (remember, its alphabetical):

one
three
two

Finally, we need to get it back into a single-line format, so we send the output into the reverse of the first tr command which replaces any newlines with spaces. Our final output looks like this:

"one three two"

Having conquered that challenge, I integrated the code fragment and observed its behavior. Oddly, I discovered that whether or not I supplied arguments, the case statement always resolved my input to be in the "*" branch rather than the "" branch. After taking a closer look, I discovered that my output was not what it appeared... In fact, the final newline had been replaced with a space by the last tr command, and my string looked like this:

"one[space]three[space]two[space]"

Because SID_LIST did not match "", the case statement selected the "*" branch instead. Feeling quite impressed with my mastery of the debugging arts, I surmised that a simple sed statement could whack my terminating space, and leave me with the desired empty string that would set my logic free. But alas, it was not to be...

I left me editor, and started playing on the command line. First, I created a simulation by setting a variable to contain a series of pretend arguments:

testbox{cgh}$ A="one two two three three four"


Next, I simulated my script's pipeline so make sure I could duplicate the problem. I surrounded the output with brackets to make the trailing space more obvious...

testbox{cgh}$ B=`echo $A | tr " " "\n" | sort | uniq | tr "\n" " "`
testbox{cgh}$ echo "[$B]"
[four one three two ]


Excellent, now we can test a fix... I put a sample string with a trailing space into a variable, and sent it into a sed command. The sed script is pretty straight-forward; search for a space character immediately before the end of the line, and replace it with nothing. This breaks down to the three divisions between slashes: [s]earch/[space]$(end of line)/replace_with_nothing/.

testbox{cgh}$ X="four one three two "
testbox{cgh}$ echo "[`echo $X | sed -e 's/ $//'`]"
[four one three two]


And behold, it worked! I now take the tested sed script, and attach it to the end of the pipeline...

testbox{cgh}$ A="one two two three three four"
testbox{cgh}$ B=`echo $A | tr " " "\n" | sort | uniq | tr "\n" " " | sed -e 's/ $//'`
testbox{cgh}$ echo "[$B]"
[]


What happened to my string? I copied and pasted the code, and it should have worked! Here is the part where we learn to trust our instincts, and not what we see. Let's revisit our input variables using The Force...

Earlier, we set $X to contain a sample set of arguments with a trailing space, and that input string worked nicely. Maybe the input changed somewhere in the pipeline to not exactly reflect the test conditions in our experiment... Here's how we can compare them:

testbox{cgh}$ echo $X | od -c
0000000 f o u r o n e t h r e e t
0000020 w o \n
0000023
testbox{cgh}$ echo $A | tr " " "\n" | sort | uniq | tr "\n" " " | od -c
0000000 f o u r o n e t h r e e t
0000020 w o
0000023


Do you see it? The difference is that our experiment's $X string is terminated by a newline character, while our pure pipeline string has lost its newline. This becomes a problem for the sed command which removes our trailing space. Sed acts when it sees an input terminator like a newline or ctrl-D character. In this pipeline, sed is never getting what it needs.

The solution is fairly simple, although not pretty. I broke this pipeline into two statements, and sent my sed script its input from an echo command rather than directly through the pipeline. This allows echo to put a newline onto the string and make sed happy. Here's what it looks like:

SID_LIST=`echo $SID_LIST | tr " " "\n" | sort | uniq | tr "\n" " "`
SID_LIST=`echo $SID_LIST | sed -e 's/ $//'`


This could be performed in other ways, my personal favorite being to reincarnate this script in Perl and eliminate all these pipelines and separate commands. But, by leaving it as-is I can keep the user base more comfortable with the language. It also serves as a great lesson for Jedi training, and so shall it remain.

Poor grammar isn't always a bad thing

If you write enough shell scripts you will eventually fall prey to your own comments. Unless you read my blog of course, in which case you will have saved hours of frustration!

Let's take a fictitious problem... You need to print the first and third columns of the /etc/passwd file so that a report can be generated correlating user IDs to user names. Being the UNIX monk that you are, you assure your management that a shell script can meet their every need, and there is really no reason to have an ODBC link from Microsoft Access to the passwd file.

You throw together some code, and it looks like this:

#!/usr/bin/ksh
nawk 'BEGIN { FS=":" }
# We don't want to print anything but
# the first and third column
{print $1,$4}' /etc/passwd
exit 0


Looks like a nice tight algorithm, well commented, and generally a job well done. You pat yourself on the back and refill your coffee, ready for the next challenge. Not so fast... First you decide to test that script, and you see the following:

testbox{cgh}$ ./comtst.ksh
./comtst.ksh[6]: syntax error at line 6 : `'' unmatched
testbox{cgh}$


But how can this be? It's a simple script, and the logic is flawless! Let's test it to be sure...

testbox{cgh}$ nawk 'BEGIN { FS=":" } {print $1,$4}' /etc/passwd
root 1
daemon 1
bin 2
sys 3
adm 4
lp 8
uucp 5
nuucp 9
ftp 60001
smmsp 25
listen 4
nobody 60001
noaccess 60002
nobody4 65534
cgh 1000


It works... What is the problem here?

It turns out that the comments in the embedded nawk code are the problem. In this case, the apostrophe in "don't" closes the opening apostrophe at the beginning of the nawk statement, and the shell interprets the code like this:

#!/usr/bin/ksh nawk 'BEGIN { FS=":" }# We don'


So what we really do it pass nawk a syntactically incorrect program. Having figured it out, we re-write the code as follows:

#!/usr/bin/ksh
nawk 'BEGIN { FS=":" }
# We do not want to print anything but
# the first and third column
{print $1,$4}' /etc/passwd
exit 0


There are two morals to this story: First, at the risk of repeating myself like a broken record, don't use multiple shells unless it's absolutely necessary because you run the risk of obscure interpretation problems. In this case, we could solve the problem by writing in Perl where there's no need to embed a second language.

The second moral is to always avoid using contractions and meta-characters in your comments. It makes for slightly longer comments, but if you scrictly avoid the temptation, it is one less thing to worry about. This example was so simple that it's not hard to locate, but if you had a complex nawk script with its own subroutines buried in a complex shell script, it can be very frustrating trying to locate the bug.

The dark side will tempt you with contractions, but now your Jedi training has equipped you to calm your mind and type out those extra few characters. Until next time, may the code be with you.

Don't Shed Your Shell

I've said it before, and will say it again; Switching interpreters in mid-code is a practice to avoid whenever possible. There are times that it can be avoided, but there's a lot of times when you can sacrifice a bit of elegance for simpler maintenance.

As with most bugs, I was recently bit by a dumb mistake. I needed the ability to lookup Solaris Resource Manager Project information using tags embedded in the description field. For example, SID=TESTDB is how I would specify an Oracle database SID. I wrote a Korn shell function called getprojbyattrib() which accomplished this very thing. Tested on its own, it worked wonderfully. When I went to integrate it with the existing Oracle start-up scripts I ran into some problems. Turned out they were easy to debug, but the root cause was my old enemy of incompatible interpreters.

This new shell library function is used to figure whether or not an SRM project is configured for a given Oracle database. If one and only one match is returned, then the database is started in a project container. Any other condition means that the database is started without SRM. To help in this cause, I embedded a counter in the function to return how many matches were found. The code in question was simple:

# Keep track of the number of projects we find while outputting
# them so the final tally can be used as a success indicator.
PRJCOUNT=0
for PRJ in $PRJLIST
do
echo "$PRJ"
PRJCOUNT=$(($PRJCOUNT+1))
done


Make note of the seventh line of code which does the incrementing. This is a Korn shell specific operation. When the calling code from the oracle startup script referenced this, it gave an error which told me that it had interpreted line #7 at "PRJCOUNT=$". This is because the Bourne shell doesn't understand the operation.

The fix is simple. Either switch the calling script to use the Korn shell interpreter because Korn is a superset of Bourne, or change the increment code to be Bourne-friendly by using either bc or expr.
PRJCOUNT=`/usr/bin/expr $PRJCOUNT + 1`

Interestingly, the library function was written with a header that specified Korn shell as its interpreter:

#!/bin/sh


This becomes irrelevant when you are sourcing functions or variables as the whole point is to have your calling shell get access to these objects.

Sp what did I do? At first I switched the calling code, but some afterthought lead me to work with the underlying Bourne shell subset so the library would be more portable. I don't really like Bourne shell as Korn is much more capable, but in this case portability is weighted more heavily than elegance.

Repeat after me: Switching interpreters in mid-code is something to be avoided whenever possible.

Solaris Date command and epoch time

I can't count over the years how often I've wanted to output a date stamp in seconds since the epoch to make duration calculations simple.

Just to prove that I'm not 100% biased towards Solaris let me point out that my Linux scripts all enjoy the ability to call /bin/date with a simple switch that outputs time in my desired format: date +%s.

In Solaris, neither /usr/bin/date nor /usr/xpg4/bin/date support output in the "seconds since epoch" format. This is what we call low hanging fruit as enhancements go. Unfortunately, the fruit still hangs.

Perl does a very nice job of handling date math and epoch conversion, but that requires a separate interpreter, and when I'm in shell I don't like to jump in an out of other interpreters. I found a pretty cool hack that seems to avoid an external interpreter, and gets me what I want...

Since we know that the system keeps track in the format we want, we need to find a utility that uses a system call... In this case I made a crazy guess and found the time() call. Here's what it looks like:

testbox# man -s 2 time
System Calls                                              time(2)

NAME
     time - get time

SYNOPSIS
     time_t time(time_t *tloc);

DESCRIPTION
     The time() function returns the value  of  time  in  seconds
     since 00:00:00 UTC, January 1, 1970.

So, making a second wild guess I assumed that our beloved /usr/bin/date command uses the time() system call. Let's take a look... If we use the truss command to check out system calls and returns we should find what we're looking for. We'll use the grep command to look for the time() call.

There's a catch though... Truss is going to dump output to stderr, and grep looks for input on stdin. Those paths won't cross. So, we need to redirect stderr into the stdout stream before piping it all over to grep.

testbox# truss /usr/bin/date 2>&1 | grep ^time
time()                                          = 1149275766

Cool! You can see the 2>&1 take stderr (2) and redirects (>) to catenate (&) with stdout (1). This cuts the 40+ lines of system calls down to the one we care about.

It's not a standard interface, so any time we use it we run the risk of any OS patch breaking our algorithm. Perl would be a safer way to go, but it does require more overhead in terms of firing up the interpreter for such a simple thing. You'll have to decide for yourself whether or not this hack is useful to you, but I think it's a good one.

To clean it up and make a bit better behaved we'll need to get rid of leading and trailing spaces, and output just what we need. Here's a quick script you can call or source...

#!/bin/sh
/usr/bin/truss /usr/bin/date 2>&1 |  nawk -F= '/^time\(\)/ {gsub(/ /,"",$2);print $2}'
exit $?

And finally, let's see it in action:

testbox# ./edate
1149276150
testbox#

So there you have it. A way to get epoch time without writing a single line of C.

Veritas Foundation Suite goes free

Veritas Volume Manager is free. I'm not joking! Check it out for yourself!. Of course, there are some restrictions involved.

• Only for Linux and Solaris x64


• <= 2 CPU cores


• Max of 4 user volumes

I'm not sure I "get" this move. Historically, there has been a philosophical battle going on in the enterprise engineering space over whether 'tis best to manage your root disks with Sun's Solaris Volume Manager (SVM) or Veritas' Volume Manager (VxVM).

On one side, VxVM had the strength of a volume management solution that scaled to infinity and beyond. It's not much more difficult to manage 1000 disks on VxVM than it is to manage 10. The corollary to this point is that if you require VxVM for your data farm, then you can reduce system architecture complexity by standardizing on use of VxVM for management of root disks as well. Simple is good, right?

On the other side, SVM is extremely easy to use, and extremely easy to recover from problems with. Given it's simple layering approach, it's also much more difficult to get into trouble with. SVM has long been a favorite with Sys Admins because it means they never have to worry about the dreaded unencapsulation dance. Again, simple is good, right?

From a purely academic standpoint, the complexity of using multiple Volume Managers seems unnecessary. In practice, I believe the opposite is true. Root disks are a very different beast than data disks. For one thing, they tend to be configured and then forgotten (barring a disk failure). In contrast, the data farm is always churning with reconfiguration or expansion. Given the opposing dynamics of these types of volumes, it makes sense to use tools with different strengths.

When a root mirror pair needs work, you want a simple and reliable way to solve the problem and move on with very little chance of making an unrecoverable mistake. This is SVM in a nutshell. Its tight OS integration, and shallow learning curve mean that even a junior SA has a great probability of pulling off that disk replacement.

Unfortunately, if you have terrabytes of disk storage hooked to your Enterprise 20k you want a volume manager that is very flexible and agile. Historically, SVM has not been that tool. Its GUI does not display huge data farms efficiently, and its volume naming conventions get unruly when the disk count grows. This is where VxVM comes in.

VxVM use abstraction and well designed (although complex) interfaces that allow mountains of data to be displayed quickly through either command line interface or GUI. Historically, it also had much more power and flexibility than Sun's offerings. This meant you needed VxVM for data farms.

The best of both worlds, in my opinion is using SVM for Operating System volumes and internal disks, and VxVM for all external storage whether SAN or DAS. If you only have limited external storage then just use SVM. Really, simple is good!

Returning to Veritas' recent change in marketing strategy, it seems that they may be trying to counter some of the arguments for using Sun's integrated solution. It's clear that ZFS is going to provide a very powerful facility for storage provisioning within Solaris 10, and SVM already supports many of the core features which used to be Veritas' key selling points. Veritas has one key advantage in that they have a time tested solution available NOW. ZFS isn't mature enough yet for the mission critical enterprise, but that's a very short term disadvantage. The OpenSolaris model means that when ZFS makes it into a hardware update of Solaris 10, it's going to be 95% there. I'll give it six months before they master that remaining 5% which wider distribution will open up.

Is the right strategy to pick to seldom-used platforms to make VxVM free on, and then limit use to four volumes on a product which is only advantageous when volumes are plentiful? I don't think so. If you have a simple system you're not going to WANT to use VxVM because SVM is so much simpler.

To me it looks like Veritas is trying to use a seeding strategy for a market that it has no chance of enticing. While Veritas has a great product, it seems that they don't fully understand their niche. I'm putting my money on SVM and ZFS.

Testing for correct usage in shell functions

Here's a simple touch you can apply to your shell scripts to aid in debugging when they grow to become monstrous and you can't remember the syntax of all your subroutines any better than you can remember the 10th digit in Pi, which happens to be 3 for those who care about such things.

Although not strictly required to take advantage of this tweak, I recommend you begin by using good headers for each subroutine. I won't go into each one, but a specific entry I always make is usage. For example, if a subroutine do_foo takes arguments arg_one and arg_two, the header would look like this:

# ------
# do_foo
# ------
# USE: doo_foo ARG_ONE ARG_TWO
# DESC: Execute foo functionality
# PRE: na
# POST: na
# ERR: na
foo () {
    ...
    ...
} #end do_foo


The line I want you to pay attention to in the above code begins with "USE:" (4th line). This line specifies the interface which a user of your code should be aware of. You are telling them that this code expects TWO arguments. Now, you can get fancy and use EBNF like syntax to identify optional arguments, but let's keep it simple for this example and just recognize that we have established an interface.

What can we do as a developer to make sure that when someone calls our code, they do not get something unexpected? We can check to make sure they follow our instructions. It's simple enough, although you can certainly take it greater depths. Let's go back to our do_foo example and put a check in place...

foo () {
    test $# -eq 2 || exit 1
    ...
    ...
} #end do_foo


Let's break down the line I just added... test lives in /usr/bin and should be a fluent part of your shell vocabulary. We are "testing" to see if the number of arguments ($#) is equal to the integer 2. If not (symbolized by ||) then we exit with non-zero status, which is the UNIX convention for something other than success. The next level of effort would include writing a shell equivalent to Perl's die subroutine. This would allow an error message to accompany the exit. We'll save that for another article.

So, what's the benefit of adding this code-bloat to our subroutine? It's common to have a function that uses optional arguments and acts differently depending on what arguments it receives. If the function expects ARG_ONE and ARG_TWO, and you call it with only ARG_ONE, it may assume that ARG_TWO is equal to "". In that case, the output may be "object not found" rather then "Whoa! You made a mistake calling me!". If you were depending on a specific output, this could cause later code blocks to break.

Here's a more specific example. If we are using the ldaplist command to check on project information, we will get two totally different sets of output if we omit a second argument. Pay particular attention to the command and arguments in the examples below:

testbox# ldaplist project
dn: solarisprojectname=srs,ou=projects,dc=mydomain,dc=com
dn: solarisprojectname=bar,ou=projects,dc=mydomain,dc=com
dn: solarisprojectname=foo,ou=projects,dc=mydomain,dc=com
dn: solarisprojectname=group.staff,ou=projects,dc=mydomain,dc=com
dn: solarisprojectname=default,ou=projects,dc=mydomain,dc=com
dn: solarisprojectname=noproject,ou=projects,dc=mydomain,dc=com
dn: solarisprojectname=user.root,ou=projects,dc=mydomain,dc=com


In contrast, what we REALLY wanted was only one line that matches our criteria, not the whole set of data.

testbox# ldaplist project solarisprojectname=user.root
dn: solarisprojectname=user.root,ou=projects,dc=mydomain,dc=com


If we use an argument checker, the error woudl be caught immediately rather than passing on a long list of irrelevant data to whatever we do next. In this case it's particularly ugly because both outputs are identically formatted. Maybe you'd find the problem quickly, maybe you wouldn't.

When your code gets to be hundreds of lines long and you need to start debugging obscure behavior, it can save you a lot of time to write self-policing code. Chances are that if you make a simple mistake calling that subroutine it will fail immediately rather than doing the wrong thing in a hard to find way. A line of prevention is worth an hour of debugging!

Using syslog with Perl

I recently had an occasion to write a fairly simple Perl script that checks for rhosts files in any home directory which is configured on a system. Nothing fancy, but very useful. After getting through the file detection logic I was left with the question, what now? Should I write a custom log file? Should I call /usr/bin/logger?

As always, I looked for precedents and standard facilities. The first thing that came to mind was syslog. And of course, the fact that I was using Perl led me to believe that I wasn't going to need to execute an external process (the "duct tape hack" as I call it). I view the shell as another language, and something never really feels right when I need to embed one language within another. Don't even get me started about embedding big awk scripts inside shell scripts... That's going to be a future topic.

The duct tape method is bad for a number of reasons. There is overhead associated with forking and executing a new child process from your main script. If you are running awk and sed, or other tools thousands or millions of times against a file then you are forcing Solaris to execute far more system calls than necessary. By keeping it all inside Perl and using modules, you can let the interpreter do the work, and realize a good part of the efficiency that C system programming gives you. I'll save the specifics of this for a later time - we need to dig into the syslog example.

In this case I quickly found the standard Sys::Syslog module. This little gem makes it a snap to log output. I won't go into the Solaris syslog facility here, but suffice it to say that you'll need to arrive at your intended Facility and Priority before going farther. For my purposes I went with User and LOG_NOTICE.

To begin with, we need to include some libraries...

use Sys::Syslog;


When we want to set up the connection with syslog we do the following:

openlog($progname, 'pid', 'user');


The above line specifies that we will use the 'user' facility, which is typically what you should be using if you don't have a specific reason to go with one of the other options. It also specifies that we want to log the pid of the logging process with each entry. Logging the pid is a convention that isn't always necessary, but I like it. The first part, $progname is a variable that stores the name of the script. This deserves a little extra attention.

Since I'm known to change the name of my scripts on occasion I don't like to hard code the name. In shell scripts I usually set a progname variable using /usr/bin/basename with the $0 argument. $0 always contains the first element in the array of command line variables. So, if I called a script named foo with the arguments one, two, three, the command would look something like this:

# /home/me/foo one two three

The resulting array $* would be:

[0:/home/me/foo][1:one][2:two][3:three]

To identify our program name we want the first array element. However, we don't want all that extra garbage of the path. It makes for a messy syslog. The basename UNIX utility helps us to prune the entry. Here's an example in shell:

$ basename /home/me/foo
foo

If we want to do the equivalent in Perl without spawning an external process we can use the File::Basename module. Again, with a simple include at the top of our script this function becomes available to us:

use File::Basename;

Now we can put it all together and create an easily referenced identity check:

my $progname=basename("$0");


Why don't we just hard code the script name? After all, not everyone likes to refactor their code for fun. Besides the idea that we want our code to be maintenance free, there are times when one set of code may be called from links which have different names than the primary body. For example, let's assume that the script foo performs three functions: geta, getb, and getc. To make it easier to call these functions we want to be able to call these directly without duplicating code. Here's how we could do that:

# ls -l ~/bin
-r-xr-xr-x 1 root root 5256 Jun 8 2004 /usr/local/bin/foo
# ln ~/bin/foo ~/bin/geta
# ln ~/bin/foo ~/bin/getb
# ln ~/bin/foo ~/bin/getc

We can now call any of geta,getb,getc and actually call foo. With some simple logic blocks based on what $programe evaluates to we are able to create a convenient interface to a multi-functional program with centralized code. Nice! But I digress - let's get back to looking at syslog...

We have opened a connection to the syslog, and now is the moment of truth. Let's write a syslog entry...

syslog($priority, $msg);


Let's recap... I used a facility of user, and a priority of notice. I want to record the pid, and write a message. What does this look like when its executed?

May 25 11:01:25 testbox rhostck[833]: rhosts file found at /u01/home/cgh


That was really easy, and it's much cleaner than executing the external logger utility because it's all inside Perl.

A plethora of ldapsearches...

If you're going to deploy a directory service for Solaris systems, and you are really lucky, your server and clients will all be using a Solaris version greater than 9. LDAP works nicely in 9, but it's a bit of a transition release. Only in Solaris 10 is Sun's commitment to LDAP clear. Let's take a look at one of the more frustrating examples of Solaris 9's transitionary status: The ldapsearch command.

ldapsearch comes in many different flavors. First is the native Solaris version which lives in /usr/bin. On Solaris 9 this version does not support SSL (-Z option). In Solaris 10 SSL is nicely supported through this client. Next we have the iPlanet flavor which lives in a dark and gloomy path: /usr/iplanet/ds5/shared/bin. This is installed by default with Solaris 9 and happily supports SSL despite its gloomy path. But wait, there's still one more! After installing the JES Directory Server you will find one more flavor of ldapsearch living in /usr/sadm/mps/admin/v5.2/shared/bin. Now that's an intuitive path. This last flavor will only be on your server, but I'd hate to leave it out of the fun.

As if having too many to choose from isn't enough, two of the ldapsearch flavors require proper setting of the LD_LIBRARY_PATH variable. When a dynamically linked binary requires a library that lives somewhere other than the system default (usually /usr/lib variants) it needs the LD_LIBRARY_PATH variable to tell it where to look.

Here's an example of a binary that needs the extra help from LD_LIBRARY_PATH:

testbox$ /usr/sadm/mps/admin/v5.2/shared/bin/ldapsearch
ld.so.1: ldapsearch: fatal: libldap50.so: open failed: No such file or directory
Killed

So what happened? Let's take a closer look...

testbox$ truss /usr/sadm/mps/admin/v5.2/shared/bin/ldapsearch
execve("/usr/sadm/mps/admin/v5.2/shared/bin/ldapsearch", 0xFFBFFB54, 0xFFBFFB5C) argc = 1
resolvepath("/usr/lib/ld.so.1", "/usr/lib/ld.so.1", 1023) = 16
resolvepath("/usr/sadm/mps/admin/v5.2/shared/bin/ldapsearch", "/usr/sadm/mps/admin/v5.2/shared/bin/ldapsearch", 1023) = 46
stat("/usr/sadm/mps/admin/v5.2/shared/bin/ldapsearch", 0xFFBFF928) = 0
open("/var/ld/ld.config", O_RDONLY) Err#2 ENOENT
stat("../libldap50.so", 0xFFBFF430) Err#2 ENOENT
stat("../lib/libldap50.so", 0xFFBFF430) Err#2 ENOENT
stat("../../lib/libldap50.so", 0xFFBFF430) Err#2 ENOENT
stat("../../../lib/libldap50.so", 0xFFBFF430) Err#2 ENOENT
stat("../../../../lib/libldap50.so", 0xFFBFF430) Err#2 ENOENT
stat("../lib-private/libldap50.so", 0xFFBFF430) Err#2 ENOENT
stat("/usr/lib/libldap50.so", 0xFFBFF430) Err#2 ENOENT
ld.so.1: ldapsearch: fatal: libldap50.so: open failed: No such file or directory
write(2, " l d . s o . 1 : l d a".., 81) = 81
lwp_self() = 1

Here we can see Solaris trying to find the required dynamically linked library, libldap50.so. It traverses 8 directories, each time returning the ENOENT key which intuitively means "ERROR - No entity found". So, job #1 is finding that library and acquainting it with the binary that's lost its way...

testbox$ grep libldap50.so /var/sadm/install/contents
/usr/appserver/lib/libldap50.so f none 0755 root bin 380348 45505 1052289104 SUNWasu
/usr/dt/appconfig/SUNWns/libldap50.so f none 0755 root sys 450716 23095 1032825102 SUNWnsb
/usr/iplanet/ds5/lib/libldap50.so f none 0755 root bin 361976 55632 1013353620 IPLTdsu
/usr/lib/mps/libldap50.so f none 0755 root bin 392416 44988 1100692806 SUNWldk
/usr/lib/mps/sparcv9/libldap50.so f none 0755 root bin 433976 29179 1100692807 SUNWldkx

In this case, we know that the needed library is going to be used with the JES ldapsearch, so we'll guess that appserver's offering isn't quite what we want. /usr/iplanet looks tempting, and will probably work, but what we want is the /usr/lib/mps directory which is distributed with the Sun LDAP C SDK.

So now that we've found the missing library, let's plug it into the LD_LIBRARY_PATH and see what happens. I'm using the Korn shell, so if you're a C-Shell type you'll just have translate on the fly.

testbox$ export LD_LIBRARY_PATH=/usr/lib/mps:/usr/lib/mps/sasl2
testbox$ sudo /usr/sadm/mps/admin/v5.2/shared/bin/ldapsearch [...]
version: 1
dn: dc=foo,dc=com
objectClass: top
objectClass: domain
objectClass: nisDomainObject
dc: apps
nisDomain: foo.com

It worked! (You didn't doubt me did you?) You may have noticed that I actually added two paths. After fixing the first missing library you would have dicovered a second missing one which was identified and fixed the same way. I love a problem with multiple layers... Especially when layer #2 is the same solution I needed to peel layer #1. The other thing to note is that I abridged the command line for ldapsearch. Executing an ldap query with SSL can be like writing a book so I cut it short.

So, not only do you need to pick the right ldapsearch flavor, but you also need to set LD_LIBRARY_PATH accordingly. If you are using the Solaris native versions you don't need to do anything. But for JES and iPlanet verions, here's what you need:

• iPlanet: LD_LIBRARY_PATH=/usr/lib/mps


• JES: LD_LIBRARY_PATH=/usr/lib/mps:/usr/lib/mps/sasl2

So which one should you use? Here's a quick flow to make that decision. If you are using Solaris 10, just go with /usr/bin/ldapsearch. It does everything without any hassle. If you are on 9, then a decision emerges. If you have an SSL-secured directory server you can not use /usr/bin/ldapsearch. Typically, you will use the iPlanet version on Solaris 9, and if you are on the server itself, go with the JES version.

So there you have it, a lot of hassle can be saved by deploying on Solaris 10 rather than 9. Most of what you'll need the Directory for will be handled by Solaris internals, so you won't need ldapsearch, for example, to authenticate users against the Directory Server. Where you will need ldapsearch is if you are storing custom entries in the directory, or executing a special query against it.

Solaris Naming Services: The good, the bad, and the ugly

One of the first things that needs to be done in streamlining an Enterprise is establishing a centralized naming service. Of course there are some fringe reasons not to, but the bottom line is that the IT world is continuously being asked to do more with less. If you need to manually update the passwords for 100 users on 100 systems once each month you are either a great candidate for life in a Skinner box or you are failing to do more with less.

My experience has always been that the risk of someone propagating a catastrophic change across a data center is much lower than that of someone making a subtle manual error that goes undetected for months until it rears its ugly head. When you know that your finger hovers over the big red button that controls the fate of your buddy's on-call pager you get religion fast and become very careful about what you propagate. Centralization and automation are good. Manual effort is bad. So, we've established that we want to do this. So, what is a Solaris site to do next?

There are a few naming services to choose from. In our case, common choices include: File propagation, NIS, NIS+, and LDAP. Within LDAP there are a few different implementations now that the industry has begun to move in that direction. Microsoft's Active Directory, OpenLDAP, and Sun's Java Enterprise System Directory Server. Again, going by experience, the best path is usually the one which aligns with your primary vendor's offerings. If you're running a Windows shop, use AD. If you're running a Solaris shop, use Sun's Directory Server. If you run both, it's a different question altogether, but I would be inclined to create a layered solution involving both AD and Sun DS. But, I digress. Let's take a look at the options...

File propagation isn't as bad as it sounds unless you are doing it manually. There are many bolt-on solutions for implementing the functionality of rdist. It can be done over SSH, and with a surprising degree of control. You can also use SSH with some script-fu to push scripts to be executed, or use a product like CF Engine. While not a bad solution, it tends to be a bit heavier in the maintenance and integration time. It's also somewhat limited in that it can push, but you aren't really centralizing. Systems inevitably get left out or miss pushes which results in more custom coding to make queues, and pretty soon you're on your way to reinventing a management framework. Ugh. File pushing is a good solution within its scope, but I'm not a fan when it comes to a general centralized administration scheme.

NIS is a grand old favorite. The environment I cut my teeth on included a Sparc 10 workstation hooked up to a QIC tape drive which acted as a NIS server for a building full of Sparc IPX workstations. I found it fairly easy to manage, rock solid, and generally a great technology. Unfortunately, it also included a whole new world of security gaps that while acceptable for a non critical workstation environment were totally inappropriate to a world class data center housing mission-critical data. Even if it weren't inherently insecure, Sun announced the EOL of NIS, so the end is near. Given that, I suggest that NIS is probably not the best technology to invest in these days although Linux has a good implementation that will allow it to continue for some time.

Our next option is NIS+. Have you actually supported a NIS+ environment? Yuck. After the joy of NIS I had expected a simple evolution, but it felt more like assuming that if you speak English you can speak Russian. While it fixed the gaps in NIS security, it was always annoying trying to figure out which key was broken and how to get it initialized again. Although Sun has not EOL'd NIS+ yet, it's clear that it is not a technology whose momentum is being maintained. The writing is on the wall. NIS+ again proves to be a non starter.

Finally we arrive at LDAP, and for the reasons above, We look at Sun's Directory Server. I'm currently working on a Directory infrastructure for a large site, and have been both impressed and frustrated with the product. I'll be explaining this dichotomy in more depth in future installments, but a few months into the project, I'm confident that I made the right recommendation.

Why? First of all the updates for this product come from the same place all the other Sun updates come through. Trying to keep track of multiple product update sites is a non-value-add activity. Second, LDAP is insanely flexible. You can store just about anything in a directory, and access it via shell script, OS client, Perl script, and more. Why is this cool? Its much easier than storing anything in an Oracle database because you won't need to gain a new certification in relational databases to take advantage of this new piece of your infrastructure. A systems engineer or advanced admin can do everything they need. Third, DS is part of the Java Enterprise System. JES is a stack of middleware products which are all nicely integrated with the OS and support. By giving us a stack of common and world-class components Sun has given systems engineers the opportunity to start including a deeper application awareness into their scope with an easy product to standardize on. And the final reason to go with DS and JES: It's free to download and use. Before you put it into production, USE it. Put it on your x86 machines, put it on your Sparc machines. Hack it, tweak it, learn it, know it, master it before you go live. Sun has just leveled the playing field. Now you can learn a world class product without capitol investment because Sun's on-line documentation is second to none. One last note: After you've enjoyed Sun's generous new distribution model, buy a production support contract or JES subscription. Don't go into production with unsupported software. You know better, and your customers expect that you'll advise them to do the right thing. Besides, I'm sure you didn't go into production with an unsupported Linux server, right? After all, you were too smart to fall for that whole "Linux is free" story weren't you? Nothing is free.

And for my final act I'd like to mention the paradox that continues to twist my mind into little pretzels. NIS and NIS+ were OS integrated components that were free components of a great Operating Environment. Sun has always been about the large scale net infrastructure, and the integrated NIS servers echoed that intent. With the JES Directory Server Sun has gone beyond the site-scale naming services with a product that can store just about anything and scale it to your wildest dreams. Think consolidation of eBay and Amazon identities and DS isn't breaking a sweat - it's that good. But in the mean time, there's a lot of sites that need a centralized naming service for their tightly secured data centers that will never share identities with the outside world, and never cross 1000 users. That's like having a Ferrari that never gets out of first gear.

What fills the gap between 100 and infinity? Fortunately, from a technical perspective JES DS scales in nice increments, and can deployed for a single site's centralized naming services on reasonably sized hardware. But there's a catch: Support will cost you. Even if all you are doing is Solaris native client support, the DS costs money to obtain a software support contract. Sun, what are you thinking here? Kill NIS and NIS+, tell everyone to go to LDAP, and then neglect to include it in the Operating System support? Crazy! Native OS support should be included with Solaris just as Active Directory is free with the Windows Server support. Charge us more money when we ask you for support outside native clients, but don't' make us pay for what you provide for free with NIS+.

JES is a great product, I'm glad we're using it in our project, but I was really disappointed when I discovered this Dilbertian marketing twist. There's still hope; SRM used to be an expensive add-on that's now a part of the OS. Given enough time, Sun usually does the right thing.

In the beginning there was SunOS...

I'm enjoying a renewal of enthusiasm for Solaris, and this posting marks the beginning of a blog tracking the things I'm working on and exploring. I've use Solaris professionally for more than ten years from both Systems Administration and Systems Engineering perspectives. Six of those years I spent working in various roles at Sun Microsystems; a journey I'm grateful for as I learned more in those years than I would have elsewhere.

For a significant portion of those six years I was increasingly frustrated by how much ground Linux made on Solaris' lead. Linux has its place, and I'm certainly not anti-Linux or anti-OSS. In fact, quite the opposite. I've been tinkering with Linux since 1992, and use it regularly at not-for-profit site I volunteer at. For many years it was primary workstation (Fedora / Red Hat) so I've spent years observing Linux in both a server and workstation role.

I'm no longer using the Linux workstation which served me the past few years. I've fallen in love with Mac OS-X and never looked back. I don't expect to write much about Mac OS because in itself it doesn't stand out. That's the whole point - after using my Mac I talk more about what I've done than how I did it. Interesting paradigm shift.

With the advent of Solaris 10 I firmly believe the Solaris has taken another evolutionary leap which seem to leverage the best of both Open and Closed source development models. I think this evolution is important as it shows what can be done when extremism is curbed for practicality. Solaris 10 found a sweet spot which I hope to write more about.

I'm now working as a consultant for a large UNIX shop in my area and working on some exciting technologies. My primary interest is in large scale UNIX infrastructure, so expect to see a lot of content focused on things like Directory Server, Consolidation, Automation, Resource Management, RBAC, and much more.

So there you have it... A brief introduction to me, and the intended content of this blog. I've been notoriously lousy at keeping blogs up to date, but not for lack of good intent.