Solaris Jedi

Cool or Embarrassing?

2011-02-25T20:46:00.000-05:00


10:24:35 up 1000 days,  2:43,  1 user,  load average: 0.11, 0.13, 0.09

I admit it, my first reaction is, "How cool is THAT?". But then this little voice in my head says, "Yeah, but that means it's been an awfully long time without a kernel patch."

ndd-nettune.xml not quite right (from DSEE7 docs)

2010-06-03T09:23:00.000-05:00

If you try to follow the current revision of docs for Sun Directory Server Enterprise Edition 7 (or for that matter, the DSEE 6.x as well) you will find references to installing a SMF manifest called ndd-nettune.xml. In the DSEE7 process, it comes in the Sun Directory Server Enterprise Edition 7.0 Deployment Planning Guide.

When you install the provided manifest, and run svccfg validate on it, you are slapped across the face with a useless error indicating that the file can not be parsed. Hmm, thanks. I'm the first to admit that my XML skills are basic; I'm not a full time software developer, I just play one on TV.

After some digging around I decided to model the stock coreadm manifest that is extremely similar. It can be found at /var/svc/manifest/system/coreadm.xml. With some minor editing, it was pretty easy to get it working. I believe the issue is in the exec_method tags not being properly terminated. It's a bit of a PITA to post XML into this blog, but if you take a look I think you'll see what I mean. The newly edited manifest imported without complaint, and I'm back on track with the installation.

I also stumbled onto a neat tool which was new to me. Not being satisfied with the lame error message svccfg spat at me, I dug up a utility called "xmllint" which has much more interesting diagnostics. I'm not sure how to interpret what it gave me, but with a little time I'm sure I could have leanred and benefitted from it. Something to file away in the Jedi library for a rainy day.

Got Webstack?

2010-06-02T08:33:00.000-05:00

Did anyone else notice that the Sun Webstack appears to have quietly disappeared from download-ability? 1.5 was released in the summer of 2009, and now it appears to be quite the maze of links leading nowhere.

I can find wikis and articles, but the links all lead to a general page and searches on oracle.com lead nowhere. Sure would be nice if they'd at least posted a "so long and thanks for all the fish" notice.

Webstack, webstack, where art thou, webstack?

To syslog, or not to syslog? That is the question.

2010-03-30T19:46:00.001-05:00

At $WORK we have a simple application which acts as a wrapper around common account management tools. Its purpose is to ensure that all account move/add/change implementations are logged with who did the work and what ticket authorized it, etc. The method for this logging is currently syslog. I'm not sure sure that I would have chosen that design, and that is the topic of this post.

In general I'm made to suffer by observing the slow decay of UNIX practitioners. That's not to say that use of UNIX is in decline, but rather that people who understand UNIX are becoming fewer and farther between. Note that in this case, I lump Linux and UNIX together. So many people are hopelessly tainted by their knowledge of Windows that it creates a prism through which the light of every operating system is split between reality and perception.

UNIX is more than an operating system. It is a development platform. It contains the tools you need to handle many application components including logging, messaging, file parsing, etc. And yet, we find people frequently turning to proprietary (or sometimes open) tool kits to solve problems the OS is well equipped to handle. It frustrates me.

Given that I'm a strong proponent of using the Operating System's features to solve problems, why wouldn't I like the idea of a local script using syslog to handle its logging? The biggest reason is that today's Solaris syslog facility remains tightly constricted in its use of facilities and levels. Yes, I'm well aware that syslog-ng can open those doors. I'm all for it! However, someone at Sun (Oracle) doesn't seem to be all for it yet, and I'm not a big fan of re-plumbing core OS features. So, until Sun sees the light and modernizes the syslog facility I believe in sticking with the standard.

One approach that may be acceptable is combining all Operational logging into a single facility. The trouble with that approach is that we have some logs which fall under tougher security policies than others, so you end up needing destination files with different attributes. So, what's the answer to the engineer's dilemma?

If an application needs to send its output for Enterprise-wide real-time processing for something like a log watcher, then it may be appropriate to use syslog in order to leverage its ability to forward log streams to syslog servers. But if you are writing a simple application which diligently generates log files for audit or troubleshooting purposes, you may be better off in the long run by writing a simple log function that dumps to a configurable destination file. Of course, your log files will be stored under /var/opt/something and will integrate with logadm(1M), right? Of course. After all, you are a Jedi...

How about OpenJET?

2010-02-28T19:11:00.001-05:00

I've had numerous occasions to "pop the hood" on JET to add functions or debug problems, and one of the things I appreciate is how it is implemented in well structured scripts. JET doesn't use complex frameworks or app servers just for the sake of adding "web 2.0" to a marketing slide. It's as simple as it needs to be. Bravo!

One thing I've noticed is that there's a HUGE amount of opportunity for enhancing JET, both in terms of optimizing existing code (increase use of functions, internal documentation, etc) but also in terms of enhancing functionality with new modules or new features in existing modules. Unfortunately, Sun (ahem, Oracle) is not really exploiting these opportunities in a timely manner. JET updates don't happen very frequently. I would expect that's due to economic strains impacting resources, and I don't expect the IT economy to suddenly produce a legion of JET developers.

Why not create an open community for the development of JET? There's all kinds of energy at OpenSolaris for next generation provisioning / packaging. If there were an OpenSolaris project for today's JET product I think we'd see it really take off. Many of us need to perform these enhancements either way, but we'd all benefit if we could leverage each others work and continue the evolution of a fantastic framework.

I'm hoping that resistance to opening JET isn't due to Sun Professional Services. JET is great, but it isn't rocket science. SunPS can always retain copyright or restrict distribution of their proprietary modules (as they always have). But the base framework screams to be open sourced, or turned into a community.

If Sun won't open the code it's tempting to try a black-box rewrite (I have no desire to break Sun's licensing - I respect their rights to code they developed!). It would certainly be a fun science project. Unfortunately, it would also be a waste of time given that an existing code base would make such a perfectly suited foundation.

So, how about it, Sun? OpenJET?

Solaris Information Center: Release Naming

2010-02-07T21:59:00.002-05:00

Why didn't I find this reference a long time ago? I even put a shell script in my home directory so I could dump releases whenever I needed it. Not any more. I'm moving that script into the cloud!

The Release Naming Matrix lists all of the Solaris releases and their associated maintenance updates. Very useful to anyone who manages a JumpStart / JET server, or maintains an Enterprise update process.

Free the Support Tools Bundle!

2009-12-23T09:48:00.008-05:00

If you aren't already familiar with the Support Tools Bundle, you probably ought to check it out. It contains many very useful tools, at least one of which you absolutely need if you support more than one Solaris server.

I consider many of these tools to be critical components of our current Solaris architecture. As such, updating the tools is a part of our regular patch process. The tools are also integrated in our JumpStart JET templates. And herein lies my frustration.

You can only get the support tools as a bundle. If I want to get the latest SNEEP, I need to download the whole bundle. It's only ~ 40MB, so I can live with that given today's bandwidth. Unfortunately, when you unzip the shiny new file you are faced with something I consider a monstrosity. A shell archive. Why?

The next design flaw we encounter is the extraction method. The shell script exits unless you run it as root. If all I want to do is extract files, why should I be root? This undermines the principle of least privilege if I just need to put files in my home directory, or /var/tmp.

So let's assume we recklessly assume the role of root and execute the shell archive. We are presented with a choice to install or extract the files. Hopefully you want those files in /var/tmp/stb because that's your only choice. Again I ask, Why? Is there some flaw in using gzipped tar balls? I'm not a big fan of using zip, but it accomplishes a similar goal and would be acceptable.

How about a simple plan? Use a gzipped tarball that extracts one directory for each product and an installer in the root. That way I can just extract it and get the product updates into my JET server without having to go through an extra step. If you are skilled enough to know why you need the tools in STB, you can handle a tar.gz file. UNIX has survived the test of time by leveraging simplicity and standards. When we get too fancy we undermine the platform's greatest strengths.

As with any feature (and use of a shell archive is indeed a feature) we should ask the question, what is the value of this extra complexity? I would suggest the answer to that question is "none". Let's whack it and get back to standards, Sun.

Recommendations for you!

2009-12-23T09:33:00.003-05:00

I just have to ask the question... Is anyone else who spends a LOT of time on Sun's web sites getting ready to rip out their fingernails when the site pops up the "Recommendations for you" box and forces you to close it? I'm a paying customer with a valid contract. I don't need to be treated like a mass marketing target.

The Internet provides great new business opportunity, and wild possibilities for creative marketing. But let's hope personal customer relationships which were once important haven't been replaced by marketing shotguns designed to bug 1,000 customers as long as one or two click on the shiny links.

Solaris Patching Made Simple

2009-11-19T09:13:00.003-05:00

Most data centers I've encountered tackled their patching strategy a long time ago. Some may have revisited it when Live Upgrade was introduced, but in general the process doesn't change much once it is created. Why? Patching isn't glorious and exciting. We tend to take it for granted when it works, and "deal with it" when it doesn't. I have to admit I have been guilty of not paying a lot of attention to the guts of Solaris patching for years because all the sites I've worked at had a process and I was busy doing other things. Until now, that is.

I'm currently tasked with designing an Enterprise patching strategy for Solaris servers. What started out as a project I considered pretty dry turned into something I'm really glad to have the opportunity to work on. Why? Because I'm excited about the approach Sun is recommending. I think a lot of the things I used to dislike about patching Sun systems are on their way out.

If you haven't already seen it, Sun's On-Line Learning Center has a new course: Solaris 10 Patching Best Practices(WS-2700-S10). It's free, so even in the current climate of slashed training budgets you can still learn the new way of approaching updates. You should be able to get through it in an average work day and still keep up with email.

For a long time sites with more advanced Sun support have been able to leverage a patch baseline known as EIS, or Enterprise Installation Standards. However, if you didn't have some form of advanced interaction with Sun, or the xVM Operations Center (xVMOC) you don't have regular access to EIS. That left you with maintenance updates/upgrades, recommended clusters, SunAlert cluster, or the "Dim Sum" approach of grabbing an analysis off a current patchdiag.xref and installing the patchlist-du-jour. Which path is the right one?

Here's what you don't want to do: Research all of Sun's white papers and best practices that remain available long after growing long in the tooth. The patching strategies and recommendations are a snarled mess of contradictions that lead to confusion, frustration, and eventually rolling your own because its better than nothing. The good news is that Sun's new training course brings some sanity to the plate.

The high level recommendation from Sun is very straight-forward. Start with the patch/package utilities updates from SunSolve to ensure your patching system is not going to introduce problems. Then install either the latest maintenance upgrade (ideally), or the latest maintenance patch set. This gives you a clean and well integrated baseline. Next, apply the SunAlert recommended cluster to attack any critical fixes that have become necessary since the last maintenance release. The training course implies that Sun plans to merge the Recommended and SunAlert clusters to reduce confusion - another great improvement.

What's great about this approach? First, it's simple. I can grab a few clusters and put together an easy to understand, easy to implement, repeatable process. Second, I'm a huge fan of the use of baselines. By minimizing the use of one-off patches we move to grabbing a baseline which includes the required fix. This means that while I'm introducing more change, I'm introducing a set of changes that went through QA at Sun. That doesn't remove my testing responsibility, but it means I'm standing on the shoulders of giants rather than hoping for the best. Even if I have a phenomenal test suite, it's not going to be as mature or comprehensive as Sun's internal processes. Third, my environment is going to be more consistent. Why? Because all the Solaris 10 servers will eventually end up on the same MU. Today I have similar patch levels on a wild assortment of MUs.

While there's a lot more to the training content, the other big point made throughout is that you need to use Live Upgrade. It's not just a feature you may want to try. It's how you should be patching Sun systems. The catch of course, is that not all systems are configured in a way that lends itself to LU. But the writing is on the wall, and my interpretation tells me I need to start (1) updating our site's reference architectures to move toward being LU-friendly, and (2) begin using LU on those systems which will support it conveniently so we start building site knowledge.

xVM OpsCenter and overbundling

2009-09-29T20:11:00.001-05:00

I've been spending a fair amount of time assessing the patching strategy on my current assignment. My primary focus is on Solaris systems, although there is a Linux population to take care of as well. My recommendation has always been to stick with vendor recommended solutions when it comes to patching because in the Enterprise it's a lot more complicated than clicking on Windows update and hoping for the best.

With that in mind, I browsed over to Sun.com to see what the latest recommendation is. xVM OpsCenter pops out in neon lights. It will even wash my dishes. For what it's capable of, I think it's possible to make an argument that the price is tolerable. Unfortunately, if you are a practitioner of Solaris and need a patching solution you may not need your dishes washed. Then what?

If you aren't going to need full blown provisioning, monitoring, audit, and other cool features you are left with precious little in the way of keeping up on what I call "oil changes". Most of the historical tools are now on their death beds, no doubt to encourage the herd to graze on xVM. Note that I'm only talking about Enterprise level patching which requires some degree of configuration management.

When you dig into xVM you see that there are two options. The basic option does very little that most sites don't already do, although it's wrapped in a nice package. I don't think it's doing anything worth the price of admission at that level though. The advanced package adds what everyone wants: patching. So, you can buy your car with or without tires.

I think this is a bad idea.

Patching is a vital component of the customer experience. It's a way to ensure that Sun doesn't have a CNN moment because a major bug was too difficult to patch and highly visible site didn't get the hole plugged in time. It's also the bane of most admins' existence. It takes a lot of time, causes our customers to suffer downtime, and occasionally takes a server to the happy hunting grounds. To be the best operating system, you need to have a great update strategy.

I have no problem with the xVM framework being an expensive Cadillac, as long as I can still buy a Chevy that does the job. In other words, as long as the Solaris operating environment includes a decent functional framework for patching, then charge all you want for xVM. Today, even with a support contract I don't have access to a proper patching framework from Sun, which means all those third party solutions start getting traction on something that ought to come from Sun.

A basic level of functionality should be part of the environment, so what would the base requirements be? Call it xVM-lite, or call it part of Solaris. Either way, here's a stab at it:

- An on-site proxy option so all hosts don't talk directly to SunSolve. Why not include it in Solaris? This would save Sun bandwidth costs and probably help them to sell some storage.

- Integration with Explorer. Wouldn't it be nice to use that same patching server as the site's Explorer repository for pre-planning patching sessions? We're talking trivial shell scripting here.

- Ability to leverage SunSolve baselines for SunAlert, Security, and Recommended bundles, as well as to manage site-specific custom patch lists.

- Basic auditing of who patched what, and when.

- No GUI necessary. Just a well thought out command line.

What's the precedent? Look at JET. Sun will offer you xVM if you want an Easy Button solution in a GUI, or you can use the JET framework. Personally, I prefer JET. It has nothing to do with the price... I just believe its a well thought out, very reliable design. What I appreciate most, is that when it comes to provisioning I have a choice, and as part of Solaris there is an included option that gets the job done.

Including patching functionality for customers with valid SunSolve entitlements would be a huge improvement in Solaris' usability. Forcing us to buy a 12 course meal when we only need lunch feels like something that happens when you let a marketing department without industry experience make key decisions.

Default routes on 7x00 series Open Storage

2009-09-14T17:27:00.001-05:00

I've been having a very enlightening time with our new 7310 Open Storage array. As a totally new product, and one that hasn't yet reached ubiquity, the normal resources are a bit shy of what I'm used to. Put simply, Google hasn't yet learned how to manage these arrays.

We're in the process of deploying a reasonably complex network scenario on ours using two link aggregations, then layering tagged VLANs for administrative access and the dedicated storage net. Each VLAN is to be redundant via IP Multi-pathing (IPMP). This configuration is just about the only option for high capacity and redundancy when you have multiple VLANs involved.

The good news is, Sun's Open Storage, or Fishworks, has a very well designed command line interface. It's quite comprehensive, and from what I can see, it allows you to lose the GUI and still have a workable device. Which is good, because I managed to decapitate the GUI, or BUI (browser user interface, as Sun calls it).

The kiss of death for the BUI came when I attempted to replace a simple datalink on nge0 with an aggregation of nge0 and nge1. In doing so the default route was removed and not replaced. No problem on the dedicated storage VLAN because it was a non-routed private subnet. Big problem on the public side where I was trying to find the BUI.

It turns out to be a simple problem to fix, but the fix itself is not very intuitive. Because the BUI is dead, you have no choice but to use the CLI. For this reason alone, I strongly encourage anyone using 7x00 series storage to make sure that EVERYTHING you implement in the BUI has an equivalent process via CLI. You never know when you'll need it.

After logging in to the BUI, head over to services --> routing. What you'll probably see is a bunch of routes for each interface, but no default route. To add the default route and reanimate the BUI you will need to create a route as follows:


7310array:configuration services routing > create
7310array:configuration services routing > set family=IPv4
7310array:configuration services routing > set destination=0.0.0.0
7310array:configuration services routing > set mask=0
7310array:configuration services routing > set gateway=192.168.1.1
7310array:configuration services routing > set interface=ipmp1
7310array:configuration services routing > commit

Note, of course, that you'll need to plug in the appropriate gateway and device according to your configuration.

If you are used to adding a default route in Solaris, it isn't all that intuitive to type in 0.0.0.0/0, and it sure as heck wasn't documented anywhere I could find. All's well that ends well though; The change immediately brought back my BUI.


7310array:configuration services routing> show
Properties:
                       = online

Routes:

ROUTE      DESTINATION                      GATEWAY         INTERFACE TYPE
route-000  0.0.0.0/0                        192.168.1.1     ipmp1     static
route-001  10.151.1.0/24                    10.1.1.46       ipmp2     dynamic
route-002  10.151.1.0/24                    10.1.1.47       ipmp2     dynamic
route-003  13.151.249.0/24                  192.168.1.46    ipmp1     dynamic
route-004  13.151.249.0/24                  192.168.1.47    ipmp1     dynamic

7310array:configuration services routing>

All is at balance in the universe. My work here is done.

A farewell to Solaris 9... Already?

2009-06-29T20:29:00.000-05:00

My flock still has a large Solaris 9 community within it. It's hard to believe its already time to start the long march to EOSL, but alas, the announcement is clear, as is the Solaris 9 Transition FAQ. The bell is ringing.

Looking back at other Solaris EOLs I seem to always recall thinking that revision had really grown long in the tooth, and the replacement OS was badly needed. In this case, Solaris 10 has a long list of what I consider "dreams come true" to make you want to upgrade. However, I have a lot of experience watching Solaris 9 boxes take some incredible abuse and keep ticking. In my mind, it may have fewer bells and whistles, but it really did its job well.

So let's raise a glass of Solaris and toast to the legacy of 5.9, and to the enterprise evolution that is 5.10 and OpenSolaris. Cheers!

OpenSolaris on the ThinkPad

2009-06-21T14:06:00.002-05:00

After a long run of just dealing with Windows on my personal latpop I have finally managed to get OpenSolaris running on it. I've had a continuous hassle with my old Wifi card that seemed to only be truly happy under Windows. After a few years of that I took a chance on a new card from eBay and found that it... WORKED!

I started out trying the latest Ubuntu desktop, which has a great library of packages available for it and fantastic integration. Unfortunatley, its driver configuration seemed to work, then send my wifi into a coma after some period of time. Didn't diagnose it. Didn't care to. My laptop isn't a science project for me, it's a tool I want to just work when I dump a new OS onto its disk.

Next stop was the one I was more excited about: OpenSolaris.

root@saphyra:~# uname -a
SunOS saphyra 5.11 snv_111b i86pc i386 i86pc Solaris
root@saphyra:~# wificonfig showstatus
linkstatus: connected
active profile: none
essid: <>
bssid: <>
encryption: wep
signal strength: medium(10)
root@saphyra:~#

Yes, that's right, it's all working. At the moment I'm able to work on a zone / LDAP project from the comfort of my couch enjoying my reborn Thinkpad T23. This thing works like a charm despite being a dinosaur by modern standards.

Before rebuilding it I had pretty much stopped using it because Windows XP was unable to boot in under 5 minutes and it took almost as long to launch an Acrobat Reader session for simple PDF stories I was reading. At the moment everything I do, including web browsing works well and is responsive under only 1GB ram and a 1.1 GHz processor. Sweet.

Solaris Web Console on Windows... Ouch.

2009-06-03T21:06:00.001-05:00

I've been spending quite a bit of time lately running the Sun Directory Service Control Center (DSCC) via the Solaris Web Console (port 6789). When I first started the project I was running Firefox on a Sun workstation. Everything was snappy, the engineer was happy.

Somehow along the way I started using my Windows box to access the console. Still running Firefox I discovered an unbelievable slowness. It takes about three clock minutes to process the initial log in. Once I'm in DSCC everything runs acceptably, but that first login is murder.

One of my co-workers stopped my cube today and suggested I try Internet Explorer. Perish the thought! How could that bloated pig possibly out-perform my Firefox browser? OK, I tried it. He was right.

Internet Explorer provides almost instantaneous response to Webconsole logins while Firefox churns its butter for three minutes. This isn't some dot-net application that's clearly Microsoft slanted. It's a Sun web application. Open stuff that would never have a Microsoft bias. I'm not running dead hardware either; This is on a sweet core-duo 1.83 GHz with 1 GB RAM. Handling an initial log in to Webconsole ought to be cake for this hardware.

My observations are based on stock out of the box configurations, so I'm sure there's some Firefox flag to tweak which will optimize it. It just seems mind boggling that a Sun Microsystems web application would perform exponentially better on Internet Explorer and unacceptably slow on Firefox.

Me? I'm going back to running the browser on my UNIX box. It's way too frustrating trying to be a UNIX Engineer via the Windows platform.

Adding UNIX users to DS6

2009-05-21T13:58:00.007-05:00

I seem to be digging up rants this week. I'm a pretty positive guy, you just wouldn't know it by reading my blog this week. I'm currently working on deploying a fresh Sun Directory Server environment using version 6.3.1. This is to replace an aging 5.2 environment that's ready to retire. Overall I've been very impressed with how much more mature and polished the new version is. A few learning curves to get through, but once I found the right way I was pleased with the product. Unfortunately, today I hit something that just can't be right. Unfortunately, it seems to be confirmed by a bunch of Google hits so I'm not the only one.

When you use Directory Services Control Center (DSCC) to add a user it doesn't provide any of the POSIX fields you need from the posixAccount class. So, your new users pretty much have a user name and a first / last name. No home directory, no user ID, no group ID, and hey... You didn't need a shell did you? Are you kidding me?

The workaround, and I use the term loosely, appears to be adding the record without the necessary information, then editing the record after it is created. You then switch the record to "text mode" and manually insert the following lines into the editable section:


objectclass: posixAccount
loginshell: /bin/ksh
homeDirectory: /home/username
uidNumber: 1234
gidNumber: 10
gecos: John Smith

Ok, so that gets us an account, but isn't it moderately annoying to have to go through all that? Why in the name of Scott McNealy didn't anyone make the wild and unruly assumption that once in a freakishly rare moon someone might use DSEE to centralize the administration of their Solaris users. After all, NIS and NIS+ are deprecated and no one digs local file editing. So, wouldn't that assumption have been somewhere around the top ten for their user requirements?

I did a quick dig to see if I could find a simple configuration file that specified what schema object(s) were used when adding a user attribute, or populating the "common objects" menu, but came up dry. I'll have to a deeper search when time allows. I know it's sitting in some XML file somewhere, but there's more than a few to look through.

So what are my options? Well, there's always the LDIF plan. Which is pretty much useless to the folks who typically manage user account maintenance. Way too error-prone. It's also pretty aggravating for day to day administration. LDIF is pretty much intended for batch loading and sitting behind various automations. I shouldn't need to write an automation solution to add simple UNIX accounts since that capability was standard in the 5.x Directory Servers.

Another option is to use Sun's Directory Editor which is part of DSEE. This path leads to some entertainment as well. If you try to download DSE, the web form will not let you select a platform, and thus prevents you from downloading the component. So, you need to download the ZIP distribution of DSEE instead. Then you just need to deploy Sun's Application Server, or Tomcat. Yeah, just what I needed - another component. Doesn't webconsole already sit on an app server? The best part is, DSE is left over from the 2005Q1 JES distribution from what I can see. Obviously, not a high priority for maintenance. Very encouraging indeed.

So, while Sun's Directory Server continues to be a phenomenal data repository it appears that Sun views its user base as being application / identity developers rather then the legions of system administrators / engineers out there trying to implement a well supported central management strategy. Come on guys and gals, it's not that hard to make us happy. Lose the web 2.0 bling and give us core functionality. Hmm, then add the bling back in! The DSCC interface really is very nice, but what good is a hot car without a steering wheel?

ldaplist: Why so much white space?

2009-05-19T20:48:00.001-05:00

Sometimes little things drive me nuts. So nuts, it's almost tempting to get into some code and make it right. Of course, that would have absolutely no return on investment for a singificant amount of hassle, but I have to admit I think about it from time time. What has rubbed me the wrong way?

The complete lack of either [1] aesthetic engineering, or [2] use of traditional 80x24 console screens as experienced by the developers of the ldaplist utility. It's as if someone had just finished a grade school term paper when they wrote the output format. Here's the default output:


testbox# ldaplist
dn: cn=Directory Administrators, dc=example,dc=com

dn: cn=nsAccountInactivationTmp,dc=example,dc=com

dn: ou=Timezone,dc=example,dc=com

dn: automountMapName=auto_home,dc=example,dc=com

dn: automountMapName=auto_direct,dc=example,dc=com

dn: automountMapName=auto_master,dc=example,dc=com

dn: ou=projects,dc=example,dc=com

dn: ou=group-ldap,dc=example,dc=com

dn: automountMapName=auto_shared,dc=example,dc=com

dn: ou=SolarisAuthAttr,dc=example,dc=com

dn: ou=SolarisProfAttr,dc=example,dc=com

dn: ou=people,dc=example,dc=com

dn: ou=group,dc=example,dc=com

dn: ou=rpc,dc=example,dc=com

dn: ou=protocols,dc=example,dc=com

dn: ou=networks,dc=example,dc=com

dn: ou=netgroup,dc=example,dc=com

dn: ou=printers,dc=example,dc=com

dn: ou=hosts,dc=example,dc=com

dn: ou=services,dc=example,dc=com

dn: ou=ethers,dc=example,dc=com

dn: ou=profile,dc=example,dc=com

dn: ou=aliases,dc=example,dc=com

Forty-seven lines? That takes up WAY too many lines and provides no value for the white space incurred, not to mention requiring me to scroll my terminal window when I'm on the console. This actually annoys me enough that I run the command this way:


testbox# ldaplist | sed '/^$/d'

dn: cn=Directory Administrators, dc=example,dc=com
dn: cn=nsAccountInactivationTmp,dc=example,dc=com
dn: ou=Timezone,dc=example,dc=com
dn: automountMapName=auto_home,dc=example,dc=com
dn: automountMapName=auto_direct,dc=example,dc=com
dn: automountMapName=auto_master,dc=example,dc=com
dn: ou=projects,dc=example,dc=com
dn: ou=group-ldap,dc=example,dc=com
dn: automountMapName=auto_shared,dc=example,dc=com
dn: ou=SolarisAuthAttr,dc=example,dc=com
dn: ou=SolarisProfAttr,dc=example,dc=com
dn: ou=people,dc=example,dc=com
dn: ou=group,dc=example,dc=com
dn: ou=rpc,dc=example,dc=com
dn: ou=protocols,dc=example,dc=com
dn: ou=networks,dc=example,dc=com
dn: ou=netgroup,dc=example,dc=com
dn: ou=printers,dc=example,dc=com
dn: ou=hosts,dc=example,dc=com
dn: ou=services,dc=example,dc=com
dn: ou=ethers,dc=example,dc=com
dn: ou=profile,dc=example,dc=com
dn: ou=aliases,dc=example,dc=com

Ahhh, that's better. And at 1/2 the screen real estate I rarely need to scroll. Come on, what on Earth would motivate someone to add extra newlines to an output like this? Next thing you know they'll offer CSS templates so your output can have the right "user experience" complete with standard fonts.

Ok, I feel better now... Really. I'm ok.

#@$@#$# Spammers

2009-05-05T21:34:00.006-05:00

Gotta love those spammers. I wish I could talk the way they write - it would be entertaining at a party to sound like one of the Cylon Hybrids. I'm going to go out on a limb and assume there at least a few Sci-fi fans out there reading a blog like this one.

The real purpose of this post is to apologetically announce that I've turned on comment moderation to keep everything clean after a long wave of spammers hit me. I'm not big on censoring, so rest assured that if you post a rational comment I will be happy to release it and continue encouraging dialog.

JET and the Recommended Cluster

2009-05-05T20:29:00.006-05:00

JET is bugging me. I'm a sort of pack rat when it comes to installation media, and that extends to patch sets. Hey, you never know you might get a request to Jumpstart Solaris 2.4, right? Ok, I'm not really that bad.

But, you may well be using a certain recommended cluster for a certain OS, and then need to jump a box to test the next recommended cluster right? Surely, it's not necessary to make a global change to your production server build configuration when implementing a test cluster?

As far as I can tell, base_config's use of recommended clusters is not handled in a manner that encourages good revision management. For each OS major revision (e.g., 10, 9, 8) there can be one cluster. For example, in today's JET software if we were using /export/install as our JET media base there would be a directory called /export/install/patches. Under that we can store one patch cluster for each major OS revision:


/export/install/patches/10_Recommended
/export/install/patches/9_Recommended
/export/install/patches/8_Recommended

That works nicely until the next recommended cluster is released. At that point in time you can no longer have a repeatable build process because you need to replace the single instance of each OS with the new cluster. Not a good plan. We want our patch configuration to be configured in the template so that it's managed, and can be under source code control. Managing patch configurations outside the template is pretty much impossible to audit.

Here's an alternative approach I think would be a step in the right direction: Create a hierarchy to organize recommended clusters:


/export/install/patches/recommended/5.10/sparc/2009-04-22
/export/install/patches/recommended/5.10/sparc/2009-01-foo
/export/install/patches/recommended/5.10/x86
/export/install/patches/recommended/5.9/sparc/2009-04-22
/export/install/patches/recommended/5.9/x86/2009-01-foo

We need to be able to add recommended clusters in the same way we add other products. I'd like to see a new command called "list_recommended_clusters" which would have an output something like this:


# list_recommended_clusters
Version                              Location
------                              ---------------
5.10_sparc_200901         /export/install/patches/recommended/sparc/5.10_sparc_200901
5.10_sparc_200902         /export/install/patches/recommended/sparc/5.10_sparc_200901
5.10_sparc_200903         /export/install/patches/recommended/sparc/5.10_sparc_200901
5.9_sparc_200901           /export/install/patches/recommended/sparc/5.10_sparc_200901
5.9_sparc_200902           /export/install/patches/recommended/sparc/5.10_sparc_200901
5.9_sparc_200903           /export/install/patches/recommended/sparc/5.10_sparc_200901

These clusters could then be specified in the JET template using a variable like base_config_recommended_cluster. In addition, the check routine used during a make_client invocation would ensure that the directory exists, and perhaps ensure that each patch on the patch_list was represented. Bingo! Now we can use good revision control to manage the integration of patch clusters with our server build process.

But I think we can take it one step farther. How about we add the ability to include arbitrary patch sets? Here's a first cut at how it could work: We start by creating a patch repository. Say, /export/install/patch_repo. Under that directory we may have subdirectories for 5.10, 5.9, etc. Patches are simply added to that directory by copying them into place. Nothing fancy. The nice thing about this approach is in its economy of space.

The recommended clusters will have a lot of overlap between them, with the potential for storing the same patch in many different directories. By having one patch repository, we simply store each necessary patch one time and refer to it in a patch_order file. It would be trivial to write a few scripts that could operate or query on a set of patches according to a certain patch list, or perhaps cull out patches not referenced in any current patch lists. I could give or take this feature. There are some good arguments to be made for just storing each patch set and ignoring the storage space. I'm ok with either approach, and even happier if this flexibility were accounted for.

Having established a patch repository, we now need a place to manage patch lists. These would be in typical patch_order formatted lists; No need to reinvent the wheel. Each would need to be named with a unique identifier. For example,


patch_order_5.10_2009q1
patch_order_5.9_dev-servers
patch_order_5.10_test01

These patch lists could then be specified within base_config as an alternative to using the Recommended clusters. Why?

The site has a known incompatibility with a patch or two in the common cluster.

The site wants to deploy other patches in the early part of the install as part of a managed list rather than manual entries in a template (e.g., custom_patches).

Using these lists allows a configuration to be frozen in time for configuration management, and allows a convenient record of exactly what a server was deployed with

I think these would be some very beneficial enhancements to the JET framework. I'd like to work on some of them, but I wanted to get the idea out there before I got wrapped up in something else and forgot about it. I'd be interested in hearing any thoughts on this topic - especially if someone has a better idea!

By the way, I do know about EIS baselines. But I think it's pretty rare for any enterprise to never have need for managing custom patch sets. It would be great if JET could come through with some help in this space.

Finding those pesky HBA cards

2009-03-30T18:42:00.000-05:00

I was given a mission yesterday of finding how many host bus adapter (HBA) cards were in a set of servers. At first glance it seemed like an easy task, but then I remembered that Solaris servers never had a nice convenient output to tell us what card is in what slot in a way that normal humans could benefit from. It's sort of like playing charades; You have to put together a bunch of clues. Here's how I went about it.

The first place I stopped was prtdiag. That's my go-to configuration summary in most cases. Here's a subset of what I saw (probably going to look bad unless your browser is really stretched...):


                                Bus  Max
            IO   Port Bus       Freq Bus  Dev,
FRU Name    Type  ID  Side Slot MHz  Freq Func State Name                              Model
----------  ---- ---- ---- ---- ---- ---- ---- ----- --------------------------------  ----------------------
/N0/IB6/P1  PCI   25   B    4    100  100  1,0  ok    SUNW,qlc-pci1077,141.1077.141.2/+ QLA2462
/N0/IB6/P1  PCI   25   B    4    100  100  1,1  ok    SUNW,qlc-pci1077,141.1077.141.2/+ QLA2462
/N0/IB6/P1  PCI   25   A    6    100  100  2,0  ok    SUNW,qlc-pci1077,141.1077.141.2/+ QLA2462
/N0/IB6/P1  PCI   25   A    6    100  100  2,1  ok    SUNW,qlc-pci1077,141.1077.141.2/+ QLA2462
/N0/IB7/P1  PCI   27   B    4    100  100  1,0  ok    SUNW,qlc-pci1077,141.1077.141.2/+ QLA2462
/N0/IB7/P1  PCI   27   B    4    100  100  1,1  ok    SUNW,qlc-pci1077,141.1077.141.2/+ QLA2462
/N0/IB7/P1  PCI   27   A    6    100  100  2,0  ok    SUNW,qlc-pci1077,141.1077.141.2/+ QLA2462
/N0/IB7/P1  PCI   27   A    6    100  100  2,1  ok    SUNW,qlc-pci1077,141.1077.141.2/+ QLA2462
/N0/IB8/P0  PCI   28   A    3    100  100  1,0  ok    SUNW,qlc-pci1077,141.1077.141.2/+ QLA2462
/N0/IB8/P0  PCI   28   A    3    100  100  1,1  ok    SUNW,qlc-pci1077,141.1077.141.2/+ QLA2462
/N0/IB8/P1  PCI   29   B    4    100  100  1,0  ok    SUNW,qlc-pci1077,141.1077.141.2/+ QLA24
/N0/IB8/P1  PCI   29   B    4    100  100  1,1  ok    SUNW,qlc-pci1077,141.1077.141.2/+ QLA24
/N0/IB8/P1  PCI   29   A    6    100  100  2,0  ok    SUNW,qlc-pci1077,141.1077.141.2/+ QLA2462
/N0/IB8/P1  PCI   29   A    6    100  100  2,1  ok    SUNW,qlc-pci1077,141.1077.141.2/+ QLA2462
/N0/IB9/P1  PCI   31   B    4    100  100  1,0  ok    SUNW,qlc-pci1077,141.1077.141.2/+ QLA2462
/N0/IB9/P1  PCI   31   B    4    100  100  1,1  ok    SUNW,qlc-pci1077,141.1077.141.2/+ QLA2462
/N0/IB9/P1  PCI   31   A    6    100  100  2,0  ok    SUNW,qlc-pci1077,141.1077.141.2/+ QLA2462
/N0/IB9/P1  PCI   31   A    6    100  100  2,1  ok    SUNW,qlc-pci1077,141.1077.141.2/+ QLA2462

Of course, there were a great many other lines, but this is what the Fibre Channel card lines look like. Of course, I picked this out because I recognized the QLC driver. Not sure what someone would do if they didn't know that. In this case, there were 18 lines with this output. This indicates there are 9 cards because each slot was represented twice (two ports on each device). This was supported by me being reasonably sure that we had dual-ported cards on this server.

The next place I looked for confirmation was prtconf. This output tends to be more complete, but far more verbose, and generally annoying to get summaries from. To be more precise, the output contains a lot of information...


foobox: prtconf -v | wc -l
    9358

That was a complete moment of frustration. The output was too busy and didn't look helpful. Note to self: Why is this not simple? I'm looking for a simple answer, not an excuse to write a Nawk script. No matter how I skinned the output I ended up with 18 matching lines. I'm right back at the prtdiag output.

My last stop was a more obscure one, but a tool which is very helpful: prtpicl. Ok, I'll admit, this one is still ugly.


foobox: prtpicl -v | wc -l
   11183

But, at this point I just wanted to get it done, so I dug in a little bit and checked out what it had to say. The easily parsed format provides a convenient Vendor ID and Device ID for each connected device. That's good news because those PCI IDs are easy to look up on the Internet. Knowing our site standards I was able to identify the Vendor ID of the cards we order and look for them:


foobox: prtpicl -v | egrep -e '0x1077' | grep -v subsystem
                  :vendor-id     0x1077
                  :vendor-id     0x1077
                  :vendor-id     0x1077
                  :vendor-id     0x1077
                      :vendor-id         0x1077
                  :vendor-id     0x1077
                  :vendor-id     0x1077
                  :vendor-id     0x1077
                  :vendor-id     0x1077
                      :vendor-id         0x1077
                  :vendor-id     0x1077
                  :vendor-id     0x1077
                  :vendor-id     0x1077
                  :vendor-id     0x1077
                  :vendor-id     0x1077
                  :vendor-id     0x1077
                  :vendor-id     0x1077
                  :vendor-id     0x1077
                  :vendor-id     0x1077
                  :vendor-id     0x1077

Please, no comments about how this could be done in a Perl one-liner. We're going to ignore the indented items because they belong to a different hierarchy of data. If we count up the leftmost indented items we see again there are 18 instances of PCI devices with the relevant vendor ID. So, is this a port 0, port 1, deal which requires me to divide by two?

Again, I'm not sure because the output is cryptic. Yes, I know there are ways to make sense of it with hardware knowledge, but let's assume we're dealing with an average SA, and not a device driver developer.

The last tool I tried is a device path decoder which is sort of an unsupported toy developed inside Sun. I don't know where we obtained it, but we happened to have it here so I ran the path_to_inst file through it. What did it tell me? That I had nine of the HBA cards in the box. It had a very simple, easy to read format which used indentation to clearly show the system's layout.

So, it looks like prtdiag was the most direct way to surmise an answer. I would like to see Solaris give me a hardware diagnostic which provides a physical model rather than a logical one. Just tell me there is a card in slot 4 with its vendor / device ID. I don't care to sort out its ports. I just want the device. There are plenty of other tools which provide the logical view, or device driver hierarchy.

Solaris LDAP Integration Void

2009-02-09T17:30:00.003-05:00

Yikes, that was a harsh post title from a self-proclaimed advocate of Sun's products. I can't count the number of times I've had conversations with people about two related topics: First, how critical it is that sites begin to adopt LDAP and stop managing boxes independently. Second, how immature the administrative side of Sun's LDAP is.
It appears that Ben Rockwood, a much respected voice in the OpenSolaris community, has observed the same.

These topics each deserve a series of posts because they are complex. I mean it. Until you've tried, its hard to understand the documentation dichotomy of Sun's Directory Server Enterprise Edition. The best way I can describe it would be to imagine you have been asked to learn English given a dictionary as your only resource.

There is phenomenal depth to the documentation in form of resource guides. In other words, once you "get it" you can do anything with Sun's documentation. The number of concepts you need to master to deploy LDAP in an Enterprise is staggering, and the number of real-world cases available from Google is small. You really need a few weeks of Instructor Lead Training, but how many companies are on that track these days? Not too many. There are a few outdated books as well, but they only get you to the starting gate for a basic environment.

So now let's assume that you have learned the system and properly architected your Directory Servers. Your next challenge is managing the data. I worked on a project which integrated Oracle instances with Solaris Resource Manager (SRM). The central LDAP project ID repository allowed us to ensure no Project IDs were duplicated around the environment, and minimized the amount of management associated with application migrations. Seems simple, right?

The first issue we encountered was that there is no facility for entering records into the Directory. Don't even talk to me about the documented solution of using Sun Management Console (SMC). It's cute for local files, but it is worthless for naming services, and even Sun's solution center thinks its insane to try using it. No, really. I opened a case, and they asked my why I would ever try to use it.

There should be a set of CLI interfaces for managing this data. Period. Its a simple thing, and by now the Directory Services have been around long enough that this is sorely over due. They should follow the standard usage model that tools like useradd or usermod provide. People understand this, and the precedent should be respected.

The only other option is the Directory Editor. You pick a third party one, or a Sun one. But in the end you are responsible for reverse-engineering whether a directory attribute is a list, or a collection of attributes. This is not appropriate. For standard Solaris maps like netmasks, auto_master, hosts, etc. there should be interface dialogs which provide reasonable levels of sanity checking. I shouldn't need to scan through cryptic attributes. What's even more scary is the idea of handing over a full directory editor to say, someone on the first tier help desk who may not fully understand how terrifying it would be to make the wrong right-click.

This was a bit of a rant, but it is primarily intended to scream out in support of Ben's post. This is a huge opportunity to improve Solaris' administrative scalability and I think all too often LDAP projects get dropped during internal evaluations because the local staff has too many issues getting it working.

Kerberos and the SCSECA Curriculum

2008-11-20T16:27:00.001-05:00

I remember when I first took the Network Administrator (SCNA) Exam back in the Solaris 7 days, and I was frustrated by the depth of NIS / NIS+ content. NIS was widely used back in the day, and fairly intuitive. However, NIS+ was a bit of a niche, and its use dropped off like a rock on the Solaris 7 era. I think people really failed to enjoy all those key exchanges and inherant troubleshooting.

Long after NIS and NIS+ services were deprecated by the coming promise of LDAP their place in the curriculum was maintained. But of course, I learned it and passed the exam. Having recently passed the SCNA again for Solaris 10 I was pleased with its content. I was convinced that Sun had brought the canon into the modern era. Good stuff. But just when I thought it was safe...

I'm now finishing up my prep for the Sun Certified Security Administrator (SCSECA) and am finding myself frustrated by the presence of Kerberos on the SCSECA test curriculum.

Will the number of sites using Kerberos please raise their hands? Ah ha! We now know the answer to the question, "What is the sound of one hand clapping?". Ok, it's more than one, I know. It's not very many though... I'm really hoping that when I sit down to the test the questions are written to a depth proportional to the installed base.

I think there's a lot of great content that can be included on a Solaris security exam in place of esoteric solutions like Kerberos. I'd like to see the bulk of the SCSECA content focus on an SA's ability to implement and evaluate impact of the various checks in the CIS Solaris 10 Benchmark. The key of course is "evaluate" more than "implement." I'm amazed at how many people flip through checklists without understanding the implications of these reconfigurations, and I think the SCSECA content is a great opportunity to fix that problem.

But that's ok. I'll brush up on my Kerberos and maintain my historical acumen.

Capturing output from format

2008-09-23T21:21:00.000-05:00

Ever need to obtain the contents of the format command for other processing in a shell or Perl script? It's fairly simple to do, but the command's behavior is a bit counter-intuitive and makes for an interesting case.

When you run the format command it lists the disks, then issues a prompt asking you to select one of the enumerated devices. It does not provide an option for existing the command at that point. So, we need to appease this interface oddity by passing a "0" to the command, which will arbitraily select the first disk from the list. This should work in any case excepting a diskless client.

The format command looks for its input from a file descriptor known as STDIN, or standard input. The way we queue up entries in STDIN is using the good old echo command. Altogether it looks like this:


root@testbox# /usr/bin/echo 0 | /usr/sbin/format
Searching for disks...

AVAILABLE DISK SELECTIONS:
       0. c1t0d0 
          /pci@7c0/pci@0/pci@1/pci@0,2/LSILogic,sas@2/sd@0,0
       1. c1t1d0 
          /pci@7c0/pci@0/pci@1/pci@0,2/LSILogic,sas@2/sd@1,0

Specify disk (enter its number): selecting c1t0d0
[disk formatted]
/dev/dsk/c1t0d0s0 is part of SVM volume stripe:d10. Please see metaclear(1M).
/dev/dsk/c1t0d0s1 is part of SVM volume stripe:d11. Please see metaclear(1M).
/dev/dsk/c1t0d0s5 is part of SVM volume stripe:d15. Please see metaclear(1M).
/dev/dsk/c1t0d0s7 contains an SVM mdb. Please see metadb(1M).

FORMAT MENU:
        disk       - select a disk
        type       - select (define) a disk type
        partition  - select (define) a partition table
        current    - describe the current disk
        format     - format and analyze the disk
        repair     - repair a defective sector
        label      - write label to the disk
        analyze    - surface analysis
        defect     - defect list management
        backup     - search for backup labels
        verify     - read and display labels
        save       - save new disk/partition definitions
        inquiry    - show vendor, product and revision
        volname    - set 8-character volume name
        !     - execute , then return
        quit
format>
root@testbox#

The problem with this is it captured more than we want in the output. We don't need a menu, and we don't need to know about selecting c1t0d0 since that's already enumerated in the first disk list. To edit this stream of test, we'll need a stream editor... Can you guess what it's called? Sed. Let's modify the command to squelch out some of the noise.


root@testbox# /usr/bin/echo 0 | /usr/sbin/format 2>&1 | sed -e '/^Specify disk/,$d'

AVAILABLE DISK SELECTIONS:
       0. c1t0d0 
          /pci@7c0/pci@0/pci@1/pci@0,2/LSILogic,sas@2/sd@0,0
       1. c1t1d0 
          /pci@7c0/pci@0/pci@1/pci@0,2/LSILogic,sas@2/sd@1,0
 
root@testbox#

That's better!

Google Blog Search - Algorithm Insanity?

2008-09-23T20:53:00.000-05:00

This isn't really Solaris related as much as computer science related, but today I experienced a very strange behavior from the great Google. One of my hobbies is archery, and I live in the Rochester, NY area. So, I was searching blogs on Google with the following string: "rochester NY archery". Seems pretty benign, right? Apparently, it's more like looking for Dick's Sporting Goods at www.d_c_s.com.

The number one hit for Rochester, NY archery is: "Club Intoxicated Girls". Actually, almost all of them were blog spam hits. That's incredibly frustrating. It's also a bit surprising because in my experience the SPAM heuristics in Gmail are second to none. Interesting times we live in.

A quick way to check UDP ports on Solaris

2008-09-04T21:00:00.003-05:00

Ever need a quick way to check what UDP connections are active on your Solaris server? I recently had to validate a scanner's report that we had an unnecessary service running on UDP port 177. Unfortunately, Solaris does not yet ship with lsof as a standard tool, so it requires the use of netstat(1M).


root# netstat -an -P udp

UDP: IPv4
   Local Address        Remote Address      State
-------------------- -------------------- ----------
      *.123                               Idle
127.0.0.1.123                             Idle
13.129.6.168.123                          Idle
      *.111                               Idle
      *.*                                 Unbound
      *.32771                             Idle
...
Active UNIX domain sockets
Address  Type          Vnode     Conn  Local Addr      Remote Addr
6001f6c18f8 dgram      6001fa6eb40 00000000 /var/vx/isis/vea_portal 
6001f6c1c88 stream-ord 6001f6a4180 00000000 /var/run/.inetd.uds

Not too painful at all. Turns out that scan must have been an intermittent service, or a false-positive because I didn't turn up any trace of it, but it did give me a chance to reacquaint myself with a useful incantation of netstat.

Solaris available on Dell Servers

2008-08-28T19:32:00.000-05:00

I have to admit I was surprised, albeit pleasantly, when I saw a post on c0t0d0.org indicating that Solaris 10 is now an order option on certain Dell servers.

Of course Solaris has been available on best in class Sun x64 hardware for some time now, but the mainstream world doesn't follow Sun's products in the same way they do Dell. In a sense, I think this going to be a better channel for advertising than revenue, although I really hope its beneficial for both.

There will now be a lot of Dell customers who see Solaris on their order options, and I believe this will make a larger group of consumers think about Solaris where previously they had no occasion to.

Regardless of the outcome, it feels good to see Sun opening up a new channel and I sincerely wish both Dell and Sun success with it.

Repairing file permissions: pkgchk -f

2008-08-05T20:41:00.001-05:00

I was recently testing a process for repartitioning root disks which requires booting on an alternate disk, then copying and restoring data to the primary disk. I used ufsdump for this because of its excellent handling of some of UFS' nuances. The downside is that if you don't use ufsrestore frequently, you will be asked a nonintuitive question at the end of the operation. Yes, yes, a quick trip to the man pages would have helped. Unfortunately, I was being a bit cavalier at the time, and since it was a lab machine I thought little of it.

Turns out I should have thought a little harder. I ended up restoring data wonderfully, but pretty much toasted the system because all files were owned by root, with group other. Good in some places, not so good in others. Prognosis: rejump the server? Naah.

Sun published a Blueprint way back in 1999 which I think all system administrators should read. Someday this information will save your butt. Repairing File Ownership and Mode by Richard Elling.

I had forgotten about the "-f" option to pkgchk, which is described in this document. This option will attempt to correct any file system attributes such that they align with the registry's entries. This won't help things outside the OS, but it will restore sanity to an OS full of toasted attributes. The recommendation is to boot CD-ROM or network, then mount the root file system on /a, and run a pkgchk -R /a -f. I found that simply booting single-user and running pkgchk -f did the trick. Your mileage may vary.

I don't think there would have been any other practical approach short of re-jumping the box to restore all of the lost attributes, so it is with great enthusiasm that I recommend keeping "pkgchk -f" in your tool bag.

nslookup: Rumors of my death have been greatly exaggerated

2008-07-30T20:59:00.000-05:00

I'm currently working through the Sun Learning Connection (on-line / web-based training) to review the curriculum for my Sun Certified Network Administrator (SCNA) update examination. I'm a big fan of Sun's web based training as a study tool because it has always done a great job of preparing me for my certifications.

One of the interesting pieces of content I passed through indicated that in Solaris 10 the nslookup command has been deprecated. Dig is now included in Solaris, and according to the WS-3002-S10 course, is the preferred tool for querying DNS information. I remember when this same fascination with dig sped through the Linux distributions I used to use as well. I would type "nslookup _____" and the OS would dutifully reply that I really ought to be using dig, but here's my reply. You know what? I don't need my OS to tell me what I want. I just need it to do what I ask.

Fortunately, despite the menacing overtone of this training curriculum's message, I have yet to find a warning message come out of my Solaris servers. Dig is indeed included in Solaris, which is a great thing. It is certainly a more detailed tool for diagnosing DNS queries, and I'm thrilled to see Solaris inclusion of industry standard DNS tools.

But let's return once again to that hint about deprecating nslookup... Let's say I just want to see what the name service is returning for a given lookup. I'm just looking for right or wrong, not a detailed and cryptic report to gaze through. Here's the dig command and output for a reverse-lookup:


testbox# dig @192.168.1.2 foo.edu -x 192.168.2.1

; <<>> DiG 9.2.4 <<>> @192.168.1.2 two.edu -x 192.168.2.1
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1174
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0


;; QUESTION SECTION:
;two.foo.                       IN      A

;; AUTHORITY SECTION:
foo.edu.                10800   IN      SOA sys22.foo.edu. 
root.sys22.foo.edu. 2005010101 3600 1800 6048000 86400


;; Query time: 11 msec
;; SERVER: 192.168.1.2#53(192.168.1.2)
;; WHEN: Wed Jan 12 08:07:30 2005
;; MSG SIZE  rcvd: 72


;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR,
id: 1982
;; flags: qr rd ra; QUERY: 1, ANSWER: 1,
AUTHORITY: 2, ADDITIONAL: 0


;; QUESTION SECTION:
;1.2.168.192.in-addr.arpa.      IN      PTR


;; ANSWER SECTION:
1.2.168.192.in-addr.arpa. 86400 IN      PTR    sys21.foo.edu.

;; AUTHORITY SECTION:
2.168.192.in-addr.arpa. 86400   IN      NS     sys23.foo.edu.
2.168.192.in-addr.arpa. 86400   IN      NS     sys22.foo.edu.


;; Query time: 6 msec
;; SERVER: 192.168.1.2#53(192.168.1.2)
;; WHEN: Wed Jan 12 08:07:30 2005
;; MSG SIZE  rcvd: 109

Whoa. That was a lot to digest. Now, REALLY QUICK... Go find out what the hostname is for the queried IP. Yeah, sorry, you took too long tracing through all that. Now lets' look at the nslookup approach:


testbox# nslookup 192.168.2.1
Server:         192.168.2.1
Address:        192.168.2.1#53

1.2.168.192.in-addr.arpa        name = sys22.foo.edu.

Yep, that's a bit more efficient.

The moral of the story is that UNIX includes many tools, each of which serves a specific purpose it is (usually) optimized for. I'd hate to think that my future basic DNS queries would be serviced by unwieldy dig output. I'm thrilled that if I run into a more serious DNS issue I can call on dig to help me, but replacing nslookup completely with dig would be like replacing gEdit with OpenOffice Writer. The completely wrong philosophy.

To borrow from Mark Twain, "The rumours of nslookup's death are greatly exaggerated!"

Setting Terminal Title

2008-07-07T09:59:00.000-05:00

Now that I'm spending a lot of time working on zones I've found myself needing to keep my desktop better organized so I can quickly find the zone and host I need amongst a slew of terminals. I like to keep things simple, so I went with a little shell script that sets the title of a window on demand. Here's what I ended up with:


if [ -x /bin/zonename ]; then
   # if we are on a box that supports zones, include zone info in title
   /bin/echo "\033]0;`/bin/hostname` [`/bin/zonename`]\007\c"
else
   # handle non-zone platforms by omitting the zone name
   /bin/echo "\033]0;`/bin/hostname`\007\c"
fi

This will update the gnome-terminal, or xterm title bar with "hostname [zonename]" on a platform that supports zones (as determined by the presence and executable attribute of /bin/zonename). If a host does not have that executable available and executable (such as pre-Solaris 10) it will simply print the hostname.

True to traditional UNIX' abbreviated nature I named the script stt, short for "set terminal title" and placed in my $HOME/bin directory for convenience. Now when I log in to a host, if I'll be in there for a while I just type 'stt' and my window is properly adorned.

A simple extension of this script would be to include the function in a shell's profile and inject it into the PS1 variable so that it is executed after each command. This would allow the title to update dynamically with each command. Haven't messed with that approach yet as this has scratched my itch quite well.

No space left on device? (metainit)

2008-06-17T15:41:00.000-05:00

Here comes another rant about error messages. I was rebuilding a server today that uses SVM to manage som SAN storage which gives a home to four very nice Solaris zones. I began by issuing a metainit command to build a concat/stripe device from these two SAN devices...


testbox{lvm}$ sudo metainit -f d100
metainit: testbox: /etc/lvm/md.tab line 72: c4t6006048000018775125753594D433742d0s7: No space left on device

What?!?! I took a quick look at partitioning...


Part      Tag    Flag     Cylinders         Size            Blocks
  0 unassigned    wm       0                0         (0/0/0)            0
  1 unassigned    wm       0                0         (0/0/0)            0
  2     backup    wu       0 - 56653       25.93GB    (56654/0/0) 54387840
  3 unassigned    wm       1 -     3        1.41MB    (3/0/0)         2880
  4 unassigned    wm       4 - 56653       25.93GB    (56650/0/0) 54384000
  5 unassigned    wm       0                0         (0/0/0)            0
  6 unassigned    wm       0                0         (0/0/0)            0
  7          -    wu       0 - 56653       25.93GB    (56654/0/0) 54387840

Ok, so the partition exists. What the heck is wrong?

In my absent minded hurry to get this trivial task completed I made an undiscipined assumption that both devices which are to comprise d100 have the same underlying VTOC. It turns out they did not. One of them was set up to use slice 4, and the other slice 7.

So, I issued a quick command to synchronize them using the traditional prtvtoc | fmthard tango, then edited the /etc/lvm/md.tab file to accomodate the s4 slice when defining d100. This time it worked nicely.

But come on, "no space left on device?" What kind of an error message is that? How about something more like, "specified slice does not exist." Technically, a storage device of size zero would have no space available, but there sure are more direct ways to express that concept.

The Evolution of Email

2008-06-05T17:59:00.000-05:00

Have you ever stopped to ask yourself what benefits have been derived by the evolution of email from the days of ASCII text to our modern world where Microsoft Word can act as the email editor?

Fortunately I don't need to ponder this question any longer. Today I received an email which simply would not have had the same impact back in the old days of low-tech correspondence.

The email started out with the following, which is a direct quote:

Starting IMMEDIATELY - ZERO TOLERANCE for any and all non compliance of the following process!
...

It looks pretty menacing in ASCII text, but thanks to Microsoft Exchange and its mind-blowing capabilities to allow more effective self-expression I was able to receive that motivational phrase in a 24-point underlined red font.

I have to admit, it's difficult to fully realize the gravity of the phrase without gratuitous aesthetic enhancement. Let's face it, it would take a PowerPoint attachment to more effectively intimidate me.

The Unconventional Explorer

2008-06-04T17:33:00.000-05:00

The habit of Sun's explorer dumping output to /opt/SUNWexplo/output makes me wince a bit. In all fairness, I think the documentation could be seen as technically inconclusive, but in spirit I believe a more correct solution is not difficult to derive.

Consulting the Solaris 10 System Administration Guide: Devices and File Systems we find a concise chart of default Solaris file systems and their raison d'etra. Three specific entries jump out at me as being relevant to this topic:

/opt: Optional mount point for third-party software. On some systems, the /opt directory might be a UFS file system on a local disk slice.

/var: System files and directories that are likely to change or grow over the life of the local system. These include system logs, vi and ex backup files, and uucp files.

root(/): The top of the hierarchical file tree. The root (/) directory contains the directories and files that are critical for system operation, such as the kernel, the device drivers, and the programs used to boot the system. The root (/) directory also contains the mount point directories where local and remote file systems can be attached to the file tree.

Considering these practices, it makes perfect sense that explorer is installed in /opt/SUNWexplo. So far, so good. On the systems we deploy at my current place of employment, the /opt file system is part of the root file system, which means that Explorer is dumping output at ~ 5mb per shot onto the root file system.

All things considered, it's pretty benign considering we use either 72 or 146 GB boot drives. But as Solaris Jedi, we look to the harmony and availability of the system, and Explorer is definitely creating a disturbance in the force by dumping volitile files into a subdirectory within /opt. What if someone wrote a script to manage the contents of that output directory and made a little error in their code? What file system would you want it compartmentalized within? Would you want the potential of filling root, or filling a less critical file system? Methinks there must be a better way.

As in most dilemmas, I tend to look for precedents. Where would we find a traditional location in the standard Solaris file system that might be used to spool (hint, hint) volatile files which might grow over time? I would immediately look to /var. There are two immediate paths I see as being preferential to /opt/SUNWexplo/output.

The first option would be /var/spool/explo. This would follow a convention that aligns with out use of a local explorer agent. The servers here produce an explorer on a regular file which is immediately shipped to a central (on-site) repository. The most recent explorer is typically left on the system and the history is managed at the repository. This makes the output directory a traditional spool directory, and as such a perfect fit for /var/spool/explo.

Where this may not be as intuitive is the case of an environment where explorers are retained on the host rather than collected and managed centrally. In that case, the explorers are better described as log files than spools. Intuition brings me to the use of /var/opt/SUNWexplo/output for this case. It's close to the legacy directory structure of the tool, which makes the solution marginally more intuitive than using a spool directory. It also follows the rarely observed SYSV standard of pairing optional software installed in /opt with a directory in /etc/opt, /usr/opt, and /var/opt. I'm not a fan of this specific model when taken to its literal implementation, but it's worth noting.

So, which one is best? As noted earlier, it depends. If I were a member of Sun's Explorer engineering team and needed to pick one consistent location with the intent of minimizing discontent I would select /var/opt/SUNWexplo/output. It is intuitive in the largest set of configurations, and doesn't break any rules. My secondary recommendation would be to create a symbolic link to redirect /opt/SUNWexplo/output for backwards compatibility over the next few years until it could be phased out.

Now I'm left wondering what interesting problems I might create in the data center if I put together a change package that implemented this very model... Nothing is ever as simple or benign as it appears on the surface.

Complex commands with sudo

2008-05-01T17:23:00.000-05:00

I've heard all the excuses for why someone issued a "sudo su -" command, and instantiated a shell that no longer tracked their actions. Of course we can argue about how to configure sudo so that problem goes away, but what if you have a lenient sudoers configuration?

The problem usually occurs when you need to redirect output. For example:


# tar cvf - /etc/ | gzip -c > /protected_dir/etc_backup.tgz

Or, the one which I just used, and reminded me that this deserves a quick posting:


# m4 somefile.m4 > newfile.cf

Both of these will fail if the target directory is one that your user ID does not have permission to write to. In many cases, the frustrated SA will simply use sudo to "su" to the root user and perform the command there. But we Solaris Jedi know that this is simply a temptation of the dark side pulling at a time when you need to get work done.

The right thing to do is create a subshell that executes the command. Returning to the above examples, the right instantiation would be:


# sudo sh -c "m4 somefile.mc > somefile.cf"
# sudo sh -c "tar cvf - /etc | gzip -c > /protected_dir/etc_backup.tgz"

Works like a charm. That being said, I'm much more an advocate for using RBAC on Solaris, but I'm going to fight the power of scope creep on this posting and stick with sudo.

Open Engineering: snmpXdmid follow-up

2008-03-11T12:13:00.001-05:00

Just for kicks I did a search on Google for the same problem I encountered only a few days ago: How to disable snmpXdmid in Solaris 10. The first time I searched for this information I found a wealth of Solaris 8 and 9 information, but very little about Solaris 10, and nothing about the alleged bug in SMF.

After finding the solution, I posted to this blog documenting the answer. Having given the Googlebots a little time to work their magic, I returned to the scene of the crime and entered the following search query: "disable snmpXdmid Solaris 10". SolarisJedi shows up in the #3 position for that query with all the information necessary for remediation. It feels pretty good knowing that someone else might get to complete a job in five minutes rather than five hours.

While I was riding the warm-fuzzy, I started thinking about how many large Corporations with legions of skilled SAs and Engineers maintain private knowledge bases rather than using public resources. I'm not talking about internal problems and proprietary issues - I'm talking about solving problems related to the generic off the shelf products they leverage. Let's face it, there isn't much proprietary about sendmail and DNS other than perhaps some parameters that are easily scraped clean.

Companies like Sun Microsystems have really paved the way of the future by encouraging their employees to blog, and trusting that proper standards of professionalism will be maintained. I believe Sun recognized that many of the problems they generate revenue from solving are based on the Internet's ability to act as a research assistant. I'd like to see more IT professionals invest back in the community.

One of the points in the System Administrator's Code of Ethics, a joint statement by LOPSA, USENIX, and SAGE, is the following:

RESPONSIBILITY TO THE COMPUTING COMMUNITY: I will cooperate with the larger computing community to maintain the integrity of network and computing resources.

Sometimes the definition of "network and computing resources" is one of hardware and software, but I suspect that other times it ought to apply to the operators of those resources since you cannot have one without the other.

Die Hard: disabling snmpXdmid on Solaris 10 (dmi)

2008-03-03T18:47:00.000-05:00

On a recent server build project we ran into a security scan that surprised us with a mandate that snmpXdmid be disabled. The alleged vulnerability is based on a buffer overflow that originated in the days of Solaris 8 as documented in CIAC Information Bulleting l-065 and SunSolve Security bulletin #00207. The details aren't important to this story other than finding it entertaining to respond to a Solaris 8 vulnerability on a Solaris 10 build. I'll save my thoughts on the corporate world's implementation of automated scanning for another post.

Our normal JumpStart image was configured about a year and a half ago, and in it we addressed the problem. Of course, none of us could remember what we did, and it turns out to not be the easiest thing to extract from Google. The process is pretty straight forward once you find it...


# svcadm disable dmi

Next, I dug up a quick test case to make sure the fix worked. It's easy enough to check registered RPC services using rpcinfo.


# rpcinfo -p | grep 100249
    100249    1   udp  43483
    100249    1   tcp  42683
#

Wait a minute... I thought I turned that off! The normal behavior of the service management facility, or SMF is to immediately change the state of a service after a disable command, so I made the (false) assumption that I had disabled the wrong service. After some additional research and testing I found that I wasn't wrong. The SMF was wrong. I did a quick zone reboot, and sure enough, the service was no longer responding. This led me to conclude the DMI service was not removing its registration with the portmapper (svc:/network/rpc/bind:default).

The next step on the path is to look at the service method that stops and starts the DMI service. All method scripts are stored in /lib/svc/method, and this one is easy to find: svc-dmi. So now we need to take a look at how it goes about stopping the service:


stop)
        /usr/bin/pkill -9 -x -u 0 -z ${_INIT_ZONENAME:=`/sbin/zonename`} \
            '(snmpXdmid|dmispd)'
        ;;

And so we come to the flaw in this method. In order to be consistent with predominant SMF behavior, this method should stop the service completely. There are two ways we can address this. We can either restart the entire portmapper, or we can be more surgical and remove the snmpXdmid registration from the portmapper. I preferred the latter since restarting the portmapper could temporarily impact other services. the code change is pretty simple:


stop)
        /usr/bin/pkill -9 -x -u 0 -z ${_INIT_ZONENAME:=`/sbin/zonename`} \
            '(snmpXdmid|dmispd)' && /usr/bin/rpcinfo -d 100249 1
        ;;

The problem, of course, is that to implement this fix I need to modify a script which is managed by the pkgadd facility, and subsequently, checksummed. I wont' get into addressing that issue right now, as the goal of this post is simply to provide breadcrumbs to other Jedi working to improve security with as little impact as possible.

The DaVinci Zone: Automating Zone Installation

2008-02-12T15:54:00.000-05:00

I love a good mystery as much as the next guy, and this one took a bit of piecing together. It's all documented, and with the proper grasp of docs.sun.com, man pages, and Google query syntax anyone can automate their zone installation. Since it took me a while to piece it together I thought I'd leave a few notes in the Jedi archives.

I'm going to leave my breadcrumbs in Perl, and focus on the workflow more than the syntax, so you won't be able to copy and paste code. If you know some Perl you chouls be able to fill in the blanks pretty easily.

The first thing we need to do is create the input stream for zonecfg. This is essentially the same things you would type if you doing it interactively, which is exactly how I derived the text I'm using.


# Open the file for writing
open(ZONECFGTMP, ">$zonecfgfile") or die "ERROR: Could not open $zonecfgfile for writing";
# Write the contents
print ZONECFGTMP "create\n";
print ZONECFGTMP "set zonepath=$zonepath/$zonename\n";
print ZONECFGTMP "add net\n";
print ZONECFGTMP "set physical=$zoneif\n";
print ZONECFGTMP "set address=$zoneip\n";
print ZONECFGTMP "end\n";
print ZONECFGTMP "exit\n";
close (ZONECFGTMP);

Next we need to create a sysidcfg file using a similar strategy... It gets a bit funky in the middle when I base some logic on whether or not the zone is entered into DNS. Solaris has what I consider a nuisance behavior during installation. If you want to configure DNS at install-time, the hostname must already be in DNS. If not, the install will revert to an interactive prompt asking if you really want to do this. To get around this, we need to FIRST determine if the zone name is in DNS. If it is, then install a sysyidcfg that reflects DNS. If not, then we need to use "none" for the sysidcfg naming service, and then install a resolv.conf file. It's kludgy, but it works.


# Make sysidcfg file (either NONE or DNS dep. on earlier check)
# File name should be mkzone.sysidcfg.ppid
$sysidcfg="$tmpdir/$thisscript.sysidcfg.$$";
open(SYSIDCFGTMP, ">$sysidcfg") or die "ERROR: Could not open $sysidcfg for writing";
   print SYSIDCFGTMP "root_password=\n";
   print SYSIDCFGTMP "system_locale=en_US\n";
   print SYSIDCFGTMP "timeserver=localhost\n";
   print SYSIDCFGTMP "timezone=US/Eastern\n";
   print SYSIDCFGTMP "terminal=vt100\n";
   print SYSIDCFGTMP "security_policy=NONE\n";
   print SYSIDCFGTMP "nfs4_domain=$mydomain\n";

   # if host not in DNS, use NONE, else use DNS.
   if ( $zonenotindns ) {
      print SYSIDCFGTMP "name_service=NONE\n";
   } else {
      print SYSIDCFGTMP "name_service=DNS {\n";
      print SYSIDCFGTMP "   domain_name=$mydomain\n";
      print SYSIDCFGTMP "   name_server=1.2.3.4, 1.2.4.4, 1.2.5.4\n";
      print SYSIDCFGTMP "   search=search.domain1, search.domain2\n";
      print SYSIDCFGTMP "}\n";
   } #end if

   print SYSIDCFGTMP "network_interface=PRIMARY {\n";
   print SYSIDCFGTMP "   hostname=$zonename\n";
   print SYSIDCFGTMP "   ip_address=$zoneip\n";
   print SYSIDCFGTMP "   netmask=255.255.255.0\n";
   print SYSIDCFGTMP "   protocol_ipv6=no\n";
   print SYSIDCFGTMP "   default_route=$zonedefroute\n";
   print SYSIDCFGTMP "}\n";
close(SYSIDCFGTMP);

If we needed to create a resolv.conf, it would look something like the example below. Note that I entered the DNS server list into an arry called @dnsserverlist. The $zonenotindns variable is set earlier in the execution when we perform an nslookup. I do this with a call to system() rather than using a separate module because it makes the code easier to distribute.


if ( $zonenotindns ) {
   $resolvdotconf="$tmpdir/$thisscript.resolvdotconf.$$";
   open(RESOLVDOTCONF,">$resolvdotconf");
   print RESOLVDOTCONF "domain $domain\n";
   foreach ( @dnsserverlist ) {
      print RESOLVDOTCONF "nameserver $_\n";
   } #end foreach
   print RESOLVDOTCONF "search $mydomain\n";
   close(RESOLVDOTCONF);

The piece that threw me for a loop was getting rid of the NFSv4 prompt. It turns out to be as simple as putting this command into the code right before the zone is booted, but after the zone is installed. Kudos to the OpenSolaris Zones and Containers FAQ for documenting it!


system("/usr/bin/touch $zonepath/$zonename/root/etc/.NFS4inst_state.domain");


Using the above files is covered well in other posts, so I won't duplicate content.  Using these details, you should be able to get your site's zone installation automated without too much trouble.



Zonecfg: removing a resource
2008-02-05T09:46:00.000-05:00
I just noticed that there aren't a whole lot of examples of removing a resource from a zone to be had in the vast caches of Google at the moment.  It's pretty simple once you understand the zonecfg syntax.  Of course, just about everything in UNIX is simple once you know how to do it!

First, we need to fire up zonecfg and look at the specifics of how our zone is configured:

cgh@testbox$ pfexec zonecfg -z testzone
zonecfg:testzone> info
zonename: testzone
zonepath: /export/zones/testzone
autoboot: true
pool:
limitpriv:
inherit-pkg-dir:
        dir: /lib
inherit-pkg-dir:
        dir: /platform
inherit-pkg-dir:
        dir: /sbin
inherit-pkg-dir:
        dir: /usr
fs:
        dir: /myapp/u01
        special: /dev/dsk/c2t5006048ACC36D646d138s0
        raw: /dev/rdsk/c2t5006048ACC36D646d138s0
        type: ufs
        options: []
net:
        address: 192.168.1.1
        physical: e1000g0
zonecfg:testzone>


In this case, the file system /myapp/u01 has a problem and is preventing the zone from rebooting.  In order to remove it we need to use the remove syntax, which requires enough parameters to uniquely identify the resource we want removed.  In this case, the dir setting of /myapp/u01 should be sufficient.

zonecfg:usa0300uz0002> remove fs dir=/uv1234/u01


A quick repeat of the info command should now display that the file system is not part of this configuration, and indeed it does.

zonecfg:testzone> info
zonename: testzone
zonepath: /export/zones/testzone
autoboot: true
pool:
limitpriv:
inherit-pkg-dir:
        dir: /lib
inherit-pkg-dir:
        dir: /platform
inherit-pkg-dir:
        dir: /sbin
inherit-pkg-dir:
        dir: /usr
net:
        address: 192.168.1.1
        physical: e1000g0


And funally, we commit the changes using the commit command. A quick call to zoneadm, and a reboot is issued, allowing our zone to successfuly reboot.


RBAC, Zone Management, and the mortal user
2008-02-04T13:45:00.000-05:00
I'm doing a lot of work with automating zone configuration at the moment, and have been using the zlogin command frequently.  Having never been a big fan of Sudo, I really wanted an excuse to dabble in RBAC and see if I could get it to work for me.  Turns out to be a very trivial thing.  In this case I wanted to be able to perform zone administration as conveniently as possible without spending a lot of time whittling down a command set - just give me quick and easy.

I started out looking for any execution attributes which may have been preconfigured for my convenience...

cgh@testbox$ grep -i zone /etc/security/exec_attr
Zone Management:solaris:cmd:::/usr/sbin/zlogin:uid=0
Zone Management:solaris:cmd:::/usr/sbin/zoneadm:uid=0
Zone Management:solaris:cmd:::/usr/sbin/zonecfg:uid=0


So now I needed to get them plugged into my user ID (I didn't want to fiddle with su-ing to a role, just wanted them in my ID).  I loaded up the /etc/user_attr file into my favorite editor (for those who are curious, I'm a vi guy) and added my name, and the profile:

adm::::profiles=Log Management
lp::::profiles=Printer Management
rroot::::auths=solaris.*,solaris.grant;profiles=Web Console Management,All;lock_after_retries=no
cgh::::profiles=Zone Management


A quick test verifies that all is well with the world:

cgh@testbox$ profiles
Zone Management
Basic Solaris User
All


And finally we give it a shot:

cgh@testbox$ zlogin testzone
zlogin: You lack sufficient privilege to run this command (all privs required)


But of course!  The use of RBAC commands seamlessly requires that you use an RBAC-aware shell such as pfcsh, pfsh, or pfksh.  But at the moment my shell is a standard ksh.  The easy way to get around this is to use the pfexec command.

cgh@testbox$ pfexec zlogin testzone
[Connected to zone 'testzone' pts/4]
Last login: Mon Feb  4 13:45:24 on pts/4
You have new mail.
root@testzone#


And there you have it.  With RBAC, it's easy to attach administrative commands to a general user ID.  Of course, this demonstration was a hack, and isn't a best practice.  Why?  Administrative commands are separated from user commands for a reason.  You dont' want a general user doing things that can impact the entire system.  

The best way to do this in most situations would be to embrace the R in RBAC and create a role for Zone Management that a user could assume to perform this work.  In my case it's a lab machine, not many people are using it, and I wanted an excuse to play with RBAC.


Basename saves the day...
2008-02-01T08:42:00.000-05:00
One of the things I like to do when setting up a Perl script is to set a variable called "thisscript".  It's essentially the $0 special variable, but with a subtle twist.  The inspiration for this article comes from forgetting the twist, and true to my mission, I am documenting my detours from the Jedi path.

I'm working on a fun script at the moment which simplifies and automates the process of deploying and configuring a zone.  Sort of a JET-lite if you will.  The script creates numerous temp files, and I prefer the following naming convention:  "tmpdir"/"name of parent script"."functional identifier"."process pid".  So, a file name may look like this:  /tmp/mysite-mkzone.sysidcfg.343224.  And here begins the oddity I ignored, and eventually fixed.

Although the script worked well, I noted the following output:

...
Cleaning up temp files...
   /tmp/./mysite-mkzone.zonecfg.2012
   /tmp/./mysite-mkzone.sysidcfg.2012
   /tmp/./mysite-mkzone.resolvdotconf.2012


Fortunately, the way UNIX inteprets a pathname, this is a perfectly legitimate albeit circuitous path value.  The "./" evaluates to the current directory and continues on its merry way.  To be more explicit, the following examples all evaluate to the same value:

/tmp/mysite-mkzone.zonecfg.2012

/tmp/./mysite-mkzone.zonecfg.2012

/tmp/././mysite-mkzone.zonecfg.2012

/tmp/./././mysite-mkzone.zonecfg.2012



But, my heightened Jedi awareness felt this extraneous path element to be disturbing the balance of the force.  Once you journey down the path of the dark side, it is difficult to return to the light.  But where was this coming from?  My first suspicion was a syntax error somewhere in a Perl string catenation.

In perl, strings are catenated with a dot operator (".").  For example, we could set up a string using catenation as follows:

$ vi catenation.pl
my $a="The quick brown fox";
my $b="jumped over the lazy dog";
my $sentence="$a" . " $b." . "\n";
print $sentence;
------
$ catenation.pl
The quick brown fox jumped over the lazy dog.
$


So, if I were to misplace a quote, it's possible that I might have included an errant period somewhere in the code.  After a scan of each use of the variable I quickly determined that my hypothesis was unlikely to have manifested itself.  I then returned to the code which set the initial variable:

my $thisscript=$0;


It was then that I remembered what I had omitted.  I don't typically have "." in my current path.  Just a habit, the result of which is another habit.  I always qualify the path to whatever I'm running. So, if I'm executing a script called mysite-mkzone in the current directory, I execute the following on the command line:

$ ./mysite-mkzone


Now, consider the preceeding wetware behavior in concert with the following software behavior:

$sysidcfg="$tmpdir/$thisscript.sysidcfg.$$";


Herein lies the problem.  Evaluating $sysidcfg we get the following: "/tmp + ./mysite-mkzone + sysidcfg + 12345"  which explains where the extraneous "./" is coming from.  So how did I fix it?  I used File::basename.  Which is a Perl equivalent of the shell basename(1) command.  It deletes any path prefix ending in "/" from a string.  In other words, it yanks the directory part of a command, leaving just the command.  To use this, I made the following trivial modification to my code:

use File::Basename;
my $thisscript=basename($0);


And the output dutifully responded as follows:

...
Cleaning up temp files...
   /tmp/mysite-mkzone.zonecfg.18129
   /tmp/mysite-mkzone.sysidcfg.18129
   /tmp/mysite-mkzone.resolvdotconf.18129


This wouldn't have happened if I'd used my normal starter tempalte, which has this variable pre-configured, but I'd made a careless decision to just go from scratch on this one.  Yet another misstep on the path, but a good lesson.


Queuing Efficiency
2008-01-18T09:26:00.000-05:00
On my way through the office door this morning I had an experience that I realized needs to be recorded so that I don't go on and rant about it in the future.  This post will allow me instead to reference it, thus saving a vicious cycle of rehashing.

In addition to my career in systems engineering, and my small photography business I take martial arts classes.  A few days ago I managed to pull a hamstring and sprain a toe on the opposite leg.  Don't get me wrong - it's worth it.  But these injuries are relevant to the story because they allow you to appreciate my inability to move quickly.  I'm not limping, I'm just moving cautiously.

The weather was cold, and the sidewalks covered in treacherous ice.  A bitter wind cut through me as I approached the building.  Moving at a determined but non-rapid pace, I noticed someone about a mile (or three) front of me, presumably intending to enter the same doors.  And then it happened.

They decided to take it upon themselves to hold the door open.  At this point they almost need a telescope or radar to even know I'm planning to go through the same door, but yet they stood there holding it open while the arctic air flooded the entry to our building.

Here's the deal people...  If someone behind you stands a chance of having the door slam in their face, then you hold it open.  If they are carrying a heavy load and don't have arms free, you can wait until they get close to open the door for them.  If it's more than five steps, you move along.  Let's stop the self-gratifying good deed of holding a door open just to see if the recipient of your good will will start to jog because they feel guilty you are letting all that cold air into the building.  If they have enough room to jog they probably don't need you to hold the door.


Got dependencies?
2008-01-16T15:45:00.000-05:00
As anyone who follows my blog has learned, I'm a packaging junkie.  I write packages for everything I deploy, and happily plunk them into my JumpStart server where they perform their duty in a predictable and maintainable way.  Does it get any better than this?  Well, actually...  It got pretty rough this week as my troubleshooting skills enjoyed a solid workout.

We are in the process of moving our standard deployment from Solaris 9 with SRM project containers to Solaris 10 Zones.  Now Zones are really surprisingly simple for the rich myriad of benefits they provide.  It takes very little time to get to the point where you can set up a test box with multiple zones in a basic Solaris environment.  But what if you don't have a basic environment?

We have packaged everything.  Most of our /etc files that aren't stock are edited by class action scripts or postinstall routines.  We've deployed hundreds of servers with this configuration and it worked flawlessly until we started deploying Zones.  Then suddenly, our zones couldn't talk to the Directory, were missing resolv.conf, and all kinds of other cascading problems.  Ugh.  Where to start?

The first thing I did was check the error messages.  Pretty clever, no?

The file /export/zones/testzone01/root/var/sadm/system/logs/install_log contains a log of the zone installation.


Ok, let's see what's in the log...

WARNING: setting mode of /var/spool/cron/crontabs/root to default mode (644)
ERROR: attribute verification of /export/zones/testzone01/root/var/spool/cro
n/crontabs/root failed
  pathname does not exist

Installation of CGHtest on zone testzone01 partially failed.


Well that's strange.  Let's see what the registry says about root's crontab:

$ grep "/var/spool/cron/crontabs/root" /var/sadm/install/contents
/var/spool/cron/crontabs/root e cron 0600 root sys 48 3760 1200431631 SUNWcsr:cronroot


The root crontab is owned by SUNWcsr.  So it turns out that when the global zone creates the sparse root zone, it tried to install CGHtest before SUNWcsr was installed.  The root of my problem was slipping a habit that I'm normally quite vigorous about enforcing.

Any time I use question marks in my prototype file to inherit attributes from an already-installed package I always, always, always stop immediately to look up that inherited object in the registry and add its parent package to a depend file.  Here's an example prototype file displaying inheritance:

$ cat prototype
i pkginfo
i depend
i copyright
i i.cron
i r.cron
d none var ? ? ?
d none var/spool ? ? ?
d none var/spool/cron ? ? ?
d none var/spool/cron/crontabs ? ? ?
e cron var/spool/cron/crontabs/root ? ? ?


This was never a problem in our traditional JumpStart environment because the JET custom packages are installed after SUNWcsu.  Fortunately, it's easy to control the order (dependency) of package installations using the depend(4) file.

$ cat depend
P       SUNWcsr Core Solaris, (Root)
$ grep depend ./prototype
i depend
$


Problem solved!  Of course, the problem never would have happened if I'd remembered my Jedi training.  It's good to be humbled on a semi-regular basis.


The nature of systems engineering
2007-10-29T07:35:00.000-05:00
I was reading an interesting passage from the Tao Te Ching this morning which, I believe, has great applicability to the nature of systems engineering.

We join spokes in a wheel,
but it is the center hole
that makes the wagon move.

We shape clay into a pot,
but it is the emptiness inside
that holds whatever we want.

We hammer wood for a house,
but it is the inner space
that makes it livable.

We work with being,
but non-being is what we use.


What does this mean to those of us who wield keyboards at the battle of the command line?  Probably more things than can be said.  To start the pondering I'd like offer two thoughts.

(1) The external perspective: Remember that the business your systems support don't think in terms of IOPS, MB/sec, or LoC.  The IT organization does not exist to amaze itself.  It exists to enable a business process.  As you learn about Perl, Zones, and ZFS, are you also learning the business those technologies support?

(2) The internal perspective:  Have you ever met an administrator or engineer whose wall is decorated with certifications, and yet you would not trust them to configure IPMP on a server you were responsible for?  Have you met anyone who could write code as fluently as you speak your native language, and yet they could not effectively translate business requirements into functionality without great effort?  As you learn the technologies you need to execute your job, are you also learning universal skills such as troubleshooting, and communication?

What other areas of our trade does this parable apply to?


Oracle's writing on the wall
2007-09-20T14:25:00.001-05:00
I received an email newsletter this morning with the headline, "Oracle support betrays a preference for Linux and x86."  Sun and Oracle seem to have a love hate relationship driven primarily by thir symbiosis rather than their ideals.  This appears to be another chapter in that long story.  

The article referenced by the newsletter mentions the fact that Oracle 11g is currently only available for Linux.  That's a very interesting move considering the size of the Oracle installed base on Solaris.  Not only the population size, but the class of customer.  More than one global enterprise is running Oracle on enterprise class Solaris hardware.

I can't help but speculate that we're leading up to a boost in Sun's emphasis on PostgreSQL.  First we saw its inclusion in the base Solaris 10 software.  This is no small thing; even compilers are distributed separately.  Postgres' own FAQ recommends use of Sun's compilers over GCC on the Sparc platform.  It's practically heresy to recommend an open source product be compiled on anything other than GCC, so again this is not to be dismissed.  Finally, I'll draw your attention to the release announcement for Solaris 10, Update 4 where enhancements to PostgreSQL DTrace probes are released.  If this doesn't look like building up a rebellion, I don't know what does.

I give Sun a lot of credit for investing heavily in PostgreSQL and bringing some serious competition to Oracle.  Evolution is based upon competition, and I'm happy to see the Sun species evolving into a new predator.


The trouble with packages and auto-pilot
2007-09-20T07:17:00.000-05:00
I stumbled into a very interesting problem and resolution this morning which I think deserves some attention.  I didn't work on the diagnosis and research, so I'm summarizing from an email thread.  We use a Citrix server to share out GNOME environments from our development server.  It's particularly nice when you're working from home and the VPN kicks you out, or if you're using public wifi and your connection is spotty.  

At some point a week or two ago people began to notice that they couldn't connect to GNOME.  This took a little while to unfold because some people keep sessions opened for  extended periods of time, but eventually we discovered that it was dead for everyone.  After eliminating license server issues there was only one thing we could come up with that had been done to the server. 

A colleague had installed a current version of FireFox on the server because Sun's desktop environment is often very slow to integrate application software updates.  He used the packages from Blastwave.org.  Note that I say packages: a plural word.  Indeed, FireFox turned out to be more than twenty packages when delivered by Blastwave.

The foundation of Blastwave is their packaging system, pkg-get.  If you have any stick time in the Linux world you're probably familiar with something like Yum, apt-get, or up2date.  These tools know how to connect to software servers through http, https, ftp, firewalls, proxies, etc.  They also know how to resolve package dependencies.  This can be very convenient on a Linux system where a single source handles the OS packaging and application packaging.

In contrast, Solaris provides pkgadd.  Pkgadd can not resolve dependencies.  It only knows how to retrieve packages from a specified URL, but does not have any ability to retrieve packages from a Sun resource.  Pkgadd is a bit antiquated by modern UNIX standards unless coupled with the Sun Connection which is not quite the same thing.  This huge gap between Linux packaging systems and Sun's pkgadd inspired Blastwave's packaging system and repository.  

Blastwave provides many packages that are provided by the Solaris OS.  The difference is that they provide more frequent and convenient updates.  If you need bleeding edge features in the tools you install, Sun's usr/sfw/* and /opt/sfw/* packages will probably not help.  I tend to think that it's more the exception than the norm to require updates that frequently.  I know there are exceptions here and there, but overall, how often do you really need a new version of wget, or gtar?  Although I love having latest and greatest "stuff", I even use the old Mozilla browser in Solaris and rarely have any problems.

When my colleague innocently asked Blastwave to install the latest FireFox package, it installed a fairly significant list of packages.  One of them was fam, the file alteration monitor.  For those who may not be familiar with FAM, it is described as follows (from the FAM web site):

GUI tools should not mislead the user; they should display the current state of the system, even when changes to the system originate from outside of the tools themselves. FAM helps make GUI tools more usable by notifying them when the files they're interested in are created, modified, executed, and removed.


We eventually discovered that fam installs an inetd service.  I don't know, or care what that service is doing.  What I do know is that I did not want a new service running.  As a result of installing the Blastwave FireFox package and its slew of dependencies we ended up with a new service running and had absolutely no warning that it was happening.  That service somehow conflicts with, and breaks GNOME.  It turns out that there is an OpenSolaris bug describing the same symptoms.

Ignoring the obvious concerns about a simple desktop web browser requiring 20 package dependencies and breaking GNOME, I have a much larger concern.  Turning up an inetd service creates a new attack vector for a server.  Whether or not that is acceptable is a question of risk management.  In many cases it doesn't matter.  In our data center, servers must pass an external probe scan to be in production and adding services requires change requests.  So for our purposes, the changes are not acceptable, and we will need to back them out.  We are also imposing a ban on blastwave within our data center servers.  It's simply not an acceptable framework for a mission critical server environment.  

Whether or not you deem it reasonable to install an inetd service to run FireFox, it's hard to justify the intuitive nature of a web browser requiring the inetd service.  Note that fam is NOT a FireFox dependency in other distribution channels.  Of course, this kind of thing can be caught with good change management using a promote-to-production path, which is how we found this issue on our development server.

While Solaris' pkgadd facility is not as convenient as some of the Limux systems, it forces you to make conscious changes to a system rather than hitting auto-pilot and hoping for the best.  I would love to see Solaris' packaging facility evolve into a tool with the capabilities of its Linux counterparts, but only for the freeware / OSS packages that are built and distributed by Sun (of which there are quite a few).  I'd also like to see the ability to configure additional repositories (such as a local server for custom packages), as long as it's not set that way out of the box.  I guess its time for me to start exploring Update Connection's capabilities.

My suggestions are as follows:  First, beware the autopilot.  Second, keep Blastwave on the workstations, and as far away as possible from the critical servers.


DVD upgrade adventures
2007-08-23T15:08:00.000-05:00
I had an irresistible opportunity to rescue an Ultra 60 workstation from a trash nap recently.  This is the sort of thing I really shouldn't do because I'm trying to reduce my data center footprint.  On the other hand, it's such a cool workstation that I had to do it.  This box was reported to be unable to boot, but I'm pretty good with hardware repairs, so decided to go for it.

Although it took forever to get through the process, the classic method worked.  I can't count how many systems in this era seemed to have problems that turned out to be solved by reseating memory or CPUs.  I did both, and it came to life like a resuscitated drowning victim.

Next stop, storage.  I replaced the 9GB disks with 36GB disks from the unused half of my D1000 array.  This was going too easy.  As I was poking around the drive bay I noticed that the cable had been removed from the CD-ROM.  Not a good sign.  Tracing to the other end of that ribbon I noticed that someone must have been having a bad day as it was half ripped from the daughter board's crimping.  Confirmed ugliness.

Being the fatal optimist I grabbed my tool kit and carefully pressed the ribbon back down onto its pins.  Next stop, the drive bay.  I reconnected he CD-ROM thinking that it might work... Nope.  This one had a bad case of indigestion and spit out any disks I inserted.  What's worse, once it spit them out, the drive tray could not be closed.  Stick a fork in it - it's toasted.

I borrowed a Sun DVD from my 420r just to test out the SCSI channel, and successfully loaded Solaris 10, so it looks like the drive needs to be replaced.  Next stop: eBay.  I picked up a Pioneer DVD-302, which is one of the few remaining SCSI DVD options out there.  I could have bought a Sun DVD, but they are all grey, and this case is beige.   Can't compromise the aesthetics. (I'm really in bad shape, aren't I?).  The drive arrived, looking shiny and new.  I managed to get the thing installed, but it's not happy.  

Booting from a DVD results in error messages like "Short read.   0x0 chars read".  Eventually the retries end, and it complains about errors finding interpreter, and "Elf64 read error".  Booting from a CD-ROM gets a little farther along before it spits out "incomplete read- retrying", and "vn_rdwr failed with error 0x5".  Oddly, it does seem to be working once the OS is loaded, so this appears to be an incompatibility at the OBP level.

What annoyed me the most in this whole exercise was not finding anything in an hour of Google searches that indicated anyone had even attempted such an upgrade.  I know there are quite a few U60s still kicking around out there, and I'd have to think their owners would be looking for DVD capability and higher speeds.  I must have thought wrong.  If you happen to be reading this post and have experience with a SCSI DVD-ROM being bootable in a Sun Ultra workstation I'd love to hear about it.

I guess I'll just have to keep looking for a beige Sun DVD-ROM on eBay, but so far the pickings are slim.  Wish me luck.


IBM Sees the Light?
2007-08-16T13:51:00.000-05:00
Wow, I didn't see this coming.

IBM and Sun today jointly announced that IBM will offer and fully support Solaris on their compatible hardware lines.  This raises some interesting dust clouds.

What does this mean to AIX, IBM's flagship UNIX?  Personally, I think it means little.  Sun supports Windows and Linux on their hardware, but those of use who have been with Sun for a long time still prefer Sparc in most cases.  I believe the same will be true of IBM and Solaris.

How will Solaris compete with the investment IBM has already made in optimizing their previously supported operating environments?  There's no way it will be on the same level right out of the gates, but when you consider the OpenSolaris model, it becomes clear that IBM will not have to jump through hoops to make it happen, and I believe they will.  When IBM announced that they would be supporting Linux it was initially a bit of a surprise because of the inherent undermining ot AIX.  And yet, they have contributed some incredible advances to Linux's abilities in the enterprise data center.  I think of IBM as the mature mentor that helped Linux to grow up.

Now Solaris is no padawan looking for a master to study under, so that makes for a different game.  But there's no question in my mind that IBM will have a serious group of Jedi coders participating openly and actively in the OpenSolaris community, and that can only help Sun and Solaris.

Of course, the down side is the precident this sets leading towards too broad a foundation.  Using Linux as an example we see a massive code base that tries to support as much hardware as possible.  There are basic laws of software engineering, just as there are laws of physics, and the more lines of code you have, the more potential you have for bugs, integration issues, and regression failures.  Doesn't matter how good your developers, the probability still goes up.  I'd hate to see Solaris supporting everything Linux does; I'd like it to stay focused in its sweet spot of quality hardware, which despite my preferences, I think IBM hardware is in alignment with.

What does this mean for Linux in the Enterprise?  Well, I think Linux has a tough climb ahead of it as it stares up the cliff at Solaris' backside.  Linux was developed on PCs by people who aren't typically in an enterprise.  You could argue that the coders went to Linux because they coudln't afford at home what they had at work, but the bottom line is still the same.  

Linux does not have a lot of "stick time" on servers built at the scale of Sun's high end servers like the Enterprise 20k.  On the other hand, Solaris has been running on multiprocessor systems since before Linux was a twitch in Linus Torvald's ear.  You have to spend time working with servers that have 20GB of RAM and 64 processors before you can even anticipate the kinds of problems that occur.  Linux just doesn't have that kind of time in a data center.  I'm not saying they can't get there, I'm just saying you have to pay your dues to provide stability at the high end.

Keeping all that in mind, put yourself in IBM's shoes.  AIX is not gaining market share, although its a rock solid enterprise class operating environment.  Linux brought IBM a huge customer base, and helped them to sell Intel hardware.  Unfortunately, it didn't really put them in the data center where they belong.  Along comes Solaris with the openness of Linux, and the opportunity to leverage it quickly - just as they did with Linux.  But this time, they start at the upper end of scalability and bypass that climb altogether.  Where would you put your resources in the long run?

Wearing my purely speculative hat, I think this announcement was a big strike against Linux in the Enterprise, and a foreshadowing of Solaris' long term viability.  As more and more products come on-line with the Internet, data centers are only going to grow.  And as that continues to happen, consolidation will be the only way to drive utilization up and costs down.  The natural extension of this prophesy is that the operating environment that scales best and stays stable is going to be the evolutionary top of the food chain.  And I think that Solaris will be in that seat.


IPMP, anyone?
2007-08-16T05:59:00.001-05:00
While scanning various news feeds this morning I ran into a story regarding a computer breakdown at the LAX airport.  The first article suggested the problem was a network card failure, and the second article suggested the problem was a switch failure.  

In either case, the result was 17,000 - 20,000 (varies by atricle) international passengers being stranded for a fairly significant duration.  But wait, it gets better...  "The system was restored about nine hours later, only to give out again late Sunday for about 80 minutes, until about 1:15 a.m. Monday."  Two failures, both stopping passengers at an incredibly busy airport.

I'd like to offer my consulting services to LAX for free, and recommend that they move an obviously critical function over to servers running the Solaris operating environment where they can enjoy the benefits of IP MultiPathing (IPMP).  A properly architected system would have had redundant switches, and multiple network interfaces, each connected to a unique switch.  The failures indicated would have cause no interruption to service.  This is server design 101.

What, you may ask, would be the cost of this highly advcanced architecture?  Well, of course it depends on the cost of the switches you run because you'd need two, but on the server side its free, and included with Solaris.  I run IPMP on the servers in my basement, and my wife can assure any who may ask, my IT budget is far less than that of the mighty LAX airport.


When will Wall Street wake up?
2007-08-03T10:03:00.000-05:00
In case you had any lingering doubts as to whether Sun has been doing the right thing by embracing the Open Source model, take a moment to peruse an entry from Jonathan Schwartz' Blog.  I'll quote the part that caught my attention:

As you may have seen, we've announced our fourth quarter and full fiscal year results ... We grew revenue, expanded gross margins, streamlined our operating expenses - and closed the year with an 8% operating profit in Q4, more than double what some thought to be an aggressive target a year ago.

We did this while driving significant product transitions, going after new markets and product areas, and best of all, while aggressively moving the whole company to open source software (leading me to hope we can officially put to rest the question, "how will you make money?"). 


It is extremely frustrating to me that public companies must deal with putting their fate in the hands of a group of analysts who have such limited understanding of the ecosystems of information technology.  Wall street has been so timid about Sun since the bubble burst, largely because their fear of the past clouds their ability to see the future (or the present, for that matter).

Today Sun has the best product portfolio I've ever seen.  They also have the financial metrics to prove their strategy is good.  I have invested more than ten years of my life in their products, and I can say with no hesitation that I plan to continue that investment for the next ten years as well.  The only question I have is when the rest of the industry will catch up.


Picking a terminal server
2007-07-27T21:09:00.000-05:00
My plans to equip the lab with older, but solid equipment has been going very well thus far.  It's not cheap, but it's going to be very functional.  The two Netra X1 servers are doing a great job, and I'm really enjoying having a LOM.  I wish my "big iron" 420R had a LOM, but a Sun serial port still beats an x86 BIOS program.  And what could be cooler than accessing those serial LOM devices through a terminal server?  (Yes, I suppose a modern Sun server with an Ethernet LOM would be cooler, but don't burst my bubble, ok?).

So now that I've accumulated these boxes and am beginning to use them on a regular basis, you can imagine that patching wasn't far behind.  Patching is one of many activities where a console connection comes in pretty handy.  To make a long story short, I quickly grew tired of trucking my laptop downstairs, attaching a serial cable to it, and then performing an elaborate contortion routine to find the LOM port in the back of my rack while pressing my face through cobwebs.  Been there before?   Yes.  I have decided that I need a terminal server.

So, what is my ideal terminal server?  Well, there's a few requirements.  It must be a quiet, low power device - no giant noisy fans need apply.  I need an 8-port device, but 16 would give me room to grow if the price is right.  I don't care too much about security protocols - this is a home lab that sits behind a firewall, and all my systems can be reprovisioned from a flash archive in a heartbeat.  Should be easy right?

The first thing I learned is there are a LOT of 32-48 port high end (not old!) term servers available, primarily Cyclades devices.  These look like Ferraris to me, and I dream of winning an auction for about $50 and attaching that puppy to my rack.  Not going to happen...  The next thing I noticed is a bunch of really old Xyplex and Perle devices.  These rack up, but I read a bunch of horror stories, and got the idea from a few USENET postings that they are loud.  I found a few other older devices, but they all had something that didn't seem right to me.  It was time to get drastic...

I went with plan "C".  In this case, the C stands for Cisco.  Turns out that with some auction patience, a properly equipped Cisco 2509 (8 port) or 2511 (16 port) can be had with cables for around $150 or less.  That's right at my pain threshold, but acceptable given what it provides.  This solution appears to be hit or miss with the issue of spontaneous break signals halting the Sparc machines, which usually happens if the TS powers down, but the kbd command can be used to configure an alternate break sequence and avoid the issue.  

The other appealing feature seems to be that I can configure reverse-telnet.  This would allow me to run a command like "telnet termserver 2001" to get to port 1.  Much more convenient than authenticating to a termserver and navigating annoying menus.  And finally, being a full size 19" box I can rack it up without coming up with some combination of plywood and duct-tape.  Suh-weet.

The downside?  Well, ssh would be more cool than Telnet, but I can swallow my pride.  Who knows?  Maybe there's a Cisco update that would provide this.  It might be a loud device.  I have no diea.  Another issue which decrements the coeficient of cool: It requires an AUI adapter to convert to an Ethernet RJ45 port.  On the other hand, there's probably a lot of new SAs in the world who would look at that like a vintage muscle car...  "Whoa - is that a REAL aui adapter, dude?  You're must be hard core."  Um, yeah.  Maybe not.  Although the loudness and power consumption concern me, I think I can live with these issues if it works, which I'm reasonably confident it will.

Now, to set up an eBay search and begin the hunt...


Learning to think in Z
2007-07-26T20:25:00.000-05:00
In the traditional disk mounting world we had a device uner the /dev directory which is mounted on a (aptly named) mount point.  For example:

# mount /dev/dsk/c0t2d0s0 /export/install


On a large database server you might see the common convention of mounting disks with /uXX names...

# ls -1d /u*
/u01
/u02
/u03
/u04


This is the frame of reference I used when walking into the building of my new JumpStart server.  My goal was to stick as close as possible to standard mount points.  The first file system was to be mounted on /export/install.  The second file system would serve as my home directory, and I didn't much care where it lived since I'd use the auto mounter.

The default zfs configuration is to mount a complete pool under its pool name.  I tried to be creative in coming up with a naming convention, but slipped into mediocrity with a "z##" name.  Hey, I'm tired of seeing /u##; It's amazing what a difference one letter can make in spicing up a server.  Having come up with my name, I created the pool from my second disk:

# zpool create z01 c0t2d0 
# zfs create z01/install
# zfs create z01/home
# Hmm, why not make my home its own fs?
# zfs create z01/home/cgh


Wow.  That was easy!

But now there's a sort of a problem.  I can't quite get past seeing the JumpStart directory under /z01.  It's not intuitive there.  The world of Solaris sysadmins looks for JumpStart files in /export/install.  So, how can we get this sweet ZFS file system to show up where I want it?  Turns out this is pretty easy as well.

# zfs set mountpoint=/export/install z01/install


It even unmounts and remounts the file system for me.  Oh yes, I'm a fan at this point.

One thing that's interesting is that once you move a mountpoint from its default, it can be easy to "loose" that file system.  For example, if I list the contents of z01 at this point, I only see home.  "install" no longer shows up there because its mounted on /export/install.  In this example it's hard to loose anything, but on a large production server there could be many pools and many file systems.  As you would expect, there's an easy command to list the file systems and their mount point:

# zfs list
NAME                   USED  AVAIL  REFER  MOUNTPOINT
z01                   1.61M  36.7G  26.5K  /z01
z01/home              1.49M  36.7G  1.45M  /z01/home
z01/home/cgh          35.5K  36.7G  35.5K  /z01/home/cgh
z01/install           28.5K  36.7G  28.5K  /export/install


I decided to leave the z01/home in place and just repoint the auto-mounter.  From zero to "get it done!" in about 20 minutes with some play time.  I love it.


First impressions of ZFS
2007-07-26T20:10:00.000-05:00
If you're anything like me, you cling to that which you know while yearning for that which you haven't yet dabbled in.  Tonight was a small victory for my self discipline, and a great example of why I think I'm going to be good friends with ZFS.

I've been mentally moving forward with a new JumpStart server layout for a while now.  This server would have very little need for horsepower with storage space being what I really needed.  It's main purpose is to help me consistently provision lab environments here at home for projects.  I ended up selecting a Netra X1, which is very inexpensive on eBay.  It's a nice low power draw platform that has plenty of power, and one less common feature among the Sun lines:  IDE (PATA) drives.  Yes, I mean that in a good way.

I was able to load it up with a 40gb boot drive and 120gb data disk to house install media images, flash archives, home directories, and some crude backups for the rest of the lab environment.  The cost of a SCSI disk in that size is insane by comparison, and would provide no advantage for the tiny demand it would be charged with.  I jumpstarted the hardware from another Sun machine, then loaded the Jupmstart Enterprise Toolkit (JET) and prepared to boogie.

Ahh, but now the moral dilemma rears its ugly head.  How to manage that data disk?  I haven't spent much time playing with Solaris Volume Manager (SVM) soft partitions, but enough to know it was a snap and would do the job.  On the other hand, I've been twitching to learn ZFS, and this could be just the excuse I needed to get started.

The hard part about this decision was deciding whether or not I perceived ZFS to be an abyss, or a simple technology.  I can't count the number of times I've done something silly like saying, "Oh sure, we could write a quick Perl script to do that."  Only to find that two months later I'd grossly underestimated the complexity.  I'm a chronic and pathological optimist.

I'm happy to report ZFS was painless and a pleasure to use.  I'm still in shock from the simplicity.  This is fun...  I don't miss Linux at all.


Inconsistency in prtdiag output
2007-07-16T10:48:00.000-05:00
I've been doing a lot of work recently writing Perl scripts to mine data from local Explorer repositories.  It's a phenominal resource as a sort of RAW input to a configuration DB, and with Perl it's a snap to pull out data.  My latest excecise was pretty trivial.  I need to yank out the memory size field from prtdiag for each system, then dump it into an XML feed that serves one of our databases.

The information resides in the prtdiag-v.out file, and looks something like this:

fooserver{sysconfig}$ more ./prtdiag-v.out
System Configuration:  Sun Microsystems  sun4u Sun Fire E20K
System clock frequency: 150 MHz
Memory size: 65536 Megabytes


So, we throw together a little Perl script that does this:

sub get_memory_size {
   my $explodir=shift();
   my $prtdiagfile="$explodir/sysconfig/prtdiag-v.out";
   my $line;
   my $memsize;

   if ( -e "$prtdiagfile" ) {
      open(PRTDIAG,$prtdiagfile);
      while () {
         chomp;
         last if ( $_ =~ /^Memory size:\s/ );
      };
      close(PRTDIAG);
      s/Memory size:\s//g; # Kill the label
      s/\s+$//;  # Remove any trailing whitespace
      return $_;
   } else {
      # We did noit find the prtdiag file.
      return 0;
   } #end if

} #end get_memory_size


No problem!

Then I put together a simple loop to check what I'd found...  Now help me understand why this can't be simple and consistent?  Here's some of the variety:

[2GB]
[6144 Megabytes]
[512MB]


Can't we just agree to use either MB or GB?  Or if we're in a verbose frame of mind, Megabytes or Gigabytes.  My response is to normalize the exceptions I can locate so that it comes out consistently with GB or MB, but I wonder whether this will remain a stable interface?

What I find even more entertaining is a daydream of an engineering team sitting around a table having a serious debate about changing the output from Megabytes to MB.  With such a controversial topic, I'd imagine the debate was heated.


Sun Certification: To dig, or not to dig? (part 1 of 2)
2007-06-17T13:09:00.000-05:00
Certification can be an almost religious debate amongst the technical community.  One faction believes whole heartedly that the measure of a technologist is his ambition, and list of accomplishments.  The other camp believes to death that certifications are a demonstration of professional commitment and a common ground from which to base skill assessments.  I am currently a Sun Certified System Administrator (SCSA) for Solaris 7 and Solaris 9, as well as a Sun Certified Network Administrator (SCNA) for Solaris 7.  I have been studying avidly for the upgrade exam which will add Solaris 10 to my SCSA listing which I hope to pass soon so I can reclaim the studying hours for other more interesting tasks.

I think just about everyone who has been in the field for a reasonable length of time has encountered the certification specter...  You know, the guy who has his Masters degree in CS or IS/IT, MCSE, CCNA, SCSA, and a few others tossed in for good measure.  They look like a lesser god on paper, but then you notice that once they log on to a system they can't write a script to save their life, and forget that shutting off the SSH daemon during business hours is a bad thing.  These academic savants are a big reason why certifications have a bad name.  In my mind they demonstrate the basis for the phrase, "just because you CAN doesn't mean you SHOULD."  A certification, in my mind is a commitment to understand the best practices and core tools within a product and apply that knowledge actively to your solutions and daily work.  A classic example is the proper use of init scripts - something that the majority of system administrators I have crossed paths with never learned.  This information is found easily in the Solaris System Administration documentation collection, so why is no one practicing it?  In this case, it has nothing to do with it being a bad practice to follow...  It's just a topic people do not bother to understand beyond the minimum required to make it work.

On the other hand, I have known many top-notch Solaris professionals who are not certified.  They can run circles around me in both theory and practice, but never took the additional step.  I don't respect them any less because they have demonstrated a commitment to their field through practice.  What I don't respect is the "average" SA who believes they could write the kernel scheduler in half the lines of code, but hasn't accomplished anything more advanced than setting up Apache Virtual Servers and using Veritas' Volume Manager to unencapsulate a root disk.  

I've listened to this type of person lecturing from their soapbox about how they don't need a certification to prove their skills.  Uh huh.  But it might take the edge off the cowboy hat, and create a spark of thought-discipline.  You see, being certified does not mean that you have to practice everything you learned.  It means you have taken the time to understand in depth one way of doing things.  The alternative is spending no time studying, and simply absorbing that which you cross paths with.

Another reason certifications have a bad name is that they do not address the real world.  Exactly how could a one-hour exam possibly compress all the operational knowledge one gathers by the time they are ready to be certified?  By now anyone reading my blog should be free of an doubt that I love Sun Microsystems.  Having reminded you of that point first, I will now say that I am not a fan of Sun's certification strategy in the Solaris Operating System Track.  I am basing my study on Sun's Web Based training curriculum, which I find generally outstanding as a substitue for instructor lead education.  My beef is not with the vehicle, but the curriculum.

As an example, I would estimate that one third of the training materials consumed my time with how to accomplish a task in the Solaris Management Console (SMC).  SMC is an interesting idea which flew about as well as a snail tied to a brick.  It's not all bad, but it's not all that useful.  I don't mind the option to use a GUI, but the amount of time spent on it in the curriculum is rediculous when considered against the amount of use SMC gets in the real world.  

Is it good to know how to use SMC?  Of course!  Especially for it's ability to manage local accounts (but it stinks for network information systems like NIS+ or LDAP).  But let's not worry about memorizing all of its menus and screens.  One of UNIX advantages is its ability to be remotely managed over a serial connection.  I'd never hire a UNIX SA who couldn't do his job proficiently over a 9600-8-n-1 connection.

Here's another sore spot for me...  One of Solaris 10's most incredible features is ZFS.  I have not begun to expand in my mind the full effect it will have on the industry, and it's not just a series of commands to memorize - it's an entirely new way to manage storage.  And yet, there is NO coverage of it on the Solaris 10 exam.  Are you KIDDING me?

Thankfully zones are covered, and I'm told that the exam had a good number of related questions.  However, the coverage isn't very deep, and sticks to the commands more than the theory.  That's unfortunate because it's easy to look up a man page, but hard to design a well thought out consolidation platform.  I'd say that sentence sums up my thoughts on certification strategies on many levels.

Resource management is another feature which seems conspicuously absent from the certification curriculum.  Although it is very complex (aren't zones as well?), it is a very powerful Solaris feature which I believe is a competitive advantage for Sun.  So why not expect a certified administrator to know how to use it?  The idea isn't to make everyone feel good with a title on their business card, it is to demonstrate that someone has differentiated themselves by demonstrating a defined level of skill.

What else would be important for an SA to have cursory knowledge of?  DTRACE, any one?  I don't expect every competent Solaris administrator to be able to write advanced D scripts, or memorize the seemingly infinite number of probes available in Solaris, but for the love of McNealy, can't we even expect them to know what kind of problem it solves?  Can't we even establish what a probe is, and why Solaris is WAY ahead of Linux in that respect?

Finally, the emphasis on memorizing obscure command line options really grates on me.  This is really what undermines the technical merit of Sun's Solaris Certifications.   There are so many commands and concepts that deserve coverage, it seems a shame to take up space with questions like, "What is the option you must give to ufsdump in order to ensure /etc/dumpdates is updated when usng UFS snapshots."  I don't know anyone practicing in the field who wouldn't look that up in the man pages even if they THOUGHT they knew the option.

I could go on, but I think the point is made:  The exam is lacking in strategic substance.  As a result, most folks describe it as a memorization excercise.  That's a shame, because the exam COULD be a differentiating ground for Solaris professionals as well as a way for Sun to ensure the compelling features of Solaris are being leveraged to their fullest.  And yet, I'm getting ready to take my third SCSA exam.  Why?

I maintain the currency of my Solaris Certifications because I believe a professional seeks to understand standards in their field, whether good or bad.  As a professional Solaris system architect, the SCSA and SCNA exams are at the core of my practice whether I choose to follow or deviate from their content.  I also believe that a certification tells my customers (or employers) that I demonstrate a certain level of competence, even if the bar is not as high as I would like to see it.  

I believe deeply in the importance of standards and certifications as a vehicle to advancing the maturity of Systems Engineering practices as applied to system administration.  And although Sun's certifications are not there yet, I will continue to support them for what they do provide, and what I hope they will provide in the future; A vehicle to advance the maturity of the industry.

Part 2 of this article will discuss my recommendations for improving Sun's Certifications.  As usual I have a few ideas up my sleeves.  Stay tuned...


Who's on first?  Identifying port utilization in Solaris
2007-05-21T15:12:00.000-05:00
Setting up Apache2 on Solaris 10 is normally about as challenging as brushing your teeth.  But in this case, I was humbled by an unexpected troubleshooting adventure.  I needed to transfer a TWiki site from an Apache server running on Solaris 9 to an Apache2 server running on Solaris 10.  Sounds pretty straight forward, but I abandoned discipline at one point in the game and that detour came back to bite me.

I started carelessly thinking that to make things simple I would just use the legacy apache.  This would save any initial headaches with module incompatibilities (if any existed).  So, I started out copying the config file in place and trying to start the daemons.  It didn't work, and after a few minutes of fiddling with the new httpd.conf I changed course.  My reasoning went something like this, "If I'm going to spend much time fiddling, I might as well fiddle with Apache2 and have something better than I started with."  And so it began.

I stopped the legacy Apache daemons and followed a similar process with Apache2, ending with the same result:  No daemons.  I did some fiddling and located a minor typo I'd made in the configuration which is not of consequence to this story.  I issued a "svcadm restart apache2" command.  Yeeha!  Now I had five httpd processes just chomping at the bit for a chance to serve those Wiki pages.

Or did I?  It turned out that no matter what I did with my web browser remotely or locally I couldn't get a response.  So, I tried a quick telnet to port 80 to see what there was to see...  And of course I received a response, so all must be well.  Somewhere in my troubleshooting process I made two mistakes:

First, I didn't remove the httpd.conf file from /etc/apache, which means the legacy Apache starts up and conflicts with Apache2 on a reboot.  I've already written an article that goes into some detail about why the current legacy Apache's integration isn't ideal, so I won't expand on my frustration in this one.  This problem was quickly solved, and could have been avoided if I had adhered to my Jedi training.

Second, I assumed that when I directed a Telnet session to port 80 it was reaching the Apache2 server.  In fact, it was not.  I shut down the Apache2 server and again issues the Telnet command to port 80.  Surprise!  The same greeting appeared.  So, some process on the system had claimed port 80 before Apache could do so.  Now...  To find it!

Linux distributions typically ship with the lsof utility.  This provides a quick and convenient way to identify what process is using what TCP port.  Solaris doesn't have lsof in the integrated Open Source software (/usr/sfw) or the companion CD (/opt/sfw).  It's not hard to obtain and compile, but it's just inconvenient enough that I'm inclined not to do it.  My next logical question became, "what is the Solaris way to accomplish my goal?".

Solaris has no way to natively solve this issue without a shell script.  There are a number of similar scripts available on-line through a quick Google search.  None are particularly complex, but complex enough that you wouldn't want to write them every time you need it.  Here's what I ended up with:

#!/bin/sh

if [ `/usr/xpg4/bin/id -u` -ne 0 ]; then
   echo "ERROR: This script must run as root to access pfiles command."
   exit 1
fi

if [ $# -eq 1 ]; then
   port=$1
else
   printf "which port?> "
   read port
   echo "Searching for processes using port $port...";
   echo
fi

for pid in `ps -ef -o pid | tail +2`
do
   foundport=`/usr/proc/bin/pfiles $pid 2>&1 | grep "sockname:" | egrep "port: $port$"`
   if [ "$foundport" != "" ];
   then
      echo "proc: $pid, $foundport"
   fi
done

exit 0


When executed, it will produce output similar to the following.  Note that it requires root permissions to traverse the proc directories...

cgh@testbox{tmp}$ sudo ./portpid 80
proc: 902,      sockname: AF_INET 0.0.0.0  port: 80
        sockname: AF_INET 192.168.1.4  port: 80
        sockname: AF_INET 127.0.0.1  port: 80

A quick "ps -ef " command told be that our Citrix server was to blame for the port conflict...


cgh@testbox{tmp}$ ps -ef |  nawk '$2 ~ /^902$/ {print $0}'
 ctxsrvr   902     1   0   May 18 ?           7:00 /opt/CTXSmf/slib/ctxxmld


Ah ha!  Problem solved.  I'd like to see the Solaris engineering team add a "p" command, or an option to an existing command to make this functionality a standard part of Solaris.  Another option would be to integrate the Linux syntax for the fuser command to make this possible.


Apache in Solaris 10:  3 Simple Things I Would Change
2007-05-18T09:11:00.000-05:00
The Apache legacy run control script in Solaris 10 (/etc/init.d/apache)  provides an excellent example of a few practices to avoid when writing init scripts.

Take a look at the code snippet below:

if [ ! -f ${CONF_FILE} ]; then
       exit 0
fi


Are you kidding me?  Of course this is easy to debug, but  let's look at what it does anyway:  If the configuration file is missing, when you ask to start Apache, and it will exit with a code of zero when it doesn't find the /etc/apache/httpd.conf file.  In case you didn't catch the first four words of this paragraph I'll repeat them.  Are you kidding me?

Here's a simple improvement...

if [ ! -f ${CONF_FILE} ]; then
       echo "ERROR: ${CONF_FILE} not found.  Exiting."
       exit 1
fi


The first change was to exit with a non-zero status.  Zero is the UNIX standard exit code representing successful completion.  If the configuration file is missing and you request a startup, it should NOT exit with a zero status.

The second change is to provide a concise error message indicating why the exit code is going to be zero.  There is no benefit to bolstering the cryptic nature of UNIX.  In my mind the best systems are designed such that a tired SA at 4AM has a reasonable chance of accurate debug and corrective action.

Having said all this, the reason the code is necessarily convoluted because the not-yet-configured service has an active set of init scripts in the run control directories.

cgh@testbox{etc}$ ls -i /etc/init.d/apache     21813 /etc/init.d/apache*
cgh@testbox{etc}$ find /etc/rc?.d -inum 2813
/etc/rc0.d/K16apache
/etc/rc1.d/K16apache
/etc/rc2.d/K16apache
/etc/rc3.d/S50apache
/etc/rcS.d/K16apache


So the root cause of our problem is that someone decided to make it easy for someone who doesn't understand the Solaris Run Control facility to start Apache by simply creating the httpd.conf file.  Is that really a good idea?  I would argue that for many reasons it's a bad practice.  If a service is not configured to run, it should not be active in any run level.

The third detail I would change is Solaris' default behavior of installing active sym-links in the legacy rc directories, and instead use an SMF manifest that adheres to standards.

None of this impacts the otherwise excellent web server that Sun has integrated into their OS, and I'm grateful that Sun has provided it in their standard OS rather than leaving it to the semi-integrated Companion CD.  I woudl, however, like to see that integration brought up to Jedi standards.

5/21/07 Postscript:  I probably should have made it clear that the Apache2 server is implemented nicely using SMF, and is probably what you ought be to using on Solaris 10 if you've decided to forego the JES Web Server.  I don't think that excuses the older Apache server from maintaining Jedi discipline, but it does move the issue a bit toward the background.


Turn off the LAMP and Reuse Acronyms
2007-05-07T19:30:00.000-05:00
I've never been a fan of the LAMP acronym because it's too restrictive. It gives the impression that to be socially responsible in the Linux community one needs to be a LAMP developer.

In this month's Linux Journal magazine I  found an article explaining one perspective on why PostgreSQL is a more desirable database than MySQL.  I've had the exact same thought process for years now.  Truth be known, I also prefer Perl development to PHP, and I prefer running the stack on Solaris over Linux.  I guess that SAPP doesn't have the same sexy ring as LAMP.  There's probably an odd trademark thing with an ERP company as well.

Now before you get too bent out of shape, I am aware that the acronym has some poetic license with it, and people often swap Perl and PHP, and in theory any other letter can be swapped out.  Why invent a new acronym that doesn't convey the real idea when a perfectly good acronym already exists?

There is nothing wrong with simply stating that an application is built on an Open Source Stack.  The acronym OSS (Open Source Software) is well known and conveys a lot more than LAMP.  It stands for a methodology rather than a point solution, and embraces the foundation that made "LAMP" so successful.  Why limit yourself to MySQL and PHP?  Wouldn't you be more valuable as an architect capable of leveraging the most appropriate components Open Source has to offer?



Another Round with the Laptop
2007-04-17T19:37:00.000-05:00
Having recently switched a bunch of older Fedora Core servers to CentOS 4, I became very excited to see the announcement that CentOS 5 has been released.  I am very pleased with the polish and stability of CentOS 4 and thought 5 would be a perfect update for my (then) Ubuntu laptop.  My laptop is a fairly old IBM Thinkpad T23 - certainly not bleeding edge hardware.

The display was maxed out at 800x600 rather than the native 1400x1050 I'm accustomed to and my NetGear wifi card was recognized, but not functional.  It comes as no surprise that none of my Thinkpad buttons worked.  I spent the past 10 years of my hacking career tweaking Linux boxes, and I was using X windows in the days when you actually risked toasting your monitor with a bad config.  So, yes, I could make X work.  And I have gone through the Windows wifi card firmware dance, so yes, I could make the Wifi card work as well.  I could also recompile the kernel and add Thinkpad buttons and ACPI events.  Yuck.  My rebuild just turned into a lot of research and work.

On a whim I thought I'd try Fedora Core 6 just in case my problems stemmed from the more conservative approach CentOS takes.  Same problems, although I love the eye candy in Fedora - their art is great.

I didn't bother to try Solaris x86 because I know it won't detect any of my special hardware.  Someday I'd love to be able to use Solaris outside of work, but that day hasn't come yet on the desktop.

So here I am, back with Ubuntu.  It detects everything and works right out of the box.  Within an hour I had a fully functional laptop, and despite my 1 GHz CPU in a 5GHz world it performs great.  I suppose a year from now I'll try it again since I prefer a Red Hat based distribution to a Debian base.  But what matters most comes down to two words: "It Works."

Thanks, Ubuntu.


Does Your Virualization Strategy Have a Blind Spot?
2007-04-01T12:47:00.000-05:00
I just finished reading a very interesting article about virtualization.  It described the test results of running two sample web applications under virtual and physical environments.  The idea was simply to check how the virtualization affected the application's performance.  The results are interesting.

Before I go further, let's clarify that this article is about Windows IIS and Win2k3 server, so it is not about Solaris, or any other flavor UNIX beyond the fact that the underlying OS for VMWare in this case was CentOS.  None the less I believe the story's moral is highly applicable to any operating environment, definitely including UNIX.

There is a lot of detail in the article, but if we can assume their methodology to be relevant, we can reduce it down to a short extract from its conclusion:  "a virtualized server running a typical web application may experience a 43% loss of total capacity when compared to a native server running on equivalent hardware."

Wow.

Of course, I have no reason to think that this function carries over directly to Zones, Xen, or even VMWare running Linux.  To believe this without research would be to abandon one's Jedi training.  It would be a very valuable experiment to try such an excercise though, because I have no reason (at this moment) to think it would not have some relevance.

The big point here I want to make is that I would bet there are a great many sites out there who dilligently track low hanging fruit metrics like CPU utilization, and use that metric both in planning and asessing their virtualization projects.  Server "A" runs an application and on average sits at 10% utilization.  Lets call it 100Mhz and find it a home on the consolidation farm.  Not so fast.

The problem is that if you completely ignore business metrics, you will fail to identify this article's identified flaw in virtualization.  Hey, we moved from 8 servers down to one, and its at 80% all the time.  Success!  But what if that new server is running at 80% and only processing 50% of the volume it used to?  I would suggest that the article I referenced might be a great reason to reflect on your site's metric stratgey and see if you have the ability to accurately asess a consolidaton stratgy.  If not, you could be making an expensive transition.


On Proper Use of English
2007-03-06T15:21:00.000-05:00
While my mind was entering a virtual screen-saver (yes, oddly enough my mind seems to run a GUI over its CLI interface) I overheard a conversation in the next cubicle. A project manager said, "...and then we give it to the engineers for solutioning". I started to think about how often I'd heard that expression and quickly realized the count was quite high.

Shortly after that exciting revelation (much more exciting than the status bar I watched during an in-progress installation) I recalled another expression I'm finding has become prevalent on our site: The act of "dialoguing". I'm not completely sure of the spelling of this colloquialism, but I'm hearing it all the time.

If I'm understanding them correctly, my job as an engineer entails dialoguing with customers to gain requirements which I can use in the solutioning process. Ain't that it?

From the Merriam Webster Dictionary, we learn that "solution" is actually a noun. Just to remove any ambiguity from my position on this matter, let me also state that a noun in this scenario cannot moonlight as a verb and still retain its dignity.

Continuing on our lexical journey, the MW Dictionary also has an entry for dialogue. It's simply amazing to find such an artifact considering how few people have been able to study this elusive part of speech. But, in the interest of open sharing of knowledge I'd like to share what it contains. Dialog is also a noun. Amazing!

Having solved this perplexing grammar mystery I can now return to designing a solution to the problem of how to virtualize our Directory Service. I hope you've enjoyed this dialogue.


JET: Controlling custom_files with a custom extension
2007-02-28T20:48:00.000-05:00
Any site running Sun hardware with more than one system should be looking at JumpStart to ensure that systems can be rebuilt consistently.  the corollary to this is that any site running JumpStart environment should be using Sun's Jumpstart Enterprise Toolkit (JET).  JET provides a consistent framemwork for accomplishing most common tasks, and a consistent framework to write extensions within.  Standards and discipline are good.

One of the modules which comes with JET is called simly enough, custom.  The custom module allows you specify either packages or files which should be added to a server during any of N predetermined reboots.  This allows you to ensure that a change which requires a reboot can be made prior to a dependent process being started.  Sounds good so far.

Following a recent Solaris 9 server build I was perusing the system for problems by auditing log files.  In the messages file I discovered some lines indicating that a Kerberos problem was rearing its ugly head:
Kerberos mechanism library initialization error: No profile file open.
Our site does not use Kerberos, so it had to be a recent configuration change - not surprising considering we had just updated the patch set.  After some research I arrived at BugID 5020096.  This bug indicates that the issue can be resolved by removing some offending lines from /etc/krb5/krb5.conf.

This should be easy enough to fix in future builds.  Just add the modified krb5.conf to the JET template's custom_files variable, and we'll be in good shape.  Ahh, not so fast.  How will we know what the file originally contained?  A true Solaris Jedi will always manage an audit trail of his activities.  If I were making the change manually I would copy the file to file.orig, or file.datestamp.  Automation is not an excuse for abandoning discipline.

The trouble with JET is that its custom module's functionality for installing files is limited to two operations: overwrite or append.  Overwrite simply clobbers any file which may exist.  For example, to install the /etc/motd file I would palce my custom file in the configured JET file location, then add a line like this to the JET template:
custom_files_1="motd:o:/etc/motd"

motd is a fairly harmless little file, but knowing little about Kerberos, I dind't want to blindly whack the original file.  The right solution to this problem lies in creating a simple extension to the JET toolkit.  I began by examining the code from the custom module.  Two modules specifically are relevant to this project:  install, and postinstall.  Within them is a simple case statement which handles the "o" or "a" functionality:

case ${mode} in
   a) case ${fn2} in
      /etc/hosts)   JS_merge_hosts ${filefound};;
                     *)  JS_cat ${filefound} ${ROOTDIR}${fn2}
      ;;
      esac;;
   o) JS_cp ${filefound} ${ROOTDIR}${fn2};;
esac


So, when I use an "o" in my custom_files module, it called JS_cp.  I now needed to find the library which contains these core functions.  Eventually, a colleague and I traced it back to /opt/SUNWjet/utils/lib.  Looking at the JS_cp function revealed exactly what I expected: a simple copy routine wrapped in some voodoo.

Feeling a bt optimistic, I copied JS_cp to JS_cp_preserve and modified the code a bit so it would first check to see if the destination file exists, and if so, backup the file with a datestamp.  Once the backup was in place, the original copy operation was performed.  This was very trivial shell scripting.  Here's what I ended up with:

if [ "$#" != "2" ]; then
        JS_error "`basename $0`: Illegal Arguments. Usage:  "
fi

JS_FROM=$1
JS_TO=$2

JS_display "Copying file `echo ${JS_FROM} | sed -e \"s?^${SI_CONFIG_DIR}/??\"` to ${JS_TO}"

if [ -f ${JS_TO} ] ; then
   datestamp="`/usr/bin/date +%Y%m%d`"
   /bin/cp -p ${JS_TO} ${JS_TO}.jet.${datestamp}
   case $? in
      0) # Success
         JS_display "Successfully preserved ${JS_TO}.jet.${datestamp}"
         ;;
      1) # Failure
         JS_display "WARNING: Failed to preserve original file ${JS_TO}"
         ;;
   esac
fi

/bin/cp -p ${JS_FROM} ${JS_TO}

if [ "$?" != "0" ]; then
        JS_error "JS_cp:\t\tError occured while copying ${JS_FROM} to ${JS_TO}"
fi


Next, I returned to the install and postinstall code, and modified the case statements to accept a "b" operation (b for backup).  I then executed a test Jump and was very pleased to see my JET extension had worked!  I can now have custom_files install the workaround krb5.conf, and maintain a backup of the original.  Here's the modified code:

case ${mode} in
   a) case ${fn2} in
      /etc/hosts) JS_merge_hosts ${filefound};;
               *) JS_cat ${filefound} ${ROOTDIR}${fn2};;
      esac;;
   o) JS_cp ${filefound} ${ROOTDIR}${fn2}
   b) JS_cp_preserve ${filefound} ${ROOTDIR}${fn2};;
esac


Note that you need to make this modification in both /opt/SUNWjet/Products/custom/isntall and postinstall.

Now, all I need to do it specify something in the custom_files module like this:
custom_files_N="krb5.workaround:b:/etc/krb5/krb5.conf"

And I will get a clean backup of the original file.  Such a simple tweak - I hope the Sun folks who maintain JET will add something similar.  While some limitations of JET can be frustrating, its intuitive layout and ease of extension make it something I grow more fond of each time I use it.


Frustration with Solaris Packages
2007-02-01T09:30:00.000-05:00
I have a love / hate relationship with Solaris packaging. When you need to crank out a simple package I find it much easier to deal with than RPM. I also like the simple file system or streams based stucture vs. the binary mode of RPM. All things considered, it gets the job done, and has been a tremendous help in standardizing our provisioning system. There are, however, a few things that I'm not crazy about.

In Sun's model, the package is used to release functionality while the patch is a vehicle to fix existing functionality. If I want to add feature X to my software, I need to release a new package version. In contrast, if feature X is broken in a package then I need to release a patch. Seems simple on the surface.

One place this model gets sketchy is if I have the following situation: Package FOO needs to be updated to a new revision, but package BAR depends on it, and is required for system operations. In this case I need to first remove package BAR, then update package FOO, and finally, reinstall package BAR. In my mind this causes an unjustified level of system disruption. An RPM or dpkg based system would use an update option to perform this in-place. I'm told that there's an "in place upgrade" capability in the Solaris packaging system, but I haven't yet discovered it or found it documented. I will be looking though.

I have also noticed documentation gaps in the use of patches. Sun does provide instructions on how to produce a patch-package, but they omit naming conventions. Clearly, it would be a bad thing to produce package 123456-01 and then have Sun release the same one. This conflict could be very disruptive to a patch process. It seems that by selecting an upper range (ie 90001-01) you can have safety similar to selecting a 10.0.0.0 network address. I'd feel quite a bit better if Sun woudl explicitly define this range so we'd know it was safe. In the interim, I've been fixing bugs by creating minor revisions of packages rather than using patches.

The last point I wanted to touch on in this article is the use of package prototypes. In packaging nomenclature, a prototype file is the list of files included in the package, and their ownership and permission attributes. Here's an example of a prototype I'm durrently working for a custom sendmail solution:

d none etc 0755 root sys
d none etc/mail 0755 root mail
f none etc/mail/foo-client-v10sun.cf 0644 root bin
f none etc/mail/foo-server-v10sun.cf 0644 root bin
d none usr 0755 root sys
d none usr/lib 0755 root bin
d none usr/lib/mail 0755 root mail
d none usr/lib/mail/cf 0755 root mail
f none usr/lib/mail/cf/proto.m4 0444 root mail
f none usr/lib/mail/cf/foo.m4 0644 root mail
f none usr/lib/mail/cf/foo-client-v10sun.mc 0644 root mail
f none usr/lib/mail/cf/foo-server-v10sun.mc 0644 root mail


Pay particular attention to what I call placeholder lines. Those are lines in the prototype referring to directories which this package depends on, but are really part of another package by virtue of already being registered. Of course, a directory like /usr is chocked full of nested package dependencies:

# grep ^/usr /var/sadm/install/contents  head
/usr d none 0755 root sys FJSVvplu SUNWctlu SUNWcsr TSBWvplu SUNWocfd SUNWncft SUNWGlib SUNWgcmn SUNWGtku SUNWctpls SUNWxwdv SUNWpl5u SUNWcpp FJSVcpc SUNWopl5p FJSVcpcx FJSVmdb FJSVmdbx IPLTadman SUNWowbcp SUNWpamsc SUNWpamsx SUNWpcmcu IPLTdsman SUNWadmj SUNWmcdev SUNWjsnmp SUNWtftp SUNWbsu SUNWpd SUNWsckmu SUNWpdx SUNWpiclh SUNWuxflu SUNWuxfl1 SUNWeurf SUNW1251f SUNWuxfl2 SUNWuxfl4 SUNWuxfle SUNWmgapp SUNWrmui SUNWpiclx SUNWpl5p SUNWTcl SUNWjpg SUNWTiff SUNWTk SUNWaccu SUNWaclg SUNWadmap SUNWpng SUNWpool SUNWpoolx SUNWant SUNWrcmdc SUNWpppd SUNWpppdu SUNWpppdt SUNWpppdx SUNWpppg SUNWfns SUNWsadml SUNWapct SUNWascmn SUNWasac SUNWqosu SUNWjaf SUNWjmail SUNWxsrt SUNWxrgrt SUNWxrpcrt SUNWiqfs SUNWiqjx SUNWiqu SUNWiquc SUNWiqum SUNWjaxp SUNWasu SUNWasdem SUNWrmodu SUNWrmwbx SUNWrpm SUNWrsg SUNWfnsx SUNWrsgx SUNWdfbh SUNWsadmi SUNWi15cs SUNWsadmx SUNWi1cs ... (lines omitted)


That was a tiny fraction of the list...

I don't think there is anything wrong with declaring a package as being dependent on a pre-existing directory, but I have a problem with how easy it is for a new package to overwrite the intende dattributes of that directory. Note that in my custom package's prototype I need to declare the attributes for /usr. This typocally means that I need to look at a clean operating system on the platform I intend to deploy on (ie - consistent Solaris revision) and pick the attributes from there.

I'd like to see the packaging facility accept a prorotype entry that has no attributes, and instead inherit the attributes from the package which initially registered the directory. This would minimize the chances of stray patches and packages conflicting with intended system permissions.

Having spent all this time complaining, let me end on a positive note by reinforcing how much efficiency we have gained by moving from tarballs and custom scripts to version controlled packages. I'd do it again in a heartbeat. I'm hoping Jedi discipline will eventually reverse the chaos inherent to the current packaging architecture.


Perl: Beware when counting array elements
2006-12-20T14:51:00.000-05:00
Sometimes it's the simple things that put a halt to my productivity.  In fact, it's rarely the big things.  Master Yoda gave us an excellent phrase to consider when addressing a programming problem:  "Judge me by my size do you?  And well you should NOT!".

When I need to check how big an array is in Perl, it's quite intuitive to access that value using the $#array_name special variable.  It works like a charm.  But what happens when the array you need to count elements in is a referenced array?

A typical de-reference operation would be @$array_reference.  This gives us the underlying array.  Forgetting my Jedi training, I allowed the Dark Side to i$nfluence me as I flailed through syntactic permutations.


$$#array_reference
@$#array_reference
$#{$array_reference}


Much to my dismay, these futile expressions continued to plague me.  I then focused my energies, and put the blast shield down.

...From WikiQuote

    Ben: Remember a Jedi can feel the Force flowing through him.
    Luke: You mean it controls your actions?
    Ben: Partially, but it also obeys your commands.
    Han: Hokey religions and ancient weapons are no match for a good blaster at your side, kid.
    Luke: [to Han] You don't believe in the Force, do you?
    Han: Kid, I've flown from one side of this galaxy to the other. I've seen a lot of strange stuff, but I've never seen anything to make me believe there's one all-powerful Force controlling everything. There's no mystical energy field controls my destiny! It's all a lot of simple tricks and nonsense.
    Ben: I suggest you try it again Luke. This time, let go your conscious self and act on instinct.
    [Ben places a helmet on Luke's head with the blast-shield down to blind him]
    Luke: But with the blast shield down I can't see a thing!
    Ben: Your eyes can deceive you. Don't trust them. Stretch out with your feelings.
    [Luke calmly evades and deflects three pulses from the remote, successfully using the Force]
    Han: I call it luck.
    Ben: In my experience, there's no such thing as luck.
    Han: Look. Good against remotes is one thing. Good against the living…that's something else.
    Ben: [to Luke] You've taken your first step into a larger world. 


I wrote a quick code fragment to isolate the problem and experiment with it.  In doing so, I was reminded of the true nature of dereferencing and array contexts.


#!/usr/bin/perl -w
use strict;

my @a = qw(one two three four five six);
my $aref = \@a;

print "Number of elements in the array: $#a\n";
print "Number of elements in the array (ref): " . @$aref . "\n";


Looking at this code, ask yourself what values will be produced in each print statement.  With carelessness likened to use of a blaster, one may assume they will produce the same integer.  In fact they will not; A fact whose presence will be known through the Force.  Let's take a closer look at these expressions.

@a is a literal array.  The $# special variable is a direct vector to the last element used by that array.  Knowing that Perl indexes its arrays beginning at zero, we then know this line will print the integer 5.

@$aref is a de-referencing operation.  It represents the literal array, rather than the scalar that points to it.  In this case, we are printing the array in scalar context, which will evaluate to the number of elements in the array: 6.

A simple problem with a not-so-simple explanation.  What if your code had thousands of elements in it...  Would your test plan have covered this condition when you decided to use referenced arrays rather than literals in one of your code branches?  Remember that the Jedi Code tells us ...there is no ignorance; there is knowledge.  Code, and learn.


Packaging delinquency in 3rd party software
2006-10-20T20:02:00.000-05:00
We're working hard at the moment on migrating unmanaged scripts and solutions into Solaris pkgadd format, a.k.a. "packages".  This has a number of benefits.  First, it avoids the need to have complicated manual installation routines;  By including preinstall / postinstall scriptsinstallation can be automated.  Second, it is easy to know what revision of a software environment is installed right down to the installation process.  Third, it ensures that not only can I ensure software is installed with the ight attributes and in the right places, but also that I can validate at a later time things are still as intended using pkgchk.  I refer to this as managed files vs. unmanaged files.

The easy part was packaging our custom site scripts.  We standardized on a hierarchy under /opt which contained bin, man, etc, lib, and sbin subdirectories.  We then inventoried the unmanaged scripts and determined which were still valid and which could be discarded.  The "keepers" were then incorperated into packages which were fairly simple overall.

Phase two has been an asessment of what unmanaged files for third party applications are being deployed.  We found a lot of opportunity here and decided to start with the utility software like monitoring, security, and other non-revenue generating software.  Here is where we have been uncovering nothing less than a mess.

For some reason, third party software providers in the UNIX space seem determined to make it impossible to manage their files.  We've seen many interesting perversons of best practices that I thought woudl be interesting to collect in one place.

One product choose to adopt a package management solution called LSM, which I believe stands for Linux Software Manager.  Note that this solution is for Solaris which has a perfectly good vendor provided and supported standard for software management.  It turns out to be quite a technical feat to reversen engineer the format of LSM and directly convert to packaged.

Another product did not use any software management, but went so far as to encrypt their pre-installation bundles so as to make it impossible to install via a standards-based management system.  This really blew my mind.  What could be of such critical intellectual property in an installation routine that it justified encryption?  And wouldn't the real IP be available once the software was installed anyway?

We routinely encounter software that has highly interactive installation processes that become cumbersome to integrate into packages because on each new release the routines would need to be re-ported to pre/post install scripts.  The idea of managing software is to reduce work - not increase maintenance.

The biggest thorn in our side is Oracle.  It's deployed everywhere in our environment and is intalled manually each time because we're told that's just how its done, and from observation it seems to be in the PITA bucket as far as automation goes.  Contrast this to PostgreSQL which as of Solaris 10 (6/06) is integrated into the Operating Environment in clean packages.

So here's a message to all you third party software developers who provide Solaris solutions:  Sun publishes an excellent guide to software packaging that any reasonably technical person could use to master the process in a few hours.  Let me summarize a few key points in advance:

(1) You don't need to have a conversation to install software.  Just copy the files, then configure it later.

(2) Sometimes I don't want to have a conversation.  Let me put the answers in a file and feed that instead.

(3) Some of your customers have too many systems to install manually.

(4) Don't use a non-standard solution when the OS vendor provides a perfectly usable solution.


To err is human
2006-10-19T20:20:00.000-05:00
I am writing this post as a catharsis and purification.  A centering of my spiritual engineering energy that may otherwise be out of balance.  Three days ago I made a typo which eliminated the /etc directory on a fairly important server.  It was amazing how long that server continued to plug away after being lobatomized.  Let me take you through the story as I relive the moment, and ensure that I learn from it.

Like many of the tasks I juggle, this was to be a short time-slice effort.  I needed a distraction from a longer term project, and wanted to bite off a small piece of something that didn't require significant thought.  Part of our Jumpstart environment deploys a tar archive to the client which is later unpacked and massaged by a custom script.  My task was to eliminate the usr/local/etc directory from that archive and than recreate it.  As my fingers systematically hit the keys, one extraneous finger made its imprint on the keyboard.

"r" "m"  "-" "r"  "."  "/" "e" "t" "c".

The world slowed down as my finger hit enter, and I felt my heart stop beating.  I believe I actually flat-lined that morning.  Could it be?  Had I really deleted /etc?  Yes. I had.  The command I entered was: "rm -r . /etc".  I removed the current working directory and the server's /etc directory.

Why was I using elevated privileges for mundane work?  The tarball had root-owned files in it.  This is a downfall of our approach at the moment.  When using pkgadd format, anyone can own the files which are given attributes at installation time.  This makes day to day maintenance much safer.  Ironically, I was editing the archive because I had just created a package to replace the files I was deleting.  It was almost as if the prior bad practice were vomiting on me as I excercised it from the server.

Fortunately we had an excellent SA on hand to boot from CD and restore the missing file system, and it was back in business a relatively short while later.  Eningeering nad operations are segregated in duties at my current site, so I was unable to clean up my own mess.  A very humbling experience indeed, and this is what it taught me:

(1) Mirrored operating system disks are a good thing, but they don't protect you from human error propogating mistakes across both disks.  While I've been a bit critical of maintaining a third contingency disk, there are other similar solutions which I have a heightened respect for.

(2) Whenever executing commands using RBAC, sudo, or the root account, count to three before hitting enter.  No matter how much longer it takes to get your work done, no matter how good you are with UNIX, and no moatter how long it has been since you made a mistake, counting to three will always be quicker than restoring a file system from tape.


Sun loves Oracle, Sun loves PostgreSQL
2006-10-18T19:15:00.000-05:00
Sun and Oracle have announced they will work together for another ten years.  Not only that, but there's a new bundle in town that includes Oracle Enterprise with Sun servers.  I haven't exactly figured out what it means to have software included for free that will require a support contract; Is that still free?  But there were words indicating that processor count may not be relevant and that's probably where the savings lie.  Maybe it's just saving you download time?

I don't really care about the pricing because big companies don't seem to hesitate to throw down dollars for Oracle licensing.  What made this interesting to me was that Solaris, as of the 6/06 update now includes PostgreSQL natively - and there's no catches there.  If you perform a "full distribution" install you already have an RDBMS.  What's more, if you want to take that database into the critical waters of the production pool Sun will offer their world-class software support which means the company that knows their own operating system better than onyone else will also know the RDBMS sitting on top of it.  Tres chique, n'est pas?

I'd imagine with Oracle's market share Sun has to play nice in the short term, but I give them a lot of credit for including PostgreSQL and picking a side.  Right or wrong, in the age of mediocrity they made a decision.  PostgreSQL is a phenominal database that competes aggressively with Oracle in many venues.  

I'm fascinated with what the future will hold for relational databases on Solaris now that Sun has picked a side.  This isn't just another open source database running on Linux farms - this battle will take place in the big data centers that Linux is just starting to scratch the surface of.  I love Linux as much as the next guy, but how many sites do you know of running systems as large as an Enterprise 25K with Linux under the hood?  Not too many - it's not in the heritage of Linux kernel - at least not yet.

So, where will this take Postgres?  Methinks Oracle had better keep close tabs on Postgres over the next five years or so.


Adventures on eBay
2006-09-28T11:15:00.000-05:00
I'm finally creating a Sparc based lab outside my place of employment.  I want to make sure I can work on projects without any hint of conflict of interest of proprietary intellectual capital nonsense.  My work is for hire from 8-5, but my ideas remain my own, and I want to be able to implement them outside of work with freedom.

I've been using an old AMD based system with Solaris x86 for a while now, but since my real interest is in Enterprise Engineering I really wanted to have some Sparc based equipment and at least one disk array.  This will open the doors to many projects that would only be feasible on expensive modern x86 hardware.  Funny how those same features have been on Sparc hardware since the early days...

For budget reasons I've gone with a v240 as my workhorse server.  It doesn't have the sweet LOM that many Sparc system enjoy, but it does have four CPUs and four GB RAM.  That will give me plenty of horsepower to run a few Oracle databases and work on some Zone projects.  A v440 would have been ideal for my goals, but the price point on those systems is WAY too high for the investment returns I'll be looking at in the short-term.

I have the v240 server in my office right now, and have been playing with it a bit before it goes to the basement rack.  After attaching a console cable and booting I was met with a hostname from the test.aol.com domain.  I was a bit surprised to find that AOL doesn't wipe their disks before sending servers to auction, but after a quick look around I discovered that there isn't really anything on the disks anyway; just a Solaris 8 OS image.  I almost skipped checking it out...  I have zero interest in cracking, least of all a Solaris 8 image.  I did have a small interest in their best practices, but I'm far more interested in my own projects, so the box will be getting a fresh load of Solaris 10 (6/06) this weekend.

My storage array of choice is the D1000.  I'm not going to do much that requires massive expansion or throughput.  I just need a bunch of disks I can put into ZFS, share for a cluster, and run databases on.  The D1000 has a 12 disk Ultra-SCSI backplane that provides plenty of throughput for my needs.  And hey, there's not much worry about driver obsolescence.  It's such a simple device that it's just going to work as long as Solaris continues to support SCSI.  The D200 is very cool, but I just couldn't find a value-add for my project list, so I went with budget.  I won the bid for this unit this morning, so I'll have it in a week or so.  Then I need to fill it with disks.

I upgraded my measly 16 port 10mb SuperStack II hub to a 24 port Superstack II Switch.  This will ensure the servers have plenty of bandwidth to talk amongst themselves.  I'm particulary proud of this purchase - only $1 plu s shipping.  Works for me.

A few white papers from now I'm planning to add a second v240 for work on SunCluster and alternative clustering solutions.  I'll also be adding a pair of Netra T1 AC200 servers for a directory services project. 

And that should just about round out the new data center.  I'm amazed at how much capability old servers have.  CPU technology has progressed so much faster than the remaining system bottlenecks that many of today's systems simply will never show significant CPU utilization.  For me, this means that if I'm willing to run servers at a level that makes them sweat, I can accomplish the same work for a fraction of the cost.  This is the premise for a white paper I'll be working on that explores the use of old enterprise class hardware in not-for profit or small business shops.  Stay tuned!


Shells and the path of standards
2006-09-27T11:36:00.001-05:00
It seems that shells evoke almost as much controversy as religion.  Everyone has a favorite, and for each favorite, there's another that would sooner grind their knuckles on the back of a circuit board than follow.

As long as I've been using Solaris systems I've never found common ground with the C-shell.  Its programming capabilities are more limited compared to other options and its syntax is just too far from standards to be worth learning to me.  But C is a standard UNIX language, you say.  How can I call C-shell non-standard?

I'm a systems engineer, not an application developer (and now's not the time to debate that grey area!).  If I were to write a script to traverse the entire Operating Environment, and calculate a distribution for the various shell I think we'd find 97% written in Bourne shell, 2% written in ksh, and 1% written in Bash.  Please remember that I'm talking about Solaris here, and not Linux.  When in Rome, do as the Romans do.

Because I create a LOT of shell scripts, I'm also painfully aware of Bourne shell's limitations.  Not the least of which is its anemic data structure capability.  For that reason, I started to use the Korn shell, or ksh.  I have used ksh on Solaris for about 10 years both as my interactive shell and as my programming shell.  It's still painful for data structures when compared to Perl, but it's MUCH nicer to work with than the Bourne Subset.

In addition to my Solaris career, I have a life in a parallel universe as a Linux engineer, volunteering at a local not-for-profit organization.  There we use Linux for servers because Solaris was too expensive when we chose the initial environment.  In the Linux world, the Bourne Again Shell, or Bash, is the de-facto standard for good reason.  It's a very capable shell with great programming constructs, and a pleasant demeanor for interactive shell use.  I dig it.

In fact, I dug it so much that I recently changed my Solaris interactive shell to it.  Bash has been included in Solaris for quite a while now, and I consider it a standard-enough component that it's safe to learn.  I love the file completion and history browsing with tab and arrow keys.  It's so much more intuitive than the equivalent in ksh.  I've been happily bashing around for a few months on Solaris, but recently did an about face in my thinking while reading up on Role Based Access Control, or RBAC.

My current site has a sudo to RBAC conversion on the roadmap, which should be a great project.  In the RBAC world, you assume a role to complete an activity that is outside your default privileges.  Those roles will have one of three shells:  pfcsh, pfsh, or pfksh.  Did you notice that Bash is not amongst them?  Not for the foreseeable future.

So, in a Solaris-standard world that leverages the powerful RBAC facility, you will not have the option of working in a bash compatible shell if you want to perform an activity outside your normal rights.  That's enough reason for me to drop the bells and whistles of Bash and go back to ksh.  I'll probably miss it for a few weeks before I forget completely.

This experience proved to be yet another reminder for me that staying on the path of standards is full of temptations.  While Bash is stable and common, it is not a native Solaris standard.  That's fine for an end-user or software developer that operates outside the OS internals, but as a systems engineer my job is to live in the internals, and at this point in time Korn shell seems to give me the best functionality, programming, and standards compliance.


Revision control bites...  Unless you train it!
2006-09-21T08:11:00.000-05:00
I've recently begun the task of placing my world under revision control.  Keeping track of edits, versions, builds, etc. had become too much of a chore considering that Solaris has a facility to do it for me.  The Source Code Control System, or SCCS, is the Solaris standard for revision control in just about every release I'm familiar with.  There are many other revision control system out there, each with pros and cons.  Popular alternatives include RCS, CVS, and Subversion.  In my case, I want to choose a system that I know will be available in all reelases of Solaris on all sites, even in galaxies far, far away.  SCCS is the only one that meets my key criteria.

Having chosen my platform for version control, I begain moving projects into its protective custody.  All went well for the first few weeks, and I began to develop the the comfort level that usually preceeds a problem.  The most recent project I put under SCCS control is a package with a simple preinstall script.  The script is responsible for checking for the existance of a file prior to package installation, and making a backup before overwriting it.  The backup is named FILENAME.PACKAGENAME.DATESTAMP.  To implement this datestamp I set a variable in a backup subroutine as follows:

backup_file () {
   DATESTAMP=`date +%Y%m%d%H%M`
   test -f "${1}" && /usr/bin/cp ${1} ${1}.CGHfoopkg.${DATESTAMP}
   return ${?}
} #end backup_file


After testing successfully on my development machine, I checked the file back in (sccs delget) and rebuilt the package.  When I transferred the package to my staging server, the preinstall failed miserably with strange syntax originating at lines two and three of the above segment.

The first thing I questioned was whether there is some difference in versions of the time command between my development server and staging server, but it only took a second to prove that theory null and void.

After some mucking around, I discovered that the problem only exists when the source code was checked in to SCCS.  This quickly lead me to realize that SCCS was expanding keywords in my DATESTAMP variable.

SCCS keywords allow you to have SCCS dynamically insert a revision, filename, check-in time, and other metadata when a file is checked in.  By placing this into my checked out code:
#
# SCCS Revision Control:
#       %M%     %I%     %H% %T%
#


I end up with this when the code is checked-in:
#
# SCCS Revision Control:
#       preinstall     1.5     09/21/06 09:01:46
#


The problem is, SCCS by default does not stop at the header.  It goes all the way through the code.  In this case, it was hitting my DATESTAMP variable and changing its value, thus breaking my backup_file function.

It took some digging to find the solution, but I finally discovered the sccs admin command.  Using this command I was able to specify that only the first ten lines of the file should be considered for keyword expansion using the following command:
# sccs admin -fs10 preinstal


After making this change, and checking the file back in, my problems were gone.  The source stayed clean, and I was once again blissfully coding.  My newly trained SCCS repository has been behaving wonderfully ever since.


To install links, or not to install links...  That is the Question!
2006-09-18T08:32:00.000-05:00
I'm working on a package at the moment that integrates Oracle 9i / 10g startups with Solaris Resource Manager (SRM).  One of its functions is providing init scripts to start and stop the database and listeners during reboots.  Seems simple at first, but I ran into an interesting dilemma.

At my current site, Oracle is not an automated installation, and not part of our Jumpstart framework.  The challenge this presents is that we do not want to install active run control links for software that is not yet installed on the system, but we want to be able to install these packages during the jumpstart.

Ideally, I'd like to see the rc scripts registered via pkgadd so they can be easily identified, and cleanly removed if the package is de-installed.  In the end, I had to compromise, but I think it turned out pretty safe because I followed standards in use of those rc / init scripts and used a hard link.

Although it may sound odd at first glance, I chose not to include the rc links in my package prototype.  The links are installed in a disabled mode (pre-pended underscore character) by a postinstall script.  I chose this route because it creates a placeholder for the startup order I selected rather than hoping whoever does the installation gets it right.  This eliminates inconsistency and human error.

Next, I created a preremove script.  The preremove works by issuing a find command on the /etc/rc?.d scripts that looks for any files with an inode matching the init script's inode.  If proper convention for using hard-links was followed, this method will find the associated rc scripts even if their startup order or link name are changed over the life of the system.

LINKS=`find /etc/rc?.d -inum ${INITSCRIPTINODE} -print`

The final touch was to create a simple script that can activate all the sym-links at once.  It leverages the preremove's inode hunting strategy  and finds entries that have an initial underscore, and renames them using a simple Sed expression:

NEWLINKNAME=`echo $LINKNAME | sed -e 's/^_//'`

I'll be adding a reverse-function for the enabler script that can disable as well, and this will live in our NFS-accessible admin scripts repository for convenience.  No need to distribute software that isn't critical to operation when the network is down.

The only downfall to my solution is not being able to search for the rc links in the Solaris software registry (/var/sadm/install/contents).  This would have allowed someone unfamiliar with the solution to identify its components.  The problem with registering RC links is that they change a lot over time and it's very difficult to keep the registry current.  A link may change its name, start order, may be active, or inactive.

The effort required to manage that kind of a dynamic file would have been almost as large as the project I was packaging, so I decided to keep my scope tight and assume that as an acceptable risk.  When we move from Solaris 9 to Solaris 10 this problem will be eliminated by integration with SMF, so no point in getting wrapped around the axles.


Auditing challenged with SSH
2006-09-13T20:12:00.000-05:00
I recently had a real dilemma thrown at me by the security team at a site I work on.  The site's policy dictates that no authentication should take place without a password.  Any exceptions require both a business case justification on file, and an expiration.  This presents a real challenge with SSH.  SSH allows public key authentication as an alternative to passwords, and the private key can be created without a passphrase.  In addition, there is no way to enforce a key expiration at a server level (at least, not that I've been able to find).

SSH supports two primary means of authentication.  Password authentication is essentially the same process that occurs through Telnet, except we use a secure tunnel instead.  Public Key authentication bypasses the system's password channel completely.  At first glance, it's easy to say, "so disable public key authentication and be done with it."

There are two benefits to using public key authentication with SSH.  First, you have the ability to confiigure your private key (residing on the client) without a passphrase.  In doing so, you increase the risk of private key compromise, but enable passwordless authentication.  This makes batch jobs much easier because it elminiates an interactive session.  

The second benefit is that public key authentication is a two-factored authentication.  It combines something you have, a private key, and something you know: a passphrase.  As such, it's much more secure than traditional password based authentication which onyl uses single factor authentication.  Even if someone compromises your private key file, it can't be used unless they know the passphrase, which SHOULD be a complex phrase.

The problem is that there is no way to control who on a server is authorized to use public key encryption, no way to enforce passphrase complexity, and no way to expire a public key.  I could create a public / private key pair on a machine under my desk that isn't subject to production data center security scans and audits. Let's next say that I decided not to use a passphrase, or that the passphrase was weak and easily guessed.  Next, assume I'm the DBA for the company's Oracle Fincancials encironment.  It would require very little effort for someone to compromise my under-desk system and gain passwordless access to the company's critical systems.  This is a low-risk, high payoff scenario that people who know what they're doing would be likely to attempt.

Another issue with the public key encryption capabilities of SSH is that you can enter any passphrase (or no passphrase).  With my system password, I can install a crack module into the Pluggable Authentication Module (PAM) stack and enforce very complex passwords.  Unfortunately, because key pairs are generated on clients, there is no way to enforce sanity at the server level.

After some research I determined that the version of Sun SSH which ships with Solaris 9 is far less capable than the OpenSSH releases one could obtain and build.  Using OpenSSH it is possible to move the default $HOME/.ssh to another base directory (like /var).  In doing so, it is much easier to create a root controlled environment where someone can not use the authorized_keys file unless authroized by a superuser.  Unfortunately, the maintenance issues created by doing this are not justified, and our policy is to stick with vedor provided and supported software in all but the most extenuating circumstances.  Under Solaris 9, there's just no safe and auditable way to allow publick key authentication.  In an ideal world, I should be able to not just configure public key authentication at a server leve, but specify which keys were respected by the server's SSH daemon.

There is no solution I can see to the issue of enforcing passphrase complexity, or auditing use of non-interactive key pairs because that part of the process is handled entirely by the client.  It's very difficult to convnice a security officer that key generated on an uncontrolled device can be trusted for authentication against Sarbox servers.

Our decision, much to my dismay, was to disable public key authentication site-wide.  I feel like we're throwing the baby out with the bath water, but at the same time I understand the need to audit system access, and be able to enforce policy.  I'm anxious for our Solaris 9 fleet to turn over to Solaris 10 so we can begin using the more capable version of SSH it includes.


AppleCare not up to Sun Service
2006-09-08T20:01:00.000-05:00
One of the big reasons I was so excited to move my desktop environment to Mac OS X was its underlying UNIX operating environment.  Being a UNIX guy, I'm well aware of how well instrumented it is, and how surgically it can (often) be debugged.  This, of course is in contrast to the Windows world where it has become not only common, but almost accepted that troubleshooting step #1 is to reboot and sacrifice a chicken.

Over time my G5 was starting to crash with increasing frequency.  At first it was once in a rare while, although I was still surprised that it happened.  Recently it accelerated to the point where it crashed almost once per day.  Given how much I paid to have rock solid hardware, and AppleCare behind it, this was not acceptable to me.  So, I finally grew tired of my Mac's grey text-box of death and called my friendly AppleCare representative.

I began by telling my story, and adding a detail I felt was critical.  Each time the system crashes I generated a crash dump report, and the stack trace always pointed back to the USB driver.  I grabbed the text below from another site as an example, but it's very similar to what I was seeing.


1 Unresolved kernel trap(cpu 0): 0x300 - Data access DAR=0xdeadbeef PC=0x0e692550
2 Latest crash info for cpu 0:
3    Exception state (sv=0x0EB5DA00)
4       PC=0x0E692550; MSR=0x00009030; DAR=0xDEADBEEF; DSISR=0x42000000; LR=0x0E692530;
        R1=0x081DBC20; XCP=0x0000000C (0x300 - Data access)
5       Backtrace:
6          0x0E6924A8 0x00213A88 0x00213884 0x002141D4 0x00214830
           0x00204CB0 0x00204C74
7       Kernel loadable modules in backtrace (with dependencies):
8          com.apple.dts.driver.PanicDriver(1.0)@0xe691000
9             dependency: com.apple.iokit.IOUSBFamily(1.9.2)@0xed9c000
10 Proceeding back via exception chain:
11    Exception state (sv=0x0EB5DA00)
12       previously dumped as "Latest" state. skipping...
13    Exception state (sv=0x0EB64A00)
14       PC=0x00000000; MSR=0x0000D030; DAR=0x00000000;
                   DSISR=0x00000000; LR=0x00000000; R1=0x00000000; XCP=0x00000000 (Unknown)


Note that USB line?  I was seeing it in EVERY crash.  This tends to be something worth investigating.  In my case I have a USB card reader attached to the USB ports in the bottom of my CinemaDisplay, and a Palm Pilot USB cable plugged into the front of my case.  Another observation I made was that each time my system crashed it was essentially idle.  It usually happened at night, or while I was at work.  I would return to the sound of a jet engine coming from that silver box.

My first suggestion was that we look at the panic logs and try to identify the faulty components, but this didn't get too much traction.  AppleCare is set up so that if the basic rubber stamp checks (slightly better than a reboot, but not by much) fail, they redirect you to a local store.  In my case this wasn't appealing.  It's a 40 minute drive from here, and the issue is intermittant.  I could end up being without my Mac for more than a week if things went well.  

So, we went through and erased all the caches and preferences, then reset the NVRAM.  My system was brought back to factory specs, although I really did almost nothing abnormal to it.  I don't use funky extensions or other hacks; I use mainstream well supported stuff.

Much to my surprise, the system has been stable since the activities.  I'll be the first to eat my words, but I'm not used to voodoo troubleshooting.  This was like chemo-therapy where we just bombard the system in hopes of getting all the cancerous code.  I'm used to working in a surgical environment where we see CPU 0 corrupting data on an interval that indicates it needs replacing.

As much as I'm a hopeless fan of Solaris, I have to say that I don't think it's a huge quality difference between Sun and Apple that gives me ths uneasy feeling about this experience; I think it's the quality of Sun Service.  They are used to dealing with mission critical servers more than art-critical desktops.  No offense to the Mac world - I'm one of you...  But it's a very different world.


Initology 101: A lesson in proper use of Solaris run control scripts
2006-08-31T11:35:00.000-05:00
Starting and stopping applications through init scripts ought to be a simple thing that doesn't cause much debate, but in fact its just the opposite.  I routinely see servers with functional but non-standard artifacts nested in the rc directories.  I also hear many justifications for these configurations; some reasonable, others somewhat less so.  But in the end, I believe that a systems engineering approach to using init scripts will filter the options, and this article intends to do just that.

There are three specific conventions that I want to address:  

1. Which run levels should be used for starting and stopping typical applications.
2. Should a symbolic link (sym-link) or hard-link be used?
3. How should a link be disabled

Let's begin with identifying the correct run levels to start and stop a common application.  By common application I mean something that is not a core part of the operating system, but rather in the application layer that depends on the operating environment's core features.  Oracle and web servers are common examples of what I consider common applications.  Knowing that the Solaris Operating Environment has well defined run level states, the first step is to consult the docs.sun.com web site for your particular Solaris version and refer to those definitions.  Let's take the case of Solaris 9 (9/05) which is that last release in the Solaris 9 series.  I am not going to address Solaris 10 in this context because it uses the new Service Management Facility as part of the new Predictive Self Healing feature to replace init scripts.

According to the Solaris 9 (9/04) System Administration Guide: Basic Administration Section 8: Run Levels and Boot Files  We have the following run levels and explanations:


Run Level Description
0 Shut down all processes and power down to ok> prompt (sparc).
S Run as a single user with some file systems mounted and accessible.
1 Administrative state with access to all file systems, but no user logins permitted
2 Multi-user state.  For normal operations. Multiple users can access the system and all file system. All daemons are running except for the NFS server daemons.
3 Multi-user state: For normal operations with NFS resources shared. This is the default run level for the Solaris environment.
4 Alternative MU state.  This is not used by Solaris, but is available for site customization if needed.  I recommend NOT using it.
5 Power down after shutting down all processes.
6 Reboot the system.


In theory, we need to consider that a system may transition from any run level to any other run level.  This means that when the system enters run level S, if our application is running, we need to ensure it is stopped.  The same thing goes for 0, 1, and 2.  Run level three is the conventional system state associated with end user applications being loaded.  Putting this into practice, we will need to install the following links to fully integrate with Solaris' run levels:

/etc/rc0.d/K##fooapp
/etc/rc1.d/K##fooapp
/etc/rc2.d/K##fooapp
/etc/rc3.d/S##fooapp
/etc/rcS.d/K##fooapp


These will ensure that our application is started in run level 3, and stopped in any other run level.  This contrasts with what I see in most data centers where rc scripts are installed to run level 2 or 3 for start up, and 0 for shut down.  While this approach can work for reboots it has a down fall.  How many times have you been told that before patching you need to reboot a server into single user mode?  This is because kill scripts are not installed for all applications for all run level transitions.  I still advocate rebooting into single user mode to be safe, but in a perfect world this would not be necessary.

Having selected the run control directories, you are now ready to put the links in place.  But wait!  You have another decision to make.  Should you use a symbolic link or a hard link?  There are all kinds of reasons for and against either method if you approach the question from an emotional standpoint.  However, as a Solaris Jedi, you do not allow your emotions to control you.  You look for standards.

Referring again to the docs.sun.com web site, we return to the Solaris 9 (9/04) System Administration Guide: Basic Administration.  This time, to Section 8, How to Add a Run Control Script.  The examples on the page clearly show how to use the ln command to create a hard link.  This is where the discussion should end.  You didn't write Solaris, and you didn't do the integration testing.  You are disciplined, and you follow standards; This is the way of the Jedi.

I have heard numerous arguments for using sym-links in place of hard links, and I believe each of them stems from not fully understanding how UNIX file system inodes work, and how Solaris commands can be used to understand them.  Using the "ls -i" command you can prove that the files reference the same inode, and are thus the same.

cgh@soleil{/etc/rc0.d}# ls -li /etc/rc3.d/S90samba 
      9731 -rwxr--r--   6 root     sys          324 Jan 14  2006 /etc/rc3.d/S90samba*
cgh@soleil{/etc/rc0.d}# ls -li /etc/init.d/samba 
      9731 -rwxr--r--   6 root     sys          324 Jan 14  2006 /etc/init.d/samba*
cgh@soleil{/etc/rc0.d}# 


Notice the first field in each record shows the integer, 9731?  That is the inode number.  The next field to attend to is the third.  In this case, a "6" for each record.  This refers to the link count, or number of links that point to the same piece of data.

Another approach to observing all rc links associated with an init script is to use the find command to search a branch of the file system for the inode number matching the init script.  Let's look at the standard Samba service included with Solaris 10.  We know from the prior example that inode #9731 references the samba script.  The following command will seek out all of the hard links:

cgh@soleil{/etc/rc0.d}# find /etc/rc?.d -inum 9731
/etc/init.d/samba
/etc/rc0.d/K03samba
/etc/rc1.d/K03samba
/etc/rc2.d/K03samba
/etc/rc3.d/S90samba
/etc/rcS.d/K03samba
cgh@soleil{/etc/rc0.d}#


If these link were symbolic the task would not be as simple, and we would not have the benefit of a link counter to ensure the integrity of our boots.

The last facet of initology I want to discuss is proper convention for disabling an init script on a Solaris server.  As with the above examples, the correct process comes right out of the Basic Administration Guide, Section 8.  The init scripts only process files that begin with an "S" or a "K".  I most often see the upper-case letter replaced with lower case.  The number two method I've observed is to remove the links altogether, leaving (hopefully) the init script in place.

The correct process for disabling an init script is almost always to prepend an underscore.  The underscore stands out clearly in the list while lower cases characters tend to have less contrast next to the upper case entries.   It sounds trivial, but how goood is your eye sight at 3am after your pager goes off?  Another benefit is the grouping of all disabled scripts in the directory listing so you can tell at a glance what is turned off.  Finally, by not removing it altogether we can preserve the ordering of the scripts, which is some cases is critical.  Take a look at the example below, and hopefully my suggestions will be apparent:

cgh@soleil{/etc/rc3.d}# ls -l
total 44
-rw-r--r--   1 root     sys         1285 Jan 21  2005 README
-rwxr--r--   6 root     sys          474 Jan 21  2005 S16boot.server*
-rwxr--r--   6 root     sys         1649 Jan  8  2005 S50apache*
-rwxr--r--   6 root     sys         5840 Jan 29  2004 S52imq*
-rwxr-xr-x   1 root     sys          491 Apr 10 12:49 S75seaport*
-rwxr--r--   6 root     sys          685 Jan 21  2005 S76snmpdx*
-rwxr--r--   6 root     sys         1125 Jan 21  2005 S77dmi*
-rwxr--r--   6 root     sys          344 Jan 21  2005 S80mipagent*
-rwxr--r--   6 root     sys          513 May 15 19:21 S81volmgt*
-rwxr-xr-x   5 root     sys         2225 Apr 10 12:49 S82initsma*
-rwxr--r--   5 root     sys          824 May 26  2004 S84appserv*
-rwxr--r--   6 root     sys          324 Jan 14  2006 S90samba*
-rw-r--r--   1 root     root           0 Aug 31 21:31 _S92foodb
-rw-r--r--   1 root     root           0 Aug 31 21:31 _S95fooapp


 Henceforth, you will properly integrate your scripts with the entire run level facility using hard links.  When those magical links need to be disabled you will prepend underscores to them.  You are now a master of the Solaris init scripts, and ready to carry this knowledge to others.  You are also ready to explore the Solaris 10 SMF and enjoy all that it has to offer.


Spotlight on Richard McDougall
2006-08-23T20:06:00.000-05:00
If you haven't yet visited Richard McDougall's Blog you should fire up your browser and head over.  I had the pleasure of meeting Richard at a SunUP Network Conference in Singapore where we were both giving presentations.  We met up again at later conferences in Sydney Australia and Boston in the same scenario, and he hit it out of the park every time he got in front of customers.  It's very rare to meet someone as brilliant as Richard who is also so down to Earth and generous with his knowledge; he is a true Jedi Master in the land of Solaris. 

One characteristic you observe early in Richard's presentations is his enthusiasm for Solaris and its potential.  This article on Chip Multi-Threading is a classic example of his style, and was what inspired me to write this entry.  I remember him speaking about some of Sun Volume Manager's (SVM) new (at the time) features which were specifically designed to address the reasons customers had chosen Veritas Volume Manager.  Rather than attacking the message with the technical nuts and bolts, he hit on a few topics and delivered the message that Sun was listening.  I am certain that more people re-examined SVM after his delivery than any speeds and feeds preso would have motivated.  A true Jedi master delivers important messages without patronizing through understanding the intended recipients.

The other item I want to draw your attention to is his new set of books: Solaris Internals and Solaris(TM) Performance and Tools: DTrace and MDB Techniques for Solaris 10 and OpenSolaris (Hardcover) which were just delivered to me from Amazon.  First of all, I hate poorly bound books.  I buy books to use as reference manuals - tools of my trade.  These books feel like professional tools that you will appreciate returning to.  Remember buying that Calculus book in college?  The one that weighs 25 lbs?  This is that book.  I love it!  They are expensive, but good books aren't cheap, and the investment the author made in sharing his skills isn't cheap either.  The first edition of Solaris Internals is well known as the authoritative reference on Solaris plumbing, and with all of the exciting changes Solaris 10 brings, this book is timely.  I'm anxious to dig into it and post a review, but in the mean time please check the books out.  This page has more information, and sample content.

I'd like to start paying tribute to some of the Jedi Masters I've benefited from, and this post serves as the first.  Please take a moment to read Richard's Blog.  Check back frequently - if he posts it, you should know about it.  And if you need a diversion from Solaris, he's also a great photographer.


The verdict: Ubuntu Linux is a keeper
2006-08-20T13:55:00.000-05:00
I'm writing now from the keyboard of my reborn laptop.  Having just completed the installation and configuration of Ubuntu Linux on it, and happily retired its installation of Windows XP.  Since this blog is really about Sun Solaris and Systems Engineering I don't want to spend too much time talking about this from a technical standpoint.  It does have relevance to the theme as we all need a portable means of working on Solaris systems.  If you use UNIX as your primary operating environment, you know how awkward it is to depend on Windows as your interface to the systems you support.

So far, Unbuntu "just works" with no headaches at all;  It auto-detects and configures my Netgear WG511 "G" Network Card, and can successfully enter and exit both hibernate and suspend modes.  These were the two big headaches for me under Fedora Core.  I am really impressed that the special volume and mute keys worked as well.  Those used to require installing a separate Thinkpad buttons package called tpb.  The boot screens look slick, the theme is very clean and coherant, and the desktop is clean and EMPTY.  I love that!  I give it two thumbs up.  It looks like I'm finally going to learn the Debian flavor Linux after years of being a die-hard Red-Hat camper.

Now let me clarify this position; I'm not changing my opinion about Mac OS being the ultimate desktop.  But, I can obtain an old IBM Thinkpad T20 for a fraction of the cost of a PowerBook or MacBook.  I wouldn't want to process my photographs on it, but for a tool that lets me perform systems work, and use typical Office Software, I'm very happy.


Microsoft's Genuine Advantage
2006-08-19T21:00:00.000-05:00
I have an older IBM Thinkpad T23 laptop which I purchased after sending most of my scrapyard through eBay.  I bought it with the intention of running 90% Linux, and occasionally using Windows when I need some odd utility, or have to connect to something that only speaks Windows.  The T23 is a rock solid machine, and being far from the bleeding edge, also has pretty decent hardware compatibility.  With 1GB of RAM and a 1GHz CPU, the machine is plenty fast for it's intended role as a terminal web browser, email client, and occasional OpenOffice platform.  I bet the most used application on it was gnome-terminal if I really analyzed the accounting records; Nothing stressful.

When I first loaded Fedora Linux it was a simple process to get the machine useable.  Useable and optimial turned out to be divided by a full-strength, adult size, bang-a-roo of a headache.  The little things like getting it to play MP3s didn't phase me too much.  In fact, taken one by one the entire list isn't anything that can't be handled.  The problem is that I'm tired of having to handle things.  I just want my computers to let me do what I want without HAVING to hack.  I'd rather hack by choice than for base survival.

I was able to get Wireless ethernet working after some digging, but what really sent me over the edge was ACPI.  My power consumption was awful, and getting it to be even close to Windows proved as complex as tuning 100 Oracle instances fighting for the resources of a SparcStation 5.  Not fun at all.  Eventually, I decided that despite my inability to mentally mesh with Window's gears I would dump Linux and stick to the main stream.

I bought a copy of Windows XP Pro off eBay, complete with hologram media, funky sticker, and all of those gimmicky little things they do to make ou think you're getting something official and important.  I downloaded all the updates, I filled out the registration, I did all the things someone would do when they are an IT professional who wants to be legitimate.  After using it for about a year with no issues, including the "Windows Genuine Advantage" thingy which used to think I has a legitimate copy.

Today, after a long hiatus, my laptop was booted and it informed me that Windows Genuine Advantage had changed its mind.  Warning boxes were popping up left and right, and graciously giving me the opportunity to "purchase genuine Windows".  You know what?  I already did.  It was shrink wrapped, and had so many gimmicky little security things that it was gaudy.  And now you want to give me an opportunity to do it again?  No thanks.  From a quick Google search it looks like I'm not the first person to be annoyed.

I did notice that my system clock was really goofed up, and I've heard that the validation process involves hardware checks, so maybe something in my configuration triggered it.  I don't know, but frankly I don't care.  I don't want to know why it happened.  I'm going back to a world that doesn't include helpful paper clips and other rediculous instantiations of a help system.

Since my battery is nearly dead, I've decided not to worry about ACPI.  Windows is being scrapped tonight and I'm going to either run Ubuntu or Fedora linux.  I'm not crazy about Solaris on the desktop because it has less standard productivity software, and the updates for non-Solaris software are not convenient.  I'm the #1 fan for servers, but on the desktop I'm a Linux guy until I can afford a PowerBook or MacBook.

I'll post more about the final choice I make, but I felt it necessary to document this eve of liberation for all to see.  And now I must end this post as I have a date with fdisk to catch...


IM Ruining Grammar?
2006-08-03T21:26:00.000-05:00
This one is a bit off theme, but I couldn't resist.  Apparently it has been found that the prolonged use of IM does not truly impair one's grammatical ability.  Thank the University of Toronto for this piece of knowledge...

Are you serious?  It's not grammar that gets hurt when IM is abused.  It's one's social skills.  The article I mention above is really talking about kids who get carried away, but living in a cube farm, I've seen the adult version as well.  At some point, we've all been guilty of IM'ing someone close enough to hit with a paper airplane.

Between email, voicemail, wikis, and IM, not to mention remote work, just about everything is driving a wedge between developing the personal relationships that foster good working environments.  When I've met someone in person I immediately feel more at ease trusting them for the role they will play in a project.

Body language plays a HUGE role in our ability to communicate effectively, and to dismiss it for the "efficiency" of electronic communication is naive at best.  The next time you need to talk to someone, walk to the other side of the building, or schedule the time to drive to their site.  You'll be glad you did, and they probably will too.


The demise of Linux
2006-08-03T20:37:00.000-05:00
I read a quick editorial which suggests that Ubuntu Linux is going to be the downfall of Red Hat.  The premise being that as Red Hat grew more commercial, and abandoned its community in favor of its stockholders, the sys-admins who used Red Hat as their desktop OS drifted away from the Red Hat camp.  As Ubuntu came into being, and did so in a very strong way, those SA-types who were driven away from Red Hat will now want to put what they are more familiar with (Ubuntu) on their servers when they have the choice.

This whole discussion brought me right back down memory lane.  I started using Linux before Red Hat existed with a few early Slackware distributions.  I remember writing all those 3" diskettes - somewhere around 80 of them by the time I burned the X-windows distribution as well.  Shortly after came Red Hat, and at that time I was helping to set up the first Linux environment at SUNY Plattsburgh.  We switched over from Slackware to Red Hat, and loved it.

I ended up sticking with Red Hat for the next 10 years.  It was the OS of choice when I lead a project to build servers for our local Boy Scouts of America council, and remained there until Red Hat went totally commercial, burning the bridges out from under us.  Make no mistake, I was extremely disappointed with their decision.  We switched over to Fedora Core, and for the most part it has been a smooth transition.  Despite its big red "DEVELOPMENT" stamp, Fedora has been very good to our availability.  In fact, at the moment we've got an impressive uptime on an FC2 system:

# uptime
 21:41:15 up 412 days,  2:37,  2 users,  load average: 0.00, 0.00, 0.00


But thinking back on the experience I have an entirely different memory of what direction I was forced in, and where I wanted to be.  The places I've worked would not consider using Linux for their production environments.  Linux has made some great in roads t the corporate world, but there aren't a lot of Fortune 500 companies running their SAP central instance on Linux.  I'm sorry, it's just not happening.  HPUX, AIX, and Solaris are king in the land of mission-critical highly scalable UNIX servers.

Although Solaris never made a particularly compelling desktop, it's what I've wanted to use on every server I've ever built.  It's rock-solid, well documented, and very cohesive.  When you use Solaris and Sun products, you rarely get the impression that 10,000 individual developers all tried to do it "their way" when the final build was cast in stone.  What always stopped me was cost.  Solaris x86 had pathetic support and commitment in the past.  It's so incredibly painful to migrate between operating environments that I never wanted to risk Solaris x86 being yanked - which it was.

The second big barrier was cost.  If you went with Sparc, you had to have money.  Lots of money.  Oodles of money!  What I had was a basement full of x86 architecture hardware, and the not-for-profits I volunteer at had the same.  There was simply no funding for shiny Sun hardware no matter how badly we wanted it.

And then the sleeping giant awoke.  After being pummelled by the dot-com crash, Sun figured out what went wrong, and executed one of the m most amazing feats of corporate intertia changes I've ever seen.  In a very short time frame, support for Solaris x86 was restored at a full commitment level.  And it was made free.  Then they continued to make their Java Enterprise System free to download and use as well.

So, the making of a fantastic Linux in Ubuntu may hurt Red Hat, but it's not what will deliver the killing blow.  Red Hat has an opportunity right now to try to pull off a corporate inertia swing of Sun's magnitude.  They need to restore faith in the community restore the religion they destroyed and find some kind of innovation to draw people back in.  Solaris has done all of this and created an affordable support model that doesn't intimidate the small businesses who were once driven to Linux.

The first blood has been drawn by Solaris, but the second wound is far deeper.  This second wound is bleeding internally and missing a lot of coverage.  Mac OS-X is the killer desktop.  If you have a reason to be using UNIX on a desktop, then using anything other than Mac OS-X is a tough sell in my book.  Hardware is a bit more expensive, sure.  But it's the best of every world, and solid as a rock.  It doesn't hurt that it looks great either.

A recent seminar I attended talked about business models and knowing when to have the guts to drop a design.  The idea was that you need to look at things you're developing and ask whether or not they give you a long-term sustainable advantage.  I have to use that same litmus test to examine Linux.  In the server world I can get free and open Solaris which is out-innovating Linux in my observation.  And on the desktop, while Linux continues to improve, it's not even close to Mac OS-X.

In the end, these observations mean little to the tech-hobbyist who loves Linux for its religion.  But in the business world, religion doesn't make IT choices.  Competitive advantage does.


Solaris - the Wine
2006-07-23T14:19:00.000-05:00
  
   soalriswine.jpg  
  Originally uploaded by cghubbell. 
I believe this may be one of the best system tools for evenings when a long day of UNIX has left the Force unbalanced in your mind.  I stumbled on this wine while perusing a liquor store in Horseheads, NY.  I hardly drink at all these days, but definitely enjoy a nice glass of wine when I unwind after a long day.  If you're into this kind of thing and enjoy seeing Solaris outside the data center, check out the Solaris Winery!



Did you hear what I sed?
2006-07-21T13:43:00.000-05:00
When battling the dark side of UNIX, it is critical that you not let your eyes betray your instincts.  Windows teaches you to trust what you see, which is in itself a good reason to be wary.  Today's lesson will involve our old friend awk, and a not so well known friend, od (octal dump).

I was working on a section of code, which decided whether or not arguments were passed by checking in a case statement for an empty string, or anything other than an empty string.  It looks like this:

case "$SID_LIST" in
   "" ) # No arguments passed, go with default.
        echo "Stopping all configured Oracle databases."
        su oracle -c "$ORACLE_HOME/bin/dbshut"
        ;;
   *  ) # SID list paased - pass it on to dbshut
        echo "Stopping specified Oracle database(s)."
        su oracle -c "$ORACLE_HOME/bin/dbshut $SID_LIST"
        ;;
esac


This structure comes from the Oracle 10g dbshut script which I'm applying some customizations to.  As a result, I'm trying not to completely restructure the script.  If I were to write it myself, I'd be more tempted to put this in an if statement, and test for a null string (test -z).  But, since I'm working with someone else's code, I'm trying to stick to minimizing my impact.

If you call this particular script with arguments (an argument is a token that follows the command, like do_something RED BLUE) I detect the extra arguments from the command line, and put them into a variable called SID_LIST as follows:

ACTION=$1       # Assign first argument to action
shift;          # shift arg pointer past $1 (action)
SID_LIST="$*"   # Assign any remaining args to the argument list


So, when I call the script with a command like, "dbshut TESTDBA TESTDBB" I expect to see SID_LIST end up with the values "TESTDBA TESTDBB".  Good enough!  But what if someone repeats an argument?  We don't want to iterate through arguments we have already processed, so I decided to add my own personal garnish of ensuring the list is unique.  And this little detour is where the fun began...

The modification I made looked like this:

SID_LIST=`echo $SID_LIST | tr " " "\n" | sort | uniq | tr "\n" " "`


Let's break this down into logical steps:
First, translate any spaces into newlines because the next commands in the pipeline will expect to see things in multi-line form.  This turns "one two one three" into:

one
two
one
three

Next, sort the output alphabetically to ensure similar items are immediately next to each other, which is necessary for the following piece of the command.  Now we send the sorted list to a program called uniq which removes duplicates.  The output now looks something like this (remember, its alphabetical):

one
three
two

Finally, we need to get it back into a single-line format, so we send the output into the reverse of the first tr command which replaces any newlines with spaces.  Our final output looks like this:

"one three two"

Having conquered that challenge, I integrated the code fragment and observed its behavior.  Oddly, I discovered that whether or not I supplied arguments, the case statement always resolved my input to be in the "*" branch rather than the "" branch.  After taking a closer look, I discovered that my output was not what it appeared...  In fact, the final newline had been replaced with a space by the last tr command, and my string looked like this:

"one[space]three[space]two[space]"

Because SID_LIST did not match "", the case statement selected the "*" branch instead.  Feeling quite impressed with my mastery of the debugging arts, I surmised that a simple sed statement could whack my terminating space, and leave me with the desired empty string that would set my logic free.  But alas, it was not to be...

I left me editor, and started playing on the command line.  First, I created a simulation by setting a variable to contain a series of pretend arguments:

testbox{cgh}$ A="one two two three three four"


Next, I simulated my script's pipeline so make sure I could duplicate the problem.  I surrounded the output with brackets to make the trailing space more obvious...

testbox{cgh}$ B=`echo $A | tr " " "\n" | sort | uniq | tr "\n" " "`
testbox{cgh}$ echo "[$B]"
[four one three two ]


Excellent, now we can test a fix...  I put a sample string with a trailing space into a variable, and sent it into a sed command.  The sed script is pretty straight-forward; search for a space character immediately before the end of the line, and replace it with nothing.  This breaks down to the three divisions between slashes: [s]earch/[space]$(end of line)/replace_with_nothing/.

testbox{cgh}$ X="four one three two "
testbox{cgh}$ echo "[`echo $X | sed -e 's/ $//'`]"
[four one three two]


And behold, it worked!  I now take the tested sed script, and attach it to the end of the pipeline...

testbox{cgh}$ A="one two two three three four"
testbox{cgh}$ B=`echo $A | tr " " "\n" | sort | uniq | tr "\n" " " | sed -e 's/ $//'`
testbox{cgh}$ echo "[$B]"
[]


What happened to my string?  I copied and pasted the code, and it should have worked!  Here is the part where we learn to trust our instincts, and not what we see.  Let's revisit our input variables using The Force...

Earlier, we set $X to contain a sample set of arguments with a trailing space, and that input string worked nicely.  Maybe the input changed somewhere in the pipeline to not exactly reflect the test conditions in our experiment...  Here's how we can compare them:

testbox{cgh}$ echo $X | od -c
0000000   f   o   u   r       o   n   e       t   h   r   e   e       t
0000020   w   o  \n
0000023
testbox{cgh}$ echo $A | tr " " "\n" | sort | uniq | tr "\n" " " | od -c
0000000   f   o   u   r       o   n   e       t   h   r   e   e       t
0000020   w   o    
0000023


Do you see it?  The difference is that our experiment's $X string is terminated by a newline character, while our pure pipeline string has lost its newline.  This becomes a problem for the sed command which removes our trailing space.  Sed acts when it sees an input terminator like a newline or ctrl-D character.  In this pipeline, sed is never getting what it needs.

The solution is fairly simple, although not pretty.  I broke this pipeline into two statements, and sent my sed script its input from an echo command rather than directly through the pipeline.  This allows echo to put a newline onto the string and make sed happy.  Here's what it looks like:

SID_LIST=`echo $SID_LIST | tr " " "\n" | sort | uniq | tr "\n" " "`
SID_LIST=`echo $SID_LIST | sed -e 's/ $//'`


This could be performed in other ways, my personal favorite being to reincarnate this script in Perl and eliminate all these pipelines and separate commands.  But, by leaving it as-is I can keep the user base more comfortable with the language.  It also serves as a great lesson for Jedi training, and so shall it remain.


Poor grammar isn't always a bad thing
2006-07-19T17:22:00.000-05:00
If you write enough shell scripts you will eventually fall prey to your own comments.  Unless you read my blog of course, in which case you will have saved hours of frustration!

Let's take a fictitious problem...  You need to print the first and third columns of the /etc/passwd file so that a report can be generated correlating user IDs to user names.  Being the UNIX monk that you are, you assure your management that a shell script can meet their every need, and there is really no reason to have an ODBC link from Microsoft Access to the passwd file.

You throw together some code, and it looks like this:

#!/usr/bin/ksh
nawk 'BEGIN { FS=":" }
   # We don't want to print anything but 
   # the first and third column
   {print $1,$4}' /etc/passwd
exit 0


Looks like a nice tight algorithm, well commented, and generally a job well done.  You pat yourself on the back and refill your coffee, ready for the next challenge.  Not so fast...  First you decide to test that script, and you see the following:

testbox{cgh}$ ./comtst.ksh
./comtst.ksh[6]: syntax error at line 6 : `'' unmatched
testbox{cgh}$ 


But how can this be?  It's a simple script, and the logic is flawless!  Let's test it to be sure...

testbox{cgh}$ nawk 'BEGIN { FS=":" } {print $1,$4}' /etc/passwd
root 1
daemon 1
bin 2
sys 3
adm 4
lp 8
uucp 5
nuucp 9
ftp 60001
smmsp 25
listen 4
nobody 60001
noaccess 60002
nobody4 65534
cgh 1000


It works...  What is the problem here?

It turns out that the comments in the embedded nawk code are the problem.  In this case, the apostrophe in "don't" closes the opening apostrophe at the beginning of the nawk statement, and the shell interprets the code like this:

#!/usr/bin/ksh nawk 'BEGIN { FS=":" }# We don'


So what we really do it pass nawk a syntactically incorrect program.  Having figured it out, we re-write the code as follows:

#!/usr/bin/ksh
nawk 'BEGIN { FS=":" }
   # We do not want to print anything but 
   # the first and third column
   {print $1,$4}' /etc/passwd
exit 0


There are two morals to this story:  First, at the risk of repeating myself like a broken record, don't use multiple shells unless it's absolutely necessary because you run the risk of obscure interpretation problems.  In this case, we could solve the problem by writing in Perl where there's no need to embed a second language.

The second moral is to always avoid using contractions and meta-characters in your comments.  It makes for slightly longer comments, but if you scrictly avoid the temptation, it is one less thing to worry about.  This example was so simple that it's not hard to locate, but if you had a complex nawk script with its own subroutines buried in a complex shell script, it can be very frustrating trying to locate the bug.

The dark side will tempt you with contractions, but now your Jedi training has equipped you to calm your mind and type out those extra few characters.  Until next time, may the code be with you.


Don't Shed Your Shell
2006-07-11T12:09:00.000-05:00
I've said it before, and will say it again;  Switching interpreters in mid-code is a practice to avoid whenever possible.  There are times that it can be avoided, but there's a lot of times when you can sacrifice a bit of elegance for simpler maintenance.

As with most bugs, I was recently bit by a dumb mistake.  I needed the ability to lookup Solaris Resource Manager Project information using tags embedded in the description field.  For example, SID=TESTDB is how I would specify an Oracle database SID.  I wrote a Korn shell function called getprojbyattrib() which accomplished this very thing.  Tested on its own, it worked wonderfully.  When I went to integrate it with the existing Oracle start-up scripts I ran into some problems.  Turned out they were easy to debug, but the root cause was my old enemy of incompatible interpreters.

This new shell library function is used to figure whether or not an SRM project is configured for a given Oracle database.  If one and only one match is returned, then the database is started in a project container.  Any other condition means that the database is started without SRM.  To help in this cause, I embedded a counter in the function to return how many matches were found.  The code in question was simple:

   # Keep track of the number of projects we find while outputting
   # them so the final tally can be used as a success indicator.
   PRJCOUNT=0
   for PRJ in $PRJLIST
   do
      echo "$PRJ"
      PRJCOUNT=$(($PRJCOUNT+1))
   done


Make note of the seventh line of code which does the incrementing.  This is a Korn shell specific operation.  When the calling code from the oracle startup script referenced this, it gave an error which told me that it had interpreted line #7 at "PRJCOUNT=$".  This is because the Bourne shell doesn't understand the  operation.

The fix is simple.  Either switch the calling script to use the Korn shell interpreter because Korn is a superset of Bourne, or change the increment code to be Bourne-friendly by using either bc or expr.  
PRJCOUNT=`/usr/bin/expr $PRJCOUNT + 1`

Interestingly, the library function was written with a header that specified Korn shell as its interpreter:

#!/bin/sh


This becomes irrelevant when you are sourcing functions or variables as the whole point is to have your calling shell get access to these objects.

Sp what did I do?  At first I switched the calling code, but some afterthought lead me to work with the underlying Bourne shell subset so the library would be more portable.  I don't really like Bourne shell as Korn is much more capable, but in this case portability is weighted more heavily than elegance.

Repeat after me:  Switching interpreters in mid-code is something to be avoided whenever possible.


Solaris Date command and epoch time
2006-06-02T16:11:00.000-05:00
I can't count over the years how often I've wanted to output a date stamp in seconds since the epoch to make duration calculations simple. 

Just to prove that I'm not 100% biased towards Solaris let me point out that my Linux scripts all enjoy the ability to call /bin/date with a simple switch that outputs time in my desired format: date +%s.

In Solaris, neither /usr/bin/date nor /usr/xpg4/bin/date support output in the "seconds since epoch" format.  This is what we call low hanging fruit as enhancements go.  Unfortunately, the fruit still hangs.

Perl does a very nice job of handling date math and epoch conversion, but that requires a separate interpreter, and when I'm in shell I don't like to jump in an out of other interpreters.  I found a pretty cool hack that seems to avoid an external interpreter, and gets me what I want...

Since we know that the system keeps track in the format we want, we need to find a utility that uses a system call...  In this case I made a crazy guess and found the time() call.  Here's what it looks like:


testbox# man -s 2 time
System Calls                                              time(2)

NAME
     time - get time

SYNOPSIS
     time_t time(time_t *tloc);

DESCRIPTION
     The time() function returns the value  of  time  in  seconds
     since 00:00:00 UTC, January 1, 1970.

So, making a second wild guess I assumed that our beloved /usr/bin/date command uses the time() system call.  Let's take a look...  If we use the truss command to check out system calls and returns we should find what we're looking for.  We'll use the grep command to look for the time() call.

There's a catch though... Truss is going to dump output to stderr, and grep looks for input on stdin.  Those paths won't cross.  So, we need to redirect stderr into the stdout stream before piping it all over to grep.  


testbox# truss /usr/bin/date 2>&1 | grep ^time
time()                                          = 1149275766


Cool!  You can see the 2>&1 take stderr (2) and redirects (>) to catenate (&) with stdout (1).  This cuts the 40+ lines of system calls down to the one we care about.

It's not a standard interface, so any time we use it we run the risk of any OS patch breaking our algorithm.  Perl would be a safer way to go, but it does require more overhead in terms of firing up the interpreter for such a simple thing.  You'll have to decide for yourself whether or not this hack is useful to you, but I think it's a good one.

To clean it up and make a bit better behaved we'll need to get rid of leading and trailing spaces, and output just what we need.  Here's a quick script you can call or source...


#!/bin/sh
/usr/bin/truss /usr/bin/date 2>&1 |  nawk -F= '/^time\(\)/ {gsub(/ /,"",$2);print $2}'
exit $?


And finally, let's see it in action:

testbox# ./edate
1149276150
testbox#

So there you have it.  A way to get epoch time without writing a single line of C.


Veritas Foundation Suite goes free
2006-06-02T10:43:00.000-05:00
Veritas Volume Manager is free.  I'm not joking! Check it out for yourself!.  Of course, there are some restrictions involved.

Only for Linux and Solaris x64

<= 2 CPU cores

Max of 4 user volumes


I'm not sure I "get" this move.  Historically, there has been a philosophical battle going on in the enterprise engineering space over whether 'tis best to manage your root disks with Sun's Solaris Volume Manager (SVM) or Veritas' Volume Manager (VxVM).

On one side, VxVM had the strength of a volume management solution that scaled to infinity and beyond.  It's not much more difficult to manage 1000 disks on VxVM than it is to manage 10.  The corollary to this point is that if you require VxVM for your data farm, then you can reduce system architecture complexity by standardizing on use of VxVM for management of root disks as well.  Simple is good, right?

On the other side, SVM is extremely easy to use, and extremely easy to recover from problems with.  Given it's simple layering approach, it's also much more difficult to get into trouble with.  SVM has long been a favorite with Sys Admins because it means they never have to worry about the dreaded unencapsulation dance.  Again, simple is good, right?

From a purely academic standpoint, the complexity of using multiple Volume Managers seems unnecessary.  In practice, I believe the opposite is true.  Root disks are a very different beast than data disks.  For one thing, they tend to be configured and then forgotten (barring a disk failure).  In contrast, the data farm is always churning with reconfiguration or expansion.  Given the opposing dynamics of these types of volumes, it makes sense to use tools with different strengths.

When a root mirror pair needs work, you want a simple and reliable way to solve the problem and move on with very little chance of making an unrecoverable mistake.  This is SVM in a nutshell.  Its tight OS integration, and shallow learning curve mean that even a junior SA has a great probability of pulling off that disk replacement.

Unfortunately, if you have terrabytes of disk storage hooked to your Enterprise 20k you want a volume manager that is very flexible and agile.  Historically, SVM has not been that tool.  Its GUI does not display huge data farms efficiently, and its volume naming conventions get unruly when the disk count grows.  This is where VxVM comes in.

VxVM use abstraction and well designed (although complex) interfaces that allow mountains of data to be displayed quickly through either command line interface or GUI.  Historically, it also had much more power and flexibility than Sun's offerings.  This meant you needed VxVM for data farms.  

The best of both worlds, in my opinion is using SVM for Operating System volumes and internal disks, and VxVM for all external storage whether SAN or DAS.  If you only have limited external storage then just use SVM.  Really, simple is good!

Returning to Veritas' recent change in marketing strategy, it seems that they may be trying to counter some of the arguments for using Sun's integrated solution.  It's clear that ZFS is going to provide a very powerful facility for storage provisioning within Solaris 10, and SVM already supports many of the core features which used to be Veritas' key selling points.  Veritas has one key advantage in that they have a time tested solution available NOW.  ZFS isn't mature enough yet for the mission critical enterprise, but that's a very short term disadvantage.  The OpenSolaris model means that when ZFS makes it into a hardware update of Solaris 10, it's going to be 95% there.  I'll give it six months before they master that remaining 5% which wider distribution will open up.

Is the right strategy to pick to seldom-used platforms to make VxVM free on, and then limit use to four volumes on a product which is only advantageous when volumes are plentiful?  I don't think so.  If you have a simple system you're not going to WANT to use VxVM because SVM is so much simpler.

To me it looks like Veritas is trying to use a seeding strategy for a market that it has no chance of enticing.  While Veritas has a great product, it seems that they don't fully understand their niche.  I'm putting my money on SVM and ZFS.


Testing for correct usage in shell functions
2006-05-30T12:07:00.000-05:00
Here's a simple touch you can apply to your shell scripts to aid in debugging when they grow to become monstrous and you can't remember the syntax of all your subroutines any better than you can remember the 10th digit in Pi, which happens to be 3 for those who care about such things.

Although not strictly required to take advantage of this tweak, I recommend you begin by using good headers for each subroutine.  I won't go into each one, but a specific entry I always make is usage.  For example, if a subroutine do_foo takes arguments arg_one and arg_two, the header would look like this:

# ------
# do_foo
# ------
# USE:  doo_foo ARG_ONE ARG_TWO
# DESC: Execute foo functionality
# PRE:  na
# POST: na
# ERR:  na
foo () {
    ...
    ...
} #end do_foo


The line I want you to pay attention to in the above code begins with "USE:" (4th line).  This line specifies the interface which a user of your code should be aware of.  You are telling them that this code expects TWO arguments.  Now, you can get fancy and use EBNF like syntax to identify optional arguments, but let's keep it simple for this example and just recognize that we have established an interface.  

What can we do as a developer to make sure that when someone calls our code, they do not get something unexpected?  We can check to make sure they follow our instructions.  It's simple enough, although you can certainly take it greater depths.  Let's go back to our do_foo example and put a check in place...

foo () {
    test $# -eq 2 || exit 1
    ...
    ...
} #end do_foo


Let's break down the line I just added...  test lives in /usr/bin and should be a fluent part of your shell vocabulary.  We are "testing" to see if the number of arguments ($#) is equal to the integer 2.  If not (symbolized by ||) then we exit with non-zero status, which is the UNIX convention for something other than success.  The next level of effort would include writing a shell equivalent to Perl's die subroutine.  This would allow an error message to accompany the exit.  We'll save that for another article.

So, what's the benefit of adding this code-bloat to our subroutine?    It's common to have a function that uses optional arguments and acts differently depending on what arguments it receives.  If the function expects ARG_ONE and ARG_TWO, and you call it with only ARG_ONE, it may assume that ARG_TWO is equal to "".  In that case, the output may be "object not found" rather then "Whoa!  You made a mistake calling me!".  If you were depending on a specific output, this could cause later code blocks to break.

Here's a more specific example.  If we are using the ldaplist command to check on project information, we will get two totally different sets of output if we omit a second argument.  Pay particular attention to the command and arguments in the examples below:

testbox# ldaplist project
dn: solarisprojectname=srs,ou=projects,dc=mydomain,dc=com
dn: solarisprojectname=bar,ou=projects,dc=mydomain,dc=com
dn: solarisprojectname=foo,ou=projects,dc=mydomain,dc=com
dn: solarisprojectname=group.staff,ou=projects,dc=mydomain,dc=com
dn: solarisprojectname=default,ou=projects,dc=mydomain,dc=com
dn: solarisprojectname=noproject,ou=projects,dc=mydomain,dc=com
dn: solarisprojectname=user.root,ou=projects,dc=mydomain,dc=com


In contrast, what we REALLY wanted was only one line that matches our criteria, not the whole set of data.

testbox# ldaplist project solarisprojectname=user.root
dn: solarisprojectname=user.root,ou=projects,dc=mydomain,dc=com


If we use an argument checker, the error woudl be caught immediately rather than passing on a long list of irrelevant data to whatever we do next.  In this case it's particularly ugly because both outputs are identically formatted.  Maybe you'd find the problem quickly, maybe you wouldn't.

When your code gets to be hundreds of lines long and you need to start debugging obscure behavior, it can save you a lot of time to write self-policing code.  Chances are that if you make a simple mistake calling that subroutine it will fail immediately rather than  doing the wrong thing in a hard to find way.  A line of prevention is worth an hour of debugging!


Using syslog with Perl
2006-05-25T13:02:00.000-05:00
I recently had an occasion to write a fairly simple Perl script that checks for rhosts files in any home directory which is configured on a system.  Nothing fancy, but very useful.  After getting through the file detection logic I was left with the question, what now?  Should I write a custom log file?  Should I call /usr/bin/logger?

As always, I looked for precedents and standard facilities.  The first thing that came to mind was syslog.  And of course, the fact that I was using Perl led me to believe that I wasn't going to need to execute an external process (the "duct tape hack" as I call it).  I view the shell as another language, and something never really feels right when I need to embed one language within another.  Don't even get me started about embedding big awk scripts inside shell scripts...  That's going to be a future topic.

The duct tape method is bad for a number of reasons.  There is overhead associated with forking and executing a new child process from your main script.  If you are running awk and sed, or other tools thousands or millions of times against a file then you are forcing Solaris to execute far more system calls than necessary.  By keeping it all inside Perl and using modules, you can let the interpreter do the work, and realize a good part of the efficiency that C system programming gives you.  I'll save the specifics of this for a later time - we need to dig into the syslog example.

In this case I quickly found the standard Sys::Syslog module.  This little gem makes it a snap to log output.  I won't go into the Solaris syslog facility here, but suffice it to say that you'll need to arrive at your intended Facility and Priority before going farther.  For my purposes I went with User and LOG_NOTICE.

To begin with, we need to include some libraries...

use Sys::Syslog;


When we want to set up the connection with syslog we do the following:

openlog($progname, 'pid', 'user');


The above line specifies that we will use the 'user' facility, which is typically what you should be using if you don't have a specific reason to go with one of the other options.  It also specifies that we want to log the pid of the logging process with each entry.  Logging the pid is a convention that isn't always necessary, but I like it.  The first part, $progname is a variable that stores the name of the script.  This deserves a little extra attention.

Since I'm known to change the name of my scripts on occasion I don't like to hard code the name.  In shell scripts I usually set a progname variable using /usr/bin/basename with the $0 argument.  $0 always contains the first element in the array of command line variables.  So, if I called a script named foo with the arguments one, two, three, the command would look something like this:

# /home/me/foo one two three

The resulting array $* would be: 

[0:/home/me/foo][1:one][2:two][3:three]

To identify our program name we want the first array element.  However, we don't want all that extra garbage of the path.  It makes for a messy syslog.  The basename UNIX utility helps us to prune the entry.  Here's an example in shell:

$ basename /home/me/foo
foo

If we want to do the equivalent in Perl without spawning an external process we can use the File::Basename module.  Again, with a simple include at the top of our script this function becomes available to us:

use File::Basename;

Now we can put it all together and create an easily referenced identity check:

my $progname=basename("$0");


Why don't we just hard code the script name?  After all, not everyone likes to refactor their code for fun.  Besides the idea that we want our code to be maintenance free, there are times when one set of code may be called from links which have different names than the primary body.  For example, let's assume that the script foo performs three functions: geta, getb, and getc.  To make it easier to call these functions we want to be able to call these directly without duplicating code.  Here's how we could do that:

# ls -l ~/bin
-r-xr-xr-x   1 root     root        5256 Jun  8  2004 /usr/local/bin/foo
# ln ~/bin/foo ~/bin/geta
# ln ~/bin/foo ~/bin/getb
# ln ~/bin/foo ~/bin/getc

We can now call any of geta,getb,getc and actually call foo.  With some simple logic blocks based on what $programe evaluates to we are able to create a convenient interface to a multi-functional program with centralized code.  Nice!  But I digress - let's get back to looking at syslog...

We have opened a connection to the syslog, and now is the moment of truth.  Let's write a syslog entry...

syslog($priority, $msg);


Let's recap...  I used a facility of user, and a priority of notice.  I want to record the pid, and write a message.  What does this look like when its executed?

May 25 11:01:25 testbox rhostck[833]: rhosts file found at /u01/home/cgh


That was really easy, and it's much cleaner than executing the external logger utility because it's all inside Perl.


A plethora of ldapsearches...
2006-05-23T10:09:00.000-05:00
If you're going to deploy a directory service for Solaris systems, and you are really lucky, your server and clients will all be using a Solaris version greater than 9.  LDAP works nicely in 9, but it's a bit of a transition release.  Only in Solaris 10 is Sun's commitment to LDAP clear.  Let's take a look at one of the more frustrating examples of Solaris 9's transitionary status: The ldapsearch command.

   ldapsearch comes in many different flavors.  First is the native Solaris version which lives in /usr/bin.  On Solaris 9 this version does not support SSL (-Z option).  In Solaris 10 SSL is nicely supported through this client.  Next we have the iPlanet flavor which lives in a dark and gloomy path: /usr/iplanet/ds5/shared/bin.  This is installed by default with Solaris 9 and happily supports SSL despite its gloomy path.  But wait, there's still one more!  After installing the JES Directory Server you will find one more flavor of ldapsearch living in /usr/sadm/mps/admin/v5.2/shared/bin.  Now that's an intuitive path.  This last flavor will only be on your server, but I'd hate to leave it out of the fun.

   As if having too many to choose from isn't enough, two of the ldapsearch flavors require proper setting of the LD_LIBRARY_PATH variable.  When a dynamically linked binary requires a library that lives somewhere other than the system default (usually /usr/lib variants) it needs the LD_LIBRARY_PATH variable to tell it where to look.

   Here's an example of a binary that needs the extra help from LD_LIBRARY_PATH:

testbox$ /usr/sadm/mps/admin/v5.2/shared/bin/ldapsearch
ld.so.1: ldapsearch: fatal: libldap50.so: open failed: No such file or directory
Killed

  So what happened?  Let's take a closer look...

testbox$ truss /usr/sadm/mps/admin/v5.2/shared/bin/ldapsearch
execve("/usr/sadm/mps/admin/v5.2/shared/bin/ldapsearch", 0xFFBFFB54, 0xFFBFFB5C)  argc = 1
resolvepath("/usr/lib/ld.so.1", "/usr/lib/ld.so.1", 1023) = 16
resolvepath("/usr/sadm/mps/admin/v5.2/shared/bin/ldapsearch", "/usr/sadm/mps/admin/v5.2/shared/bin/ldapsearch", 1023) = 46
stat("/usr/sadm/mps/admin/v5.2/shared/bin/ldapsearch", 0xFFBFF928) = 0
open("/var/ld/ld.config", O_RDONLY)             Err#2 ENOENT
stat("../libldap50.so", 0xFFBFF430)             Err#2 ENOENT
stat("../lib/libldap50.so", 0xFFBFF430)         Err#2 ENOENT
stat("../../lib/libldap50.so", 0xFFBFF430)      Err#2 ENOENT
stat("../../../lib/libldap50.so", 0xFFBFF430)   Err#2 ENOENT
stat("../../../../lib/libldap50.so", 0xFFBFF430) Err#2 ENOENT
stat("../lib-private/libldap50.so", 0xFFBFF430) Err#2 ENOENT
stat("/usr/lib/libldap50.so", 0xFFBFF430)       Err#2 ENOENT
ld.so.1: ldapsearch: fatal: libldap50.so: open failed: No such file or directory
write(2, " l d . s o . 1 :   l d a".., 81)      = 81
lwp_self()                                      = 1

Here we can see Solaris trying to find the required dynamically linked library, libldap50.so.  It traverses 8 directories, each time returning the ENOENT key which intuitively means "ERROR - No entity found".  So, job #1 is finding that library and acquainting it with the binary that's lost its way...

testbox$ grep libldap50.so /var/sadm/install/contents
/usr/appserver/lib/libldap50.so f none 0755 root bin 380348 45505 1052289104 SUNWasu
/usr/dt/appconfig/SUNWns/libldap50.so f none 0755 root sys 450716 23095 1032825102 SUNWnsb
/usr/iplanet/ds5/lib/libldap50.so f none 0755 root bin 361976 55632 1013353620 IPLTdsu
/usr/lib/mps/libldap50.so f none 0755 root bin 392416 44988 1100692806 SUNWldk
/usr/lib/mps/sparcv9/libldap50.so f none 0755 root bin 433976 29179 1100692807 SUNWldkx

  In this case, we know that the needed library is going to be used with the JES ldapsearch, so we'll guess that appserver's offering isn't quite what we want.  /usr/iplanet looks tempting, and will probably work, but what we want is the /usr/lib/mps directory which is distributed with the Sun LDAP C SDK.

   So now that we've found the missing library, let's plug it into the LD_LIBRARY_PATH and see what happens.  I'm using the Korn shell, so if you're a C-Shell type you'll just have translate on the fly.

testbox$ export LD_LIBRARY_PATH=/usr/lib/mps:/usr/lib/mps/sasl2
testbox$ sudo /usr/sadm/mps/admin/v5.2/shared/bin/ldapsearch [...]
version: 1
dn: dc=foo,dc=com
objectClass: top
objectClass: domain
objectClass: nisDomainObject
dc: apps
nisDomain: foo.com

  It worked!  (You didn't doubt me did you?) You may have noticed that I actually added two paths.  After fixing the first missing library you would have dicovered a second missing one which was identified and fixed the same way.  I love a problem with multiple layers...  Especially when layer #2 is the same solution I needed to peel layer #1.  The other thing to note is that I abridged the command line for ldapsearch.  Executing an ldap query with SSL can be like writing a book so I cut it short.

  So, not only do you need to pick the right ldapsearch flavor, but you also need to set LD_LIBRARY_PATH accordingly.  If you are using the Solaris native versions you don't need to do anything.  But for JES and iPlanet verions, here's what you need:

iPlanet: LD_LIBRARY_PATH=/usr/lib/mps

JES: LD_LIBRARY_PATH=/usr/lib/mps:/usr/lib/mps/sasl2


So which one should you use?  Here's a quick flow to make that decision.  If you are using Solaris 10, just go with /usr/bin/ldapsearch.  It does everything without any hassle.  If you are on 9, then a decision emerges.  If you have an SSL-secured directory server you can not use /usr/bin/ldapsearch.  Typically, you will use the iPlanet version on Solaris 9, and if you are on the  server itself, go with the JES version.

So there you have it, a lot of hassle can be saved by deploying on Solaris 10 rather than 9.  Most of what you'll need the Directory for will be handled by Solaris internals, so you won't need ldapsearch, for example, to authenticate users against the Directory Server.  Where you will need ldapsearch is if you are storing custom entries in the directory, or executing a special query against it.


Solaris Naming Services: The good, the bad, and the ugly
2006-05-22T22:59:00.000-05:00
One of the first things that needs to be done in streamlining an Enterprise is establishing a centralized naming service.  Of course there are some fringe reasons not to, but the bottom line is that the IT world is continuously being asked to do more with less.  If you need to manually update the passwords for 100 users on 100 systems once each month you are either a great candidate for life in a Skinner box or you are failing to do more with less.  

My experience has always been that the risk of someone propagating a catastrophic change across a data center is much lower than that of someone making a subtle manual error that goes undetected for months until it rears its ugly head.  When you know that your finger hovers over the big red button that controls the fate of your buddy's on-call pager you get religion fast and become very careful about what you propagate.  Centralization and automation are good.  Manual effort is bad.  So, we've established that we want to do this.  So, what is a Solaris site to do next?

There are a few naming services to choose from.  In our case, common choices include: File propagation, NIS, NIS+, and LDAP.  Within LDAP there are a few different implementations now that the industry has begun to move in that direction.  Microsoft's Active Directory, OpenLDAP, and Sun's Java Enterprise System Directory Server.  Again, going by experience, the best path is usually the one which aligns with your primary vendor's offerings.  If you're running a Windows shop, use AD.  If you're running a Solaris shop, use Sun's Directory Server.  If you run both, it's a different question altogether, but I would be inclined to  create a layered solution involving both AD and Sun DS.  But, I digress.  Let's take a look at the options...

File propagation isn't as bad as it sounds unless you are doing it manually.  There are many bolt-on solutions for implementing the functionality of rdist.  It can be done over SSH, and with a surprising degree of control.  You can also use SSH with some script-fu to push scripts to be executed, or use a product like CF Engine.  While not a bad solution, it tends to be a bit heavier in the maintenance and integration time.  It's also somewhat limited in that it can push, but you aren't really centralizing.  Systems inevitably get left out or miss pushes which results in more custom coding to make queues, and pretty soon you're on your way to reinventing a management framework.  Ugh.  File pushing is a good solution within its scope, but I'm not a fan when it comes to a general centralized administration scheme.

NIS is a grand old favorite.  The environment I cut my teeth on included a Sparc 10 workstation hooked up to a QIC tape drive which acted as a NIS server for a building full of Sparc IPX workstations.  I found it fairly easy to manage, rock solid, and generally a great technology.  Unfortunately, it also included a whole new world of security gaps that while acceptable for a non critical workstation environment were totally inappropriate to a world class data center housing mission-critical data.  Even if it weren't inherently insecure, Sun announced the EOL of NIS, so the end is near.  Given that, I suggest that NIS is probably not the best technology to invest in these days although Linux has a good implementation that will allow it to continue for some time.

Our next option is NIS+.  Have you actually supported a NIS+ environment?  Yuck.  After the joy of NIS I had expected a simple evolution, but it felt more like assuming that if you speak English you can speak Russian.  While it fixed the gaps in NIS security, it was always annoying trying to figure out which key was broken and how to get it initialized again.  Although Sun has not EOL'd NIS+ yet, it's clear that it is not a technology whose momentum is being maintained.  The writing is on the wall.  NIS+ again proves to be a non starter.

Finally we arrive at LDAP, and for the reasons above, We look at Sun's Directory Server.  I'm currently working on a Directory infrastructure for a large site, and have been both impressed and frustrated with the product.  I'll be explaining this dichotomy in more depth in future installments, but a few months into the project, I'm confident that I made the right recommendation.  

Why?  First of all the updates for this product come from the same place all the other Sun updates come through.  Trying to keep track of multiple product update sites is a non-value-add activity.  Second, LDAP is insanely flexible.  You can store just about anything in a directory, and access it via shell script, OS client, Perl script, and more.  Why is this cool?  Its much easier than storing anything in an Oracle database because you won't need to gain a new certification in relational databases to take advantage of this new piece of your infrastructure.  A systems engineer or advanced admin can do everything they need.  Third, DS is part of the Java Enterprise System.  JES is a stack of middleware products which are all nicely integrated with the OS and support.  By giving us a stack of common and world-class components Sun has given systems engineers the opportunity to start including a deeper application awareness into their scope with an easy product to standardize on.  And the final reason to go with DS and JES:  It's free to download and use.  Before you put it into production, USE it.  Put it on your x86 machines, put it on your Sparc machines.  Hack it, tweak it, learn it, know it, master it before you go live.  Sun has just leveled the playing field.  Now you can learn a world class product without capitol investment because Sun's on-line documentation is second to none.  One last note:  After you've enjoyed Sun's generous new distribution model, buy a production support contract or JES subscription.  Don't go into production with unsupported software.  You know better, and your customers expect that you'll advise them to do the right thing.  Besides, I'm sure you didn't go into production with an unsupported Linux server, right?  After all, you were too smart to fall for that whole "Linux is free" story weren't you?  Nothing is free.

And for my final act I'd like to mention the paradox that continues to twist my mind into little pretzels.  NIS and NIS+ were OS integrated components that were free components of a great Operating Environment.  Sun has always been about the large scale net infrastructure, and the integrated NIS servers echoed that intent.  With the JES Directory Server Sun has gone beyond the site-scale naming services with a product that can store just about anything and scale it to your wildest dreams.  Think consolidation of eBay and Amazon identities and DS isn't breaking a sweat - it's that good.  But in the mean time, there's a lot of sites that need a centralized naming service for their tightly secured data centers that will never share identities with the outside world, and never cross 1000 users.  That's like having a Ferrari that never gets out of first gear.

What fills the gap between 100 and infinity?  Fortunately, from a technical perspective JES DS scales in nice increments, and can deployed for a single site's centralized naming services on reasonably sized hardware.  But there's a catch:  Support will cost you.  Even if all you are doing is Solaris native client support, the DS costs money to obtain a software support contract.  Sun, what are you thinking here?  Kill NIS and NIS+, tell everyone to go to LDAP, and then neglect to include it in the Operating System support?  Crazy!  Native OS support should be included with Solaris just as Active Directory is free with the Windows Server support.  Charge us more money when we ask you for support outside native clients, but don't' make us pay for what you provide for free with NIS+.  

JES is a great product, I'm glad we're using it in our project, but I was really disappointed when I discovered this Dilbertian marketing twist.  There's still hope; SRM used to be an expensive add-on that's now a part of the OS.  Given enough time, Sun usually does the right thing.


In the beginning there was SunOS...
2006-05-22T11:54:00.000-05:00
I'm enjoying a renewal of enthusiasm for Solaris, and this posting marks the beginning of a blog tracking the things I'm working on and exploring.  I've use Solaris professionally for more than ten years from both Systems Administration and Systems Engineering perspectives.  Six of those years I spent working in various roles at Sun Microsystems; a journey I'm grateful for as I learned more in those years than I would have elsewhere.

For a significant portion of those six years I was increasingly frustrated by how much ground Linux made on Solaris' lead.  Linux has its place, and I'm certainly not anti-Linux or anti-OSS.  In fact, quite the opposite. I've been tinkering with Linux since 1992, and use it regularly at not-for-profit site I volunteer at.  For many years it was primary workstation (Fedora / Red Hat) so I've spent years observing Linux in both a server and workstation role.

I'm no longer using the Linux workstation which served me the past few years.  I've fallen in love with Mac OS-X and never looked back.  I don't expect to write much about Mac OS because in itself it doesn't stand out.  That's the whole point - after using my Mac I talk more about what I've done than how I did it.  Interesting paradigm shift.  

With the advent of Solaris 10 I firmly believe the Solaris has taken another evolutionary leap which seem to leverage the best of both Open and Closed source development models.  I think this evolution is important as it shows what can be done when extremism is curbed for practicality.  Solaris 10 found a sweet spot which I hope to write more about.

I'm now working as a consultant for a large UNIX shop in my area and working on some exciting technologies.  My primary interest is in large scale UNIX infrastructure, so expect to see a lot of content focused on things like Directory Server, Consolidation, Automation, Resource Management, RBAC, and much more.

So there you have it...  A brief introduction to me, and the intended content of this blog.  I've been notoriously lousy at keeping blogs up to date, but not for lack of good intent.

Run Level	Description
0	Shut down all processes and power down to ok> prompt (sparc).
S	Run as a single user with some file systems mounted and accessible.
1	Administrative state with access to all file systems, but no user logins permitted
2	Multi-user state. For normal operations. Multiple users can access the system and all file system. All daemons are running except for the NFS server daemons.
3	Multi-user state: For normal operations with NFS resources shared. This is the default run level for the Solaris environment.
4	Alternative MU state. This is not used by Solaris, but is available for site customization if needed. I recommend NOT using it.
5	Power down after shutting down all processes.
6	Reboot the system.