No space left on device? (metainit)

Here comes another rant about error messages. I was rebuilding a server today that uses SVM to manage som SAN storage which gives a home to four very nice Solaris zones. I began by issuing a metainit command to build a concat/stripe device from these two SAN devices...

testbox{lvm}$ sudo metainit -f d100
metainit: testbox: /etc/lvm/md.tab line 72: c4t6006048000018775125753594D433742d0s7: No space left on device


What?!?! I took a quick look at partitioning...

Part Tag Flag Cylinders Size Blocks
0 unassigned wm 0 0 (0/0/0) 0
1 unassigned wm 0 0 (0/0/0) 0
2 backup wu 0 - 56653 25.93GB (56654/0/0) 54387840
3 unassigned wm 1 - 3 1.41MB (3/0/0) 2880
4 unassigned wm 4 - 56653 25.93GB (56650/0/0) 54384000
5 unassigned wm 0 0 (0/0/0) 0
6 unassigned wm 0 0 (0/0/0) 0
7 - wu 0 - 56653 25.93GB (56654/0/0) 54387840


Ok, so the partition exists. What the heck is wrong?

In my absent minded hurry to get this trivial task completed I made an undiscipined assumption that both devices which are to comprise d100 have the same underlying VTOC. It turns out they did not. One of them was set up to use slice 4, and the other slice 7.

So, I issued a quick command to synchronize them using the traditional prtvtoc | fmthard tango, then edited the /etc/lvm/md.tab file to accomodate the s4 slice when defining d100. This time it worked nicely.

But come on, "no space left on device?" What kind of an error message is that? How about something more like, "specified slice does not exist." Technically, a storage device of size zero would have no space available, but there sure are more direct ways to express that concept.

The Evolution of Email

Have you ever stopped to ask yourself what benefits have been derived by the evolution of email from the days of ASCII text to our modern world where Microsoft Word can act as the email editor?

Fortunately I don't need to ponder this question any longer. Today I received an email which simply would not have had the same impact back in the old days of low-tech correspondence.

The email started out with the following, which is a direct quote:

Starting IMMEDIATELY - ZERO TOLERANCE for any and all non compliance of the following process!
...

It looks pretty menacing in ASCII text, but thanks to Microsoft Exchange and its mind-blowing capabilities to allow more effective self-expression I was able to receive that motivational phrase in a 24-point underlined red font.

I have to admit, it's difficult to fully realize the gravity of the phrase without gratuitous aesthetic enhancement. Let's face it, it would take a PowerPoint attachment to more effectively intimidate me.

The Unconventional Explorer

The habit of Sun's explorer dumping output to /opt/SUNWexplo/output makes me wince a bit. In all fairness, I think the documentation could be seen as technically inconclusive, but in spirit I believe a more correct solution is not difficult to derive.

Consulting the Solaris 10 System Administration Guide: Devices and File Systems we find a concise chart of default Solaris file systems and their raison d'etra. Three specific entries jump out at me as being relevant to this topic:

• /opt: Optional mount point for third-party software. On some systems, the /opt directory might be a UFS file system on a local disk slice.


• /var: System files and directories that are likely to change or grow over the life of the local system. These include system logs, vi and ex backup files, and uucp files.


• root(/): The top of the hierarchical file tree. The root (/) directory contains the directories and files that are critical for system operation, such as the kernel, the device drivers, and the programs used to boot the system. The root (/) directory also contains the mount point directories where local and remote file systems can be attached to the file tree.

Considering these practices, it makes perfect sense that explorer is installed in /opt/SUNWexplo. So far, so good. On the systems we deploy at my current place of employment, the /opt file system is part of the root file system, which means that Explorer is dumping output at ~ 5mb per shot onto the root file system.

All things considered, it's pretty benign considering we use either 72 or 146 GB boot drives. But as Solaris Jedi, we look to the harmony and availability of the system, and Explorer is definitely creating a disturbance in the force by dumping volitile files into a subdirectory within /opt. What if someone wrote a script to manage the contents of that output directory and made a little error in their code? What file system would you want it compartmentalized within? Would you want the potential of filling root, or filling a less critical file system? Methinks there must be a better way.

As in most dilemmas, I tend to look for precedents. Where would we find a traditional location in the standard Solaris file system that might be used to spool (hint, hint) volatile files which might grow over time? I would immediately look to /var. There are two immediate paths I see as being preferential to /opt/SUNWexplo/output.

The first option would be /var/spool/explo. This would follow a convention that aligns with out use of a local explorer agent. The servers here produce an explorer on a regular file which is immediately shipped to a central (on-site) repository. The most recent explorer is typically left on the system and the history is managed at the repository. This makes the output directory a traditional spool directory, and as such a perfect fit for /var/spool/explo.

Where this may not be as intuitive is the case of an environment where explorers are retained on the host rather than collected and managed centrally. In that case, the explorers are better described as log files than spools. Intuition brings me to the use of /var/opt/SUNWexplo/output for this case. It's close to the legacy directory structure of the tool, which makes the solution marginally more intuitive than using a spool directory. It also follows the rarely observed SYSV standard of pairing optional software installed in /opt with a directory in /etc/opt, /usr/opt, and /var/opt. I'm not a fan of this specific model when taken to its literal implementation, but it's worth noting.

So, which one is best? As noted earlier, it depends. If I were a member of Sun's Explorer engineering team and needed to pick one consistent location with the intent of minimizing discontent I would select /var/opt/SUNWexplo/output. It is intuitive in the largest set of configurations, and doesn't break any rules. My secondary recommendation would be to create a symbolic link to redirect /opt/SUNWexplo/output for backwards compatibility over the next few years until it could be phased out.

Now I'm left wondering what interesting problems I might create in the data center if I put together a change package that implemented this very model... Nothing is ever as simple or benign as it appears on the surface.

Complex commands with sudo

I've heard all the excuses for why someone issued a "sudo su -" command, and instantiated a shell that no longer tracked their actions. Of course we can argue about how to configure sudo so that problem goes away, but what if you have a lenient sudoers configuration?

The problem usually occurs when you need to redirect output. For example:

# tar cvf - /etc/ | gzip -c > /protected_dir/etc_backup.tgz


Or, the one which I just used, and reminded me that this deserves a quick posting:

# m4 somefile.m4 > newfile.cf


Both of these will fail if the target directory is one that your user ID does not have permission to write to. In many cases, the frustrated SA will simply use sudo to "su" to the root user and perform the command there. But we Solaris Jedi know that this is simply a temptation of the dark side pulling at a time when you need to get work done.

The right thing to do is create a subshell that executes the command. Returning to the above examples, the right instantiation would be:

# sudo sh -c "m4 somefile.mc > somefile.cf"
# sudo sh -c "tar cvf - /etc | gzip -c > /protected_dir/etc_backup.tgz"


Works like a charm. That being said, I'm much more an advocate for using RBAC on Solaris, but I'm going to fight the power of scope creep on this posting and stick with sudo.

Open Engineering: snmpXdmid follow-up

Just for kicks I did a search on Google for the same problem I encountered only a few days ago: How to disable snmpXdmid in Solaris 10. The first time I searched for this information I found a wealth of Solaris 8 and 9 information, but very little about Solaris 10, and nothing about the alleged bug in SMF.

After finding the solution, I posted to this blog documenting the answer. Having given the Googlebots a little time to work their magic, I returned to the scene of the crime and entered the following search query: "disable snmpXdmid Solaris 10". SolarisJedi shows up in the #3 position for that query with all the information necessary for remediation. It feels pretty good knowing that someone else might get to complete a job in five minutes rather than five hours.

While I was riding the warm-fuzzy, I started thinking about how many large Corporations with legions of skilled SAs and Engineers maintain private knowledge bases rather than using public resources. I'm not talking about internal problems and proprietary issues - I'm talking about solving problems related to the generic off the shelf products they leverage. Let's face it, there isn't much proprietary about sendmail and DNS other than perhaps some parameters that are easily scraped clean.

Companies like Sun Microsystems have really paved the way of the future by encouraging their employees to blog, and trusting that proper standards of professionalism will be maintained. I believe Sun recognized that many of the problems they generate revenue from solving are based on the Internet's ability to act as a research assistant. I'd like to see more IT professionals invest back in the community.

One of the points in the System Administrator's Code of Ethics, a joint statement by LOPSA, USENIX, and SAGE, is the following:

RESPONSIBILITY TO THE COMPUTING COMMUNITY: I will cooperate with the larger computing community to maintain the integrity of network and computing resources.

Sometimes the definition of "network and computing resources" is one of hardware and software, but I suspect that other times it ought to apply to the operators of those resources since you cannot have one without the other.

Die Hard: disabling snmpXdmid on Solaris 10 (dmi)

On a recent server build project we ran into a security scan that surprised us with a mandate that snmpXdmid be disabled. The alleged vulnerability is based on a buffer overflow that originated in the days of Solaris 8 as documented in CIAC Information Bulleting l-065 and SunSolve Security bulletin #00207. The details aren't important to this story other than finding it entertaining to respond to a Solaris 8 vulnerability on a Solaris 10 build. I'll save my thoughts on the corporate world's implementation of automated scanning for another post.

Our normal JumpStart image was configured about a year and a half ago, and in it we addressed the problem. Of course, none of us could remember what we did, and it turns out to not be the easiest thing to extract from Google. The process is pretty straight forward once you find it...

# svcadm disable dmi


Next, I dug up a quick test case to make sure the fix worked. It's easy enough to check registered RPC services using rpcinfo.

# rpcinfo -p | grep 100249
100249 1 udp 43483
100249 1 tcp 42683
#


Wait a minute... I thought I turned that off! The normal behavior of the service management facility, or SMF is to immediately change the state of a service after a disable command, so I made the (false) assumption that I had disabled the wrong service. After some additional research and testing I found that I wasn't wrong. The SMF was wrong. I did a quick zone reboot, and sure enough, the service was no longer responding. This led me to conclude the DMI service was not removing its registration with the portmapper (svc:/network/rpc/bind:default).

The next step on the path is to look at the service method that stops and starts the DMI service. All method scripts are stored in /lib/svc/method, and this one is easy to find: svc-dmi. So now we need to take a look at how it goes about stopping the service:

stop)
/usr/bin/pkill -9 -x -u 0 -z ${_INIT_ZONENAME:=`/sbin/zonename`} \
'(snmpXdmid|dmispd)'
;;


And so we come to the flaw in this method. In order to be consistent with predominant SMF behavior, this method should stop the service completely. There are two ways we can address this. We can either restart the entire portmapper, or we can be more surgical and remove the snmpXdmid registration from the portmapper. I preferred the latter since restarting the portmapper could temporarily impact other services. the code change is pretty simple:

stop)
/usr/bin/pkill -9 -x -u 0 -z ${_INIT_ZONENAME:=`/sbin/zonename`} \
'(snmpXdmid|dmispd)' && /usr/bin/rpcinfo -d 100249 1
;;


The problem, of course, is that to implement this fix I need to modify a script which is managed by the pkgadd facility, and subsequently, checksummed. I wont' get into addressing that issue right now, as the goal of this post is simply to provide breadcrumbs to other Jedi working to improve security with as little impact as possible.

The DaVinci Zone: Automating Zone Installation

I love a good mystery as much as the next guy, and this one took a bit of piecing together. It's all documented, and with the proper grasp of docs.sun.com, man pages, and Google query syntax anyone can automate their zone installation. Since it took me a while to piece it together I thought I'd leave a few notes in the Jedi archives.

I'm going to leave my breadcrumbs in Perl, and focus on the workflow more than the syntax, so you won't be able to copy and paste code. If you know some Perl you chouls be able to fill in the blanks pretty easily.

The first thing we need to do is create the input stream for zonecfg. This is essentially the same things you would type if you doing it interactively, which is exactly how I derived the text I'm using.

# Open the file for writing
open(ZONECFGTMP, ">$zonecfgfile") or die "ERROR: Could not open $zonecfgfile for writing";
# Write the contents
print ZONECFGTMP "create\n";
print ZONECFGTMP "set zonepath=$zonepath/$zonename\n";
print ZONECFGTMP "add net\n";
print ZONECFGTMP "set physical=$zoneif\n";
print ZONECFGTMP "set address=$zoneip\n";
print ZONECFGTMP "end\n";
print ZONECFGTMP "exit\n";
close (ZONECFGTMP);


Next we need to create a sysidcfg file using a similar strategy... It gets a bit funky in the middle when I base some logic on whether or not the zone is entered into DNS. Solaris has what I consider a nuisance behavior during installation. If you want to configure DNS at install-time, the hostname must already be in DNS. If not, the install will revert to an interactive prompt asking if you really want to do this. To get around this, we need to FIRST determine if the zone name is in DNS. If it is, then install a sysyidcfg that reflects DNS. If not, then we need to use "none" for the sysidcfg naming service, and then install a resolv.conf file. It's kludgy, but it works.

# Make sysidcfg file (either NONE or DNS dep. on earlier check)
# File name should be mkzone.sysidcfg.ppid
$sysidcfg="$tmpdir/$thisscript.sysidcfg.$$";
open(SYSIDCFGTMP, ">$sysidcfg") or die "ERROR: Could not open $sysidcfg for writing";
print SYSIDCFGTMP "root_password=\n";
print SYSIDCFGTMP "system_locale=en_US\n";
print SYSIDCFGTMP "timeserver=localhost\n";
print SYSIDCFGTMP "timezone=US/Eastern\n";
print SYSIDCFGTMP "terminal=vt100\n";
print SYSIDCFGTMP "security_policy=NONE\n";
print SYSIDCFGTMP "nfs4_domain=$mydomain\n";

# if host not in DNS, use NONE, else use DNS.
if ( $zonenotindns ) {
print SYSIDCFGTMP "name_service=NONE\n";
} else {
print SYSIDCFGTMP "name_service=DNS {\n";
print SYSIDCFGTMP " domain_name=$mydomain\n";
print SYSIDCFGTMP " name_server=1.2.3.4, 1.2.4.4, 1.2.5.4\n";
print SYSIDCFGTMP " search=search.domain1, search.domain2\n";
print SYSIDCFGTMP "}\n";
} #end if

print SYSIDCFGTMP "network_interface=PRIMARY {\n";
print SYSIDCFGTMP " hostname=$zonename\n";
print SYSIDCFGTMP " ip_address=$zoneip\n";
print SYSIDCFGTMP " netmask=255.255.255.0\n";
print SYSIDCFGTMP " protocol_ipv6=no\n";
print SYSIDCFGTMP " default_route=$zonedefroute\n";
print SYSIDCFGTMP "}\n";
close(SYSIDCFGTMP);


If we needed to create a resolv.conf, it would look something like the example below. Note that I entered the DNS server list into an arry called @dnsserverlist. The $zonenotindns variable is set earlier in the execution when we perform an nslookup. I do this with a call to system() rather than using a separate module because it makes the code easier to distribute.

if ( $zonenotindns ) {
$resolvdotconf="$tmpdir/$thisscript.resolvdotconf.$$";
open(RESOLVDOTCONF,">$resolvdotconf");
print RESOLVDOTCONF "domain $domain\n";
foreach ( @dnsserverlist ) {
print RESOLVDOTCONF "nameserver $_\n";
} #end foreach
print RESOLVDOTCONF "search $mydomain\n";
close(RESOLVDOTCONF);


The piece that threw me for a loop was getting rid of the NFSv4 prompt. It turns out to be as simple as putting this command into the code right before the zone is booted, but after the zone is installed. Kudos to the OpenSolaris Zones and Containers FAQ for documenting it!

system("/usr/bin/touch $zonepath/$zonename/root/etc/.NFS4inst_state.domain");


Using the above files is covered well in other posts, so I won't duplicate content. Using these details, you should be able to get your site's zone installation automated without too much trouble.

Zonecfg: removing a resource

I just noticed that there aren't a whole lot of examples of removing a resource from a zone to be had in the vast caches of Google at the moment. It's pretty simple once you understand the zonecfg syntax. Of course, just about everything in UNIX is simple once you know how to do it!

First, we need to fire up zonecfg and look at the specifics of how our zone is configured:

cgh@testbox$ pfexec zonecfg -z testzone
zonecfg:testzone> info
zonename: testzone
zonepath: /export/zones/testzone
autoboot: true
pool:
limitpriv:
inherit-pkg-dir:
dir: /lib
inherit-pkg-dir:
dir: /platform
inherit-pkg-dir:
dir: /sbin
inherit-pkg-dir:
dir: /usr
fs:
dir: /myapp/u01
special: /dev/dsk/c2t5006048ACC36D646d138s0
raw: /dev/rdsk/c2t5006048ACC36D646d138s0
type: ufs
options: []
net:
address: 192.168.1.1
physical: e1000g0
zonecfg:testzone>


In this case, the file system /myapp/u01 has a problem and is preventing the zone from rebooting. In order to remove it we need to use the remove syntax, which requires enough parameters to uniquely identify the resource we want removed. In this case, the dir setting of /myapp/u01 should be sufficient.

zonecfg:usa0300uz0002> remove fs dir=/uv1234/u01


A quick repeat of the info command should now display that the file system is not part of this configuration, and indeed it does.

zonecfg:testzone> info
zonename: testzone
zonepath: /export/zones/testzone
autoboot: true
pool:
limitpriv:
inherit-pkg-dir:
dir: /lib
inherit-pkg-dir:
dir: /platform
inherit-pkg-dir:
dir: /sbin
inherit-pkg-dir:
dir: /usr
net:
address: 192.168.1.1
physical: e1000g0


And funally, we commit the changes using the commit command. A quick call to zoneadm, and a reboot is issued, allowing our zone to successfuly reboot.

RBAC, Zone Management, and the mortal user

I'm doing a lot of work with automating zone configuration at the moment, and have been using the zlogin command frequently. Having never been a big fan of Sudo, I really wanted an excuse to dabble in RBAC and see if I could get it to work for me. Turns out to be a very trivial thing. In this case I wanted to be able to perform zone administration as conveniently as possible without spending a lot of time whittling down a command set - just give me quick and easy.

I started out looking for any execution attributes which may have been preconfigured for my convenience...

cgh@testbox$ grep -i zone /etc/security/exec_attr
Zone Management:solaris:cmd:::/usr/sbin/zlogin:uid=0
Zone Management:solaris:cmd:::/usr/sbin/zoneadm:uid=0
Zone Management:solaris:cmd:::/usr/sbin/zonecfg:uid=0


So now I needed to get them plugged into my user ID (I didn't want to fiddle with su-ing to a role, just wanted them in my ID). I loaded up the /etc/user_attr file into my favorite editor (for those who are curious, I'm a vi guy) and added my name, and the profile:

adm::::profiles=Log Management
lp::::profiles=Printer Management
rroot::::auths=solaris.*,solaris.grant;profiles=Web Console Management,All;lock_after_retries=no
cgh::::profiles=Zone Management


A quick test verifies that all is well with the world:

cgh@testbox$ profiles
Zone Management
Basic Solaris User
All


And finally we give it a shot:

cgh@testbox$ zlogin testzone
zlogin: You lack sufficient privilege to run this command (all privs required)


But of course! The use of RBAC commands seamlessly requires that you use an RBAC-aware shell such as pfcsh, pfsh, or pfksh. But at the moment my shell is a standard ksh. The easy way to get around this is to use the pfexec command.

cgh@testbox$ pfexec zlogin testzone
[Connected to zone 'testzone' pts/4]
Last login: Mon Feb 4 13:45:24 on pts/4
You have new mail.
root@testzone#


And there you have it. With RBAC, it's easy to attach administrative commands to a general user ID. Of course, this demonstration was a hack, and isn't a best practice. Why? Administrative commands are separated from user commands for a reason. You dont' want a general user doing things that can impact the entire system.

The best way to do this in most situations would be to embrace the R in RBAC and create a role for Zone Management that a user could assume to perform this work. In my case it's a lab machine, not many people are using it, and I wanted an excuse to play with RBAC.

Basename saves the day...

One of the things I like to do when setting up a Perl script is to set a variable called "thisscript". It's essentially the $0 special variable, but with a subtle twist. The inspiration for this article comes from forgetting the twist, and true to my mission, I am documenting my detours from the Jedi path.

I'm working on a fun script at the moment which simplifies and automates the process of deploying and configuring a zone. Sort of a JET-lite if you will. The script creates numerous temp files, and I prefer the following naming convention: "tmpdir"/"name of parent script"."functional identifier"."process pid". So, a file name may look like this: /tmp/mysite-mkzone.sysidcfg.343224. And here begins the oddity I ignored, and eventually fixed.

Although the script worked well, I noted the following output:

...
Cleaning up temp files...
/tmp/./mysite-mkzone.zonecfg.2012
/tmp/./mysite-mkzone.sysidcfg.2012
/tmp/./mysite-mkzone.resolvdotconf.2012


Fortunately, the way UNIX inteprets a pathname, this is a perfectly legitimate albeit circuitous path value. The "./" evaluates to the current directory and continues on its merry way. To be more explicit, the following examples all evaluate to the same value:

• /tmp/mysite-mkzone.zonecfg.2012


• /tmp/./mysite-mkzone.zonecfg.2012


• /tmp/././mysite-mkzone.zonecfg.2012


• /tmp/./././mysite-mkzone.zonecfg.2012

But, my heightened Jedi awareness felt this extraneous path element to be disturbing the balance of the force. Once you journey down the path of the dark side, it is difficult to return to the light. But where was this coming from? My first suspicion was a syntax error somewhere in a Perl string catenation.

In perl, strings are catenated with a dot operator ("."). For example, we could set up a string using catenation as follows:

$ vi catenation.pl
my $a="The quick brown fox";
my $b="jumped over the lazy dog";
my $sentence="$a" . " $b." . "\n";
print $sentence;
------
$ catenation.pl
The quick brown fox jumped over the lazy dog.
$


So, if I were to misplace a quote, it's possible that I might have included an errant period somewhere in the code. After a scan of each use of the variable I quickly determined that my hypothesis was unlikely to have manifested itself. I then returned to the code which set the initial variable:

my $thisscript=$0;


It was then that I remembered what I had omitted. I don't typically have "." in my current path. Just a habit, the result of which is another habit. I always qualify the path to whatever I'm running. So, if I'm executing a script called mysite-mkzone in the current directory, I execute the following on the command line:

$ ./mysite-mkzone


Now, consider the preceeding wetware behavior in concert with the following software behavior:

$sysidcfg="$tmpdir/$thisscript.sysidcfg.$$";


Herein lies the problem. Evaluating $sysidcfg we get the following: "/tmp + ./mysite-mkzone + sysidcfg + 12345" which explains where the extraneous "./" is coming from. So how did I fix it? I used File::basename. Which is a Perl equivalent of the shell basename(1) command. It deletes any path prefix ending in "/" from a string. In other words, it yanks the directory part of a command, leaving just the command. To use this, I made the following trivial modification to my code:

use File::Basename;
my $thisscript=basename($0);


And the output dutifully responded as follows:

...
Cleaning up temp files...
/tmp/mysite-mkzone.zonecfg.18129
/tmp/mysite-mkzone.sysidcfg.18129
/tmp/mysite-mkzone.resolvdotconf.18129


This wouldn't have happened if I'd used my normal starter tempalte, which has this variable pre-configured, but I'd made a careless decision to just go from scratch on this one. Yet another misstep on the path, but a good lesson.

Queuing Efficiency

On my way through the office door this morning I had an experience that I realized needs to be recorded so that I don't go on and rant about it in the future. This post will allow me instead to reference it, thus saving a vicious cycle of rehashing.

In addition to my career in systems engineering, and my small photography business I take martial arts classes. A few days ago I managed to pull a hamstring and sprain a toe on the opposite leg. Don't get me wrong - it's worth it. But these injuries are relevant to the story because they allow you to appreciate my inability to move quickly. I'm not limping, I'm just moving cautiously.

The weather was cold, and the sidewalks covered in treacherous ice. A bitter wind cut through me as I approached the building. Moving at a determined but non-rapid pace, I noticed someone about a mile (or three) front of me, presumably intending to enter the same doors. And then it happened.

They decided to take it upon themselves to hold the door open. At this point they almost need a telescope or radar to even know I'm planning to go through the same door, but yet they stood there holding it open while the arctic air flooded the entry to our building.

Here's the deal people... If someone behind you stands a chance of having the door slam in their face, then you hold it open. If they are carrying a heavy load and don't have arms free, you can wait until they get close to open the door for them. If it's more than five steps, you move along. Let's stop the self-gratifying good deed of holding a door open just to see if the recipient of your good will will start to jog because they feel guilty you are letting all that cold air into the building. If they have enough room to jog they probably don't need you to hold the door.

Got dependencies?

As anyone who follows my blog has learned, I'm a packaging junkie. I write packages for everything I deploy, and happily plunk them into my JumpStart server where they perform their duty in a predictable and maintainable way. Does it get any better than this? Well, actually... It got pretty rough this week as my troubleshooting skills enjoyed a solid workout.

We are in the process of moving our standard deployment from Solaris 9 with SRM project containers to Solaris 10 Zones. Now Zones are really surprisingly simple for the rich myriad of benefits they provide. It takes very little time to get to the point where you can set up a test box with multiple zones in a basic Solaris environment. But what if you don't have a basic environment?

We have packaged everything. Most of our /etc files that aren't stock are edited by class action scripts or postinstall routines. We've deployed hundreds of servers with this configuration and it worked flawlessly until we started deploying Zones. Then suddenly, our zones couldn't talk to the Directory, were missing resolv.conf, and all kinds of other cascading problems. Ugh. Where to start?

The first thing I did was check the error messages. Pretty clever, no?

The file /export/zones/testzone01/root/var/sadm/system/logs/install_log contains a log of the zone installation.


Ok, let's see what's in the log...

WARNING: setting mode of /var/spool/cron/crontabs/root to default mode (644)
ERROR: attribute verification of /export/zones/testzone01/root/var/spool/cro
n/crontabs/root failed
pathname does not exist

Installation of CGHtest on zone testzone01 partially failed.


Well that's strange. Let's see what the registry says about root's crontab:

$ grep "/var/spool/cron/crontabs/root" /var/sadm/install/contents
/var/spool/cron/crontabs/root e cron 0600 root sys 48 3760 1200431631 SUNWcsr:cronroot


The root crontab is owned by SUNWcsr. So it turns out that when the global zone creates the sparse root zone, it tried to install CGHtest before SUNWcsr was installed. The root of my problem was slipping a habit that I'm normally quite vigorous about enforcing.

Any time I use question marks in my prototype file to inherit attributes from an already-installed package I always, always, always stop immediately to look up that inherited object in the registry and add its parent package to a depend file. Here's an example prototype file displaying inheritance:

$ cat prototype
i pkginfo
i depend
i copyright
i i.cron
i r.cron
d none var ? ? ?
d none var/spool ? ? ?
d none var/spool/cron ? ? ?
d none var/spool/cron/crontabs ? ? ?
e cron var/spool/cron/crontabs/root ? ? ?


This was never a problem in our traditional JumpStart environment because the JET custom packages are installed after SUNWcsu. Fortunately, it's easy to control the order (dependency) of package installations using the depend(4) file.

$ cat depend
P SUNWcsr Core Solaris, (Root)
$ grep depend ./prototype
i depend
$


Problem solved! Of course, the problem never would have happened if I'd remembered my Jedi training. It's good to be humbled on a semi-regular basis.

The nature of systems engineering

I was reading an interesting passage from the Tao Te Ching this morning which, I believe, has great applicability to the nature of systems engineering.

We join spokes in a wheel,
but it is the center hole
that makes the wagon move.

We shape clay into a pot,
but it is the emptiness inside
that holds whatever we want.

We hammer wood for a house,
but it is the inner space
that makes it livable.

We work with being,
but non-being is what we use.

What does this mean to those of us who wield keyboards at the battle of the command line? Probably more things than can be said. To start the pondering I'd like offer two thoughts.

(1) The external perspective: Remember that the business your systems support don't think in terms of IOPS, MB/sec, or LoC. The IT organization does not exist to amaze itself. It exists to enable a business process. As you learn about Perl, Zones, and ZFS, are you also learning the business those technologies support?

(2) The internal perspective: Have you ever met an administrator or engineer whose wall is decorated with certifications, and yet you would not trust them to configure IPMP on a server you were responsible for? Have you met anyone who could write code as fluently as you speak your native language, and yet they could not effectively translate business requirements into functionality without great effort? As you learn the technologies you need to execute your job, are you also learning universal skills such as troubleshooting, and communication?

What other areas of our trade does this parable apply to?

Oracle's writing on the wall

I received an email newsletter this morning with the headline, "Oracle support betrays a preference for Linux and x86." Sun and Oracle seem to have a love hate relationship driven primarily by thir symbiosis rather than their ideals. This appears to be another chapter in that long story.

The article referenced by the newsletter mentions the fact that Oracle 11g is currently only available for Linux. That's a very interesting move considering the size of the Oracle installed base on Solaris. Not only the population size, but the class of customer. More than one global enterprise is running Oracle on enterprise class Solaris hardware.

I can't help but speculate that we're leading up to a boost in Sun's emphasis on PostgreSQL. First we saw its inclusion in the base Solaris 10 software. This is no small thing; even compilers are distributed separately. Postgres' own FAQ recommends use of Sun's compilers over GCC on the Sparc platform. It's practically heresy to recommend an open source product be compiled on anything other than GCC, so again this is not to be dismissed. Finally, I'll draw your attention to the release announcement for Solaris 10, Update 4 where enhancements to PostgreSQL DTrace probes are released. If this doesn't look like building up a rebellion, I don't know what does.

I give Sun a lot of credit for investing heavily in PostgreSQL and bringing some serious competition to Oracle. Evolution is based upon competition, and I'm happy to see the Sun species evolving into a new predator.

The trouble with packages and auto-pilot

I stumbled into a very interesting problem and resolution this morning which I think deserves some attention. I didn't work on the diagnosis and research, so I'm summarizing from an email thread. We use a Citrix server to share out GNOME environments from our development server. It's particularly nice when you're working from home and the VPN kicks you out, or if you're using public wifi and your connection is spotty.

At some point a week or two ago people began to notice that they couldn't connect to GNOME. This took a little while to unfold because some people keep sessions opened for extended periods of time, but eventually we discovered that it was dead for everyone. After eliminating license server issues there was only one thing we could come up with that had been done to the server.

A colleague had installed a current version of FireFox on the server because Sun's desktop environment is often very slow to integrate application software updates. He used the packages from Blastwave.org. Note that I say packages: a plural word. Indeed, FireFox turned out to be more than twenty packages when delivered by Blastwave.

The foundation of Blastwave is their packaging system, pkg-get. If you have any stick time in the Linux world you're probably familiar with something like Yum, apt-get, or up2date. These tools know how to connect to software servers through http, https, ftp, firewalls, proxies, etc. They also know how to resolve package dependencies. This can be very convenient on a Linux system where a single source handles the OS packaging and application packaging.

In contrast, Solaris provides pkgadd. Pkgadd can not resolve dependencies. It only knows how to retrieve packages from a specified URL, but does not have any ability to retrieve packages from a Sun resource. Pkgadd is a bit antiquated by modern UNIX standards unless coupled with the Sun Connection which is not quite the same thing. This huge gap between Linux packaging systems and Sun's pkgadd inspired Blastwave's packaging system and repository.

Blastwave provides many packages that are provided by the Solaris OS. The difference is that they provide more frequent and convenient updates. If you need bleeding edge features in the tools you install, Sun's usr/sfw/* and /opt/sfw/* packages will probably not help. I tend to think that it's more the exception than the norm to require updates that frequently. I know there are exceptions here and there, but overall, how often do you really need a new version of wget, or gtar? Although I love having latest and greatest "stuff", I even use the old Mozilla browser in Solaris and rarely have any problems.

When my colleague innocently asked Blastwave to install the latest FireFox package, it installed a fairly significant list of packages. One of them was fam, the file alteration monitor. For those who may not be familiar with FAM, it is described as follows (from the FAM web site):

GUI tools should not mislead the user; they should display the current state of the system, even when changes to the system originate from outside of the tools themselves. FAM helps make GUI tools more usable by notifying them when the files they're interested in are created, modified, executed, and removed.

We eventually discovered that fam installs an inetd service. I don't know, or care what that service is doing. What I do know is that I did not want a new service running. As a result of installing the Blastwave FireFox package and its slew of dependencies we ended up with a new service running and had absolutely no warning that it was happening. That service somehow conflicts with, and breaks GNOME. It turns out that there is an OpenSolaris bug describing the same symptoms.

Ignoring the obvious concerns about a simple desktop web browser requiring 20 package dependencies and breaking GNOME, I have a much larger concern. Turning up an inetd service creates a new attack vector for a server. Whether or not that is acceptable is a question of risk management. In many cases it doesn't matter. In our data center, servers must pass an external probe scan to be in production and adding services requires change requests. So for our purposes, the changes are not acceptable, and we will need to back them out. We are also imposing a ban on blastwave within our data center servers. It's simply not an acceptable framework for a mission critical server environment.

Whether or not you deem it reasonable to install an inetd service to run FireFox, it's hard to justify the intuitive nature of a web browser requiring the inetd service. Note that fam is NOT a FireFox dependency in other distribution channels. Of course, this kind of thing can be caught with good change management using a promote-to-production path, which is how we found this issue on our development server.

While Solaris' pkgadd facility is not as convenient as some of the Limux systems, it forces you to make conscious changes to a system rather than hitting auto-pilot and hoping for the best. I would love to see Solaris' packaging facility evolve into a tool with the capabilities of its Linux counterparts, but only for the freeware / OSS packages that are built and distributed by Sun (of which there are quite a few). I'd also like to see the ability to configure additional repositories (such as a local server for custom packages), as long as it's not set that way out of the box. I guess its time for me to start exploring Update Connection's capabilities.

My suggestions are as follows: First, beware the autopilot. Second, keep Blastwave on the workstations, and as far away as possible from the critical servers.

DVD upgrade adventures

I had an irresistible opportunity to rescue an Ultra 60 workstation from a trash nap recently. This is the sort of thing I really shouldn't do because I'm trying to reduce my data center footprint. On the other hand, it's such a cool workstation that I had to do it. This box was reported to be unable to boot, but I'm pretty good with hardware repairs, so decided to go for it.

Although it took forever to get through the process, the classic method worked. I can't count how many systems in this era seemed to have problems that turned out to be solved by reseating memory or CPUs. I did both, and it came to life like a resuscitated drowning victim.

Next stop, storage. I replaced the 9GB disks with 36GB disks from the unused half of my D1000 array. This was going too easy. As I was poking around the drive bay I noticed that the cable had been removed from the CD-ROM. Not a good sign. Tracing to the other end of that ribbon I noticed that someone must have been having a bad day as it was half ripped from the daughter board's crimping. Confirmed ugliness.

Being the fatal optimist I grabbed my tool kit and carefully pressed the ribbon back down onto its pins. Next stop, the drive bay. I reconnected he CD-ROM thinking that it might work... Nope. This one had a bad case of indigestion and spit out any disks I inserted. What's worse, once it spit them out, the drive tray could not be closed. Stick a fork in it - it's toasted.

I borrowed a Sun DVD from my 420r just to test out the SCSI channel, and successfully loaded Solaris 10, so it looks like the drive needs to be replaced. Next stop: eBay. I picked up a Pioneer DVD-302, which is one of the few remaining SCSI DVD options out there. I could have bought a Sun DVD, but they are all grey, and this case is beige. Can't compromise the aesthetics. (I'm really in bad shape, aren't I?). The drive arrived, looking shiny and new. I managed to get the thing installed, but it's not happy.

Booting from a DVD results in error messages like "Short read. 0x0 chars read". Eventually the retries end, and it complains about errors finding interpreter, and "Elf64 read error". Booting from a CD-ROM gets a little farther along before it spits out "incomplete read- retrying", and "vn_rdwr failed with error 0x5". Oddly, it does seem to be working once the OS is loaded, so this appears to be an incompatibility at the OBP level.

What annoyed me the most in this whole exercise was not finding anything in an hour of Google searches that indicated anyone had even attempted such an upgrade. I know there are quite a few U60s still kicking around out there, and I'd have to think their owners would be looking for DVD capability and higher speeds. I must have thought wrong. If you happen to be reading this post and have experience with a SCSI DVD-ROM being bootable in a Sun Ultra workstation I'd love to hear about it.

I guess I'll just have to keep looking for a beige Sun DVD-ROM on eBay, but so far the pickings are slim. Wish me luck.

IBM Sees the Light?

Wow, I didn't see this coming.

IBM and Sun today jointly announced that IBM will offer and fully support Solaris on their compatible hardware lines. This raises some interesting dust clouds.

What does this mean to AIX, IBM's flagship UNIX? Personally, I think it means little. Sun supports Windows and Linux on their hardware, but those of use who have been with Sun for a long time still prefer Sparc in most cases. I believe the same will be true of IBM and Solaris.

How will Solaris compete with the investment IBM has already made in optimizing their previously supported operating environments? There's no way it will be on the same level right out of the gates, but when you consider the OpenSolaris model, it becomes clear that IBM will not have to jump through hoops to make it happen, and I believe they will. When IBM announced that they would be supporting Linux it was initially a bit of a surprise because of the inherent undermining ot AIX. And yet, they have contributed some incredible advances to Linux's abilities in the enterprise data center. I think of IBM as the mature mentor that helped Linux to grow up.

Now Solaris is no padawan looking for a master to study under, so that makes for a different game. But there's no question in my mind that IBM will have a serious group of Jedi coders participating openly and actively in the OpenSolaris community, and that can only help Sun and Solaris.

Of course, the down side is the precident this sets leading towards too broad a foundation. Using Linux as an example we see a massive code base that tries to support as much hardware as possible. There are basic laws of software engineering, just as there are laws of physics, and the more lines of code you have, the more potential you have for bugs, integration issues, and regression failures. Doesn't matter how good your developers, the probability still goes up. I'd hate to see Solaris supporting everything Linux does; I'd like it to stay focused in its sweet spot of quality hardware, which despite my preferences, I think IBM hardware is in alignment with.

What does this mean for Linux in the Enterprise? Well, I think Linux has a tough climb ahead of it as it stares up the cliff at Solaris' backside. Linux was developed on PCs by people who aren't typically in an enterprise. You could argue that the coders went to Linux because they coudln't afford at home what they had at work, but the bottom line is still the same.

Linux does not have a lot of "stick time" on servers built at the scale of Sun's high end servers like the Enterprise 20k. On the other hand, Solaris has been running on multiprocessor systems since before Linux was a twitch in Linus Torvald's ear. You have to spend time working with servers that have 20GB of RAM and 64 processors before you can even anticipate the kinds of problems that occur. Linux just doesn't have that kind of time in a data center. I'm not saying they can't get there, I'm just saying you have to pay your dues to provide stability at the high end.

Keeping all that in mind, put yourself in IBM's shoes. AIX is not gaining market share, although its a rock solid enterprise class operating environment. Linux brought IBM a huge customer base, and helped them to sell Intel hardware. Unfortunately, it didn't really put them in the data center where they belong. Along comes Solaris with the openness of Linux, and the opportunity to leverage it quickly - just as they did with Linux. But this time, they start at the upper end of scalability and bypass that climb altogether. Where would you put your resources in the long run?

Wearing my purely speculative hat, I think this announcement was a big strike against Linux in the Enterprise, and a foreshadowing of Solaris' long term viability. As more and more products come on-line with the Internet, data centers are only going to grow. And as that continues to happen, consolidation will be the only way to drive utilization up and costs down. The natural extension of this prophesy is that the operating environment that scales best and stays stable is going to be the evolutionary top of the food chain. And I think that Solaris will be in that seat.

IPMP, anyone?

While scanning various news feeds this morning I ran into a story regarding a computer breakdown at the LAX airport. The first article suggested the problem was a network card failure, and the second article suggested the problem was a switch failure.

In either case, the result was 17,000 - 20,000 (varies by atricle) international passengers being stranded for a fairly significant duration. But wait, it gets better... "The system was restored about nine hours later, only to give out again late Sunday for about 80 minutes, until about 1:15 a.m. Monday." Two failures, both stopping passengers at an incredibly busy airport.

I'd like to offer my consulting services to LAX for free, and recommend that they move an obviously critical function over to servers running the Solaris operating environment where they can enjoy the benefits of IP MultiPathing (IPMP). A properly architected system would have had redundant switches, and multiple network interfaces, each connected to a unique switch. The failures indicated would have cause no interruption to service. This is server design 101.

What, you may ask, would be the cost of this highly advcanced architecture? Well, of course it depends on the cost of the switches you run because you'd need two, but on the server side its free, and included with Solaris. I run IPMP on the servers in my basement, and my wife can assure any who may ask, my IT budget is far less than that of the mighty LAX airport.

When will Wall Street wake up?

In case you had any lingering doubts as to whether Sun has been doing the right thing by embracing the Open Source model, take a moment to peruse an entry from Jonathan Schwartz' Blog. I'll quote the part that caught my attention:

As you may have seen, we've announced our fourth quarter and full fiscal year results ... We grew revenue, expanded gross margins, streamlined our operating expenses - and closed the year with an 8% operating profit in Q4, more than double what some thought to be an aggressive target a year ago.

We did this while driving significant product transitions, going after new markets and product areas, and best of all, while aggressively moving the whole company to open source software (leading me to hope we can officially put to rest the question, "how will you make money?").

It is extremely frustrating to me that public companies must deal with putting their fate in the hands of a group of analysts who have such limited understanding of the ecosystems of information technology. Wall street has been so timid about Sun since the bubble burst, largely because their fear of the past clouds their ability to see the future (or the present, for that matter).

Today Sun has the best product portfolio I've ever seen. They also have the financial metrics to prove their strategy is good. I have invested more than ten years of my life in their products, and I can say with no hesitation that I plan to continue that investment for the next ten years as well. The only question I have is when the rest of the industry will catch up.

Picking a terminal server

My plans to equip the lab with older, but solid equipment has been going very well thus far. It's not cheap, but it's going to be very functional. The two Netra X1 servers are doing a great job, and I'm really enjoying having a LOM. I wish my "big iron" 420R had a LOM, but a Sun serial port still beats an x86 BIOS program. And what could be cooler than accessing those serial LOM devices through a terminal server? (Yes, I suppose a modern Sun server with an Ethernet LOM would be cooler, but don't burst my bubble, ok?).

So now that I've accumulated these boxes and am beginning to use them on a regular basis, you can imagine that patching wasn't far behind. Patching is one of many activities where a console connection comes in pretty handy. To make a long story short, I quickly grew tired of trucking my laptop downstairs, attaching a serial cable to it, and then performing an elaborate contortion routine to find the LOM port in the back of my rack while pressing my face through cobwebs. Been there before? Yes. I have decided that I need a terminal server.

So, what is my ideal terminal server? Well, there's a few requirements. It must be a quiet, low power device - no giant noisy fans need apply. I need an 8-port device, but 16 would give me room to grow if the price is right. I don't care too much about security protocols - this is a home lab that sits behind a firewall, and all my systems can be reprovisioned from a flash archive in a heartbeat. Should be easy right?

The first thing I learned is there are a LOT of 32-48 port high end (not old!) term servers available, primarily Cyclades devices. These look like Ferraris to me, and I dream of winning an auction for about $50 and attaching that puppy to my rack. Not going to happen... The next thing I noticed is a bunch of really old Xyplex and Perle devices. These rack up, but I read a bunch of horror stories, and got the idea from a few USENET postings that they are loud. I found a few other older devices, but they all had something that didn't seem right to me. It was time to get drastic...

I went with plan "C". In this case, the C stands for Cisco. Turns out that with some auction patience, a properly equipped Cisco 2509 (8 port) or 2511 (16 port) can be had with cables for around $150 or less. That's right at my pain threshold, but acceptable given what it provides. This solution appears to be hit or miss with the issue of spontaneous break signals halting the Sparc machines, which usually happens if the TS powers down, but the kbd command can be used to configure an alternate break sequence and avoid the issue.

The other appealing feature seems to be that I can configure reverse-telnet. This would allow me to run a command like "telnet termserver 2001" to get to port 1. Much more convenient than authenticating to a termserver and navigating annoying menus. And finally, being a full size 19" box I can rack it up without coming up with some combination of plywood and duct-tape. Suh-weet.

The downside? Well, ssh would be more cool than Telnet, but I can swallow my pride. Who knows? Maybe there's a Cisco update that would provide this. It might be a loud device. I have no diea. Another issue which decrements the coeficient of cool: It requires an AUI adapter to convert to an Ethernet RJ45 port. On the other hand, there's probably a lot of new SAs in the world who would look at that like a vintage muscle car... "Whoa - is that a REAL aui adapter, dude? You're must be hard core." Um, yeah. Maybe not. Although the loudness and power consumption concern me, I think I can live with these issues if it works, which I'm reasonably confident it will.

Now, to set up an eBay search and begin the hunt...