Monday, June 29, 2009

A farewell to Solaris 9... Already?

My flock still has a large Solaris 9 community within it. It's hard to believe its already time to start the long march to EOSL, but alas, the announcement is clear, as is the Solaris 9 Transition FAQ. The bell is ringing.

Looking back at other Solaris EOLs I seem to always recall thinking that revision had really grown long in the tooth, and the replacement OS was badly needed. In this case, Solaris 10 has a long list of what I consider "dreams come true" to make you want to upgrade. However, I have a lot of experience watching Solaris 9 boxes take some incredible abuse and keep ticking. In my mind, it may have fewer bells and whistles, but it really did its job well.

So let's raise a glass of Solaris and toast to the legacy of 5.9, and to the enterprise evolution that is 5.10 and OpenSolaris. Cheers!

Sunday, June 21, 2009

OpenSolaris on the ThinkPad

After a long run of just dealing with Windows on my personal latpop I have finally managed to get OpenSolaris running on it. I've had a continuous hassle with my old Wifi card that seemed to only be truly happy under Windows. After a few years of that I took a chance on a new card from eBay and found that it... WORKED!

I started out trying the latest Ubuntu desktop, which has a great library of packages available for it and fantastic integration. Unfortunatley, its driver configuration seemed to work, then send my wifi into a coma after some period of time. Didn't diagnose it. Didn't care to. My laptop isn't a science project for me, it's a tool I want to just work when I dump a new OS onto its disk.

Next stop was the one I was more excited about: OpenSolaris.

root@saphyra:~# uname -a
SunOS saphyra 5.11 snv_111b i86pc i386 i86pc Solaris
root@saphyra:~# wificonfig showstatus
linkstatus: connected
active profile: none
essid: <>
bssid: <>
encryption: wep
signal strength: medium(10)
root@saphyra:~#

Yes, that's right, it's all working. At the moment I'm able to work on a zone / LDAP project from the comfort of my couch enjoying my reborn Thinkpad T23. This thing works like a charm despite being a dinosaur by modern standards.

Before rebuilding it I had pretty much stopped using it because Windows XP was unable to boot in under 5 minutes and it took almost as long to launch an Acrobat Reader session for simple PDF stories I was reading. At the moment everything I do, including web browsing works well and is responsive under only 1GB ram and a 1.1 GHz processor. Sweet.

Wednesday, June 03, 2009

Solaris Web Console on Windows... Ouch.

I've been spending quite a bit of time lately running the Sun Directory Service Control Center (DSCC) via the Solaris Web Console (port 6789). When I first started the project I was running Firefox on a Sun workstation. Everything was snappy, the engineer was happy.

Somehow along the way I started using my Windows box to access the console. Still running Firefox I discovered an unbelievable slowness. It takes about three clock minutes to process the initial log in. Once I'm in DSCC everything runs acceptably, but that first login is murder.

One of my co-workers stopped my cube today and suggested I try Internet Explorer. Perish the thought! How could that bloated pig possibly out-perform my Firefox browser? OK, I tried it. He was right.

Internet Explorer provides almost instantaneous response to Webconsole logins while Firefox churns its butter for three minutes. This isn't some dot-net application that's clearly Microsoft slanted. It's a Sun web application. Open stuff that would never have a Microsoft bias. I'm not running dead hardware either; This is on a sweet core-duo 1.83 GHz with 1 GB RAM. Handling an initial log in to Webconsole ought to be cake for this hardware.

My observations are based on stock out of the box configurations, so I'm sure there's some Firefox flag to tweak which will optimize it. It just seems mind boggling that a Sun Microsystems web application would perform exponentially better on Internet Explorer and unacceptably slow on Firefox.

Me? I'm going back to running the browser on my UNIX box. It's way too frustrating trying to be a UNIX Engineer via the Windows platform.

Thursday, May 21, 2009

Adding UNIX users to DS6


I seem to be digging up rants this week. I'm a pretty positive guy, you just wouldn't know it by reading my blog this week. I'm currently working on deploying a fresh Sun Directory Server environment using version 6.3.1. This is to replace an aging 5.2 environment that's ready to retire. Overall I've been very impressed with how much more mature and polished the new version is. A few learning curves to get through, but once I found the right way I was pleased with the product. Unfortunately, today I hit something that just can't be right. Unfortunately, it seems to be confirmed by a bunch of Google hits so I'm not the only one.

When you use Directory Services Control Center (DSCC) to add a user it doesn't provide any of the POSIX fields you need from the posixAccount class. So, your new users pretty much have a user name and a first / last name. No home directory, no user ID, no group ID, and hey... You didn't need a shell did you? Are you kidding me?

The workaround, and I use the term loosely, appears to be adding the record without the necessary information, then editing the record after it is created. You then switch the record to "text mode" and manually insert the following lines into the editable section:

objectclass: posixAccount
loginshell: /bin/ksh
homeDirectory: /home/username
uidNumber: 1234
gidNumber: 10
gecos: John Smith


Ok, so that gets us an account, but isn't it moderately annoying to have to go through all that? Why in the name of Scott McNealy didn't anyone make the wild and unruly assumption that once in a freakishly rare moon someone might use DSEE to centralize the administration of their Solaris users. After all, NIS and NIS+ are deprecated and no one digs local file editing. So, wouldn't that assumption have been somewhere around the top ten for their user requirements?

I did a quick dig to see if I could find a simple configuration file that specified what schema object(s) were used when adding a user attribute, or populating the "common objects" menu, but came up dry. I'll have to a deeper search when time allows. I know it's sitting in some XML file somewhere, but there's more than a few to look through.

So what are my options? Well, there's always the LDIF plan. Which is pretty much useless to the folks who typically manage user account maintenance. Way too error-prone. It's also pretty aggravating for day to day administration. LDIF is pretty much intended for batch loading and sitting behind various automations. I shouldn't need to write an automation solution to add simple UNIX accounts since that capability was standard in the 5.x Directory Servers.

Another option is to use Sun's Directory Editor which is part of DSEE. This path leads to some entertainment as well. If you try to download DSE, the web form will not let you select a platform, and thus prevents you from downloading the component. So, you need to download the ZIP distribution of DSEE instead. Then you just need to deploy Sun's Application Server, or Tomcat. Yeah, just what I needed - another component. Doesn't webconsole already sit on an app server? The best part is, DSE is left over from the 2005Q1 JES distribution from what I can see. Obviously, not a high priority for maintenance. Very encouraging indeed.

So, while Sun's Directory Server continues to be a phenomenal data repository it appears that Sun views its user base as being application / identity developers rather then the legions of system administrators / engineers out there trying to implement a well supported central management strategy. Come on guys and gals, it's not that hard to make us happy. Lose the web 2.0 bling and give us core functionality. Hmm, then add the bling back in! The DSCC interface really is very nice, but what good is a hot car without a steering wheel?

Tuesday, May 19, 2009

ldaplist: Why so much white space?

Sometimes little things drive me nuts. So nuts, it's almost tempting to get into some code and make it right. Of course, that would have absolutely no return on investment for a singificant amount of hassle, but I have to admit I think about it from time time. What has rubbed me the wrong way?

The complete lack of either [1] aesthetic engineering, or [2] use of traditional 80x24 console screens as experienced by the developers of the ldaplist utility. It's as if someone had just finished a grade school term paper when they wrote the output format. Here's the default output:

testbox# ldaplist
dn: cn=Directory Administrators, dc=example,dc=com

dn: cn=nsAccountInactivationTmp,dc=example,dc=com

dn: ou=Timezone,dc=example,dc=com

dn: automountMapName=auto_home,dc=example,dc=com

dn: automountMapName=auto_direct,dc=example,dc=com

dn: automountMapName=auto_master,dc=example,dc=com

dn: ou=projects,dc=example,dc=com

dn: ou=group-ldap,dc=example,dc=com

dn: automountMapName=auto_shared,dc=example,dc=com

dn: ou=SolarisAuthAttr,dc=example,dc=com

dn: ou=SolarisProfAttr,dc=example,dc=com

dn: ou=people,dc=example,dc=com

dn: ou=group,dc=example,dc=com

dn: ou=rpc,dc=example,dc=com

dn: ou=protocols,dc=example,dc=com

dn: ou=networks,dc=example,dc=com

dn: ou=netgroup,dc=example,dc=com

dn: ou=printers,dc=example,dc=com

dn: ou=hosts,dc=example,dc=com

dn: ou=services,dc=example,dc=com

dn: ou=ethers,dc=example,dc=com

dn: ou=profile,dc=example,dc=com

dn: ou=aliases,dc=example,dc=com


Forty-seven lines? That takes up WAY too many lines and provides no value for the white space incurred, not to mention requiring me to scroll my terminal window when I'm on the console. This actually annoys me enough that I run the command this way:

testbox# ldaplist | sed '/^$/d'

dn: cn=Directory Administrators, dc=example,dc=com
dn: cn=nsAccountInactivationTmp,dc=example,dc=com
dn: ou=Timezone,dc=example,dc=com
dn: automountMapName=auto_home,dc=example,dc=com
dn: automountMapName=auto_direct,dc=example,dc=com
dn: automountMapName=auto_master,dc=example,dc=com
dn: ou=projects,dc=example,dc=com
dn: ou=group-ldap,dc=example,dc=com
dn: automountMapName=auto_shared,dc=example,dc=com
dn: ou=SolarisAuthAttr,dc=example,dc=com
dn: ou=SolarisProfAttr,dc=example,dc=com
dn: ou=people,dc=example,dc=com
dn: ou=group,dc=example,dc=com
dn: ou=rpc,dc=example,dc=com
dn: ou=protocols,dc=example,dc=com
dn: ou=networks,dc=example,dc=com
dn: ou=netgroup,dc=example,dc=com
dn: ou=printers,dc=example,dc=com
dn: ou=hosts,dc=example,dc=com
dn: ou=services,dc=example,dc=com
dn: ou=ethers,dc=example,dc=com
dn: ou=profile,dc=example,dc=com
dn: ou=aliases,dc=example,dc=com


Ahhh, that's better. And at 1/2 the screen real estate I rarely need to scroll. Come on, what on Earth would motivate someone to add extra newlines to an output like this? Next thing you know they'll offer CSS templates so your output can have the right "user experience" complete with standard fonts.

Ok, I feel better now... Really. I'm ok.

Tuesday, May 05, 2009

#@$@#$# Spammers









Gotta love those spammers. I wish I could talk the way they write - it would be entertaining at a party to sound like one of the Cylon Hybrids. I'm going to go out on a limb and assume there at least a few Sci-fi fans out there reading a blog like this one.

The real purpose of this post is to apologetically announce that I've turned on comment moderation to keep everything clean after a long wave of spammers hit me. I'm not big on censoring, so rest assured that if you post a rational comment I will be happy to release it and continue encouraging dialog.

JET and the Recommended Cluster

JET is bugging me. I'm a sort of pack rat when it comes to installation media, and that extends to patch sets. Hey, you never know you might get a request to Jumpstart Solaris 2.4, right? Ok, I'm not really that bad.

But, you may well be using a certain recommended cluster for a certain OS, and then need to jump a box to test the next recommended cluster right? Surely, it's not necessary to make a global change to your production server build configuration when implementing a test cluster?

As far as I can tell, base_config's use of recommended clusters is not handled in a manner that encourages good revision management. For each OS major revision (e.g., 10, 9, 8) there can be one cluster. For example, in today's JET software if we were using /export/install as our JET media base there would be a directory called /export/install/patches. Under that we can store one patch cluster for each major OS revision:

/export/install/patches/10_Recommended
/export/install/patches/9_Recommended
/export/install/patches/8_Recommended


That works nicely until the next recommended cluster is released. At that point in time you can no longer have a repeatable build process because you need to replace the single instance of each OS with the new cluster. Not a good plan. We want our patch configuration to be configured in the template so that it's managed, and can be under source code control. Managing patch configurations outside the template is pretty much impossible to audit.

Here's an alternative approach I think would be a step in the right direction: Create a hierarchy to organize recommended clusters:

/export/install/patches/recommended/5.10/sparc/2009-04-22
/export/install/patches/recommended/5.10/sparc/2009-01-foo
/export/install/patches/recommended/5.10/x86
/export/install/patches/recommended/5.9/sparc/2009-04-22
/export/install/patches/recommended/5.9/x86/2009-01-foo


We need to be able to add recommended clusters in the same way we add other products. I'd like to see a new command called "list_recommended_clusters" which would have an output something like this:

# list_recommended_clusters
Version Location
------ ---------------
5.10_sparc_200901 /export/install/patches/recommended/sparc/5.10_sparc_200901
5.10_sparc_200902 /export/install/patches/recommended/sparc/5.10_sparc_200901
5.10_sparc_200903 /export/install/patches/recommended/sparc/5.10_sparc_200901
5.9_sparc_200901 /export/install/patches/recommended/sparc/5.10_sparc_200901
5.9_sparc_200902 /export/install/patches/recommended/sparc/5.10_sparc_200901
5.9_sparc_200903 /export/install/patches/recommended/sparc/5.10_sparc_200901


These clusters could then be specified in the JET template using a variable like base_config_recommended_cluster. In addition, the check routine used during a make_client invocation would ensure that the directory exists, and perhaps ensure that each patch on the patch_list was represented. Bingo! Now we can use good revision control to manage the integration of patch clusters with our server build process.

But I think we can take it one step farther. How about we add the ability to include arbitrary patch sets? Here's a first cut at how it could work: We start by creating a patch repository. Say, /export/install/patch_repo. Under that directory we may have subdirectories for 5.10, 5.9, etc. Patches are simply added to that directory by copying them into place. Nothing fancy. The nice thing about this approach is in its economy of space.

The recommended clusters will have a lot of overlap between them, with the potential for storing the same patch in many different directories. By having one patch repository, we simply store each necessary patch one time and refer to it in a patch_order file. It would be trivial to write a few scripts that could operate or query on a set of patches according to a certain patch list, or perhaps cull out patches not referenced in any current patch lists. I could give or take this feature. There are some good arguments to be made for just storing each patch set and ignoring the storage space. I'm ok with either approach, and even happier if this flexibility were accounted for.

Having established a patch repository, we now need a place to manage patch lists. These would be in typical patch_order formatted lists; No need to reinvent the wheel. Each would need to be named with a unique identifier. For example,

patch_order_5.10_2009q1
patch_order_5.9_dev-servers
patch_order_5.10_test01


These patch lists could then be specified within base_config as an alternative to using the Recommended clusters. Why?

  • The site has a known incompatibility with a patch or two in the common cluster.

  • The site wants to deploy other patches in the early part of the install as part of a managed list rather than manual entries in a template (e.g., custom_patches).

  • Using these lists allows a configuration to be frozen in time for configuration management, and allows a convenient record of exactly what a server was deployed with



I think these would be some very beneficial enhancements to the JET framework. I'd like to work on some of them, but I wanted to get the idea out there before I got wrapped up in something else and forgot about it. I'd be interested in hearing any thoughts on this topic - especially if someone has a better idea!

By the way, I do know about EIS baselines. But I think it's pretty rare for any enterprise to never have need for managing custom patch sets. It would be great if JET could come through with some help in this space.

Monday, March 30, 2009

Finding those pesky HBA cards

I was given a mission yesterday of finding how many host bus adapter (HBA) cards were in a set of servers. At first glance it seemed like an easy task, but then I remembered that Solaris servers never had a nice convenient output to tell us what card is in what slot in a way that normal humans could benefit from. It's sort of like playing charades; You have to put together a bunch of clues. Here's how I went about it.

The first place I stopped was prtdiag. That's my go-to configuration summary in most cases. Here's a subset of what I saw (probably going to look bad unless your browser is really stretched...):

Bus Max
IO Port Bus Freq Bus Dev,
FRU Name Type ID Side Slot MHz Freq Func State Name Model
---------- ---- ---- ---- ---- ---- ---- ---- ----- -------------------------------- ----------------------
/N0/IB6/P1 PCI 25 B 4 100 100 1,0 ok SUNW,qlc-pci1077,141.1077.141.2/+ QLA2462
/N0/IB6/P1 PCI 25 B 4 100 100 1,1 ok SUNW,qlc-pci1077,141.1077.141.2/+ QLA2462
/N0/IB6/P1 PCI 25 A 6 100 100 2,0 ok SUNW,qlc-pci1077,141.1077.141.2/+ QLA2462
/N0/IB6/P1 PCI 25 A 6 100 100 2,1 ok SUNW,qlc-pci1077,141.1077.141.2/+ QLA2462
/N0/IB7/P1 PCI 27 B 4 100 100 1,0 ok SUNW,qlc-pci1077,141.1077.141.2/+ QLA2462
/N0/IB7/P1 PCI 27 B 4 100 100 1,1 ok SUNW,qlc-pci1077,141.1077.141.2/+ QLA2462
/N0/IB7/P1 PCI 27 A 6 100 100 2,0 ok SUNW,qlc-pci1077,141.1077.141.2/+ QLA2462
/N0/IB7/P1 PCI 27 A 6 100 100 2,1 ok SUNW,qlc-pci1077,141.1077.141.2/+ QLA2462
/N0/IB8/P0 PCI 28 A 3 100 100 1,0 ok SUNW,qlc-pci1077,141.1077.141.2/+ QLA2462
/N0/IB8/P0 PCI 28 A 3 100 100 1,1 ok SUNW,qlc-pci1077,141.1077.141.2/+ QLA2462
/N0/IB8/P1 PCI 29 B 4 100 100 1,0 ok SUNW,qlc-pci1077,141.1077.141.2/+ QLA24
/N0/IB8/P1 PCI 29 B 4 100 100 1,1 ok SUNW,qlc-pci1077,141.1077.141.2/+ QLA24
/N0/IB8/P1 PCI 29 A 6 100 100 2,0 ok SUNW,qlc-pci1077,141.1077.141.2/+ QLA2462
/N0/IB8/P1 PCI 29 A 6 100 100 2,1 ok SUNW,qlc-pci1077,141.1077.141.2/+ QLA2462
/N0/IB9/P1 PCI 31 B 4 100 100 1,0 ok SUNW,qlc-pci1077,141.1077.141.2/+ QLA2462
/N0/IB9/P1 PCI 31 B 4 100 100 1,1 ok SUNW,qlc-pci1077,141.1077.141.2/+ QLA2462
/N0/IB9/P1 PCI 31 A 6 100 100 2,0 ok SUNW,qlc-pci1077,141.1077.141.2/+ QLA2462
/N0/IB9/P1 PCI 31 A 6 100 100 2,1 ok SUNW,qlc-pci1077,141.1077.141.2/+ QLA2462


Of course, there were a great many other lines, but this is what the Fibre Channel card lines look like. Of course, I picked this out because I recognized the QLC driver. Not sure what someone would do if they didn't know that. In this case, there were 18 lines with this output. This indicates there are 9 cards because each slot was represented twice (two ports on each device). This was supported by me being reasonably sure that we had dual-ported cards on this server.

The next place I looked for confirmation was prtconf. This output tends to be more complete, but far more verbose, and generally annoying to get summaries from. To be more precise, the output contains a lot of information...

foobox: prtconf -v | wc -l
9358


That was a complete moment of frustration. The output was too busy and didn't look helpful. Note to self: Why is this not simple? I'm looking for a simple answer, not an excuse to write a Nawk script. No matter how I skinned the output I ended up with 18 matching lines. I'm right back at the prtdiag output.

My last stop was a more obscure one, but a tool which is very helpful: prtpicl. Ok, I'll admit, this one is still ugly.

foobox: prtpicl -v | wc -l
11183


But, at this point I just wanted to get it done, so I dug in a little bit and checked out what it had to say. The easily parsed format provides a convenient Vendor ID and Device ID for each connected device. That's good news because those PCI IDs are easy to look up on the Internet. Knowing our site standards I was able to identify the Vendor ID of the cards we order and look for them:

foobox: prtpicl -v | egrep -e '0x1077' | grep -v subsystem
:vendor-id 0x1077
:vendor-id 0x1077
:vendor-id 0x1077
:vendor-id 0x1077
:vendor-id 0x1077
:vendor-id 0x1077
:vendor-id 0x1077
:vendor-id 0x1077
:vendor-id 0x1077
:vendor-id 0x1077
:vendor-id 0x1077
:vendor-id 0x1077
:vendor-id 0x1077
:vendor-id 0x1077
:vendor-id 0x1077
:vendor-id 0x1077
:vendor-id 0x1077
:vendor-id 0x1077
:vendor-id 0x1077
:vendor-id 0x1077

Please, no comments about how this could be done in a Perl one-liner. We're going to ignore the indented items because they belong to a different hierarchy of data. If we count up the leftmost indented items we see again there are 18 instances of PCI devices with the relevant vendor ID. So, is this a port 0, port 1, deal which requires me to divide by two?

Again, I'm not sure because the output is cryptic. Yes, I know there are ways to make sense of it with hardware knowledge, but let's assume we're dealing with an average SA, and not a device driver developer.

The last tool I tried is a device path decoder which is sort of an unsupported toy developed inside Sun. I don't know where we obtained it, but we happened to have it here so I ran the path_to_inst file through it. What did it tell me? That I had nine of the HBA cards in the box. It had a very simple, easy to read format which used indentation to clearly show the system's layout.

So, it looks like prtdiag was the most direct way to surmise an answer. I would like to see Solaris give me a hardware diagnostic which provides a physical model rather than a logical one. Just tell me there is a card in slot 4 with its vendor / device ID. I don't care to sort out its ports. I just want the device. There are plenty of other tools which provide the logical view, or device driver hierarchy.

Monday, February 09, 2009

Solaris LDAP Integration Void

Yikes, that was a harsh post title from a self-proclaimed advocate of Sun's products. I can't count the number of times I've had conversations with people about two related topics: First, how critical it is that sites begin to adopt LDAP and stop managing boxes independently. Second, how immature the administrative side of Sun's LDAP is.
It appears that Ben Rockwood, a much respected voice in the OpenSolaris community, has observed the same.

These topics each deserve a series of posts because they are complex. I mean it. Until you've tried, its hard to understand the documentation dichotomy of Sun's Directory Server Enterprise Edition. The best way I can describe it would be to imagine you have been asked to learn English given a dictionary as your only resource.

There is phenomenal depth to the documentation in form of resource guides. In other words, once you "get it" you can do anything with Sun's documentation. The number of concepts you need to master to deploy LDAP in an Enterprise is staggering, and the number of real-world cases available from Google is small. You really need a few weeks of Instructor Lead Training, but how many companies are on that track these days? Not too many. There are a few outdated books as well, but they only get you to the starting gate for a basic environment.

So now let's assume that you have learned the system and properly architected your Directory Servers. Your next challenge is managing the data. I worked on a project which integrated Oracle instances with Solaris Resource Manager (SRM). The central LDAP project ID repository allowed us to ensure no Project IDs were duplicated around the environment, and minimized the amount of management associated with application migrations. Seems simple, right?

The first issue we encountered was that there is no facility for entering records into the Directory. Don't even talk to me about the documented solution of using Sun Management Console (SMC). It's cute for local files, but it is worthless for naming services, and even Sun's solution center thinks its insane to try using it. No, really. I opened a case, and they asked my why I would ever try to use it.

There should be a set of CLI interfaces for managing this data. Period. Its a simple thing, and by now the Directory Services have been around long enough that this is sorely over due. They should follow the standard usage model that tools like useradd or usermod provide. People understand this, and the precedent should be respected.

The only other option is the Directory Editor. You pick a third party one, or a Sun one. But in the end you are responsible for reverse-engineering whether a directory attribute is a list, or a collection of attributes. This is not appropriate. For standard Solaris maps like netmasks, auto_master, hosts, etc. there should be interface dialogs which provide reasonable levels of sanity checking. I shouldn't need to scan through cryptic attributes. What's even more scary is the idea of handing over a full directory editor to say, someone on the first tier help desk who may not fully understand how terrifying it would be to make the wrong right-click.

This was a bit of a rant, but it is primarily intended to scream out in support of Ben's post. This is a huge opportunity to improve Solaris' administrative scalability and I think all too often LDAP projects get dropped during internal evaluations because the local staff has too many issues getting it working.

Thursday, November 20, 2008

Kerberos and the SCSECA Curriculum

I remember when I first took the Network Administrator (SCNA) Exam back in the Solaris 7 days, and I was frustrated by the depth of NIS / NIS+ content. NIS was widely used back in the day, and fairly intuitive. However, NIS+ was a bit of a niche, and its use dropped off like a rock on the Solaris 7 era. I think people really failed to enjoy all those key exchanges and inherant troubleshooting.

Long after NIS and NIS+ services were deprecated by the coming promise of LDAP their place in the curriculum was maintained. But of course, I learned it and passed the exam. Having recently passed the SCNA again for Solaris 10 I was pleased with its content. I was convinced that Sun had brought the canon into the modern era. Good stuff. But just when I thought it was safe...

I'm now finishing up my prep for the Sun Certified Security Administrator (SCSECA) and am finding myself frustrated by the presence of Kerberos on the SCSECA test curriculum.

Will the number of sites using Kerberos please raise their hands? Ah ha! We now know the answer to the question, "What is the sound of one hand clapping?". Ok, it's more than one, I know. It's not very many though... I'm really hoping that when I sit down to the test the questions are written to a depth proportional to the installed base.

I think there's a lot of great content that can be included on a Solaris security exam in place of esoteric solutions like Kerberos. I'd like to see the bulk of the SCSECA content focus on an SA's ability to implement and evaluate impact of the various checks in the CIS Solaris 10 Benchmark. The key of course is "evaluate" more than "implement." I'm amazed at how many people flip through checklists without understanding the implications of these reconfigurations, and I think the SCSECA content is a great opportunity to fix that problem.

But that's ok. I'll brush up on my Kerberos and maintain my historical acumen.

Tuesday, September 23, 2008

Capturing output from format

Ever need to obtain the contents of the format command for other processing in a shell or Perl script? It's fairly simple to do, but the command's behavior is a bit counter-intuitive and makes for an interesting case.

When you run the format command it lists the disks, then issues a prompt asking you to select one of the enumerated devices. It does not provide an option for existing the command at that point. So, we need to appease this interface oddity by passing a "0" to the command, which will arbitraily select the first disk from the list. This should work in any case excepting a diskless client.

The format command looks for its input from a file descriptor known as STDIN, or standard input. The way we queue up entries in STDIN is using the good old echo command. Altogether it looks like this:

root@testbox# /usr/bin/echo 0 | /usr/sbin/format
Searching for disks...

AVAILABLE DISK SELECTIONS:
0. c1t0d0
/pci@7c0/pci@0/pci@1/pci@0,2/LSILogic,sas@2/sd@0,0
1. c1t1d0
/pci@7c0/pci@0/pci@1/pci@0,2/LSILogic,sas@2/sd@1,0

Specify disk (enter its number): selecting c1t0d0
[disk formatted]
/dev/dsk/c1t0d0s0 is part of SVM volume stripe:d10. Please see metaclear(1M).
/dev/dsk/c1t0d0s1 is part of SVM volume stripe:d11. Please see metaclear(1M).
/dev/dsk/c1t0d0s5 is part of SVM volume stripe:d15. Please see metaclear(1M).
/dev/dsk/c1t0d0s7 contains an SVM mdb. Please see metadb(1M).

FORMAT MENU:
disk - select a disk
type - select (define) a disk type
partition - select (define) a partition table
current - describe the current disk
format - format and analyze the disk
repair - repair a defective sector
label - write label to the disk
analyze - surface analysis
defect - defect list management
backup - search for backup labels
verify - read and display labels
save - save new disk/partition definitions
inquiry - show vendor, product and revision
volname - set 8-character volume name
! - execute , then return
quit
format>
root@testbox#



The problem with this is it captured more than we want in the output. We don't need a menu, and we don't need to know about selecting c1t0d0 since that's already enumerated in the first disk list. To edit this stream of test, we'll need a stream editor... Can you guess what it's called? Sed. Let's modify the command to squelch out some of the noise.

root@testbox# /usr/bin/echo 0 | /usr/sbin/format 2>&1 | sed -e '/^Specify disk/,$d'

AVAILABLE DISK SELECTIONS:
0. c1t0d0
/pci@7c0/pci@0/pci@1/pci@0,2/LSILogic,sas@2/sd@0,0
1. c1t1d0
/pci@7c0/pci@0/pci@1/pci@0,2/LSILogic,sas@2/sd@1,0

root@testbox#


That's better!

Google Blog Search - Algorithm Insanity?

This isn't really Solaris related as much as computer science related, but today I experienced a very strange behavior from the great Google. One of my hobbies is archery, and I live in the Rochester, NY area. So, I was searching blogs on Google with the following string: "rochester NY archery". Seems pretty benign, right? Apparently, it's more like looking for Dick's Sporting Goods at www.d_c_s.com.

The number one hit for Rochester, NY archery is: "Club Intoxicated Girls". Actually, almost all of them were blog spam hits. That's incredibly frustrating. It's also a bit surprising because in my experience the SPAM heuristics in Gmail are second to none. Interesting times we live in.

Thursday, September 04, 2008

A quick way to check UDP ports on Solaris

Ever need a quick way to check what UDP connections are active on your Solaris server? I recently had to validate a scanner's report that we had an unnecessary service running on UDP port 177. Unfortunately, Solaris does not yet ship with lsof as a standard tool, so it requires the use of netstat(1M).

root# netstat -an -P udp

UDP: IPv4
Local Address Remote Address State
-------------------- -------------------- ----------
*.123 Idle
127.0.0.1.123 Idle
13.129.6.168.123 Idle
*.111 Idle
*.* Unbound
*.32771 Idle
...
Active UNIX domain sockets
Address Type Vnode Conn Local Addr Remote Addr
6001f6c18f8 dgram 6001fa6eb40 00000000 /var/vx/isis/vea_portal
6001f6c1c88 stream-ord 6001f6a4180 00000000 /var/run/.inetd.uds



Not too painful at all. Turns out that scan must have been an intermittent service, or a false-positive because I didn't turn up any trace of it, but it did give me a chance to reacquaint myself with a useful incantation of netstat.

Thursday, August 28, 2008

Solaris available on Dell Servers

I have to admit I was surprised, albeit pleasantly, when I saw a post on c0t0d0.org indicating that Solaris 10 is now an order option on certain Dell servers.

Of course Solaris has been available on best in class Sun x64 hardware for some time now, but the mainstream world doesn't follow Sun's products in the same way they do Dell. In a sense, I think this going to be a better channel for advertising than revenue, although I really hope its beneficial for both.

There will now be a lot of Dell customers who see Solaris on their order options, and I believe this will make a larger group of consumers think about Solaris where previously they had no occasion to.

Regardless of the outcome, it feels good to see Sun opening up a new channel and I sincerely wish both Dell and Sun success with it.

Tuesday, August 05, 2008

Repairing file permissions: pkgchk -f

I was recently testing a process for repartitioning root disks which requires booting on an alternate disk, then copying and restoring data to the primary disk. I used ufsdump for this because of its excellent handling of some of UFS' nuances. The downside is that if you don't use ufsrestore frequently, you will be asked a nonintuitive question at the end of the operation. Yes, yes, a quick trip to the man pages would have helped. Unfortunately, I was being a bit cavalier at the time, and since it was a lab machine I thought little of it.

Turns out I should have thought a little harder. I ended up restoring data wonderfully, but pretty much toasted the system because all files were owned by root, with group other. Good in some places, not so good in others. Prognosis: rejump the server? Naah.

Sun published a Blueprint way back in 1999 which I think all system administrators should read. Someday this information will save your butt. Repairing File Ownership and Mode by Richard Elling.

I had forgotten about the "-f" option to pkgchk, which is described in this document. This option will attempt to correct any file system attributes such that they align with the registry's entries. This won't help things outside the OS, but it will restore sanity to an OS full of toasted attributes. The recommendation is to boot CD-ROM or network, then mount the root file system on /a, and run a pkgchk -R /a -f. I found that simply booting single-user and running pkgchk -f did the trick. Your mileage may vary.

I don't think there would have been any other practical approach short of re-jumping the box to restore all of the lost attributes, so it is with great enthusiasm that I recommend keeping "pkgchk -f" in your tool bag.

Wednesday, July 30, 2008

nslookup: Rumors of my death have been greatly exaggerated

I'm currently working through the Sun Learning Connection (on-line / web-based training) to review the curriculum for my Sun Certified Network Administrator (SCNA) update examination. I'm a big fan of Sun's web based training as a study tool because it has always done a great job of preparing me for my certifications.

One of the interesting pieces of content I passed through indicated that in Solaris 10 the nslookup command has been deprecated. Dig is now included in Solaris, and according to the WS-3002-S10 course, is the preferred tool for querying DNS information. I remember when this same fascination with dig sped through the Linux distributions I used to use as well. I would type "nslookup _____" and the OS would dutifully reply that I really ought to be using dig, but here's my reply. You know what? I don't need my OS to tell me what I want. I just need it to do what I ask.

Fortunately, despite the menacing overtone of this training curriculum's message, I have yet to find a warning message come out of my Solaris servers. Dig is indeed included in Solaris, which is a great thing. It is certainly a more detailed tool for diagnosing DNS queries, and I'm thrilled to see Solaris inclusion of industry standard DNS tools.

But let's return once again to that hint about deprecating nslookup... Let's say I just want to see what the name service is returning for a given lookup. I'm just looking for right or wrong, not a detailed and cryptic report to gaze through. Here's the dig command and output for a reverse-lookup:

testbox# dig @192.168.1.2 foo.edu -x 192.168.2.1

; <<>> DiG 9.2.4 <<>> @192.168.1.2 two.edu -x 192.168.2.1
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1174
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0


;; QUESTION SECTION:
;two.foo. IN A

;; AUTHORITY SECTION:
foo.edu. 10800 IN SOA sys22.foo.edu.
root.sys22.foo.edu. 2005010101 3600 1800 6048000 86400


;; Query time: 11 msec
;; SERVER: 192.168.1.2#53(192.168.1.2)
;; WHEN: Wed Jan 12 08:07:30 2005
;; MSG SIZE rcvd: 72


;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR,
id: 1982
;; flags: qr rd ra; QUERY: 1, ANSWER: 1,
AUTHORITY: 2, ADDITIONAL: 0


;; QUESTION SECTION:
;1.2.168.192.in-addr.arpa. IN PTR


;; ANSWER SECTION:
1.2.168.192.in-addr.arpa. 86400 IN PTR sys21.foo.edu.

;; AUTHORITY SECTION:
2.168.192.in-addr.arpa. 86400 IN NS sys23.foo.edu.
2.168.192.in-addr.arpa. 86400 IN NS sys22.foo.edu.


;; Query time: 6 msec
;; SERVER: 192.168.1.2#53(192.168.1.2)
;; WHEN: Wed Jan 12 08:07:30 2005
;; MSG SIZE rcvd: 109


Whoa. That was a lot to digest. Now, REALLY QUICK... Go find out what the hostname is for the queried IP. Yeah, sorry, you took too long tracing through all that. Now lets' look at the nslookup approach:

testbox# nslookup 192.168.2.1
Server: 192.168.2.1
Address: 192.168.2.1#53

1.2.168.192.in-addr.arpa name = sys22.foo.edu.


Yep, that's a bit more efficient.

The moral of the story is that UNIX includes many tools, each of which serves a specific purpose it is (usually) optimized for. I'd hate to think that my future basic DNS queries would be serviced by unwieldy dig output. I'm thrilled that if I run into a more serious DNS issue I can call on dig to help me, but replacing nslookup completely with dig would be like replacing gEdit with OpenOffice Writer. The completely wrong philosophy.

To borrow from Mark Twain, "The rumours of nslookup's death are greatly exaggerated!"

Monday, July 07, 2008

Setting Terminal Title

Now that I'm spending a lot of time working on zones I've found myself needing to keep my desktop better organized so I can quickly find the zone and host I need amongst a slew of terminals. I like to keep things simple, so I went with a little shell script that sets the title of a window on demand. Here's what I ended up with:

if [ -x /bin/zonename ]; then
# if we are on a box that supports zones, include zone info in title
/bin/echo "\033]0;`/bin/hostname` [`/bin/zonename`]\007\c"
else
# handle non-zone platforms by omitting the zone name
/bin/echo "\033]0;`/bin/hostname`\007\c"
fi


This will update the gnome-terminal, or xterm title bar with "hostname [zonename]" on a platform that supports zones (as determined by the presence and executable attribute of /bin/zonename). If a host does not have that executable available and executable (such as pre-Solaris 10) it will simply print the hostname.

True to traditional UNIX' abbreviated nature I named the script stt, short for "set terminal title" and placed in my $HOME/bin directory for convenience. Now when I log in to a host, if I'll be in there for a while I just type 'stt' and my window is properly adorned.

A simple extension of this script would be to include the function in a shell's profile and inject it into the PS1 variable so that it is executed after each command. This would allow the title to update dynamically with each command. Haven't messed with that approach yet as this has scratched my itch quite well.

Tuesday, June 17, 2008

No space left on device? (metainit)

Here comes another rant about error messages. I was rebuilding a server today that uses SVM to manage som SAN storage which gives a home to four very nice Solaris zones. I began by issuing a metainit command to build a concat/stripe device from these two SAN devices...

testbox{lvm}$ sudo metainit -f d100
metainit: testbox: /etc/lvm/md.tab line 72: c4t6006048000018775125753594D433742d0s7: No space left on device


What?!?! I took a quick look at partitioning...

Part Tag Flag Cylinders Size Blocks
0 unassigned wm 0 0 (0/0/0) 0
1 unassigned wm 0 0 (0/0/0) 0
2 backup wu 0 - 56653 25.93GB (56654/0/0) 54387840
3 unassigned wm 1 - 3 1.41MB (3/0/0) 2880
4 unassigned wm 4 - 56653 25.93GB (56650/0/0) 54384000
5 unassigned wm 0 0 (0/0/0) 0
6 unassigned wm 0 0 (0/0/0) 0
7 - wu 0 - 56653 25.93GB (56654/0/0) 54387840


Ok, so the partition exists. What the heck is wrong?

In my absent minded hurry to get this trivial task completed I made an undiscipined assumption that both devices which are to comprise d100 have the same underlying VTOC. It turns out they did not. One of them was set up to use slice 4, and the other slice 7.

So, I issued a quick command to synchronize them using the traditional prtvtoc | fmthard tango, then edited the /etc/lvm/md.tab file to accomodate the s4 slice when defining d100. This time it worked nicely.

But come on, "no space left on device?" What kind of an error message is that? How about something more like, "specified slice does not exist." Technically, a storage device of size zero would have no space available, but there sure are more direct ways to express that concept.

Thursday, June 05, 2008

The Evolution of Email

Have you ever stopped to ask yourself what benefits have been derived by the evolution of email from the days of ASCII text to our modern world where Microsoft Word can act as the email editor?

Fortunately I don't need to ponder this question any longer. Today I received an email which simply would not have had the same impact back in the old days of low-tech correspondence.

The email started out with the following, which is a direct quote:

Starting IMMEDIATELY - ZERO TOLERANCE for any and all non compliance of the following process!
...

It looks pretty menacing in ASCII text, but thanks to Microsoft Exchange and its mind-blowing capabilities to allow more effective self-expression I was able to receive that motivational phrase in a 24-point underlined red font.

I have to admit, it's difficult to fully realize the gravity of the phrase without gratuitous aesthetic enhancement. Let's face it, it would take a PowerPoint attachment to more effectively intimidate me.

Wednesday, June 04, 2008

The Unconventional Explorer

The habit of Sun's explorer dumping output to /opt/SUNWexplo/output makes me wince a bit. In all fairness, I think the documentation could be seen as technically inconclusive, but in spirit I believe a more correct solution is not difficult to derive.

Consulting the Solaris 10 System Administration Guide: Devices and File Systems we find a concise chart of default Solaris file systems and their raison d'etra. Three specific entries jump out at me as being relevant to this topic:


  • /opt: Optional mount point for third-party software. On some systems, the /opt directory might be a UFS file system on a local disk slice.

  • /var: System files and directories that are likely to change or grow over the life of the local system. These include system logs, vi and ex backup files, and uucp files.

  • root(/): The top of the hierarchical file tree. The root (/) directory contains the directories and files that are critical for system operation, such as the kernel, the device drivers, and the programs used to boot the system. The root (/) directory also contains the mount point directories where local and remote file systems can be attached to the file tree.



Considering these practices, it makes perfect sense that explorer is installed in /opt/SUNWexplo. So far, so good. On the systems we deploy at my current place of employment, the /opt file system is part of the root file system, which means that Explorer is dumping output at ~ 5mb per shot onto the root file system.

All things considered, it's pretty benign considering we use either 72 or 146 GB boot drives. But as Solaris Jedi, we look to the harmony and availability of the system, and Explorer is definitely creating a disturbance in the force by dumping volitile files into a subdirectory within /opt. What if someone wrote a script to manage the contents of that output directory and made a little error in their code? What file system would you want it compartmentalized within? Would you want the potential of filling root, or filling a less critical file system? Methinks there must be a better way.

As in most dilemmas, I tend to look for precedents. Where would we find a traditional location in the standard Solaris file system that might be used to spool (hint, hint) volatile files which might grow over time? I would immediately look to /var. There are two immediate paths I see as being preferential to /opt/SUNWexplo/output.

The first option would be /var/spool/explo. This would follow a convention that aligns with out use of a local explorer agent. The servers here produce an explorer on a regular file which is immediately shipped to a central (on-site) repository. The most recent explorer is typically left on the system and the history is managed at the repository. This makes the output directory a traditional spool directory, and as such a perfect fit for /var/spool/explo.

Where this may not be as intuitive is the case of an environment where explorers are retained on the host rather than collected and managed centrally. In that case, the explorers are better described as log files than spools. Intuition brings me to the use of /var/opt/SUNWexplo/output for this case. It's close to the legacy directory structure of the tool, which makes the solution marginally more intuitive than using a spool directory. It also follows the rarely observed SYSV standard of pairing optional software installed in /opt with a directory in /etc/opt, /usr/opt, and /var/opt. I'm not a fan of this specific model when taken to its literal implementation, but it's worth noting.

So, which one is best? As noted earlier, it depends. If I were a member of Sun's Explorer engineering team and needed to pick one consistent location with the intent of minimizing discontent I would select /var/opt/SUNWexplo/output. It is intuitive in the largest set of configurations, and doesn't break any rules. My secondary recommendation would be to create a symbolic link to redirect /opt/SUNWexplo/output for backwards compatibility over the next few years until it could be phased out.

Now I'm left wondering what interesting problems I might create in the data center if I put together a change package that implemented this very model... Nothing is ever as simple or benign as it appears on the surface.