Tuesday, September 29, 2009

xVM OpsCenter and overbundling

I've been spending a fair amount of time assessing the patching strategy on my current assignment. My primary focus is on Solaris systems, although there is a Linux population to take care of as well. My recommendation has always been to stick with vendor recommended solutions when it comes to patching because in the Enterprise it's a lot more complicated than clicking on Windows update and hoping for the best.

With that in mind, I browsed over to Sun.com to see what the latest recommendation is. xVM OpsCenter pops out in neon lights. It will even wash my dishes. For what it's capable of, I think it's possible to make an argument that the price is tolerable. Unfortunately, if you are a practitioner of Solaris and need a patching solution you may not need your dishes washed. Then what?

If you aren't going to need full blown provisioning, monitoring, audit, and other cool features you are left with precious little in the way of keeping up on what I call "oil changes". Most of the historical tools are now on their death beds, no doubt to encourage the herd to graze on xVM. Note that I'm only talking about Enterprise level patching which requires some degree of configuration management.

When you dig into xVM you see that there are two options. The basic option does very little that most sites don't already do, although it's wrapped in a nice package. I don't think it's doing anything worth the price of admission at that level though. The advanced package adds what everyone wants: patching. So, you can buy your car with or without tires.

I think this is a bad idea.

Patching is a vital component of the customer experience. It's a way to ensure that Sun doesn't have a CNN moment because a major bug was too difficult to patch and highly visible site didn't get the hole plugged in time. It's also the bane of most admins' existence. It takes a lot of time, causes our customers to suffer downtime, and occasionally takes a server to the happy hunting grounds. To be the best operating system, you need to have a great update strategy.

I have no problem with the xVM framework being an expensive Cadillac, as long as I can still buy a Chevy that does the job. In other words, as long as the Solaris operating environment includes a decent functional framework for patching, then charge all you want for xVM. Today, even with a support contract I don't have access to a proper patching framework from Sun, which means all those third party solutions start getting traction on something that ought to come from Sun.

A basic level of functionality should be part of the environment, so what would the base requirements be? Call it xVM-lite, or call it part of Solaris. Either way, here's a stab at it:

- An on-site proxy option so all hosts don't talk directly to SunSolve. Why not include it in Solaris? This would save Sun bandwidth costs and probably help them to sell some storage.

- Integration with Explorer. Wouldn't it be nice to use that same patching server as the site's Explorer repository for pre-planning patching sessions? We're talking trivial shell scripting here.

- Ability to leverage SunSolve baselines for SunAlert, Security, and Recommended bundles, as well as to manage site-specific custom patch lists.

- Basic auditing of who patched what, and when.

- No GUI necessary. Just a well thought out command line.

What's the precedent? Look at JET. Sun will offer you xVM if you want an Easy Button solution in a GUI, or you can use the JET framework. Personally, I prefer JET. It has nothing to do with the price... I just believe its a well thought out, very reliable design. What I appreciate most, is that when it comes to provisioning I have a choice, and as part of Solaris there is an included option that gets the job done.

Including patching functionality for customers with valid SunSolve entitlements would be a huge improvement in Solaris' usability. Forcing us to buy a 12 course meal when we only need lunch feels like something that happens when you let a marketing department without industry experience make key decisions.

Monday, September 14, 2009

Default routes on 7x00 series Open Storage

I've been having a very enlightening time with our new 7310 Open Storage array. As a totally new product, and one that hasn't yet reached ubiquity, the normal resources are a bit shy of what I'm used to. Put simply, Google hasn't yet learned how to manage these arrays.

We're in the process of deploying a reasonably complex network scenario on ours using two link aggregations, then layering tagged VLANs for administrative access and the dedicated storage net. Each VLAN is to be redundant via IP Multi-pathing (IPMP). This configuration is just about the only option for high capacity and redundancy when you have multiple VLANs involved.

The good news is, Sun's Open Storage, or Fishworks, has a very well designed command line interface. It's quite comprehensive, and from what I can see, it allows you to lose the GUI and still have a workable device. Which is good, because I managed to decapitate the GUI, or BUI (browser user interface, as Sun calls it).

The kiss of death for the BUI came when I attempted to replace a simple datalink on nge0 with an aggregation of nge0 and nge1. In doing so the default route was removed and not replaced. No problem on the dedicated storage VLAN because it was a non-routed private subnet. Big problem on the public side where I was trying to find the BUI.

It turns out to be a simple problem to fix, but the fix itself is not very intuitive. Because the BUI is dead, you have no choice but to use the CLI. For this reason alone, I strongly encourage anyone using 7x00 series storage to make sure that EVERYTHING you implement in the BUI has an equivalent process via CLI. You never know when you'll need it.

After logging in to the BUI, head over to services --> routing. What you'll probably see is a bunch of routes for each interface, but no default route. To add the default route and reanimate the BUI you will need to create a route as follows:

7310array:configuration services routing > create
7310array:configuration services routing > set family=IPv4
7310array:configuration services routing > set destination=
7310array:configuration services routing > set mask=0
7310array:configuration services routing > set gateway=
7310array:configuration services routing > set interface=ipmp1
7310array:configuration services routing > commit

Note, of course, that you'll need to plug in the appropriate gateway and device according to your configuration.

If you are used to adding a default route in Solaris, it isn't all that intuitive to type in, and it sure as heck wasn't documented anywhere I could find. All's well that ends well though; The change immediately brought back my BUI.

7310array:configuration services routing> show
= online


route-000 ipmp1 static
route-001 ipmp2 dynamic
route-002 ipmp2 dynamic
route-003 ipmp1 dynamic
route-004 ipmp1 dynamic

7310array:configuration services routing>

All is at balance in the universe. My work here is done.