To syslog, or not to syslog? That is the question.

At $WORK we have a simple application which acts as a wrapper around common account management tools. Its purpose is to ensure that all account move/add/change implementations are logged with who did the work and what ticket authorized it, etc. The method for this logging is currently syslog. I'm not sure sure that I would have chosen that design, and that is the topic of this post.

In general I'm made to suffer by observing the slow decay of UNIX practitioners. That's not to say that use of UNIX is in decline, but rather that people who understand UNIX are becoming fewer and farther between. Note that in this case, I lump Linux and UNIX together. So many people are hopelessly tainted by their knowledge of Windows that it creates a prism through which the light of every operating system is split between reality and perception.

UNIX is more than an operating system. It is a development platform. It contains the tools you need to handle many application components including logging, messaging, file parsing, etc. And yet, we find people frequently turning to proprietary (or sometimes open) tool kits to solve problems the OS is well equipped to handle. It frustrates me.

Given that I'm a strong proponent of using the Operating System's features to solve problems, why wouldn't I like the idea of a local script using syslog to handle its logging? The biggest reason is that today's Solaris syslog facility remains tightly constricted in its use of facilities and levels. Yes, I'm well aware that syslog-ng can open those doors. I'm all for it! However, someone at Sun (Oracle) doesn't seem to be all for it yet, and I'm not a big fan of re-plumbing core OS features. So, until Sun sees the light and modernizes the syslog facility I believe in sticking with the standard.

One approach that may be acceptable is combining all Operational logging into a single facility. The trouble with that approach is that we have some logs which fall under tougher security policies than others, so you end up needing destination files with different attributes. So, what's the answer to the engineer's dilemma?

If an application needs to send its output for Enterprise-wide real-time processing for something like a log watcher, then it may be appropriate to use syslog in order to leverage its ability to forward log streams to syslog servers. But if you are writing a simple application which diligently generates log files for audit or troubleshooting purposes, you may be better off in the long run by writing a simple log function that dumps to a configurable destination file. Of course, your log files will be stored under /var/opt/something and will integrate with logadm(1M), right? Of course. After all, you are a Jedi...

Repairing file permissions: pkgchk -f

I was recently testing a process for repartitioning root disks which requires booting on an alternate disk, then copying and restoring data to the primary disk. I used ufsdump for this because of its excellent handling of some of UFS' nuances. The downside is that if you don't use ufsrestore frequently, you will be asked a nonintuitive question at the end of the operation. Yes, yes, a quick trip to the man pages would have helped. Unfortunately, I was being a bit cavalier at the time, and since it was a lab machine I thought little of it.

Turns out I should have thought a little harder. I ended up restoring data wonderfully, but pretty much toasted the system because all files were owned by root, with group other. Good in some places, not so good in others. Prognosis: rejump the server? Naah.

Sun published a Blueprint way back in 1999 which I think all system administrators should read. Someday this information will save your butt. Repairing File Ownership and Mode by Richard Elling.

I had forgotten about the "-f" option to pkgchk, which is described in this document. This option will attempt to correct any file system attributes such that they align with the registry's entries. This won't help things outside the OS, but it will restore sanity to an OS full of toasted attributes. The recommendation is to boot CD-ROM or network, then mount the root file system on /a, and run a pkgchk -R /a -f. I found that simply booting single-user and running pkgchk -f did the trick. Your mileage may vary.

I don't think there would have been any other practical approach short of re-jumping the box to restore all of the lost attributes, so it is with great enthusiasm that I recommend keeping "pkgchk -f" in your tool bag.