I've had numerous occasions to "pop the hood" on JET to add functions or debug problems, and one of the things I appreciate is how it is implemented in well structured scripts. JET doesn't use complex frameworks or app servers just for the sake of adding "web 2.0" to a marketing slide. It's as simple as it needs to be. Bravo!
One thing I've noticed is that there's a HUGE amount of opportunity for enhancing JET, both in terms of optimizing existing code (increase use of functions, internal documentation, etc) but also in terms of enhancing functionality with new modules or new features in existing modules. Unfortunately, Sun (ahem, Oracle) is not really exploiting these opportunities in a timely manner. JET updates don't happen very frequently. I would expect that's due to economic strains impacting resources, and I don't expect the IT economy to suddenly produce a legion of JET developers.
Why not create an open community for the development of JET? There's all kinds of energy at OpenSolaris for next generation provisioning / packaging. If there were an OpenSolaris project for today's JET product I think we'd see it really take off. Many of us need to perform these enhancements either way, but we'd all benefit if we could leverage each others work and continue the evolution of a fantastic framework.
I'm hoping that resistance to opening JET isn't due to Sun Professional Services. JET is great, but it isn't rocket science. SunPS can always retain copyright or restrict distribution of their proprietary modules (as they always have). But the base framework screams to be open sourced, or turned into a community.
If Sun won't open the code it's tempting to try a black-box rewrite (I have no desire to break Sun's licensing - I respect their rights to code they developed!). It would certainly be a fun science project. Unfortunately, it would also be a waste of time given that an existing code base would make such a perfectly suited foundation.
So, how about it, Sun? OpenJET?
Showing posts with label jumpstart. Show all posts
Showing posts with label jumpstart. Show all posts
Sunday, February 28, 2010
Tuesday, May 05, 2009
JET and the Recommended Cluster
JET is bugging me. I'm a sort of pack rat when it comes to installation media, and that extends to patch sets. Hey, you never know you might get a request to Jumpstart Solaris 2.4, right? Ok, I'm not really that bad.
But, you may well be using a certain recommended cluster for a certain OS, and then need to jump a box to test the next recommended cluster right? Surely, it's not necessary to make a global change to your production server build configuration when implementing a test cluster?
As far as I can tell, base_config's use of recommended clusters is not handled in a manner that encourages good revision management. For each OS major revision (e.g., 10, 9, 8) there can be one cluster. For example, in today's JET software if we were using /export/install as our JET media base there would be a directory called /export/install/patches. Under that we can store one patch cluster for each major OS revision:
That works nicely until the next recommended cluster is released. At that point in time you can no longer have a repeatable build process because you need to replace the single instance of each OS with the new cluster. Not a good plan. We want our patch configuration to be configured in the template so that it's managed, and can be under source code control. Managing patch configurations outside the template is pretty much impossible to audit.
Here's an alternative approach I think would be a step in the right direction: Create a hierarchy to organize recommended clusters:
We need to be able to add recommended clusters in the same way we add other products. I'd like to see a new command called "list_recommended_clusters" which would have an output something like this:
These clusters could then be specified in the JET template using a variable like base_config_recommended_cluster. In addition, the check routine used during a make_client invocation would ensure that the directory exists, and perhaps ensure that each patch on the patch_list was represented. Bingo! Now we can use good revision control to manage the integration of patch clusters with our server build process.
But I think we can take it one step farther. How about we add the ability to include arbitrary patch sets? Here's a first cut at how it could work: We start by creating a patch repository. Say, /export/install/patch_repo. Under that directory we may have subdirectories for 5.10, 5.9, etc. Patches are simply added to that directory by copying them into place. Nothing fancy. The nice thing about this approach is in its economy of space.
The recommended clusters will have a lot of overlap between them, with the potential for storing the same patch in many different directories. By having one patch repository, we simply store each necessary patch one time and refer to it in a patch_order file. It would be trivial to write a few scripts that could operate or query on a set of patches according to a certain patch list, or perhaps cull out patches not referenced in any current patch lists. I could give or take this feature. There are some good arguments to be made for just storing each patch set and ignoring the storage space. I'm ok with either approach, and even happier if this flexibility were accounted for.
Having established a patch repository, we now need a place to manage patch lists. These would be in typical patch_order formatted lists; No need to reinvent the wheel. Each would need to be named with a unique identifier. For example,
These patch lists could then be specified within base_config as an alternative to using the Recommended clusters. Why?
I think these would be some very beneficial enhancements to the JET framework. I'd like to work on some of them, but I wanted to get the idea out there before I got wrapped up in something else and forgot about it. I'd be interested in hearing any thoughts on this topic - especially if someone has a better idea!
By the way, I do know about EIS baselines. But I think it's pretty rare for any enterprise to never have need for managing custom patch sets. It would be great if JET could come through with some help in this space.
But, you may well be using a certain recommended cluster for a certain OS, and then need to jump a box to test the next recommended cluster right? Surely, it's not necessary to make a global change to your production server build configuration when implementing a test cluster?
As far as I can tell, base_config's use of recommended clusters is not handled in a manner that encourages good revision management. For each OS major revision (e.g., 10, 9, 8) there can be one cluster. For example, in today's JET software if we were using /export/install as our JET media base there would be a directory called /export/install/patches. Under that we can store one patch cluster for each major OS revision:
/export/install/patches/10_Recommended
/export/install/patches/9_Recommended
/export/install/patches/8_Recommended
That works nicely until the next recommended cluster is released. At that point in time you can no longer have a repeatable build process because you need to replace the single instance of each OS with the new cluster. Not a good plan. We want our patch configuration to be configured in the template so that it's managed, and can be under source code control. Managing patch configurations outside the template is pretty much impossible to audit.
Here's an alternative approach I think would be a step in the right direction: Create a hierarchy to organize recommended clusters:
/export/install/patches/recommended/5.10/sparc/2009-04-22
/export/install/patches/recommended/5.10/sparc/2009-01-foo
/export/install/patches/recommended/5.10/x86
/export/install/patches/recommended/5.9/sparc/2009-04-22
/export/install/patches/recommended/5.9/x86/2009-01-foo
We need to be able to add recommended clusters in the same way we add other products. I'd like to see a new command called "list_recommended_clusters" which would have an output something like this:
# list_recommended_clusters
Version Location
------ ---------------
5.10_sparc_200901 /export/install/patches/recommended/sparc/5.10_sparc_200901
5.10_sparc_200902 /export/install/patches/recommended/sparc/5.10_sparc_200901
5.10_sparc_200903 /export/install/patches/recommended/sparc/5.10_sparc_200901
5.9_sparc_200901 /export/install/patches/recommended/sparc/5.10_sparc_200901
5.9_sparc_200902 /export/install/patches/recommended/sparc/5.10_sparc_200901
5.9_sparc_200903 /export/install/patches/recommended/sparc/5.10_sparc_200901
These clusters could then be specified in the JET template using a variable like base_config_recommended_cluster. In addition, the check routine used during a make_client invocation would ensure that the directory exists, and perhaps ensure that each patch on the patch_list was represented. Bingo! Now we can use good revision control to manage the integration of patch clusters with our server build process.
But I think we can take it one step farther. How about we add the ability to include arbitrary patch sets? Here's a first cut at how it could work: We start by creating a patch repository. Say, /export/install/patch_repo. Under that directory we may have subdirectories for 5.10, 5.9, etc. Patches are simply added to that directory by copying them into place. Nothing fancy. The nice thing about this approach is in its economy of space.
The recommended clusters will have a lot of overlap between them, with the potential for storing the same patch in many different directories. By having one patch repository, we simply store each necessary patch one time and refer to it in a patch_order file. It would be trivial to write a few scripts that could operate or query on a set of patches according to a certain patch list, or perhaps cull out patches not referenced in any current patch lists. I could give or take this feature. There are some good arguments to be made for just storing each patch set and ignoring the storage space. I'm ok with either approach, and even happier if this flexibility were accounted for.
Having established a patch repository, we now need a place to manage patch lists. These would be in typical patch_order formatted lists; No need to reinvent the wheel. Each would need to be named with a unique identifier. For example,
patch_order_5.10_2009q1
patch_order_5.9_dev-servers
patch_order_5.10_test01
These patch lists could then be specified within base_config as an alternative to using the Recommended clusters. Why?
- The site has a known incompatibility with a patch or two in the common cluster.
- The site wants to deploy other patches in the early part of the install as part of a managed list rather than manual entries in a template (e.g., custom_patches).
- Using these lists allows a configuration to be frozen in time for configuration management, and allows a convenient record of exactly what a server was deployed with
I think these would be some very beneficial enhancements to the JET framework. I'd like to work on some of them, but I wanted to get the idea out there before I got wrapped up in something else and forgot about it. I'd be interested in hearing any thoughts on this topic - especially if someone has a better idea!
By the way, I do know about EIS baselines. But I think it's pretty rare for any enterprise to never have need for managing custom patch sets. It would be great if JET could come through with some help in this space.
Wednesday, February 28, 2007
JET: Controlling custom_files with a custom extension
Any site running Sun hardware with more than one system should be looking at JumpStart to ensure that systems can be rebuilt consistently. the corollary to this is that any site running JumpStart environment should be using Sun's Jumpstart Enterprise Toolkit (JET). JET provides a consistent framemwork for accomplishing most common tasks, and a consistent framework to write extensions within. Standards and discipline are good.
One of the modules which comes with JET is called simly enough, custom. The custom module allows you specify either packages or files which should be added to a server during any of N predetermined reboots. This allows you to ensure that a change which requires a reboot can be made prior to a dependent process being started. Sounds good so far.
Following a recent Solaris 9 server build I was perusing the system for problems by auditing log files. In the messages file I discovered some lines indicating that a Kerberos problem was rearing its ugly head:
Our site does not use Kerberos, so it had to be a recent configuration change - not surprising considering we had just updated the patch set. After some research I arrived at BugID 5020096. This bug indicates that the issue can be resolved by removing some offending lines from /etc/krb5/krb5.conf.
This should be easy enough to fix in future builds. Just add the modified krb5.conf to the JET template's custom_files variable, and we'll be in good shape. Ahh, not so fast. How will we know what the file originally contained? A true Solaris Jedi will always manage an audit trail of his activities. If I were making the change manually I would copy the file to file.orig, or file.datestamp. Automation is not an excuse for abandoning discipline.
The trouble with JET is that its custom module's functionality for installing files is limited to two operations: overwrite or append. Overwrite simply clobbers any file which may exist. For example, to install the /etc/motd file I would palce my custom file in the configured JET file location, then add a line like this to the JET template:
motd is a fairly harmless little file, but knowing little about Kerberos, I dind't want to blindly whack the original file. The right solution to this problem lies in creating a simple extension to the JET toolkit. I began by examining the code from the custom module. Two modules specifically are relevant to this project: install, and postinstall. Within them is a simple case statement which handles the "o" or "a" functionality:
So, when I use an "o" in my custom_files module, it called JS_cp. I now needed to find the library which contains these core functions. Eventually, a colleague and I traced it back to /opt/SUNWjet/utils/lib. Looking at the JS_cp function revealed exactly what I expected: a simple copy routine wrapped in some voodoo.
Feeling a bt optimistic, I copied JS_cp to JS_cp_preserve and modified the code a bit so it would first check to see if the destination file exists, and if so, backup the file with a datestamp. Once the backup was in place, the original copy operation was performed. This was very trivial shell scripting. Here's what I ended up with:
Next, I returned to the install and postinstall code, and modified the case statements to accept a "b" operation (b for backup). I then executed a test Jump and was very pleased to see my JET extension had worked! I can now have custom_files install the workaround krb5.conf, and maintain a backup of the original. Here's the modified code:
Note that you need to make this modification in both /opt/SUNWjet/Products/custom/isntall and postinstall.
Now, all I need to do it specify something in the custom_files module like this:
And I will get a clean backup of the original file. Such a simple tweak - I hope the Sun folks who maintain JET will add something similar. While some limitations of JET can be frustrating, its intuitive layout and ease of extension make it something I grow more fond of each time I use it.
One of the modules which comes with JET is called simly enough, custom. The custom module allows you specify either packages or files which should be added to a server during any of N predetermined reboots. This allows you to ensure that a change which requires a reboot can be made prior to a dependent process being started. Sounds good so far.
Following a recent Solaris 9 server build I was perusing the system for problems by auditing log files. In the messages file I discovered some lines indicating that a Kerberos problem was rearing its ugly head:
Kerberos mechanism library initialization error: No profile file open.Our site does not use Kerberos, so it had to be a recent configuration change - not surprising considering we had just updated the patch set. After some research I arrived at BugID 5020096. This bug indicates that the issue can be resolved by removing some offending lines from /etc/krb5/krb5.conf.
This should be easy enough to fix in future builds. Just add the modified krb5.conf to the JET template's custom_files variable, and we'll be in good shape. Ahh, not so fast. How will we know what the file originally contained? A true Solaris Jedi will always manage an audit trail of his activities. If I were making the change manually I would copy the file to file.orig, or file.datestamp. Automation is not an excuse for abandoning discipline.
The trouble with JET is that its custom module's functionality for installing files is limited to two operations: overwrite or append. Overwrite simply clobbers any file which may exist. For example, to install the /etc/motd file I would palce my custom file in the configured JET file location, then add a line like this to the JET template:
custom_files_1="motd:o:/etc/motd"motd is a fairly harmless little file, but knowing little about Kerberos, I dind't want to blindly whack the original file. The right solution to this problem lies in creating a simple extension to the JET toolkit. I began by examining the code from the custom module. Two modules specifically are relevant to this project: install, and postinstall. Within them is a simple case statement which handles the "o" or "a" functionality:
case ${mode} in
a) case ${fn2} in
/etc/hosts) JS_merge_hosts ${filefound};;
*) JS_cat ${filefound} ${ROOTDIR}${fn2}
;;
esac;;
o) JS_cp ${filefound} ${ROOTDIR}${fn2};;
esac
So, when I use an "o" in my custom_files module, it called JS_cp. I now needed to find the library which contains these core functions. Eventually, a colleague and I traced it back to /opt/SUNWjet/utils/lib. Looking at the JS_cp function revealed exactly what I expected: a simple copy routine wrapped in some voodoo.
Feeling a bt optimistic, I copied JS_cp to JS_cp_preserve and modified the code a bit so it would first check to see if the destination file exists, and if so, backup the file with a datestamp. Once the backup was in place, the original copy operation was performed. This was very trivial shell scripting. Here's what I ended up with:
if [ "$#" != "2" ]; then
JS_error "`basename $0`: Illegal Arguments. Usage: "
fi
JS_FROM=$1
JS_TO=$2
JS_display "Copying file `echo ${JS_FROM} | sed -e \"s?^${SI_CONFIG_DIR}/??\"` to ${JS_TO}"
if [ -f ${JS_TO} ] ; then
datestamp="`/usr/bin/date +%Y%m%d`"
/bin/cp -p ${JS_TO} ${JS_TO}.jet.${datestamp}
case $? in
0) # Success
JS_display "Successfully preserved ${JS_TO}.jet.${datestamp}"
;;
1) # Failure
JS_display "WARNING: Failed to preserve original file ${JS_TO}"
;;
esac
fi
/bin/cp -p ${JS_FROM} ${JS_TO}
if [ "$?" != "0" ]; then
JS_error "JS_cp:\t\tError occured while copying ${JS_FROM} to ${JS_TO}"
fi
Next, I returned to the install and postinstall code, and modified the case statements to accept a "b" operation (b for backup). I then executed a test Jump and was very pleased to see my JET extension had worked! I can now have custom_files install the workaround krb5.conf, and maintain a backup of the original. Here's the modified code:
case ${mode} in
a) case ${fn2} in
/etc/hosts) JS_merge_hosts ${filefound};;
*) JS_cat ${filefound} ${ROOTDIR}${fn2};;
esac;;
o) JS_cp ${filefound} ${ROOTDIR}${fn2}
b) JS_cp_preserve ${filefound} ${ROOTDIR}${fn2};;
esac
Note that you need to make this modification in both /opt/SUNWjet/Products/custom/isntall and postinstall.
Now, all I need to do it specify something in the custom_files module like this:
custom_files_N="krb5.workaround:b:/etc/krb5/krb5.conf"And I will get a clean backup of the original file. Such a simple tweak - I hope the Sun folks who maintain JET will add something similar. While some limitations of JET can be frustrating, its intuitive layout and ease of extension make it something I grow more fond of each time I use it.
Subscribe to:
Posts (Atom)
