[Infrastructures] Request For Comment: Hopeful LISA Paper Topic

[justin m. clayton]

Hello all,

Before I leap into the concepts for my paper submission, I figure that
since most of you don't know me I should begin by introducing myself and
give you an idea as to where I'm coming from: my name is Justin Clayton,
and I'm an SA at the University of Washington in Seattle. I have a devout
and sometimes strangely unnatural and irrational love for infrastructure
design, especially as it relates to UNIX systems. I first became
interested in the theory behind this field a few years ago, when Lee Damon
(LISA Guru Coordinator, member of the SAGE Ethics Committee, and all
around upstanding, if not slightly off, guy) joined our team at the
university. Together we began work on our attempt at designing a system
that would survive rain, sleet, snow, hackers, crackers, slackers, and
dissenting opinions among various research professors. We called it
Nikola. It is, in fact, largely derived from the concepts presented in
Lee's "Issues in UNIX Infrastructure Desgin" tutorial, presented at the
LISA Conference for the past several years.

I was truly naive to think that no one had thought about this problem of
managing large numbers of UNIX systems in as much detail as Lee had, and
in fact, every once in a while I re-discover the solution from someone
else's point of view. One rather memorable occurance of this was at LISA
'02, where I met Paul Anderson of the University of Edinburgh (original
author of LCFG) and Steve Traugott of our very own infrastructures.org
(and original author of isconf). I spent the majority of my downtime (and
indeed, much of the actual conference time) picking the brains of these 2
fine fellows, attempting to grasp the "bigger picture" regarding the
issues we're facing in infrastructure design theory today. It also helped
a great deal that Steve was presenting his record-shatteringly long paper
on the importance of ordering theory in configuration mangement. Indeed,
many of my thoughts and ideas for this paper proposal have been inspired
by his paper, as well as the many talks I had with both men. I was told
that I also made a number of good comments concerning the advantages and
disadvantages of method x or y, and after the conference I was encouraged
by my co-workers as well as my local SAGE chapter to submit a paper
myself on the topic.

Now, after reviewing isconf and other systems in more detail, I feel that
some of the concepts developed and/or extented for our Nikola system are
of use to the community as a whole, and am therefore requesting comments
on the ideas I will attempt to sketch out below. Some of the statements
made I will be considering for the final draft of the abstract and/or the
paper itself, so I encourage brutal honesty (or is it honest brutality?),
especially since this is my first crack at a published paper.

I have been instructed to first don my "thick skin" in preparation for
sticking my neck out for all to hack at. There, that's better.

It has been hard for me to know where to begin without first critiquing my
own work: what is Nikola? Is it a tool for developing sites, or is it
simply an implementation of our theories? Unfortunately at the moment,
it's both. (A bit more detail can be revealed by visiting its site,
http://nikola.ee.washington.edu at your leisure.) There are extremely
site-specific bits to Nikola that make it currently unportable to other
nonrelated sites as such, but a great deal of it is in fact generalized
enough to present as helpful to others both in concept and in practice: in
particular the concept of release engineering (or the "software
approach"), and the development of a system for the effective deployment
of rapidly changing configurations (perhaps too many buzzwords in that
description; more on this later). As of this writing, no real attempt has
been made to abstract Nikola into the skeleton and the flesh, as it were,
mostly because we have not had the time to do this, nor have we realized a
great benefit within our organziation beyond a purely academic one.
(However, I have been sketching out a general idea as to what this new
Nikola would look like, and given infinite time and resources, I'm sure it
would be quite impressive.)

In regards to release engineering: it has been stated that "order
matters", and I hold this as a general truism. Extending this idea a bit
more practically, ensuring that this order is properly developed,
maintained, documented, tested, and deployed should also matter to us a
great deal.  Thus the "treat your system image like software" approach: if
there is a change to the base configuration (I'll later explain in more
detail about the idea of a "base" config and how it can be "extended" to
meet various and sundry needs), whether it be a change in the way you're
providing services or simply vendor-released OS patches, increment the
version number. If these otherwise basic revision control techniques are
used and enforced on your infrastructrue, it allows for much greater
definition of what your image looks like and intends to do (having a
chagelog of sorts for your system image dev). Having this sort of
definition produces much clearer documentation, and opens the door for a
number of wonderful applications. This approach lets you release a newly
revised image to a smaller group for testing purposes, without losing
track of which machine or group of machines are the test victims in the
first place. It also allows for simple live upgrades of both test and
production systems.

Example: Nikola 1.0.0 used primarily firewalled NIS for user
authentication, and used a common installation of the OS. By 1.0.3, we had
updated the OSes we support, moved our shared filesystem development into
CVS, and set new standards for our jumpstart/kickstart areas to handle
future development and testing. In 1.1.0 we added kerberos to our systems,
paving the way for 1.2.0, which introduced LDAP as well. (We are now
starting to release 2.0.0, which incorporates new base OS releases as well
as a shift from NFS to AFS for read-only shared filesystems.) However, due
to constraints of research deadlines and/or other physical network issues,
some divisions of our organization were able to move to LDAP+Kerb more
quickly than others. This clear organization of our releases allowed us to
deploy where we needed it without leaving the other divisions out in the
cold. And since the version number is included in the motd, determining
exactly which packages and patches are installed on a given machine is as
simple as logging in.

We were also able to strictly define these transitions by the use of the
Nikola Upgrade Tool (NUT). nut is the "tool" portion of our release
engineering efforts: it is the starting point for all Nikola upgrades.
Once the path for upgrading from version x to y is defined in a file
repository, nut can easily determine which actions need to be performed in
order to declare a given machine as running the new version. nut can be
used to automatically upgrade to the latest available version or to a
specific release needed for your particular circumstances.

In regards to rapidy changing configurations: While I'm sure we'd all
thoroughly enjoy a world where absolute order could be maintained
unilaterally in the context of our own installation, we realize that this
is purely theoretical, and that even as one machine is logged into and
files are viewed, wtmp and atimes are updated, and suddenly our
theoretical utopian world comes crashing down. Having said that, we also
realize that no one actually lives in that world, and there is a certain
amount of disorder that is acceptable. In keeping with this philosophy of
"keeping it real" as it concerns our system configuration, we have another
realization to make. Though it would be in the best interests of absolute
order to increment revisions when even the slightest change is made, we
realize that some systems, especially when undergoing long-term testing,
change more rapidly than others. Furthermore, if our configuration
includes servers (which I'm sure most of yours do indeed), and if our
servers are to be placed under the same infrastructure image, then
maintaining lists that are updated regularly (such as user information or
DNS tables, potentially break the model of total and absolute
documentation in ordering.

I'm going to stop for a moment and try to explain what I mean from a
practical standpoint by using what I believe to be a major fault in
isconf. I say this with painful humility and undying respect for the
authors and maintainers of isconf, who have done much more with their
careers as SAs than I have so far accomplished in mine, but I also stand
by my claim, and I'll explain why. Though I've never used isconf in a
production environment, I have deeply combed through infrastructures.org,
and have received demonstrations and explanations as to its function. This
example assumes isconf versions 1 or 2, using the make backend, and also
assumes I correctly understand how this thing actually works.

Example: We have an isconf-managed web server running on port 80. We want
to change the port to 90 (never, ever, do this). We create another make
keyword which fires a script that replaces the old httpd.conf file with
the correct one. We later want to move it to 100 (again, silly. Think SSL
or something), so we create another keyword. However, we cannot delete the
httpd.conf sources for either the 80 and 90 keywords, because if the
machine explodes into a ball of proverbial flame and needs to be rebuilt,
isconf will want to step through every change this configuration has
undergone in its lifetime, including putting all 3 httpd.conf files into
place at certain intervals in the reinstall process. And what if the
original port change was to facilitate a new server on that port? Since
only a single machine, and not the entire data center, is being rebuilt,
we cannot assume that the fast-forward reliving of the entire history of
that box will not interfere with existing systems. What if we had a set of
DNS servers that were once AFS database servers? Should our recreation of
that node include installing core AFS packages, starting the service,
stopping the service, removing the packages, and finally installing bind?
How much of your legacy infrastructure is needed? Some of it, surely, but
not to the amount of precice detail that isconf requires.

Just to be clear, the preceding paragraph was not meant to incite a flame
war, but to make a point about an important aspect of our software
approach we've left out: preferences. Just as software vendors leave the
specifics of tweaking their product to the consumer, what we "ship" is
actually a base installation that fits on all machines in your
configuration, whether it be a workstation, a cluster node, or a mail
server. By defining characteristics of your end result for a given
machine, you can also bind individual configurations (replacing config
files, installing crontabs, etc.) to that characteristic. (I understand
that the preferences analogy is in all reality rather poor, but I hope
you've grasped the bits that fit our scenario.)

This is where both LCFG and Nikola appear rather strong, though LCFG seems
to have taken the approach of the true software engineer and made it very
OO-like, wheras we have taken the more liberal arts approach of making it
look pretty and attempting to finish developing it before lunchtime.
Nikola accomplishes this with a tool called "nuggets" (for reasons I will
deftly avoid mentioning here). It is essentially a glorified string of if
statements, because it describes files to be installed and other actions
to be performed if machine x evaluates true for keyword y. Let me give you
a possible example of a nugget controlfile:

nugget inetd.conf {
     # here we set some defaults (an else, if you will, though not always
     # used in that way)
     masterfile="inetd.conf.master"
     prefile=/usr/nikola/lib/nugget_header # says "don't edit by hand"
     delete=n
     keepold=y
     installdir=/etc
     perms=0444
     uid=0
     gid=3
     postproc="restart_inetd.sh"
     duty backupclient {
          # appends line about amandad
          postfile="inetd.conf.amanda"
     }
     arch rh71-ia32 { # rh71-ia32 = Red Hat 7.1 for Intel x86
          # Red Hat uses xinetd, so we don't even want this file
          delete=y
          keepold=n
     }
}

The "masterfile" keyword gives us the starting source for the file
(defaults to look in /usr/nikola/nuggets/<nuggetname>/), but it can be
appended to using the "prefile" or "postfile" keywords. For our if
stanzas, we have the characteristics that each individual machine stores
locally (in /var/nikola/) such as arch (os+hardware), duty (what services
are provided), as well as bunch (custom-configurable grouping, usually
based on the details of your organizational structure; much like OUs in
Active Directory, actually), or even Nikola versions (only install
kerberos files for >1.1.0, etc). You can easily see how nesting can also
be an important part of determining the logic flow.

The process of actually making this information useful begins in cron,
where at regular intervals each machine pulls down this control
information, parses what is relavent to itself, and executes the actions,
which in this case include restarting inetd. The end result is that minor
configuration changes can be made to all machines, or smaller groups of
machines, depending on the need. Any number of types of config can be
stored in this way, from an amd map to a krb.conf to a DNS zone file, and
also serves as a backup of this critical information. And since we keep
nugget sources in CVS, deploying a change for testing and then rolling
back is trivial and not uncommon. Other features of nuggets include (but
are not limited to)  wildcards via regex (arch rh\d\d-ia32 would eval true
for all versions of Red Hat for x86), frequency of the update check
(mostly for bandwidth conservation purposes), sanity checking, and listing
multiple values for keywords like "postproc" by replacing the = of the
second assignment with a +.

The end goal for any of these tools is precise documentation of your
system configuration to facilitate rapid prototyping, deployment, and
recovery of any given system state, while still attemtping to live in the
world of the practical and out of the theoretical. There is much more to
Nikola and to concepts I feel would be useful in this paper (which, by the
way, I've thought about titling "Practical Application of Infrastructure
Design Theory"). However, my email client informs me that I've now written
approximately 16K of text over the past 3 hours, and that I should
probably stop for a while and rest (yes, it said all that; pine has many
many options none of us ever knew about), so I will leave the above to
you, the reader, to rend to shreds. If I'm lucky, perhaps I may even get
some useful scraps out of all the carnage to submit to the LISA '03
conference.

Any questions, comments, concerns, suggestions, or related thoughts can be
directed to justincl@u.washington.edu and/or to the infrastructures
mailing list. I'm always interested in facilitating discussion. If you
know anyone to whom this might be of interest, please forward it along.
Additionally, if you live in the greater Seattle area, I'd be happy to
entertain any of your thoughts over lunch sometime.

Many thanks,

--
Justin Clayton
VLSI Research System Administrator
University of Washington
Electrical Engineering Dept
justincl@u.washington.edu
206/543.2523  EE/CSE 307E