From Menno.Willemse@johnguest.co.uk Mon Oct 2 17:07:29 2006 From: Menno.Willemse@johnguest.co.uk (Willemse, Menno) Date: Mon, 2 Oct 2006 17:07:29 +0100 Subject: [Infrastructures] Distributing private key information at install time Message-ID: Hello World, Here's a problem that I'm sure some of the larger sites have solved already, and that just isn't irritating enough for me to solve it here and now due to a lack of circular tuits. Since there is an infrastructure angle to this, I decided to post it here. Whenever I reinstall a machine, I set up secure shell on the box, and every time it will run ssh-keygen to make host keys. And then SSH gets all annoyed at me because the host key has changed and I have to remove it from known_hosts to assure SSH that it's allright really. Not just me, actually. Everybody has to do that for each name under which they ever accessed the box. So what you really need is a way to restore the original host keys, so that all the ssh clients will have the correct information already. This always gets me thinking: Is there a cryptographically sound way to restore the key information to the client from a file on the install server? You can't get it from an NFS server. My first instinct is to create an install image with a known private key, and to trust that key to fetch the real key at install time. But this is obviously vulnerable to the install image's key being stolen. It needs to be on an NFS server as I use NIM for AIX. For Linux, it's FAI. Now I know that the only truly secure method is to carry the key over to the machine in question on a floppy and then burn the floppy, but I'm willing to settle for "Reasonably Secure", as long as the whole method is hands-off. I may not have that many Unix boxes now, but we may get a lot of Linux workstations in the future. What is everybody else doing? Cheers, Menno Willemse -- Menno Willemse - John Guest IT Department Tel: 01895-449233 ext 290 Email: menno.willemse@johnguest.co.uk You know it's Enterprise Software when the vendor freebie is a red shirt. -- AdB - Usenet Internet communications are not secure and therefore John Guest companies do not accept legal responsibility for the contents of this message. Any views or opinions presented are solely those of the author and do not necessarily represent those of John Guest companies. From Daniel Hagerty Mon Oct 2 19:20:00 2006 From: Daniel Hagerty (Daniel Hagerty) Date: Mon, 2 Oct 2006 14:20:00 -0400 Subject: [Infrastructures] Distributing private key information at install time In-Reply-To: References: Message-ID: <17697.22608.124832.898748@perdition.linnaean.org> > Whenever I reinstall a machine, I set up secure shell on the box, > and every time it will run ssh-keygen to make host keys. And then > SSH gets all annoyed at me because the host key has changed and I > have to remove it from known_hosts to assure SSH that it's allright > really. Not just me, actually. Everybody has to do that for each > name under which they ever accessed the box. So what you really > need is a way to restore the original host keys, so that all the > ssh clients will have the correct information already. Well, there's a different problem that you have some chance of running into: what do you do if you're regenerating a key because a machine was compromised? You really do want to burn the old key at that point. You want to centralize the ssh_known_hosts file for your site to the /etc version. Ssh provides some tools for doing this, although what they give you isn't quite a 100% solution. Take a look at ssh-keyscan. From wes@umich.edu Tue Oct 3 01:41:52 2006 From: wes@umich.edu (Wesley Craig) Date: Mon, 2 Oct 2006 20:41:52 -0400 Subject: [infrastructures] state machines In-Reply-To: References: Message-ID: <9D4C5AED-798E-4942-8902-95C0F5937A4D@umich.edu> On 22 Sep 2006, at 03:28, Willemse, Menno wrote: > Without the occasional image backup, as far as I can tell, you are > stuck replaying the whole history since time began. If you simply > restore a backup, you won't have the latest changes. To get the > whole thing, you need to do both. In our environment, we're making the change with the tool that will also be used to restore, so a restore includes the latest changes. :wes From wes@umich.edu Tue Oct 3 01:53:44 2006 From: wes@umich.edu (Wesley Craig) Date: Mon, 2 Oct 2006 20:53:44 -0400 Subject: [Infrastructures] Distributing private key information at install time In-Reply-To: References: Message-ID: On 02 Oct 2006, at 12:07, Willemse, Menno wrote: > This always gets me thinking: Is there a cryptographically sound > way to restore the key information to the client from a file on the > install server? We give admins boot CDs which contain an individualized key which expires. The admin can burn this CD with a short expiration many times, if they happen to be installing a large number of machines. Typically they carry a CD with a longer term key with them, for troubleshooting. The key on the CD identifies the admin to the centralized server(s), so they are allowed to retrieve the ssh key files (among other things) that they're responsible for. Down side of the system is that CDs might be stolen. :wes From dclark@pobox.com Wed Oct 4 01:34:13 2006 From: dclark@pobox.com (Daniel Clark) Date: Tue, 3 Oct 2006 20:34:13 -0400 Subject: [Infrastructures] Distributing private key information at install time In-Reply-To: References: Message-ID: <5422d5e60610031734r57d6313ds62603fa31be07c5c@mail.gmail.com> On 10/2/06, Willemse, Menno wrote: > Hello World, > > Here's a problem that I'm sure some of the larger sites have solved already, and that just isn't irritating enough for me to solve it here and now due to a lack of circular tuits. Since there is an infrastructure angle to this, I decided to post it here. > > Whenever I reinstall a machine, I set up secure shell on the box, and every time it will run ssh-keygen to make host keys. And then SSH gets all annoyed at me because the host key has changed and I have to remove it from known_hosts to assure SSH that it's allright really. Not just me, actually. Everybody has to do that for each name under which they ever accessed the box. So what you really need is a way to restore the original host keys, so that all the ssh clients will have the correct information already. > > This always gets me thinking: Is there a cryptographically sound way to restore the key information to the client from a file on the install server? You can't get it from an NFS server. My first instinct is to create an install image with a known private key, and to trust that key to fetch the real key at install time. But this is obviously vulnerable to the install image's key being stolen. It needs to be on an NFS server as I use NIM for AIX. For Linux, it's FAI. > > Now I know that the only truly secure method is to carry the key over to the machine in question on a floppy and then burn the floppy, but I'm willing to settle for "Reasonably Secure", as long as the whole method is hands-off. I may not have that many Unix boxes now, but we may get a lot of Linux workstations in the future. > > What is everybody else doing? This is one one of the main plugins system example cases for the Bcfg2 (http://www.bcfg2.org) configuration management system. "[This problem] makes the ideal case for the plugin interface, since (a) all admins know the problem well, and (b) can easily envision a solution to it, but (c) no one does." -- Narayan Desai, primary developer of Bcfg2 >From ftp://ftp.mcs.anl.gov/pub/bcfg/papers/pay-as-you-go.pdf : """ 5.2 SSHbase SSHbase is a ssh key management system. It maintains a repository of ssh public and private keys and constructs a comprehensive ssh known hosts file. It functions entirely automatically; new keys are generated when one is requested for an unknown host. At this point, the ssh known hosts file is updated for all clients. The repository language used by this generator consists of a directory containing public and private keys for all clients. This system ensures two basic properties. First, client keys persist beyond client rebuilds. Second, client keys can be centrally revoked, in case of a system compromise. """ I think sites that use Kerberos 5 can use Simon Wilkinson's OpenSSH patches (www.sxw.org.uk/computing/patches/openssh.html) to sidestep this issue by relying on Kerberos 5 for mutual authentication instead of host keys. Of course, in that case you have the problem of keeping a client's Kerberos 5 client credentials between rebuilds. -- Daniel Joseph Barnhart Clark http://www.pobox.com/users/dclark From Menno.Willemse@johnguest.co.uk Wed Oct 4 11:05:39 2006 From: Menno.Willemse@johnguest.co.uk (Willemse, Menno) Date: Wed, 4 Oct 2006 11:05:39 +0100 Subject: [Infrastructures] Distributing private key information at install time Message-ID: Hello World, Thanks to all those who responded. The prevailing wisdom seems to be that you use a boot CD with a temporary key to do the re-install, or that you somehow keep the old keys on the system where they won't be erased by the reinstall. A different hard disk was suggested, but a USB key, floppy or a CD would probably work just as well. All of these methods are of course sensitive to the media being stolen, but that's something we'll just have to live with. I think I'll set things up so that the install image has a key in that allows you to get the proper key from the install server. The installation image will only be NFS-exported to the machines that need it, as long as they need it. This will lead to exposure while the machine downloads its install images, but so be it. I'll also have a good look at ssh-keyscan and centralising the known_hosts file. That may be another way around this problem: after a reinstall, scan the box' host key and have all other machines pull down the file on a regular basis. Cheers, Menno -- Menno Willemse - John Guest IT Department Tel: 01895-449233 ext 290 Email: menno.willemse@johnguest.co.uk There is no Cabal. Internet communications are not secure and therefore John Guest companies do not accept legal responsibility for the contents of this message. Any views or opinions presented are solely those of the author and do not necessarily represent those of John Guest companies. From andrew@operationaldynamics.com Wed Oct 4 23:22:35 2006 From: andrew@operationaldynamics.com (Andrew Cowie) Date: Thu, 05 Oct 2006 08:22:35 +1000 Subject: [Infrastructures] Training Infrastructure Teams In-Reply-To: <20060818174228.B26037@so.lanier.com> References: <20060818174228.B26037@so.lanier.com> Message-ID: <1160000555.3908.13.camel@procyon.operationaldynamics.com> --=-RrQq+IkG4WFGDMNdHrpu Content-Type: text/plain Content-Transfer-Encoding: quoted-printable On Fri, 2006-08-18 at 17:42 -0400, devick@lanier.com wrote: > The attached file is an introduction to the training process Sorry I haven't had a chance to respond to this earlier: your document, misleadingly named trainingintro.html but titled with the provocative title "The Craft" is excellent. Hiring people with technical skills is easy. Finding someone who has a sense of verve and who will embrace the wider responsibilities and obligations of their job - that's very difficult. I have often commented in my keynotes that Systems Administration, as such, is not a profession. If more people pursue their work with the diligence and spirit that is implied here, then I think we will have taken a long leap forward to changing this. ++ This is a wonderful document. I would love to use it in my own work as a resource document. Are you willing to let others use it? Your posting it here and describing it in terms of infrastructures.org implies that this is the case. If you would be able to put it under an open licence [creative commons or GFDL] then I for one would be willing to offer comments and improvements. AfC Sydney --=20 Andrew Frederick Cowie Managing Director Operational Dynamics Consulting Pty Ltd http://www.operationaldynamics.com/ Management Consultants specializing in strategy, organizational architecture, procedures to survive change, and performance hardening for the people and systems behind the mission critical enterprise. Worldwide: Sydney +61 2 9977 6866 New York +1 646 472 5054 Toronto +1 416 848 6072 London +44 207 1019201 --=-RrQq+IkG4WFGDMNdHrpu Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2 (GNU/Linux) iD8DBQBFJDQrLVETDFf2570RAvauAJ9ZqiOEx/yAH6A8LXggBXXwfPwtswCfUw9e fEvo4x1vpAXBT06aj4d5aPg= =9nie -----END PGP SIGNATURE----- --=-RrQq+IkG4WFGDMNdHrpu-- From brendan@cs.uchicago.edu Fri Oct 6 15:40:36 2006 From: brendan@cs.uchicago.edu (Brendan Strejcek) Date: Fri, 6 Oct 2006 10:40:36 -0400 Subject: [Infrastructures] Distributing private key information at install time In-Reply-To: References: Message-ID: <816b88240610060740h71da9449ib130b0a271eb5cd3@mail.gmail.com> On 10/4/06, Willemse, Menno wrote: > The prevailing wisdom seems to be that you use a boot CD with > a temporary key to do the re-install This seems unnecessarily manual to me. I don't think it would scale very well. How many machines are you managing? > or that you somehow keep the old keys on the system where they > won't be erased by the reinstall. > ... > centralising the known_hosts file. That may be another way around > this problem: after a reinstall, scan the box' host key and have all > other machines pull down the file on a regular basis. Are you using any sort of centralized configuration management system? How do you propagate configuration changes to live machines? An ssh key pair is really no different from any other bit of configuration data, and is easy to centralize. Without more information about your particular infrastructure though, it is hard to offer more concrete suggestions. Best, Brendan -- http://praksys.blogspot.com From brendan@cs.uchicago.edu Fri Oct 27 19:15:09 2006 From: brendan@cs.uchicago.edu (Brendan Strejcek) Date: Fri, 27 Oct 2006 14:15:09 -0400 Subject: [Infrastructures] Re: Distributing private key information at install time In-Reply-To: <816b88240610270836s149fdd3aw1b3727e8615969d@mail.gmail.com> References: <816b88240610060740h71da9449ib130b0a271eb5cd3@mail.gmail.com> <816b88240610270836s149fdd3aw1b3727e8615969d@mail.gmail.com> Message-ID: <816b88240610271115n83848cfs2995aebd95c3005f@mail.gmail.com> (Wesley sent me a message off-list, but said it was fine to repost to the list.) On 10/6/06, Wesley Craig wrote: > On 06 Oct 2006, at 10:40, Brendan Strejcek wrote: > > > Are you using any sort of centralized configuration management system? > > How do you propagate configuration changes to live machines? > > The question is how do you deal with a machine whose hardware you are > replacing. It's all fine to store this data centrally, but how do > you bootstrap security from a naked machine? Hooks can be added to any automated installation procedure. The security of this initial bootstrap will depend upon the particular installation method. For example, a custom install CD could easily include various security bits, such as server public keys. Another example would be a PXE boot on an isolated network where a server decides what credentials to issue to a new machine based on ethernet address. Best, Brendan -- http://praksys.blogspot.com From brendan@cs.uchicago.edu Fri Oct 27 19:27:37 2006 From: brendan@cs.uchicago.edu (Brendan Strejcek) Date: Fri, 27 Oct 2006 14:27:37 -0400 Subject: [Infrastructures] Distributing private key information at install time In-Reply-To: References: <816b88240610060740h71da9449ib130b0a271eb5cd3@mail.gmail.com> <816b88240610270836s149fdd3aw1b3727e8615969d@mail.gmail.com> Message-ID: <816b88240610271127y613d16dq58e0a0c9963786c8@mail.gmail.com> (Sorry in advance if I got some of the attribution wrong, the quotes and forwardings made it a bit complicated.) on 04 October 2006 wrote: > That's secure, but it involves walking up to the machine, which I > don't want to do. They are in locked rooms or on remote locations. > Maybe the best solution is just to leave a CD, HD partition or > floppy in the machine with its crypto-information. I use a trick to get around this. If I have to install a fresh operating system on a truly blank machine, I need to physically be at the console. In that case, however, the machine was usually just delivered and has not been deployed yet. Once a machine has been installed, it has the ability to remotely reinstall itself with several supported operating systems. I use alternative kernels and the boot loader to accomplish this. Thus, once a machine has been installed physically once (in most cases) it can be reinstalled remotely. > All security > flies out of the window as soon as someone can touch the machine > anyway. That is not totally true. They may be able to take over the machine, but they will not be able to compromise confidentiality (that is, you can encrypt data on the disk). > If you want to do hands-off, unattended installs, I suppose there > just isn't a way that's 100% secure. Yeah, true. There will likely be some trade-off between security and convenience. Best, Brendan -- http://praksys.blogspot.com