Besides being a great tv show, it’s also the name of my ipod touch. Yes, ipod touch ftw. It’s worth it, I assure you.
So I was reading a blog that I found in d-zone the other day talking about how git should be the next big unix tool, and it got me thinking. In my group at work we needed a revision control system for our various system config files. We had a couple requirements that had to be met though.
- needed to support kerberos natively
- be dead simple to add new files
- be dead simple to commit changes
- be nearly completely automated
- needs to be super way simple to set up
- needs to be able to scale bit, just in case
- recovery of b0rked files or blown up systems needs to be chuck norris easy
Well, originally I had thought let’s try putting our files in CVS. But then I said “ehh, but the subversion folks always give CVS grief for not being an adequate version control system”.
- I know CVS can support kerberos, but it takes too much headache to set up.
- You’ve got to export environment variables to use SSH (see CVS_RSH)
- Screwing around with CVSROOT is a pain in the ass. Seriously. But it’s lauded as one of those things that makes it easier to check stuff out with because you avoid the mindless violence that is involved with the raw cvs command.
- Considered inadequate by many because of all the reasons shown here
- I don’t think it can scale
Next, I thought “well, ok, can subversion do what I want”? I have quite a bit of experience with subversion because I use it for all my code versioning needs. I wasn’t quite sure if it (or I) was up to the task of setting it up though. Remember that in addition to usage and all that, other people in my group need to be able to set this software up or admin it. Let’s just say my group at work shys away from “responsibility” and “complexity” : P so if it’s too complicated for them, they’re going to be screwed if I get hit by a bus.
Subversion (at least as I use it) revolves around apache. Yes, I know that one can set it up to do the whole SSH bit, but I’ve never gone down that road because subversion also has convoluted black magic commands to accomplish the goal.
Well, so now I was in a pickle. When along came this article and git. So I downloaded it to a server and putz’d with it for about 3 or 4 hours. Got frustrated. Went home. Putz’d into the evening. Came back the next day and had the solution ready to go. And it’s awesome. Freaking awesome.
At work we call it “oobi” (named after the cartoon). It’s not really a new CST software project (god I don’t need any more of those). Instead, it’s a way of using git for version control.
oobi acts like a developer who’s collaborating with N other developers on a software project. These N developers make code (config files) and change code (the same config files). All these developers also use git and maintain their own repos on their computers. These developers commit changes to their repositories and oobi pulls changes from them to him. It’s instant version control…of anything on your filesystem.
We have N servers, and we have oobi; all run git. oobi clone’s each machine’s repo. We make changes to the git repos on the local servers and then oobi pulls from all the machines at the top of the hour.
So you might be asking why is git, in this scenario, so great.
- native kerberos authentication via ssh
- distributed and scales
- Easy cloning and pulling
- Easier recovery
Since git uses ssh by default to pull and push and clone it’s way around the universe, this means that it naturally can just use kerberos! Woot! So what we do is have a special principal that can go to our machines. We store him in a keytab and just like that, you’ve got perfect authentication and no risk of compromise via interactive login.
It scales. We don’t have a huge number of machines, but I figure we’ll gradually get more and more and more.
So you want to know how simple it is to start versioning a new system? Check this out. (The only assumption I’m making is that you’ve added the oobi principal to your root .k5login)
On the system to version, as root
- cd /
- git init
- git add path/to/my/file
- git commit -m “Initial import”
And then on oobi as the git user
- git clone ssh://myserver.domain.org/
Done.
The small amount of coding that was done for oobi is just some shell scripts that run via cron to do a “git pull” for all the repos that currently exist.
Now, what if you want to recover? Well, say your system bombs. You rebuild it, and you re-install git. Then you do the following
- git clone ssh://git@oobi/home/git/myserver.domain.org/
This gives you a directory called myserver.domain.org in your current directory. If you look inside it, you’ll see stuff like this.
hehe, I think you know where I’m going with this. To recover your system you do this.
- cd myserver.domain.org
- cp * -r /
- mv .git /
Done. Restart your services.
Even better, check this out. When you moved that .git directory in the last step, you also instantly reconfigured the system to be in the pull loop that oobi does every hour! Also, your entire revision history and logs for commits are instantly available on your new system, and you can even recover an old version of the file!
Surely you’re not serious.
Yes I’m serious. And don’t call me Shirley.
So yeah, git rules. It’s made the problem of versioning configs (or anything for that matter) completely moot. So get out there and oobify your servers. If you’d like to see how we apply it to our environment, hit me up and I’d be happy to show you what you need to do.
. useless? Yes. Likely to get me praise? Yes. hehe. It just seemed like one of those fun things to do. I’m looking around for good documentation on how to to write iPod Touch apps. The Apple docs are too wordy and don’t show enough example code. Maybe I just haven’t looked into it far enough though.