Just a quick intro to whom you are dealing with here.

• student at the University of Twente in The Netherlands
• working voluntarily as a system administrator at study association Inter-Actief for at least four years now.

I dare say I’ve seen quit a few things that can go wrong (very badly wrong too) and the most weird incompatibilities between even relatively similar platforms like FreeBSD and Debian.
Though I’m sure that there’s room for quite the improvement in our network and setup one of the things I find we’ve done quite well at Inter-Actief is our configuration management.

I’ve worked for a few companies in the IT branch over here in this kingdom, ranging from tiny startups to established names in their branch. Though those companies often have complete systems in place for documentation, versioning of code and any other expensive solution you can imagine it seems impossibly hard to find a company that applies that same idea to configuration management.

The idea

SCM (software configuration management) or SCCM (software configuration and change management) is:

• a method for organising,
• maintaining
• and tracking

all kinds of configuration changes, from updates pushed to clients to web-server configuration to pushing settings to a mobile device.

Though it can be extremely complex and very powerful to have such a solution in place most companies don’t see the need for it, don’t understand the need for it or just don’t want to spend yet another couple thousand euro’s on yet another toy for the tech-guys.

Another quite valid point made by Fred was that SCCM makes changes, every change, from top to bottom traceable, which some companies might not be all to comfortable with, it makes it easier to blame someone for a mistake too (and the manager who ordered the change).

I’ve personally encountered quite a few situations where for numerous reasons changes were made to a corporate environment which after being live for a while turned out they had broken other features of the corporate intranet or even exposed parts of it to the outside world.

Such errors happen and the consequences can be quite nasty for a company, for the poor guy or gal who made the changes. But more important in such a situation is the ability to respond rapidly, to revert the changes and then asses what went wrong and try again.

This is basically what I will try to focus in in the coming ramblings about SCM, why use SCM, how to use it, when to use it and a simple way to implement is.

Mindset

The main problem with most companies is that they don’t see the need for SCM to begin with. Because, really, we’ve got our IT-guys in the basement, they’ve configured everything, it works and because we are such an awesome company and can see in the future we’ve set up an environment which will last for at least 10 years.

Time to wake up here, that dream, ain’t gonna happen. Just as the market and the world changes, you adapt to it, and so will your environment because it will need to meet new demands by your workers. Long story short, an IT-environment is as dynamic as the company itself and just as it is important to know which changes are made to products, to visions, to a direction you’re taking it is important to be able to see which changes have been made to your IT infrastructure, what triggered them, how it has grown and possibly what went wrong where and learn that lesson.

Just to put everything into place, let’s start with a few usecases, shall we.

Welcome to Peapl, a company specialising in educational software and social engineering solutions. It has about a hundred servers ranging from content-servers, to their corporate intra- and extranet, mail-servers, a few PBX’es and so on.

Recently the company has started to use IPv6 internally as a test-phase and some changes were made to the webserver configuration to accomodate that. Though everything seems fine, a few days later someones discovers that a part of the intranet is behaving oddly and, total disaster the intranet is accessible without having to provide authentication credentials.

In such a case, the first thing you want to do is

• roll back to the ‘last good known working configuration’ and after that has been handled
• start looking for what caused the problem.

But… oh dear, where did the old configuration files go? Didn’t you create a backup? Didn’t you copy the old configuration file to the same dir in case something went wrong? Of course you did but when all seemed to be working these reserve copies were removed…

Such situations are far more common than you might think. Especially when you’re working with multiple people on some configuration changes in a big environment, a simple miscommunication can be enough to cause quite the trouble.

Another case… You’ve just tested an update in you test-environment and have made the necessary changes to the production environment to push those changes to all clients involved, say a Multivers client update to the financial department.

Next morning, their manager calls, complete disaster, the update failed, some licensing issue, now the financial department is in lockdown and they can’t do anything. Oh shi-.

In that case, it would be extremely useful if with just a few commands you could:

• revert the update
• get the old configuration for that application back
• and just tell that poor manager that the update has been undone, if everyone would be so kind to reboot, the problem will be fixed.

More importantly, it gives you the possibility to see what was changed and how that could’ve caused this problem.

Another example, but now the other way around. What better way to introduce someone to your network, explain how it works and how it has changed than having access to a history of your network configuration and changes through your SCM? Just dig in, map it out and you might actually be able to find some problems before they even occur just by connecting the dots and thinking about each consequence a change brings with it.

Sounds good, doesn’t it?

Ah, but there’s a catch. As I said earlier, most companies don’t see the need for such a solution. SCM requires a certain mindset. SCM is not some for of wicka which can do all this automagically, it doesn’t require a robe and wizard’s hat. It does however require a certain precision and the habit of documenting your changes and making sure that changes are put into your SCM solution.

Unfortunately, SCM is most often deployed as a counter measure, when it’s gone wrong a few times or it’s become clear that changes are so difficult to implement because no one actually really knows how everything works anymore and is afraid to touch the system because it just might blow up. SCM is then often forced upon a department, which makes it that more irritating because suddenly you have to accomodate to this new SCM thing and change your working habits to suit it.

Starting with SCM

The best way to start with SCM, in my opinion, is to start small. Even though it might seem completely useless in a small environment it’s the habit that matters. Getting that routine in your head that when you make changes, you document, immediately and completely. Already having such a system in place makes it a lot easier for new people to get started with that routine than when it is later on forced upon everyone from the top.

Personally I’m a bit of an over-achiever when it comes to SCM. I track anything and everything, from our servers at Inter-/Actief/ to my own server in Amsterdam, my gateway at home, even from my phone I can pull up a complete history of changes. And the weird thing is, I actually get frustrated when a SCM solution is not available, because my routine is so used to it.

So, how do we do it, this great SCM thing?

Well for starters, we need to find a solution that

• implements SCM in a way we like
• which suits our network and method of work
• preferably doesn’t cost a lot
• is relatively simple to implement in our current infrastructure
• works with all the different platforms involved

Good luck finding one… we couldn’t. The problem with pre-made SCM solutions is that they often target a certain type of network or platform. Microsoft has it’s big SCCM but try to get that to work with a Unix machine. Similarly, most providers of SCM solutions assume a certain network topology, platform prevalence or want to sell you a complete solution integrating with other products of theirs…

For those exact reasons we decided to create our own SCM. But instead of writing it from scratch we decided to use something freely available, a version control system. It gives us all the nice features of being able to track configuration changes, get full histories and diff’s of those changes, mapping it out, seeing relations between a configuration file and another and since al major version control systems support hooks it gives us the ability to do all kinds of things once we make changes to a file, from logging it to a central facility to getting all clients involved to pull a certain change-set to them to update their configuration with.

(to be continued…)