Pinderkent

Last night at the pub, a friend and colleague of mine was telling me of a recent experience he had at a company he was doing some IT work for. I think the lesson learned is a very important one, and thus I wish to share it. But first I'll describe the situation he encountered.

In the mid-1990s, the company in question built their IT operations around systems from Sun. They wrote much of their in-house code using C++, and used Oracle for their database needs. On the front-end, they used PCs running a mix of Windows NT 3.51, FreeBSD 2.x, and even OS/2, depending on the department. While that is not a unique setup by any means, what is somewhat unique is that they essentially continued to use those same systems up until 2006.

One of the main reasons why they didn't switch is because their software systems worked just fine, even if the UIs were somewhat archaic. Their software was mature and well-understood by the company's employees. They even got extremely lucky in the first place, as the developers who initially designed and implemented their software systems did so in a way that allowed for the systems to easily scale as the need arose over time.

The hardware proved to be the main instigator of change. After a decade, many of the front-end PCs they were using started to exhibit a variety of physical problems. Some had been replaced earlier, but eventually it was decided to replace them all with newer systems. However, to the best of my friend's knowledge, the back-end Sun systems were working just fine.

However, at the same time they decided to also replace the back-end systems. A variety of consultants were apparently called in to appraise the situation. For whatever reason, it was eventually decided that the new back-end systems would be built around Windows Server 2003 and SQL Server 2005. The new back-end software was to be built upon .NET, while Web-based client-side apps would be developed and used. My friend wasn't sure exactly when this effort started, but he believed it was in early 2006.

By the end of 2006, the consultants and developers deemed the new system ready to go. Over the course of the December 2006 holidays, the new systems were rolled out. It turned out to be a pretty major disaster. The first problem they ran into was a complete lack of performance. As they moved into the first weeks of 2007, their back-end systems just wouldn't scale. As an emergency fix, they ended up throwing more hardware at the problem, which did ease the burden on the existing servers somewhat. But it was in no means a permanent solution.

The front-end software systems proved to be an even bigger disaster. Many of the AJAX-based applications used Internet Explorer-specific functionality. But the IT managers of some of the front-end networks would not allow IE to be used, for security reasons. They only allowed for Firefox to be used. So the Web-based front-end software needed significant modifications right away, as well.

What was perhaps the worst failure involved the in-house users and their productivity. Large portions of the old system were built around a curses-based UI. Although it apparently wasn't very pretty, it did allow for a great deal of user productivity. One of the main complaints about the new Web-based software was that the keyboard support was quite poor, requiring the user to select input fields using the mouse, and at times even having to scroll the page to input or manipulate certain data. With the earlier system, the navigation could rapidly be performed using just the keyboard. Some of the more experienced users were apparently so efficient with the older system that their productivity was reduced to 25% of what it was before the switch.

My friend and his colleagues were called in to try to remedy the situation as best they could. The company was not willing to invest in a completely new system, but instead insisted that the old system be brought to a usable, if not optimal, state. The work is still on-going, as we speak, five months later.

There are many lessons to be learned here. The one my friend emphasized the most was that it's often a good idea to leave existing systems as they are. What they had worked, for the most part. The problems they experienced with their front-end hardware could have been easily dealt with by buying new PC systems. But the decision to replace their working server hardware, and to rewrite their existing (and well-functioning) back-end software, were obviously terrible ones. The use of AJAX and Web-based software for their front-end systems was also a poor idea.

I think some of the other major lessons are as follows:

• Use mature, well-tested, effective software (eg. Solaris, Oracle, FreeBSD).
• Avoid immature fad "technologies" like AJAX.
• Traditional applications offer more flexibility than Web-based applications.
• Always give much consideration to back-end scalability.
• Sometimes a text-based interface is far more efficient than a GUI.
• Get user feedback on software early and often.
• Maintain a reasonable level of heterogeneity, when it comes to software, hardware and vendors.

Hopefully we can all learn from these lessons made obvious by this situation. Although given my years of experience, somehow I think that won't be the case.