Pinderkent

Today I read some very true observations from Peter Harkins about the nature of aging databases. He mentions three such observations in his article, and other readers have contributed several more in the comments for that article, as well as at other sites like Hacker News. I'd like to specifically focus on his second observation, which he entitles "All Relationships Become Many-to-Many."

This is quite true. I must admit, I have seen this happen with numerous different databases in a wide variety of domains. The mere fact that it arises so often should make us reconsider how we think of certain relationships. What we often find is that real-world considerations do force a large number of relationships to become many-to-many. Although the specification indicated that wouldn't be the case, specs are often incomplete and sometimes are just plain wrong.

One option that should be considered is treating all relationships as many-to-many. Although it brings in a level of complexity, it can help avoid the after-the-fact database-level hacks that are often necessary to allow for such relationships to be stored. Arguably, the hacks are worse than the complexity brought in by always dealing with many-to-many relationships.

If our data layer treats all relationships as many-to-many, we can take advantage of this in our business logic. Here we can reduce the multiplicity of the relationships, if necessary. But if later on we find that we do need to support many-to-many relationships once we start dealing with realistic data, we will probably be in a better position to make such changes. Instead of cascading up from the database through to the UI, we're only making changes at the topmost layers of an application if the database already supports many-to-many relationships.

The more we fight with reality, the more we will run into problems like this. So perhaps assuming the "worst", that is to say that all relationships are many-to-many, is the best course of action. In will be setting us up to fail in the softest way possible, with changes happening in volatile business layer logic, rather than in the structure of our database tables.

Luke Plant has been working on a Haskell-based blogging application for a while now, and describing his progress. I was dismayed to read today that he's giving up!

One of his main complaints was with regards to the poor state of the available Haskell database interfacing libraries. He does mention Takusen, which a colleague of mine has been using for a personal project of his own. But Takusen doesn't appear useful to Luke, due to a lack of a proper MySQL backend.

It does, however, offer a SQLite backend. I think it may be more acceptable for the type of project that Luke is working on. One of the main constraints here is the limited capabilities of the Web host being used. This is where SQLite excels. It stores its database in a single file, and doesn't require a server. So one still gets the benefits of a fairly mature relational database, without the hassle of dealing with a separate server process.

Today I shared the experience a friend had with one of the companies he works with. It involved a failed transition from what was mainly a console-based suite of applications powered by Sun systems and Oracle, to an AJAX and Web-based system running on Windows, .NET and SQL Server. It seems that that entry was submitted to Reddit, and so I've been reading some of the comments that were left there.

Some of the comments were quite insightful. But there were two that I found to be just plain funny! Here is the first comment, and here is the second comment.

In those comments I am accused of being "biased against windows and ajax" and guilty of "discrimination against Windows & dot-Net". Well, I would like to address those allegations!

First of all, I've been doing this long enough to not get to attached to certain products or technologies. In the end, it's all about solutions that work to solve the client's problems in an efficient and financially-sound way. I'm glad that I get to mostly work with Solaris, HP-UX, AiX, FreeBSD, Linux and other UNIX-like systems. But I've worked with Windows many times before. As long as the job gets done properly, I really don't care what software is being used.

In this case, my feelings regarding Windows, .NET and SQL Server have absolutely no impact on the problems that that company experienced with their transition. Beyond hearing about it from my friend, I had no involvement with the development of the old system, nor any involvement with the development of the new system.

Furthermore, the fact remains that the old, UNIX- and Oracle-based system worked just fine. The new .NET-based system did not work in a suitable manner. Pointing out that certain software did not perform in an adequate fashion in a certain situation does not indicate bias. All it indicates is that the software in question was not capable of performing what needed to be done.

Likewise, the AJAX-based UI proved inefficient compared to the previous curses-based interface. That's just how things worked out. Again, my feelings, thoughts or biases would have had absolutely no impact on the situation, as I was not involved.

I do thank the two comment authors for their thoughts and opinions. However, I also urge them to be more careful in the future when accusing others of bias. Just because a particular technology fails to work in a particular situation, and this failure becomes a topic of discussion, there is not necessarily bias against that technology. Sometimes technology fails. We must admit this, and learn from such failures.

Last night at the pub, a friend and colleague of mine was telling me of a recent experience he had at a company he was doing some IT work for. I think the lesson learned is a very important one, and thus I wish to share it. But first I'll describe the situation he encountered.

In the mid-1990s, the company in question built their IT operations around systems from Sun. They wrote much of their in-house code using C++, and used Oracle for their database needs. On the front-end, they used PCs running a mix of Windows NT 3.51, FreeBSD 2.x, and even OS/2, depending on the department. While that is not a unique setup by any means, what is somewhat unique is that they essentially continued to use those same systems up until 2006.

One of the main reasons why they didn't switch is because their software systems worked just fine, even if the UIs were somewhat archaic. Their software was mature and well-understood by the company's employees. They even got extremely lucky in the first place, as the developers who initially designed and implemented their software systems did so in a way that allowed for the systems to easily scale as the need arose over time.

The hardware proved to be the main instigator of change. After a decade, many of the front-end PCs they were using started to exhibit a variety of physical problems. Some had been replaced earlier, but eventually it was decided to replace them all with newer systems. However, to the best of my friend's knowledge, the back-end Sun systems were working just fine.

However, at the same time they decided to also replace the back-end systems. A variety of consultants were apparently called in to appraise the situation. For whatever reason, it was eventually decided that the new back-end systems would be built around Windows Server 2003 and SQL Server 2005. The new back-end software was to be built upon .NET, while Web-based client-side apps would be developed and used. My friend wasn't sure exactly when this effort started, but he believed it was in early 2006.

By the end of 2006, the consultants and developers deemed the new system ready to go. Over the course of the December 2006 holidays, the new systems were rolled out. It turned out to be a pretty major disaster. The first problem they ran into was a complete lack of performance. As they moved into the first weeks of 2007, their back-end systems just wouldn't scale. As an emergency fix, they ended up throwing more hardware at the problem, which did ease the burden on the existing servers somewhat. But it was in no means a permanent solution.

The front-end software systems proved to be an even bigger disaster. Many of the AJAX-based applications used Internet Explorer-specific functionality. But the IT managers of some of the front-end networks would not allow IE to be used, for security reasons. They only allowed for Firefox to be used. So the Web-based front-end software needed significant modifications right away, as well.

What was perhaps the worst failure involved the in-house users and their productivity. Large portions of the old system were built around a curses-based UI. Although it apparently wasn't very pretty, it did allow for a great deal of user productivity. One of the main complaints about the new Web-based software was that the keyboard support was quite poor, requiring the user to select input fields using the mouse, and at times even having to scroll the page to input or manipulate certain data. With the earlier system, the navigation could rapidly be performed using just the keyboard. Some of the more experienced users were apparently so efficient with the older system that their productivity was reduced to 25% of what it was before the switch.

My friend and his colleagues were called in to try to remedy the situation as best they could. The company was not willing to invest in a completely new system, but instead insisted that the old system be brought to a usable, if not optimal, state. The work is still on-going, as we speak, five months later.

There are many lessons to be learned here. The one my friend emphasized the most was that it's often a good idea to leave existing systems as they are. What they had worked, for the most part. The problems they experienced with their front-end hardware could have been easily dealt with by buying new PC systems. But the decision to replace their working server hardware, and to rewrite their existing (and well-functioning) back-end software, were obviously terrible ones. The use of AJAX and Web-based software for their front-end systems was also a poor idea.

I think some of the other major lessons are as follows:

• Use mature, well-tested, effective software (eg. Solaris, Oracle, FreeBSD).
• Avoid immature fad "technologies" like AJAX.
• Traditional applications offer more flexibility than Web-based applications.
• Always give much consideration to back-end scalability.
• Sometimes a text-based interface is far more efficient than a GUI.
• Get user feedback on software early and often.
• Maintain a reasonable level of heterogeneity, when it comes to software, hardware and vendors.

Hopefully we can all learn from these lessons made obvious by this situation. Although given my years of experience, somehow I think that won't be the case.

As seems to have happened every year around this time for a number of years now, we have some people writing that this upcoming year will not be a good year for Linux and the open source community, and others suggesting it will be a year of major growth.

For an industry that as a whole moves extremely rapidly, I'd have to say from my experience that major technology transitions are often relatively slow. In the mid-to-late 1980s, when commercial UNIX was really starting to become widely used, there were many firms still running VMS on their VAX systems. Many others were still using systems from IBM that were purchased in the 1970s. Even into the 1990s, I worked with a number of companies that still heavily employed VMS.

I think we're still seeing such slow transitions even today. Most companies, governments and other organizations with a high degree of foresight have seen that open source software is likely the best way to go in the future. I'd have to say that they're correct. In general, the open source-based solutions I have been involved with deploying have gone far better than projects based around closed source software. There are a wide variety of factors at play here. I'd have to say that quality, cost (both installation and maintenance), and increased productivity are among the major reasons for success. And people are realizing that this is the case. We are seeing open source solutions being deployed quite often these days, even within firms where closed source solutions have been heavily entrenched.

But we shouldn't expect this transition to happen overnight. It will take time, even for those of us who have been waiting a decade, if not more. Many technical people are still ignorant about open source, for instance. They haven't yet bothered to learn about the benefits of open source systems, instead just sticking with the existing closed source operating systems and software that they are familiar with. While some of these people will soon enrich their knowledge of the software landscape, many won't.

However, I think open source systems will eventually gain a lot of momentum. Apache HTTP Server is already the most widely used web server, for instance. Sun's Java implementation has been open sourced. I know a number of companies and organizations who have opted to go with PostgreSQL over commercial offerings for their database needs. Firefox is rapidly becoming a favourite tool for use with in-house web applications. Scripting languages like Python and Ruby are really coming into their own. Much change is happening, there's no doubt about that.

I don't think we'll see any year really be considered the "year of open source" or the "year of Linux", just because the transition will be relatively gradual. Soon enough we will just find ourselves using far more open source software than closed source. It will just have happened over such a long period of time that we won't be able to pinpoint a specific year as being when everything just changed all of a sudden. Major transitions within the computing world, especially within IT, tend to be rather slow-moving, rather than abrupt.