Pinderkent

A few weeks ago, Travis Jensen posted an article about the low quality of many enterprise software systems. For those of us who have worked in such environments before, we are well aware of what he is saying. The quality of such software is often quite terrible. And he does nail a big portion of the problem: complexity.

In some domains, there is inherent complexity. We may be working with huge amounts of data, we may have to process many requests simultaneously, we may have to deal with ambiguous or unclear business rules, and so forth. But many projects also introduce unnecessary complexity at the technical level. This seems to be what Travis was focusing on, as he discusses the unnecessarily large software stack he has to work with daily.

We often see these unruly software stacks in Java-based systems. The Java development community has always been far more in favor of frameworks than many other development communities, even when working on implementing similar software systems. And even within the different types of frameworks, we often see three or more competing projects. Given that a huge amount of enterprise-grade software is developed using Java, it's no surprise that we're likely to run into systems making use of several such frameworks.

While these frameworks are often useful in smaller applications that need to be developed very quickly by a small team, they typically lose their benefit in the larger, more complex and longer-term projects we see in the enterprise. Aside from the increased complexity and the debugging difficulties they bring, as mentioned by Travis in his article, the effort necessary to integrate these frameworks often exceeds the benefit they initially bring to the table.

At some companies I've worked with, it was almost seen as a crime within the development team to write any significant amount of code for a project. The emphasis was completely on letting the framework do the work, with the developers only writing light integration code. As anyone who has worked with such frameworks knows, this often isn't possible, due to performance reasons, unusual requirements, or even the deployment environment.

I've witnessed some of these teams spent excessive amounts of time fighting to integrate the various third-party frameworks of their software stack. The code they write to integrate these components ends up being more than what they would've wrote if they'd just rolled it all by hand. And like Travis highlighted in his article, debugging can become a real nightmare. Tracking the original source of a single field displayed on the UI can become a tedious journey through Web template frameworks, the code integrating that with an underlying business layer, through to the code making up a data layer or an ORM framework, and sometimes down to hand-written SQL code. Simple bug fixes can indeed become very time-consuming tasks.

Some of those development groups have moved towards reducing their use of frameworks. If this approach is taken with an existing application, we can end up with a real mess. Some portions of the system will use some ORM or data persistence layer, while others directly invoke SQL code or stored procedures. This takes what may have already been a confusing situation, and makes it far more convoluted.

There's probably no easy solution to these problems. We may just need to accept that for larger and more complex software systems, it's better to invest in developing much of the system from scratch. While frameworks may give an initial advantage, for longer-term projects they become a nuisance, and even an additional problem to work around when modifying the system or adding new functionality. And given that many of these frameworks manage to work their way into the overall architecture in a manner that does not easily allow them to be removed later, it may be best just to not get involved with them in the first place.

It's no secret that PC hardware is continually getting faster. For most people, this is a very welcome improvement. Unfortunately, we seem to have a bad habit of throwing yet another layer of software into the mix to negate any such performance improvements. While this made sense in the past, as it often brought a huge increase in productivity, it hasn't made as much sense lately.

One such example of this was the move towards virtual machines, such as those provided by Java and .NET. Aside from a small number of microbenchmarks suggesting otherwise, the performance of real-world applications developed using these technologies took a hit relative to equivalent apps developed using languages like C or C++. But at least such environments provided a boost to programmer productivity. In most cases, this was enough to offset the slower runtime performance of the applications themselves.

Unfortunately, the Web browser has become a similar execution environment. But unlike Java and .NET, it was designed as a tool to retrieve and display hyperlinked documents. Only later was programmability tacked on haphazardly, in the form of JavaScript. This has brought a whole host of problems, ranging from security issues to extremely poor performance to the locking up of the browser itself in some cases.

We have recently seen the canvas element of HTML 5 becoming more prevalent due to it now being supported by the Safari, Firefox, Chrome and Opera Web browsers. It essentially provides a rendering surface that is accessible via JavaScript.

While this increases what can be done with the Web browser, it's clear that this functionality will be misused. One such example of this is Bespin. It claims to be "an open, extensible web-based framework for code editing that aims to increase developer productivity, enable compelling user experiences, and promote the use of open standards."

One part of Bespin is a text editor. Ben Galbraith has recently written about how the canvas element was used to develop this functionality. He does address the performance of this editor by saying, "for large classes of users, performance is really, really good."

On one hand, we shouldn't expect there to be any performance problems. Text editors are not overly complicated, nor are they particularly resource-intensive applications. We've had literally decades of experience writing text editors. Even more advanced features, like syntax highlighting, could be performed suitably fast on computers well over a decade ago. If there were any performance problems implementing a basic text editor, it would be because something has been done very, very incorrectly.

On the other hand, his description of its implementation goes a long way towards showing the wasteful overhead that is required to run such a simple application. Let's suppose we're running Bespin in Firefox on a Linux system. At the highest level, we have the JavaScript code that makes up the logic of the Bespin editor. This JavaScript code in turn is interpreted by Gecko, which also handles the manipulation of the canvas element. Since we're running Firefox on Linux, Gecko is likely using GTK+ to provide the basic UI functionality. GTK+ in turn is calling down to GDK and GLib. GDK and GLib call down to the C standard library, Xlib, other X11 libraries, possibly Cairo, and so on. The X server makes use of the C standard library, video drivers, and the Linux kernel. Finally, the Linux kernel itself accesses the actual hardware.

Only the lowest portion of that stack really provides unique functionality. When it comes to text editors, by the time we get to X we're just repeating what we had at a lower level, albeit with a slightly different API or slightly more expanded functionality. Vim, for instance, is a very full-featured command-line editor, including syntax highlighting, advanced searching, and the ability to invoke makefiles. You get more than you would with Bespin, with a fraction of the resources being needed and used.

It's clear that we've gotten stuck in a cycle of waste. We're now trying to recreate what we already had at a lower level, except with much worse performance, greater memory and CPU usage, and less functionality. And while we've seen some stunning hardware improvements over the past couple of decades, we've done a good job of eliminating them through this repetitious software reimplementation. Unfortunately, we really haven't gotten much benefit from these extra layers. For example, all we get with Bespin is a text editor that pales in comparison to Vim.

I wonder if the next step backwards will be the implementation of an HTML rendering engine written in JavaScript and targeting the HTML 5 canvas element. Running a Web browser in a Web browser may unfortunately be the next step we take to further waste our computing resources.

Frameworks have become a familiar sight within the field of software development. They exist for virtually every language, every platform, and for all sorts of domains. One area where frameworks have really taken hold is that of Web development. Some people are very much in favor of using frameworks. Others suggest that frameworks are not such a good thing. In the end, it ends up being a mix of both viewpoints.

Frameworks prove to be quite useful when initially developing a Web site. In many cases, it is unknown how much interest there will be in the site. So it proves to be very helpful when one can use a framework to help quickly get a basic version of the site functional and deployed. The effort of initially developing the Web site is minimized to better account for the greater risk. This becomes even more important if the site is a business venture, and funding is limited.

If all goes well, the Web site will become popular. This is often the point when the framework that initially was very useful starts to become a burden. One of the first problems that developers typically encounter is the need to add highly-specialized functionality to their site, only run head-first into the framework. What they want to do doesn't mesh well with the philosophy of the framework. Unfortunately, the only course of action is to either circumvent what the framework tries to enforce, or twist the implementation of the feature in a way that'll fit with the framework. Either way, this usually results in numerous hacks, which in turn lead to code that is far more difficult to debug and maintain.

If the site is really lucky, performance will start to become a problem. This is one area where frameworks really start to become a hassle. Depending on the framework and its architecture, it may not scale very well. This is especially true for frameworks that integrate with a database system of some sort. We often run into a situation where the database access, often in the case of automatically-generated queries, is suboptimal, if not outright horrid. Many times the fixes are easy, but require some custom SQL code to be written, for instance. Soon, the developers may need to be continually avoiding the framework in order to maintain a suitable, or even just reasonable, level of performance.

By the time an application or system has become more advanced and mature, we often see a trend towards moving away from whatever framework was originally used. Sometimes this is easy to do, but usually it is not. The framework has become tightly woven with the software system, typically at the most fundamental levels, including user and session management. Any attempt to rip out and replace that framework-provided functionality will quickly cascade throughout the rest of the system. The scope of the necessary changes soon becomes too great, to the point of being too risky and requiring far too many resources to adequately test.

This is a trend I have seen for years now, regardless of the programming language or environment used. Frameworks really started to become widely used as Java gained in popularity, so this is where we see a lot of these problems arising. For many organizations in the aforementioned position, they really have a limited set of choices. On one hand, they need to continually develop their software to remain competitive, but this has become difficult to do because the initial use of a framework has contributed to a convoluted, unmaintainable code base. And on the other hand, they don't have the resources to partake in the significant refactoring and rewriting necessary to fix or replace the code base they currently have.

Some have tried to move from one framework to another as a way of solving their problems. One notable example is that of CD Baby, who tried to move to Ruby on Rails from PHP, but ended up falling back on PHP after Rails proved to not be flexible enough.

The best thing to do may be to initially start with a framework, but as soon as the site proves itself viable, every effort should be taken to move away from the framework and towards custom solutions that are crafted to tackle the specific problems at hand. This allows the development team to make the best use of the framework at the start of the project when they really need it, but it also gets it out of their way early on in the lifespan of the code base, to prevent too much other code from depending on it. This compromise may not always be possible, but should likely be chosen if possible.

I saw an article today talking about a "new wave" of activity within the Perl community. Although things have been looking more positive for Perl, it's still far too early to suggest that there is a Perl resurgence in progress.

The most significant progress the Perl community has made lately is that of actually getting some traction behind Parrot and Perl 6. For most of the past decade, we saw little to nothing substantial. There was a lot of talk, but little in the way of tangible software. What was available was not production-ready by any means, which of course is unsuitable for a pragmatic language like Perl.

Pugs, a Perl 6 implementation written in Haskell, was perhaps the most useful of what was produced during what we might come to refer to as the "Perl Dark Ages" of 2001 to 2008. During 2005, it saw relatively active development, and frequent releases. Unfortunately, it lost momentum during 2006, and since then has essentially stalled. A good indication of this is its very homepage, which at the time of this writing (February 18, 2009) still says "© Copyright 2005-2007, The Pugs Contributors."

The announcement last December suggesting that we'd actually see Parrot 1.0 during the first part of 2009 was a good first step. Indeed, it appears as though the March 2009 release of Parrot 1.0 is still on track. This upcoming release will no doubt be highly publicized online, and will likely raise significant awareness of Parrot and Perl 6. Although it isn't mature yet, this release will be an important milestone in lending some legitimacy to both the Parrot team and to the Perl 6 efforts so far.

A month after the aforementioned announcement regarding Parrot 1.0, we got word that the Rakudo implementation of Perl 6 was going to become more independent of the Parrot project. Over the past month, we have seen this happen to some extent.

Without a production-ready implementation of Perl 6, we can't yet say that there is a true resurgence or "new wave" within the Perl community yet. The recent activity surrounding Parrot and Rakudo has helped put the Perl community on the right track. But we likely won't see any true and widespread excitement arise until we have a decent Perl 6 implementation readily available. Only then can module authors start writing code that makes use of Perl 6's new features, which sets the stage for framework authors and application authors to start making use of Perl 6. At that point we'll need to see the Perl community produce something truly spectacular, much like Ruby on Rails was to the Ruby community several years back. Once that happens, we can consider the Perl community to have revitalized itself.

Today I read an article suggesting that 8 GB of RAM is "only just enough" memory for a typical developer's PC running Windows. I'm not here to dispute that claim. Indeed, the RAM usages given in the article are somewhat realistic. Using recent versions of Microsoft's various software and database development tools does require a significant amount of RAM. In many ways, the need for that much RAM is unfortunate and likely quite unnecessary.

In the mid-1990s, I had the pleasure of doing software development on a NeXTstation Color. Unlike the base NeXTstation Color models, which came with 12 MB of RAM, the system I was using had been upgraded at some point to 16 MB of RAM. But by that point, it had been used by several developers before being passed on to me. Yet it was still an extremely powerful development system.

Now, it might be argued that Windows development today isn't much like the software development we did on a NeXT system 15 years ago. In some sense this is true, but for the most part very little has changed. Objective-C is relatively similar to C#. The various NeXTstep kits are fundamentally similar to the .NET standard class library. We had Project Builder for development, as well as Interface Builder for drag-and-drop GUI creation. And although it wasn't available yet when I was developing on that NeXTstation, NeXT would later introduce WebObjects for developing Web applications.

Anyone who has done any Mac OS X development lately has a general idea of what it was like to develop software on a NeXT system. It was an experience quite comparable to software development today, due to how far ahead of its time NeXT and their software offerings were. And although many people today might find it difficult to believe, we actually were able to develop relatively complex business software on a system that had only 16 MB of RAM. To put things into perspective, the article I read today claims that at least 500 times as much RAM is needed today than was needed 15 years ago. But the software we're developing today really isn't much more complex.

It's obvious why this has happened. Since then, the price of RAM has dropped considerably. It has become far cheaper to acquire more RAM than it is to pay a developer to write more efficient software. In fact, we see this reflected throughout the entire software stack. It's no secret that Windows Vista requires a significant amount of memory to run efficiently. The same goes for .NET. Developer tools like Visual Studio are also very memory-intensive. Given the platform and development tool vendor's general disdain towards writing memory-efficient software, it's no surprise that the applications developed by software developers using those tools would also consume more memory than necessary.

One thing to consider is how we now approach a simple task, such as reading and writing data files, versus how it was done 15 years ago. Today, we're likely to see some given data persisted to an XML file on disk. To work with such files, we'll need an XML parser, some way of representing the XML document in memory, custom code to tie everything together, and so forth. Fifteen years ago, we would have used a custom binary format, and written the serialization and deserialization code ourselves. Although it would have taken longer to develop, the overhead in terms of memory usage and CPU usage would have been considerably less with the custom binary format.

But to require over 500 times as much memory today is still quite excessive. I think it just goes to show how little developers today care about minimizing memory usage of their applications. The low cost of RAM no doubt helps our "cause" immensely. Today we can get away with waste that would have been considered intolerable a decade ago. But on the other hand, I think we are shortchanging ourselves to some extent. The memory we consume today, essentially out of waste and carelessness, could no doubt be better used for other purposes. Extra memory put towards even just caching data from disk is typically beneficial to system performance, even in systems with significant amounts of RAM.

Fifteen years ago, I don't think I would have believed it if somebody from now had come and told me that we'd have low-end PCs with 4 GB of RAM, typical development systems with 8 GB, and we'd actually need that much RAM to run contemporary applications. Yet here we are. And I don't even think it'd take that much more care or effort to minimize the amount of RAM we use, or at least bring it down to a more reasonable level. To have any effect, we'd need to see this initiative at all software levels, from the operating system, to any higher-level runtimes like .NET and Java, to our software development tools, and finally to the applications we develop. Unfortunately, I don't see this happening.