Pinderkent

Pain and glory from the trenches of the IT world.

Benchmarking open source software: we can't just focus on the numbers.

Posted on Sunday, January 28, 2007 at 6:24 PM.

My last article was about the performance and memory consumption of the popular open source KDE and GNOME desktops. Well, it seems that that particular article was submitted to Digg. I looked at some of the comments that people posted, and I specifically wanted to address this one comment in particular.

The comment suggested that it is improper to compare desktop environments based on qualitative, rather than quantitative, measures. That attitude is, of course, incorrect. Open source software may not have been adopted as quickly as it could have been because many developers likely do ignore qualitative data, and thus did not get an adequate idea of what the users liked or disliked.

I understand why many developers would feel more comfortable dealing with numerical data. It is, in many cases, a lot easier to work with and analyze than qualitative data. We can easily calculate various statistics and measures from that data. We can plot graphs. But we have to remember that this is of little concern to many computer and software users, especially those who are relatively non-technical.

We can only play the numbers game up to a point. Suppose for a moment that it typically takes a KDE application running on a particular system 0.5 seconds to draw a menu. It takes a GNOME application on the same system 2.0 seconds to perform a similar operation. Windows Vista takes 1.0 seconds. We could go around playing number games all we want, saying things like KDE is four times faster than GNOME when it comes to drawing menus, and GNOME is twice as slow as Windows Vista. We could even say that Windows Vista's menu drawing is twice as slow as KDE's. Regardless, such data and claims are quite meaningless.

What matters is how users find the system to work. And this will likely involve qualitative, rather than quantitative, information. For the aforementioned example, they will say things like "KDE and Windows Vista feel quick", or "GNOME's menu redrawing seems a lot slower than KDE's". Non-technical users, who tend to be the majority of computer users these days, do not sit there with a stopwatch timing how long it takes for a menu to redraw, or for some other action to be performed.

This is something that I think a lot of developers have missed, both those working on open source projects and closed source commercial developments. For the users' experience to be enjoyable, we as developers do need to pay a lot of attention to their qualitative assessments of our software. When they say it feels slow, we damn well better look into what they're talking about, even if our benchmarking data may suggest we're faster than our competitors' systems.

Furthermore, we need to use such data when deciding what to improve. If our menu drawing speed is twice as slow as that of our competitors' systems, but the users in general don't seem to mind, then it's likely not something we should bother improving. There are no doubt other features that we could better put our attention and efforts towards working on.

But I suspect that the person who posted that comment at Digg did not actually read my article, because he or she further requests some "charts and numbers". Anyone who had read my article would have been directed to this page with desktop environment memory usage data. Instead of just providing the numbers, that site also describes the methodology used to obtain such data. So I did provide some numerical data to back up my claims regarding KDE. I invite you to check out that data, and the way it was collected, for yourself. Then you can see how it correlates to my qualitative analysis of the memory usage and responsiveness of GNOME and KDE.

When we as developers are working on software, we need to consider both quantitative benchmark data, and qualitative user analysis. Only when considering both of these data sources can we truly begin to make our applications better, not just in terms of raw performance, but also in terms of satisfying our users.

Permalink: http://pinderkent.phumblog.com/post/2007/01/benchmarking_open_source_software_we_cant_just_focus_on_the_numbers
Share:

What about the performance and memory usage of KDE and GNOME?

Posted on Saturday, January 27, 2007 at 9:46 PM.

Today I read an article comparing KDE and GNOME. It covers a number of criteria, including customization, configurability, ease-of-use, and dependability. But some major criteria I really didn't see much about were those of performance and memory usage.

Let me start off by saying that I have not performed any formal benchmarking of either GNOME nor KDE. I have never really bothered to peruse the source code of either system to any great extent. What I say comes directly from my experience as a user of various versions of both desktop environments.

I have found that KDE feels more responsive than GNOME, when used on the same system. KDE applications like the Konqueror web browser, the KMail mail client, and the Kate text editor feel snappy. On the other hand, GNOME applications like the Epiphany web browser, Balsa email client, and gedit text editor do not feel as quick, and in some cases feel slow or bloated.

One thing I found with GNOME was that the menus of most applications seemed slow to redraw. I'm not completely familiar with how GNOME deals with application menus. I don't know if most applications just use the menuing facilities provided by GTK+, or if GNOME offers some extensions to what GTK+ offers. In any case, with KDE applications, say Konqueror for instance, I can open the "Location" menu, and then move between the other menus rapidly, with the other menus being drawn immediately. Trying this on the same machine with GNOME's Epiphany results in noticeable redrawing times when moving between menus, even when revisiting a menu for the seventh or eighth time.

I really don't know why certain aspects of GNOME's performance and responsiveness are so much poorer than KDE's. I don't have the time or inclination to locate and fix the problems with GNOME or GTK+. Beyond raw performance, I think we need to consider the memory usage of each environment.

A Google search turned up some actual desktop environment memory usage benchmark data for a fairly recent study. This informal study compares KDE, GNOME, XFCE and WindowMaker in a variety of situations and configurations. For what appears to be the most real-world of all the scenarios, listed under the heading of "8. Desktop's applications" in the study, KDE was found to have the lowest memory consumption. In this particular test, KDE and some typical KDE applications were found to consume 143.2 MB of RAM. GNOME and comparable GNOME applications consumed 174.8 MB. XFCE and some third-party applications comparable to the ones used for GNOME and KDE consumed 206.8 MB of RAM.

My own personal experience would mirror those results. I do tend to think that KDE and its accompanying applications are lighter than the equivalents for GNOME. This may be due to the differing levels of integration between the two projects. As shown by its download page, KDE tends to be far more integrated than GNOME. Compare the list of KDE downloads to that of the latest version of GNOME's platform and sources downloads. Keep in mind that the GNOME downloads are listed twice for each application or library, due to both gzip and bzip2 being used to compress the archives. One notable thing is that most of libraries used by many KDE applications are bundled together in the kdelibs package, while many of the GNOME libraries are spread among multiple, separate packages.

From my own development experience, applications and libraries that tend to be more tightly developed together often include less code duplication. This may play a role in the memory usage differences between KDE and GNOME. Many of the individual GNOME libraries may include functionality offered by other, separate GNOME libraries. KDE, with its more integrated applications and support libraries, may better eliminate such duplication. It would be interesting if somebody with more time on their hands looked into this hypothesis.

While the performance and memory consumption of KDE and GNOME are not the only factors to consider when comparing the two desktop environments, I think they are pretty important and must be considered. From my own experience, even minor performance deficiencies or memory usage excesses can have a profound impact on usability. Even on systems with 512 MB or 1 GB of RAM, excessive memory usage can lead to swapping, which on many systems results in hugely degraded performance. And from my personal experience, I think KDE is perhaps better than GNOME in this respect. There is even some experimental data to back up these experience-based findings.

Permalink: http://pinderkent.phumblog.com/post/2007/01/what_about_the_performance_and_memory_usage_of_kde_and_gnome
Share:

Skepticism about the productivity gains of the 3D desktop.

Posted on Thursday, January 18, 2007 at 12:02 AM.

With consumer-grade 3D graphics hardware becoming better on a daily basis, it's no wonder that people have considered the idea of a three dimensional desktop. I read an article today comparing two such desktops, Project Looking Glass and Beryl. While my experience using such systems is fairly limited, I have to express some skepticism about whether or not they bring any productivity gains.

I can see why their flashy effects and capabilities would appeal to some desktop users. Some people may consider it neat to be able to twist a window on an angle. Others may like the idea of multiple desktops being arranged around the sides of a cube. Some of the visual effects are probably entertaining. I just don't see where the productivity gains are, however.

From my experience, I'd have to say that effects like animated or wobbly windows are nothing but distracting. When there's real world to be done, jiggling windows probably won't help in any fashion.

I also don't see how arranging desktops around the sides of a cube really has any positive impact. For many years now a wide variety of X window managers and desktop environments have supported multiple desktops. On my current KDE installation, I have it set to use 16 virtual desktops. That's far more than I could get with a six-sided cube. Another thing is that switching between desktops is virtually instantaneous. There's no having to spin a cube around, or anything of those sorts, when using KDE, for instance.

Aside from putting modern video hardware to use, I really don't see much in the way of benefits from using a 3D desktop environment. The features often don't help one get work or play done more effectively, and in some cases can actually prove to be very distracting and hindering. Thankfully, 2D desktops are a subset of these 3D desktop systems. Even if there is a trend towards using such environments, those of us who much prefer 2D desktops should still be able to use them.

Permalink: http://pinderkent.phumblog.com/post/2007/01/skepticism_about_the_productivity_gains_of_the_3d_desktop
Share:

I like CDE, too.

Posted on Sunday, January 14, 2007 at 10:26 PM.

I have worked with various UNIX systems many times over the years. One of the most prevalent systems in industry has been Sun's Solaris. While I tend to use FreeBSD and KDE these days, I saw a post at Slashdot that talked of Solaris and CDE that I wanted to comment on.

I know many people who consider CDE to be too "ancient" to use, and I think it's a shame that they feel that way. No, CDE does not offer all of the modern features of desktops like GNOME, KDE and even XFCE. But I think that may be one of its strengths today. What we end up with is a desktop environment that is very lean, and not nearly as cluttered as many of today's environments.

To me, it's a great thing that Sun still ships CDE, even with Solaris 10. For some of us, it's a reminder of days long past. But when it comes to just plain getting work done, CDE is often a great environment. The distractions are minimal, and it'll even run well on older systems that are considered far too insufficient to run KDE or GNOME.

So in closing, I have to say that I hope Sun continues to ship CDE for a long time to come. Even today, when I do use Solaris, I tend to choose CDE when logging in. Part of the reason is that I don't care much for their GNOME-based Java Desktop System, but the other part, I must admit, is that I just happen to like using CDE.

Permalink: http://pinderkent.phumblog.com/post/2007/01/i_like_cde_too
Share:

NetBSD: An alternative to Xubuntu and Ubuntu Lite for machines with low specs.

Posted on Thursday, January 11, 2007 at 6:10 PM.

I know a number of people who advocate the use of Xubuntu on resource-constrained desktop systems. But there has recently been talk about how the resource usage of Xubuntu is growing at a rapid pace. Along these lines, today I read an article that considers the use of Ubuntu Lite as an alternative to Xubuntu for those with low spec computers. I haven't used Ubuntu Lite myself, but I would like to make a recommendation of an open source UNIX-like system for low-end desktop computers: NetBSD.

A quote from the NetBSD project's home page really makes clear what it is, and what its goals are: NetBSD is a free, secure, and highly portable Unix-like Open Source operating system available for many platforms, from 64-bit Opteron machines and desktop systems to handheld and embedded devices. Its clean design and advanced features make it excellent in both production and research environments, and it is user-supported with complete source. Many applications are easily available through pkgsrc, the NetBSD Packages Collection.

The goals of the NetBSD project result in NetBSD being a very high-quality system. The support for low-end platforms means that care must be taken to ensure that the kernel and userland tools and libraries use a minimal amount of system resources. Furthermore, the focus on supporting modern platforms means that NetBSD offers the capabilities one would expect from a modern UNIX-like system. That's why I think it can be used as an alternative to systems like Xubunt and Ubuntu Lite.

And when we consider its long history, dating back to when PCs were a mere fraction of the power of PCs from even a decade ago, we can see why it'd make a good choice. Compared to a 386 system, even a 300 MHz AMD K6-2 system is a powerhorse. A system like NetBSD, that even today runs suitably on a 386 or 486, is very usable on a more powerful computer. And with people considering a 1 GHz system to be "low-end" today, NetBSD makes an excellent workstation OS for such a computer.

A major feature of Debian and Ubuntu-derived Linux distributions is the extensive and friendly package management. NetBSD is quite comparable in this field. Its packages collection, pkgsrc, is very capable, complete, and up-to-date. It offers virtually all of the open source software one would expect or ever want. Support is included for all of the major open source desktops, including GNOME, KDE and XFCE.

Furthermore, i386 NetBSD offers excellent binary emulation support. This means that if you have a Linux application that is only available in binary form, you likely will be able to run it on NetBSD. In addition, NetBSD also can run binaries from a number of other systems, including x86 FreeBSD and x86 Solaris.

NetBSD is a truly remarkable and versatile system. And for many people, I think it would make a great alternative to lightweight Linux distributions like Xubuntu and Ubuntu Lite. The very philosophy of the project, that being widespread portability, will no doubt go a long way towards ensuring it remains a modern system that consumes minimal resources. If you're currently a user of a minimalistic Linux distribution that you think is beginning to get bloated, maybe you should give NetBSD a try. It may just be exactly what you're looking for.

Permalink: http://pinderkent.phumblog.com/post/2007/01/netbsd_an_alternative_to_xubuntu_and_ubuntu_lite_for_machines_with_low_specs
Share:
Feeds
  • RSS 2.0 Feed
  • Atom 2.0 Feed
Tags
Archives