Pinderkent

My current work involves working on some Web applications developed using Django. Although I've used Python much in the past, my experience with Django was quite limited. So I recently did some research to become more proficient with it, and will list below some of the Web pages that I found provided the most useful tips and tricks for when using Django.

• Some Django tips: Although an older article, it also makes some good non-Django suggestions, like installing IPython and ensuring your project has a test suite.
• Small Django tips from one newbie to another: Another older article, this one also emphasizes the need for unit testing, and gives some examples (with code) about how to go about this. It also discusses ways to manage frequent model changes during development.
• Usefull tips to start a new project with Django: A slightly dated article that summarizes how to get started with Django, and well as some suggestions for when deploying a production Django application.
• Django Tips: UTF-8, ASCII Encoding Errors, Urllib2, and MySQL: Gives useful tips about handling UTF-8 encoded strings. Although the project I'm working on thankfully didn't make the mistake of using MySQL, this article does include some tips relating to string encoding and MySQL, which may be useful for some people.
• Big list of Django tips (and some python tips too): This offers perhaps the greatest quantity of Django tips in a single page. It's quite complete, covering areas such as deployment, configuration, templating and views, the model, testing, and so forth.
• Tips for Scaling a Web App: While not completely Django-specific, it lists some good ideas for how to develop a database-backed Web application that scales well.
• Django Tips: PIL, ImageField and Unit Tests: Gives some time-saving suggestions about using the Python Imaging Library with Django and unit tests.
• Django Image Uploading: Tips and Tricks: Outlines how to upload images in Django apps, with some suggestions about how to solve some common problems.
• 10 Insanely Useful Django Tips: I think the title of this article overhypes it somewhat. The tips are useful, but they are somewhat common-sense tips, as well. Although I haven't tried it yet, this article did point me to django-debug-toolbar, which sounds like it might be useful.
• 'Practical' tips for working with Django: Includes some suggestions regarding developing custom managers, wrapping generic views, and converting text to HTML before rendering the template.
• Debugging Django: One of the more detailed articles I found suggesting some strategies for debugging Django applications.
• Django development tips: Some ideas for setting up a long-running Django development server in a UNIX-like environment using GNU Screen. More advanced users of UNIX-like systems are probably familiar with this technique, but this article is still a useful reference and tutorial for newer users.
• Django Tips - Unique Date Querysets: A quick suggestion about how to get all of the unique years and months for a data set such as the posts in a blog, or other timestamped data.
• Favorite Django Tips & Features: This thread from Stack Overflow contains a variety of user-contributed tips. Some of them suggest software to use in conjunction with Django, including Jinja2.
• Tips to keep your Django/mod_python memory usage down: Some deployment and configuration suggestions to reduce Django's memory usage when using Apache and mod_python.
• Django Doctest Tips: Some tips for testing Django applications using doctest. Suggests better ways to locate failures, to use conditionals, to check context variables, and to check content type relations.
• djangotips on Twitter: I didn't find the quality of these user-contributed tips as good as those from Stack Overflow or the other pages, but there were a few that seemed like they might be useful.
• Django Tips, Vol. 1: Contains five tips covering topics like the difference between 'blank' and 'null', displaying multiple fields on the same line in the admin, and so on.
• Django cheat sheet: Although this is a cheat sheet, and not really a Web page, this is one of the cleaner cheat sheets that I've seen. I've found it to be a useful reference so far.
• Django performance tips: This article is also older, but many of its suggestions are very sensible and apply even when not using Django, such as using separate database and Web servers if possible, using PostgreSQL, and putting as much RAM as possible into the servers.

Of course, those are just a small sample of the many useful Django resources out there. But for those new to Django, reading through the articles about may help avoid some common pitfalls, as well as offer ideas to help become more productive while getting accustomed to Django.

When unit testing is used in conjunction with a dynamic programming language, it's typical to see many unit tests whose sole purpose is to test for errors that a language employing static typing would have caught at compile time. Justin Etheredge recently discussed this, and it's something that I have written about in the past in my "Unit testing is not a substitute for static typing" article.

This isn't necessarily an argument about whether statically-typed or dynamically-typed languages are better. There is no single answer to that argument, as what's best really depends very much on the situation at hand. When we want to write software really quickly and are willing to accept a low degree of quality, then dynamic languages are quite suitable. A good example of this is a system administrator writing a short Perl script to scan through certain log files, for instance. Chances are a reasonably competent administrator will be able to write such a script with no significant errors. But once we start moving beyond that scale, we need the help of a compiler, and often static typing.

While proponents of dynamic languages are often quick to point out that such languages allow for more rapid development, they often neglect to see the greater picture. Initially writing code is a very small portion of the lifetime of a typical software product. This is especially true as the software systems get larger and more complex. Significantly more time is often spent testing the software initially, testing for regressions as changes are made to other parts of the system, and debugging user-discovered problems after it has been deployed.

Automated unit tests have become a popular way of reducing this post-initial-development burden. While using a statically-typed (and often, but not always, compiled) language, such tests are usually about testing the actual functionality of the software. More trivial checks, such as whether we're assigning textual strings to variables we expect to be purely numeric, are left to the compiler to perform. And this makes perfect sense; developers shouldn't be wasting their time writing unit tests to essentially perform checks that a compiler could do far more thoroughly and efficiently.

There have been numerous times now when I've had to work with larger software systems developed using languages like Ruby or Python. Thankfully, there have been extensive unit test suites available in the majority of those cases. But the value of those test suites is diminished by the numerous tests that would be rendered immediately unnecessary by strong, static typing. Whatever time the developers might have saved by using the dynamic languages ended up being used instead to write trivial unit tests.

This isn't to suggest that we shouldn't use dynamic languages. Like I mentioned earlier, they're often practical and suitable for very small scripts and applications. But when we're writing serious software that's expected to perform reliably, it's in everyone's best interest to use a more static language. That way the developers don't have to waste their time writing unit tests that essentially test for typos and minor programming mistakes, rather than testing the functionality of the software. Likewise, the testers don't have to file numerous bug reports about the inevitable mistakes that the programmers made, but which were missed by their unit tests. And most importantly, the end-users don't have to fall victim to typing errors that both the developers and the testers missed. In short, it just makes more sense to use static languages.

Recently, I've been working with a software development team that makes use of peer code review. They've opted to take a relatively lightweight approach, with one or two of the other developers reviewing each commit. This process has apparently worked quite well for them over several projects now. The main benefit is higher-quality software. Another benefit is that more of the undocumented knowledge about the software is spread among more of the developers. In general, their communication is much better than that of many other development teams I have worked with. I think their peer review process has helped with this.

Although they don't really adhere to the practices of test-driven development, they do also make use of a large number of automated unit tests written using a custom framework they developed in-house. Part of their development process requires that every commit include new or updated unit tests, where necessary. Another requirement of their review process is that all of the unit tests must run successfully before the commit is made, and this must be demonstrated to the reviewer.

Unfortunately, they seemed to put little to no emphasis on reviewing the actual unit test code. The reviewers would typically put much emphasis on reviewing and suggesting changes to the application code, but as long as the unit tests all passed, the code changes there were generally ignored. While ignoring the unit tests did speed up the review process, it left the team vulnerable to bad unit tests.

Early last week, one of their more important customers called in with a problem they were having with the software. For this team, it was actually quite rare for this to happen. So they immediately halted their development efforts, to focus the team on finding and solving this particular customer's problem. After some initial difficulties reproducing it, they soon enough found the case. It ended up being a relatively benign problem, and was quickly fixed.

Unfortunately, several of the developers spent an entire day investigating this problem. For a small company, waste of this sort is very detrimental. But what made it worse, however, is that an automated unit test had been written in the past to detect this very problem. Checking their code repo's logs showed that a critical assertion had been commented out of this unit test during a bug fix. Their review logs showed that the code changes for the fix had been reviewed by a couple of other developers, including the team lead. But they'd neglected to review the unit test code change, and didn't notice there was a problem with the unit test because all of the tests still ran without error.

If they had reviewed the unit test changes as part of the bug fix code review, I have no doubt that they would have caught the issue immediately. Viewing a diff of the changes made it obvious why it happened. The developer had been commenting out certain other assertions and replacing them with new assertions. But apparently this developer had done a search-and-replace on the source code file that ended up accidentally commenting out the unrelated, yet textually similar, assertion.

So I think the lesson we can take from their troubles is that automated unit tests are just as essential as application code. When it comes to peer code reviews, it's often worthwhile to put as much emphasis towards reviewing such unit test code as is put towards the app's code itself. Indeed, their peer code review policy has been updated to include mandatory reviews of any unit test changes. And although this will cause the reviews to take a little bit more time and effort, catching even one incorrectly commented line of unit test code, for instance, could save them many hours of development time, as well as prevent inconvenience to the users of their software. That will likely make it well worth the extra effort.

It's a mistake to think that unit testing is only about preventing bugs. As such, it's also a mistake to measure the usefulness of writing unit tests in terms of the number of bugs found. Unit tests are a preventative measure, both when initially writing a piece of code for the first time, and for ensuring that future changes have not broken it in some way.

Of course, I'm not the first to point this out. But it's unfortunately a concept that is missed by many in the software development field. Many such developers have difficulty thinking about the "intangible" aspects of automated unit tests. All they're immediately aware of is the time and effort it took to implement the unit tests. But they don't realize how the act of developing the unit tests made them consider to a greater extent how their unit of code would eventually be used. In proper test-driven development, the developers would get a better sense of how their unit's API should be structured. If they're good developers, they would also have put more consideration into how corner cases would be handled. And once their unit tests are implemented, the ability to run them continuously against a changing code base becomes invaluable, especially when regressions are caught rapidly.

Such developers can easily track the number of hours it took them to write the unit tests. They can easily track the number of lines of code making up those tests. But it's harder to quantify the greater familiarity and greater understanding they gain regarding the unit under test. It's also difficult to quantify the time and expense saved by catching regressions as early as possible, often within seconds of some new or modified code being committed to the source code repository. And then there's the software developer's reputation, be that developer an individual or a large company, which can be greatly improved by providing software that is generally bug free.

Often, the benefit brought by the more intangible and abstract aspects of unit testing far outweigh whatever time and effort is put into developing the unit tests initially. Remember, if a unit test that took a few minutes to write catches even one regression, the savings can be enormous. Many times in the past I've seen days upon days of effort, in terms of debugging, patch preparation, patch testing, patch deployment and customer appeasement, needed to fix a bug that could have easily been caught with three or four assertions and perhaps eight to ten lines of unit test code.

Users of of dynamic languages such as Python, Perl and PHP, are often faced with the additional task of ensuring that type-related errors do not occur in their programs at runtime. Some authors [1, 2] claim that this can be done effectively by the use of extensive automated unit tests. However, practice has shown this to just not be the case.

Automated unit tests have their place. But they should not be used to check for typing errors. Software typechecking is a mechanical task, and thus is just the sort of thing that should be delegated to a computer. And this has been done for quite some time now, in the form of statically typed programming languages. Users of such languages, such as Haskell, OCaml and Standard ML are freed from having to worry about such menial and tedious tasks. The rigorous nature of the compilers for those languages often do a far better job at checking for type errors than a human typically could.

Proponents of dynamic languages often suggest that such languages bring an increase in developer productivity. But if the developer ends up spending so much time writing unit tests to perform a task that is readily done automatically, we really don't see the supposed productivity boost. And if the developer doesn't write such unit tests, the chances of a runtime type error crashing the application are far greater. So now the developer must choose between his or her software crashing and disrupting the users' work, or spending a lot of time to write unit tests. None of the choices are reasonable, let alone optimal.

Static typing is clearly the answer. Not only does the compiler take care of the tedious task of type checking, but any inconsistencies are detected at compile-time, rather than runtime. Thus it is the developer who deals with such errors, rather than the user. Furthermore, the developer is now freed up from writing automated unit tests to check for type errors, and can instead put more emphasis on unit testing the functionality and integration of his or her code. Clearly, static typing is the only sensible, and the most efficient, route to take.