pip re-installing wrong version of a package

I recently ran into a pip gotcha that left me scratching my head for hours.

When you install a package for the first time using pip, it will download the package into a build directory, unzip it there and install from there. Even if you uninstall it, the copy in the build directory remains intact. The next time you reinstall, pip checks if there is already a version in the build. If its there, then it directly installs that without downloading it again.

This is usually fine, unless you uninstalled in order to install a different version. In that case you might be mystified as to why the same old version is installed again.

pip install pycrypto==2.3

pip uninstall pycrypto

pip install pycrypto==2.0.1 # still installs pycrypto 2.3!!

The solution to this is to simply delete the package copy from the build directory.

If you are using virtualenv, then the build directory will be a top-level folder inside the virtualenv.

Installing Python Imaging Library on Ubuntu

Continuing on setting up the VPS, its now time to install the Python Imaging Library. This is another major pain in the neck.

On Windows, PIL comes nicely bundled with everything. On Linux, it gets compiled, and the stupid part is that different bits of support get compiled in depending on what you have installed. PIL will silently skip components and say that the compile was successful. It's only when you run the application do you find out that some parts of PIL are not installed. Big, big, pain. If you are doing automated provisioning of machines, then you have to be careful that you have the right packages in place before you pip install PIL.

The two components that I am interested are PNG support and Truetype font support, because thats what we use for our app.

PNG Support

If you want PNG support, you need to have the zlib library installed.

sudo apt-get install zlib1g

sudo apt-get install zlib1g-dev

JPEG Support

JPEG requires libjpeg62

sudo apt-get install libjpeg62

sudo apt-get install libjpeg62-dev

TrueType Support

You'll need the libfreetype6 package installed

sudo apt-get install libfreetype6

sudo apt-get install libfreetype6-dev

My Django Dash 2011 experience

About Django Dash

Django Dash is an online, weekend Django hackathon. You get 48 hours to develop a Django app of some sort, from scratch.

Apart from being a lot of fun, its a great way to squeeze in a bunch of learning over a short period of time. Kausik and I set out to build a badge generation application.

(View the site here - Make My Badge)

The idea

A lot of people probably know about the badge generation python script. This is a script that takes a list of people, a badge template and generates badges for all the attendees. The badges can be printed out and laminated beforehand for attendees to pick up at the event registration.

The badge design itself is based on a simple principle (surprisingly violated by a huge number of events!) - keep the name large and easy to read from across the hall. We wanted to avoid badges where the name of the event is prominent but the name of the person is tiny. I'm also not a fan of handwritten badges. Not only do they look ugly, but the scribbling is rarely readable.

We also wanted to auto-scale the font size based on the length of the name, and to split long names into two lines so that we could use larger font sizes. These badges have been used in a number of events in Chennai, and have always been popular, with some people even preserving them as souvenirs. (See the badge in use)

One of the problems with the script is that you need some technical knowledge to generate the badge. This made it complicated for a people to generate badges for an event. It also made it difficult to generate and print badges on the fly at the registration desk for on the spot registrants.

We decided to build a web app in 48 hours to do everything that the badge generator did, but as a Django site.

Something that we wanted to do (apart from developing the site of course) was to get up to date on the current cutting edge technology in the Python/Django space. Sure we use a lot of cutting edge tools for ToolsForAgile.com but that application is now almost 3 years old. Although we do keep updating it, its a big application and updating platforms and infrastructure can take time. Plus we wanted to do a quick spike of some of the technologies so that we could take the learning back to our product. Django Dash was the perfect opportunity for that.

Here are some of the things we learned

Django deployment scenarios

When we first developed our product, the only option for python and django deployments was to use Apache with mod_python. Then the WSGI standard was finalised and mod_wsgi came along. mod_python is now officially end-of-line with no more support available for it.

Today one popular deployment configuration is to use an nginx frontend server to serve static files, with a reverse proxy to an Apache + mod_wsgi configuration to serve dynamic files.

Apart from Apache, a lot of other backend servers are becoming popular, most notably gunicorn.

Django hosting

There are now a gazillion hosting providers for Django. When we started out with ToolsForAgile.com the only option was to roll your own setup on a VPS or dedicated server. You still want to do that for complex deployment setups, but there are now any number of django specific hosts for the common scenarios - ep.io, gondor.io, Stable, Dotcloud, AppHosted, DjangoZoom. Ken Cochrane has an excellent series of blog posts with a comparison of django hosting providers.

All these services allow you to simply deploy, while they handle setting up the deployment stack, load balancing and auto scaling for you.

And of course, for the simpler use cases, there is always good old Webfaction. I've hosted various sites with them for four years, with not a single problem so far.

ep.io

ep.io and gondor.io were sponsoring Django Dash, so apart from setting up a custom stack on a Linode VPS, participants also had the choice of deploying on one of these services. We decided to try out ep.io for our submission.

Overall, I must say that I really liked ep.io. It was a bit complicated to get my head around first. It took a while to get file uploads, static files and celery set up. That was more to do with the fact that we were using ep.io for the first time. Once setup, deployment was a breeze. All we had to do was to upload the project using their command line client and ep.io would automatically provision the app with all the required services. They also have an interface where you can push your project directly via git or mercurial.

I should note here that the ep.io client only works in linux as it requires ssh to execute remote commands. Since we were on windows, it took us a fair while to hack up the client and hook it up to do its work through Putty instead.

pip

This is the first time we exclusively used pip along with a requirements.txt file to sync up the python dependencies of all the local dev environment as well as the remote server setup. It worked really well. pip rocks!

Celery

Celery is a distributed task queue. It's amazing how popular it has become in such a short span of time. For our submission, we decided to try out celery to generate the badge images asynchronously through the tasks queue.

Celery can work on top of a number of transports. Popular configurations are to use it over an AMQP broker like RabbitMQ or over Redis. We used it over RabbitMQ (which in turn requires Erlang), whereas ep.io supports celery over Redis. The nice thing is that we just had to tell ep.io that we wanted to use celery and it would set everything up, including launching the celery daemon, configuring it to use redis and what not.

On our side, we used django-celery, a Django integration of celery. With djcelery you can create a tasks.py file in your app with a bunch of task definitions which you can call asynchronously through a view.

After some initial setup hiccups, it worked like a charm, allowing us to push out the badge image generation out to the workers. When badge generation was done, it would trigger another task to zip up all the images and allow the user to download a single zip with all the badges. Everything happening through Celery. It was pretty exciting to see the whole flow in action.

Django 1.3

We used Django 1.3 for this submission. Our product is built on a version of django that is somewhere between 1.1 and 1.2.

We got a chance to play with some of the new stuff in 1.3. We got a chance to try out class based generic views, the new logging setup, TemplateResponse, some improvements to the admin, and a bunch of other stuff.

The admin interface itself is a huge section to explore. We briefly thought about doing the whole site through a custom admin interface, but we were running out of time so we shelved the idea for the time being.

Django 1.3 also has a completely new way of dealing with static files. The old django-staticfiles app has been added as a contrib package and is now the official way to handle static files. The old way had a single static directory with all the static files in it. You now have to put your static files either under an app folder, or in a global static dir. You then call the collectstatic management command to pull in all the static files into a directory that the frontend server will serve.

Similarly, there seems to have been a change in the way uploaded files are handled. Previously they used to go under the static folder. They now get a folder of their own.

What got done in 48 hours

We eventually ran out of time on all the features that we wanted. Thats not surprising, with the amount of new stuff we were using for the first time. It took almost one and a half days to setup and get comfortable with ep.io, celery, django-celery and all the changes in Django 1.3, and getting them to work properly on the local dev environment and online on ep.io.

However, since we built the whole app incrementally, we managed to get the important features in. If you go to Make My Badge you'll be able to login, generate and download a sample set of badges. You can create events, though you need to go to the admin interface to add people to the event.

What is left to do?

We ran out of time before we could integrate django-registration for new users to register. We've created a sample user through the admin interface for now. We also wanted to be able to upload a list of event attendees through a CSV upload form, but couldn't put that in. As a workaround, you need to add people through the admin interface.

Apart from those two main features, there were lots of UI design work that we wanted to do which we couldn't finish.

Where are we going from here?

Once the event judging is over, we have a bunch of enhancements to add. The CSV upload feature, for example, got done about 2 minutes past the deadline. We want to put that in. Integration with django-registration is also ready on the development environment.

We want to eventually add integration with a payment system. Charge Rs.1 per badge generated or something like that.

Overall Impressions

Django Dash was awesome fun. Although we couldn't complete everything we sure did learn a lot of new things that we can take back into other projects. And now that we know the tech, I'm pretty certain we can have a similar app complete in 48hrs.

I was just going through some of the teams, and here are some submissions that I found. Remember, all these were done by scratch in 48 hours! - Drawn by, Family Feed, ProposalMatic, FutureFI, Goal Rally, Django Lint, ConsoliTweet, Courtside, Linky, Git Awesome, Stardust, Set With Me, Libman, Show Offfr, SmartLinky, Staste, Django Docs, Code War, Grepo, Codr Space, My Img, Gearoscope

Test Driven Development in Python

Here are the slides for my talk at Pycon India this weekend. The talk was on doing test driven development in Python and it looked at 3 frameworks - unittest, py.test and nose.

Test Driven Development With Python

View more documents from Siddhi.

Pattern Matching With PEAK-Rules

I came across this interesting article on Pattern matching in Python. The article asks: Can we recreate in Python the pattern matching semantics present in languages like Haskell and Erlang.

For those not familiar with the way pattern matching works, I've copied the example from the article above.

%% handle(Path, Method)
handle("/", _) ->
  not_a_resource;
handle(Path, 'PUT') ->
  create_new_resource(Path);
handle(Path, 'POST') ->
  update_resource(Path);
handle(Path, 'GET') ->
  retrieve_resource(Path);
handle(_, _) ->
  invalid_request.

What the above code does is to return "not a resource" if you call the handle function with the path parameter as "/" and any method. If you call handle with any path and method parameter as "GET" then it calls retrieve_resource(Path) and so on.

PEAK-Rules

A method to replicate this in Python was given using match objects, but I thought hey why go through all this trouble when PEAK-Rules does most of this for us?

Multimethod Dispatch

PEAK-Rules is a library that enables multimethod dispatch in Python. Those with an OO background will recognise a specific instance of this in method overloading. When an overloaded method is called, execution can go to different method implementations depending upon the type of the parameters passed. In generic multimethod dispatch, you can route execution based on any criteria that you define. PEAK-Rules brings this sort of generic multimethod dispatch to Python.

An Example

So lets take a concrete example. Our goal is to rewrite the above Erlang code in Python using two libraries: PEAK-Rules and an add-on called prioritized_methods that allows prioritised method ordering.

First, you'll need to get install PEAK-Rules and prioritized_methods. You can pick them up from the links given or if you've got setuptools, then you can just easy_install them.

Then type out the following code:

>>> from peak.rules import abstract, when
>>> from prioritized_methods import prioritized_when
>>> @abstract()
... def handle(path, method):
...     pass
...
>>> @prioritized_when(handle, "path == '/'", prio=1)
... def not_a_resource(path, method):
...     print "not a resource"
...
>>> @when(handle, "method == 'GET'")
... def get_resource(path, method):
...     print "getting", path
...
>>> @when(handle, "method == 'PUT'")
... def create_resource(path, method):
...     print "creating", path
...
>>> @when(handle, "method == 'POST'")
... def update_resource(path, method):
...     print "updating", path
...

Here is what is happening:

We first define an abstract function handle(path, method). Do this by placing the @abstract() decorator on it.

Now consider this snippet:

>>> @when(handle, "method == 'GET'")
... def get_resource(path, method):
...     print "getting", path
...

This defines one implementation for the handle function. It says, when the handle function is called, and the condition given is True (in this case method == "GET"), then call the implementation given below (here: the get_resource function).

Similarly we define the other implementations to be called on some other conditions.

The only thing left is the usage of @prioritized_when. Now, when a call is made to handle("/", "GET"), we see that the condition for not_a_resource as well as get_resource are satisfied. Which implementation should be called? In this case, we use the @prioritized_when decorator and set the priority to 1. This tells the system to give priority to this implementation in case of conflict in match.

Here is how the output looks:

>>> handle("/", "GET")
not a resource
>>> handle("/", "POST")
not a resource
>>> handle("/home", "PUT")
creating /home
>>> handle("/home", "POST")
updating /home
>>> handle("/home", "GET")
getting /home

Pretty cool! The best part of this is that you can dispatch on virtually any condition. While the resulting code is a little more verbose than the Erlang example, its not too bad and it does the job well.

Yahoo Groups running on Python?

I tried to post a message today and go this error:

Must watch videos from DjangoCon

I was recently going through some of the videos from DjangoCon. Two must watch videos are Cal Henderson's talk titled "Why I Hate Django" and Mark Ramm's talk "A Turbogears guy on what Django should learn from Zope". Both Cal and Mark bring in interesting outside perspectives that I hadn't really considered before. Both talks run around an hour each and are well worth watching if you have the time.

Screencast: Using Amazon S3 with Django

Many thanks to Balaji and CSS for organising the second Amazon Web Services meet in Chennai and to Jinesh for being able to make it. (Also check out my live blog of the event on twitter, starting here)

As a part of this meet, I had prepared a demo showing how to build a simple gallery application using Django and Amazon's S3 service. I've recorded it as a screencast and uploaded it on ShowMeDo. It's really simple to use S3 with Django, so take a look.

This screencast uses the Boto python library for accessing AWS.

Get the Flash Player to see this movie.

This video originally comes from here at ShowMeDo from the Python category.

Why doesn't Django find my unit tests?

I ran into a Django gotcha today. I had an app for which it was simply not running the unit tests. The tests were there in the correct location. Django was correctly running the tests for all the other apps. Why was it not running the tests for this one? I checked settings.py to see if the app was in INSTALLED_APPS. It was. It worked fine through the browser, and the test code looked good. What was worse, it was working when I last ran it before the weekend. Why did it suddenly stop running?

I got my answer after some digging around in the Django source. It appears that you must have models.py in the app for tests to run. It so happened that this app has no models, only some views that do some calculations. Since models.py was empty, I had deleted it and didn't think too much about it. Whoops! The tests stopped running.

Once I had that figured out, the solution was simple - recreate the empty models.py, and the tests started running again.

This whole thing is really unintuitive though. Who would have thought that removing an empty file would cause the tests to stop running? There doesn't seem to be any connection between them at all.

In a broader sense, any piece of code that uses django.db.models.get_apps to get a list of installed apps is likely to run into this problem. Don't be surprised if you remove an empty models.py and then something breaks and you are left scratching your head as to what exactly happened.

This is exactly the kind of unintuitive "magic" that we Pythonistas hate :) Explicit is better than implicit etc.

What I don't understand is that the list of installed apps *IS* explicit. It's sitting there called INSTALLED_APPS in settings.py. Why does Django go about hunting through the models when it could just read this value? Any clues?

Has Django reached the tipping point?

Antonio Cangiano thinks so. The numbers used to come to this conclusion are not exactly scientific, but there's no doubt that Django is getting bigger (and better) everyday.

Screencast on many to many relationships between Django models

My third introductory screencast for Django. It follows on from the first screencast and the second one.

Get the Flash Player to see this movie.

This video originally comes from here at ShowMeDo from the Python category.

Cricinfo XML feed of live scores

Did you know that Cricinfo publishes an RSS feed containing live scores of matches in progrss? I had no idea either until I came across this python script on bangpypers that uses the feed to display live scores on the desktop. Very interesting.

Looking for Django developers?

Check out Djangogigs.com. Djangogigs is a new website for matching Django developers with Django jobs. If you are looking out for some Django work, add yourself to the site.

Django Screencasts

A few people have been asking for introductory Python + Django material, so I thought I'd just link up to the two intro screencasts that I did for Django.

Learn Django: Create a Wiki in 20 minutes

This is a beginners screencast that assumes no previous knowledge of Django. We illustrate the basics of creating a web app in Django by developing a simple wiki application.

Learn Django: Extending the wiki with wikiwords and search

Following on where the first screencast left off, this screencast introduces some more django features such as template inheritance, filters and newforms library.

Using python to teach kids programming

Horst Jens uses Python to teach kids how to program at Profikids in Vienna. In the following screencast, we see one of the kids, Leo, program a simple GUI button box. The screencast is in German, but it has English subtitles. Check it out below -

Get the Flash Player to see this movie.

This video originally comes from here at ShowMeDo from the Python category.

Django screencast: Template inheritance, filters and newforms

Following up on my first Django screencast, I've put up the second screencast in the Learn Django series. Like the first one, its hosted on showmedo.com and you can view the screencast here.

The second screencast extends the wiki application we built in the first part by adding a menu, support for auto-linking of wikiwords and a feature to search through the database. In the process, you'll learn about some cool Django features such as template inheritance, custom filter libraries and Django newforms. Django newforms in particular is a very cool form handling library that can really cut down on boilerplate form generation and validation code.

Interested? Check out the screencast.

Event Badge Generator released under MIT License

The event badge generator python script has been released under the MIT License: More info, Download

If you attended Proto.in, you would have noticed the cool preprinted badges for all the attendees. (If not, check out the pics here and here) We had almost 400 people, and no way was I going to manually create each badge, type and place all the text, reduce the size if it didn't fit, split names into two lines if required and all that. So I ended up creating a python script to take a badge template and a list of names and create badges for each one.

I was talking about this script at the lightning talks at BCB4, when Brad Allen from the Dallas Python User Group mentioned how it might be useful for other events as well (mainly with reference to PyCon). Therefore, the script has now been released under an MIT license.

As it stands now, there are some hardcoded variables here and there and some work needs to be done. If you are a python coder, you can probably find your way around and change what you want to change. Sometime in the future I'll parametrise it so that it is relatively easy for non-python coders to use it as well. Eventually, I'm hoping it reaches a stage where it can be used by non-tech event managers.

For now though, it can only be used by those who can read and write basic python. If you are interested in using it or playing around with it, check out the project wiki page.

My first ShowMeDo video

ShowMeDo is a website started by Ian Ozsvald and Kyran Dale. The idea is to have a collection of technical screencasts on a variety of subjects. I just got my first screencast hosted at ShowMeDo. It is a re-recorded version of my earlier screencast on doing a wiki in Django, this time with a voice over. Have a look here.

Django Code Coverage Followup

teenage mutant ninja hero coders builds on my post on code coverage with django and shows how you can get the same effect by creating an alternate test runner. This way, you can get code coverage without having to muck around in the django source.

Pretty cool. I'll have to try this out.

Link

Generating sentences using Markov chains

So, a bunch of us - Aswin, Kausik, Moyeen, Sagaro - got together at Aswin's house on Sunday to have a python codekata. We did an intro to Python session, and we wrote a program to generate sentences using Markov chains.

I really like this program. Not only is it interesting to write and run but it makes for a very nice language intro. I'll be doing this session again as a workshop at Bangalore BarCamp 4 at the end of this month.

The final version of the program is given below. The example below uses sample text from Alice in Wonderland, downloaded from Project Gutenberg. Remember that this is the final version — In the actual kata, a number of variations were written before arriving here.

import random

def getLines(filename):
 return [line[0:-1] for line in open(filename).readlines()]

def getWords(lines):
 words = []
 for line in lines:
  words.extend(line.split())
 return words

def createProbabilityHash(words):
 numWords = len(words)
 wordCount = {}
 for word in words:
  if wordCount.has_key(word):
   wordCount[word] += 1
  else:
   wordCount[word] = 1
 
 for word in wordCount.keys():
  wordCount[word] /= 1.0 * numWords
 return wordCount

def getRandomWord(wordCount):
 randomValue = random.random()
 cumulative = 0.0
 for word in wordCount:
  cumulative += wordCount[word]
  if cumulative > randomValue:
   return word

# replace with a large text sample. Here we are using Alice in Wonderland
# from Project Gutenberg
words = getWords(getLines("alice.txt"))

wordMap = {}
previous = (words[0], words[1])
for word in words[2:]:
 if wordMap.has_key(previous):
  wordMap[previous].append(word)
 else:
  wordMap[previous] = [word]
 previous = (previous[1], word)

for word in wordMap.keys():
 probabilityHash = createProbabilityHash(wordMap[word])
 wordMap[word] = probabilityHash

previous = ("The", "next") # The starting words
numWords = 100 # The number of words to print

print previous[0], previous[1],
for i in range(numWords):
 word = getRandomWord(wordMap[previous])
 print word,
 if word.endswith("."):
  print "\n"
 previous = (previous[1], word)