pip re-installing wrong version of a package

I recently ran into a pip gotcha that left me scratching my head for hours.

When you install a package for the first time using pip, it will download the package into a build directory, unzip it there and install from there. Even if you uninstall it, the copy in the build directory remains intact. The next time you reinstall, pip checks if there is already a version in the build. If its there, then it directly installs that without downloading it again.

This is usually fine, unless you uninstalled in order to install a different version. In that case you might be mystified as to why the same old version is installed again.

pip install pycrypto==2.3

pip uninstall pycrypto

pip install pycrypto==2.0.1 # still installs pycrypto 2.3!!

The solution to this is to simply delete the package copy from the build directory.

If you are using virtualenv, then the build directory will be a top-level folder inside the virtualenv.

Installing Python Imaging Library on Ubuntu

Continuing on setting up the VPS, its now time to install the Python Imaging Library. This is another major pain in the neck.

On Windows, PIL comes nicely bundled with everything. On Linux, it gets compiled, and the stupid part is that different bits of support get compiled in depending on what you have installed. PIL will silently skip components and say that the compile was successful. It's only when you run the application do you find out that some parts of PIL are not installed. Big, big, pain. If you are doing automated provisioning of machines, then you have to be careful that you have the right packages in place before you pip install PIL.

The two components that I am interested are PNG support and Truetype font support, because thats what we use for our app.

PNG Support

If you want PNG support, you need to have the zlib library installed.

sudo apt-get install zlib1g

sudo apt-get install zlib1g-dev

JPEG Support

JPEG requires libjpeg62

sudo apt-get install libjpeg62

sudo apt-get install libjpeg62-dev

TrueType Support

You'll need the libfreetype6 package installed

sudo apt-get install libfreetype6

sudo apt-get install libfreetype6-dev

Setting up Postgres on Ubuntu 10.04

We've just migrated our Django build server to a new box, and I'm currently installing all the necessary dependencies. Even though I've done this many times, I still keep forgetting the steps to do it properly, so this blog post will document the steps for future reference.

PostgreSQL

Installing postgres is a major pain, because there are so many annoying small things to do to get it working properly.

First, before you do anything you want to ensure that you set up your locale to UTF-8.

locale-gen en_US.UTF-8

update-locale LANG=en_US.UTF-8

If you forget to do this and its not setup by default, then postgres will use ASCII which is probably not what you want. If you sill forget this (yes, I forgot), and you haven't put in any data yet, then you can drop and recreate your postgres cluster.

pg_dropcluster --stop 8.4 main

pg_createcluster --start -e UTF-8 8.4 main

Okay, with that out of the way..

Installing Postgres

The latest version of Postgres on Ubuntu 10.04 is Postgres 8.4

sudo apt-get install postgresql-8.4

You'll also need the dev package to compile psycopg2 later

sudo apt-get install libpq-dev

And don't forget the python dev packages to compile psycopg2

sudo apt-get install python-dev

Setting up Postgres

By default postgres is configured to use your system users for authentication. If you want to use a specific user/password combination, you'll need to change this.

Open up /etc/postgresql/8.4/main/ph_hba.conf and change

local all all ident

to

local all all md5

Then restart postgres.

Create a postgres user and database

These are the two commands for creating a user and database. If you intend to run the Django unit tests, then dont forget to give CREATEDB permission for the user.

CREATE USER username WITH PASSWORD 'password' CREATEDB;

CREATE DATABASE db_name OWNER username ENCODING 'UTF-8';

Compiling psycopg2

You should be able to pip install psycopg2. Note that it needs to be compiled, so you should have build-essential package installed beforehand.

sudo apt-get install build-essential

pip install psycopg2

There is a gotcha here: psycopg2 version 2.4.2 is NOT compatible with Django 1.3. Either use the trunk version of Django or use psycopg2 version 2.4.1.

pip install psycopg2==2.4.1

Troubleshooting

1. make or gcc is not found: install the build-essential package
2. postgres gives the error "ERROR: new encoding (UTF8) is incompatible with the encoding of the template database (SQL_ASCII)": You haven't set the locale properly before installing postgres. See the top of the post for setting the locale
3. Django gives the error "Got an error creating the test database: permission denied to create database": If you want to run unit tests, the user must have the CREATEDB permission
4. postgres gives the error FATAL: Ident authentication failed for user "username": You need to edit the pg_hba.conf file to turn off ident authentication and set it to md5 instead
5. psycopg2 gives the error 'PyType_GenericAlloc' undeclared (first use in this function): Install the python-dev package
6. psycopg2 gives the error pg_config: command not found: Install the libpg-dev package

My Django Dash 2011 experience

About Django Dash

Django Dash is an online, weekend Django hackathon. You get 48 hours to develop a Django app of some sort, from scratch.

Apart from being a lot of fun, its a great way to squeeze in a bunch of learning over a short period of time. Kausik and I set out to build a badge generation application.

(View the site here - Make My Badge)

The idea

A lot of people probably know about the badge generation python script. This is a script that takes a list of people, a badge template and generates badges for all the attendees. The badges can be printed out and laminated beforehand for attendees to pick up at the event registration.

The badge design itself is based on a simple principle (surprisingly violated by a huge number of events!) - keep the name large and easy to read from across the hall. We wanted to avoid badges where the name of the event is prominent but the name of the person is tiny. I'm also not a fan of handwritten badges. Not only do they look ugly, but the scribbling is rarely readable.

We also wanted to auto-scale the font size based on the length of the name, and to split long names into two lines so that we could use larger font sizes. These badges have been used in a number of events in Chennai, and have always been popular, with some people even preserving them as souvenirs. (See the badge in use)

One of the problems with the script is that you need some technical knowledge to generate the badge. This made it complicated for a people to generate badges for an event. It also made it difficult to generate and print badges on the fly at the registration desk for on the spot registrants.

We decided to build a web app in 48 hours to do everything that the badge generator did, but as a Django site.

Something that we wanted to do (apart from developing the site of course) was to get up to date on the current cutting edge technology in the Python/Django space. Sure we use a lot of cutting edge tools for ToolsForAgile.com but that application is now almost 3 years old. Although we do keep updating it, its a big application and updating platforms and infrastructure can take time. Plus we wanted to do a quick spike of some of the technologies so that we could take the learning back to our product. Django Dash was the perfect opportunity for that.

Here are some of the things we learned

Django deployment scenarios

When we first developed our product, the only option for python and django deployments was to use Apache with mod_python. Then the WSGI standard was finalised and mod_wsgi came along. mod_python is now officially end-of-line with no more support available for it.

Today one popular deployment configuration is to use an nginx frontend server to serve static files, with a reverse proxy to an Apache + mod_wsgi configuration to serve dynamic files.

Apart from Apache, a lot of other backend servers are becoming popular, most notably gunicorn.

Django hosting

There are now a gazillion hosting providers for Django. When we started out with ToolsForAgile.com the only option was to roll your own setup on a VPS or dedicated server. You still want to do that for complex deployment setups, but there are now any number of django specific hosts for the common scenarios - ep.io, gondor.io, Stable, Dotcloud, AppHosted, DjangoZoom. Ken Cochrane has an excellent series of blog posts with a comparison of django hosting providers.

All these services allow you to simply deploy, while they handle setting up the deployment stack, load balancing and auto scaling for you.

And of course, for the simpler use cases, there is always good old Webfaction. I've hosted various sites with them for four years, with not a single problem so far.

ep.io

ep.io and gondor.io were sponsoring Django Dash, so apart from setting up a custom stack on a Linode VPS, participants also had the choice of deploying on one of these services. We decided to try out ep.io for our submission.

Overall, I must say that I really liked ep.io. It was a bit complicated to get my head around first. It took a while to get file uploads, static files and celery set up. That was more to do with the fact that we were using ep.io for the first time. Once setup, deployment was a breeze. All we had to do was to upload the project using their command line client and ep.io would automatically provision the app with all the required services. They also have an interface where you can push your project directly via git or mercurial.

I should note here that the ep.io client only works in linux as it requires ssh to execute remote commands. Since we were on windows, it took us a fair while to hack up the client and hook it up to do its work through Putty instead.

pip

This is the first time we exclusively used pip along with a requirements.txt file to sync up the python dependencies of all the local dev environment as well as the remote server setup. It worked really well. pip rocks!

Celery

Celery is a distributed task queue. It's amazing how popular it has become in such a short span of time. For our submission, we decided to try out celery to generate the badge images asynchronously through the tasks queue.

Celery can work on top of a number of transports. Popular configurations are to use it over an AMQP broker like RabbitMQ or over Redis. We used it over RabbitMQ (which in turn requires Erlang), whereas ep.io supports celery over Redis. The nice thing is that we just had to tell ep.io that we wanted to use celery and it would set everything up, including launching the celery daemon, configuring it to use redis and what not.

On our side, we used django-celery, a Django integration of celery. With djcelery you can create a tasks.py file in your app with a bunch of task definitions which you can call asynchronously through a view.

After some initial setup hiccups, it worked like a charm, allowing us to push out the badge image generation out to the workers. When badge generation was done, it would trigger another task to zip up all the images and allow the user to download a single zip with all the badges. Everything happening through Celery. It was pretty exciting to see the whole flow in action.

Django 1.3

We used Django 1.3 for this submission. Our product is built on a version of django that is somewhere between 1.1 and 1.2.

We got a chance to play with some of the new stuff in 1.3. We got a chance to try out class based generic views, the new logging setup, TemplateResponse, some improvements to the admin, and a bunch of other stuff.

The admin interface itself is a huge section to explore. We briefly thought about doing the whole site through a custom admin interface, but we were running out of time so we shelved the idea for the time being.

Django 1.3 also has a completely new way of dealing with static files. The old django-staticfiles app has been added as a contrib package and is now the official way to handle static files. The old way had a single static directory with all the static files in it. You now have to put your static files either under an app folder, or in a global static dir. You then call the collectstatic management command to pull in all the static files into a directory that the frontend server will serve.

Similarly, there seems to have been a change in the way uploaded files are handled. Previously they used to go under the static folder. They now get a folder of their own.

What got done in 48 hours

We eventually ran out of time on all the features that we wanted. Thats not surprising, with the amount of new stuff we were using for the first time. It took almost one and a half days to setup and get comfortable with ep.io, celery, django-celery and all the changes in Django 1.3, and getting them to work properly on the local dev environment and online on ep.io.

However, since we built the whole app incrementally, we managed to get the important features in. If you go to Make My Badge you'll be able to login, generate and download a sample set of badges. You can create events, though you need to go to the admin interface to add people to the event.

What is left to do?

We ran out of time before we could integrate django-registration for new users to register. We've created a sample user through the admin interface for now. We also wanted to be able to upload a list of event attendees through a CSV upload form, but couldn't put that in. As a workaround, you need to add people through the admin interface.

Apart from those two main features, there were lots of UI design work that we wanted to do which we couldn't finish.

Where are we going from here?

Once the event judging is over, we have a bunch of enhancements to add. The CSV upload feature, for example, got done about 2 minutes past the deadline. We want to put that in. Integration with django-registration is also ready on the development environment.

We want to eventually add integration with a payment system. Charge Rs.1 per badge generated or something like that.

Overall Impressions

Django Dash was awesome fun. Although we couldn't complete everything we sure did learn a lot of new things that we can take back into other projects. And now that we know the tech, I'm pretty certain we can have a similar app complete in 48hrs.

I was just going through some of the teams, and here are some submissions that I found. Remember, all these were done by scratch in 48 hours! - Drawn by, Family Feed, ProposalMatic, FutureFI, Goal Rally, Django Lint, ConsoliTweet, Courtside, Linky, Git Awesome, Stardust, Set With Me, Libman, Show Offfr, SmartLinky, Staste, Django Docs, Code War, Grepo, Codr Space, My Img, Gearoscope

Why your startup should not start with a Freemium biz model

I'm reading Ash Maurya's book Running Lean that I picked up from AppSumo's Lean Startup bundle [The bundle gives you $6000 worth of ebooks and web app subscriptions for only $99. Its a total bargain, but only 3 days to go before the offer expires, so get it soon - use this link and I'll get a $10 credit :)].

There is a section on the freemium pricing model and why startups should not start with freemium. This part really resonated with me because we fell into many of these problems early on. The reasons against freemium are:

• Low or no conversions: Startups often give away too much in the free plan, leaving users with no need to upgrade
• Long validation cycle: Conversion rates on upgrades are low, so its hard to validate the pricing model
• Focusing on the wrong metric: Causes a premature shift in focus from building the right product for paying customers to new user sign up
• Low signal to noise ratio: Free users may give feedback that paying users dont care about
• Free user's aren't free: It costs time and money to support free users

Do you think freemium works? What are your thoughts?

This post is a part of the selected archive.

Why New Ideas Can Be Bad

This is a great post from Tim Kastelle: Why New Ideas Can Be Bad which ties in with my blog post from last year: Too many people are ideating, not enough are executing.

Tim references a talk by Scott Belsky at the 99% conference. Here is what Belsky says:

When ideas are new, we have lots of excitement and energy. However, once we settle into trying to make the idea real, the levels of both excitement and energy go down – it starts to feel more like work. How do we respond to this?

According to Belsky, the natural response is to look for the excitement of a new idea again – and succumbing to this temptation is deadly.

Read the whole thing.

This post is a part of the selected archive.