Mobile Monday Chennai

Some updates on Mobile Monday. It's coming up this Sunday, 6th May at the TeNeT hall in IIT Madras.

Time: 10 AM

Venue:

TeNeT - ADI Seminar Hall,
CSD Block, 3rd Floor,
IIT Madras.

Check out the wiki for the speaker and attendee lists: http://www.momochennai.com/

Do you buy comics?

If you do, then Fantagraphics Books is giving a 20% discount on all their books.

Unlike other discounts, this one is different and rather unique. To avail of the discount, you have to add their user to your network in one of the popular social networking sites: MySpace, Flickr, Orkut, Bebo, Tribe.net, Yahoo 360 or AIM Pages. Once you add them to your network, just mention your nickname and the website and you get 20% off all books purchased.

I took the opportunity to order Volume 1 and 2 of Usagi Yojimbo, Like a Velvet Glove Case in Iron, a couple of new issues of Mome and Bête Noire #1.

Upgrade to new blogger

I upgraded to the new blogger interface yesterday, which means that I finally have access to labels. Over the week I'll be labeling most of the 300+ posts in the archives. If you are reading this via a feed, it may happen that a whole bunch of old posts lands up in your feedreader. If that happens you can just ignore it as its only the labeling going on.

Chennai Air Show Photographs

I've uploaded a bunch of photographs from the Chennai Air Show that happened at Marina Beach yesterday. The photos are here:

http://www.flickr.com/photos/siddhi/tags/airshow/show/

Code coverage for your Django code

One of the things that I like about Django is that it allows you to run unit tests on the code with relative ease. So I have a bunch of unit tests now, and I was looking to expand it. For that I first needed to know which parts of the code had good coverage and which parts had no coverage. I needed to integrate code coverage with the unit testing framework. In the end, it involved modifying a few files. This is what I did:

1. First, I downloaded Ned Batchelder's coverage.py module.
2. Next, get this script to colorize the coverage output. I saved it as coverage_color.py
3. Put both files somewhere on the python path. I put it at Python24\Lib\site-packages
4. Now we need to modify the django unit test runner to include coverage. Head over to your django\test directory and edit the simple.py file
5. At the top, add the following line:

   import os, coverage, coverage_color

6. Scroll down to the run_tests function. You will see a line like this:

       unittest.TextTestRunner(verbosity=verbosity).run(suite)
       destroy_test_db(old_name, verbosity)

7. Modify it to read like this:

       coverage.start()
       unittest.TextTestRunner(verbosity=verbosity).run(suite)
       coverage.stop()
       if not os.path.exists(settings.COVERAGE_DIR):
           os.makedirs(settings.COVERAGE_DIR)
       for module_string in settings.COVERAGE_MODULES:
           module = __import__(module_string, globals(), locals(), [""])
           f,s,m,mf = coverage.analysis(module)
           fp = file(os.path.join(settings.COVERAGE_DIR, module_string + ".html"), "wb")
           coverage_color.colorize_file(f, outstream=fp, not_covered=mf)
           fp.close()
       coverage.erase()
       destroy_test_db(old_name, verbosity)

8. What that does is to record the coverage when running the tests. It then creates a directory for putting the HTML output, creates the colorized version of the source and dumps it into the output directory. At the end of everything, it cleans up the coverage data
9. We now need to configure the COVERAGE_DIR and COVERAGE_MODULES settings. Open your Django settings.py file and add the following lines:

   COVERAGE_DIR = "scripts/build/coverage" # Where the HTML output should go
   COVERAGE_MODULES = ["apps.catalyst.views", "apps.catalyst.models"] # The modules that you want colorized

10. Save and run your Django unit tests. After running, you will have a html file for each module in the specified directory with the colorized coverage output

Check out the output: The lines in red were not executed by any of the unit tests. Looks like there is a function that I'm not testing at all!

Mobile Monday Chennai

Update: See here for the latest update.

Varun Krishnan is organising the first Mobile Monday event in Chennai. Mobile Monday is something like Barcamp in the sense that its a tech gathering, except that the event is only 3 hours long and the topics revolve around mobility.

Mobile Monday, as its name states, is supposed to be on a Monday, usually after work. The upcoming Chennai event, however, will be on a Sunday (some people call this Mobile Monday on a Sunday or MMS). Mobile Monday is also supposed to be once a month, usually the first Monday of the month. The Chennai edition will be once a quarter instead. If there is demand, the frequency will be increased.

So here are the details

Date: April 22nd, Sunday (2 weeks from today) Update: The date has been changed to sometime in May
Time: 10am to 1pm
Venue: Mobile Worx office

The schedule is three tech talks, 20 minutes each and three product demos, 20 minutes each followed by lunch.

I've signed up for a talk on Location Based Services. I'll be talking about a couple of projects that I did while I was in Singapore.

If you are interested in giving a talk, head over to the Speakers wiki page and add your name there.

Barcamp Bangalore 3 Roundup

I was in Bangalore on Saturday and Sunday attending the third Barcamp to be held there. Here are bits from the event that I found interesting. This is a long post, you have been warned :) Note that the following is not in chronological order.

Python Conference

I met a few guys from the BangPypers groups - Anush, Anand, Kushal, Swaroop and Sidu. Apart from the Python conversations, I also asked if anyone knew what was happening on the planned Python conference. No one had much information, so I'm not really sure whats happening. Is the Bangalore python conference going to happen?

The Knowledge Foundation

The Knowledge Foundation is the name of the group that is behind various unconferences (plus Proto.in) in Chennai. We had a discussion on how to spread the unconference movement to other cities and other non-tech settings. While Barcamps have been a big success in India, there has not been much activity outside of a few tech events like Barcamps and Mobile Mondays. TKF has been working on that in Chennai where apart from Barcamp we've had Blogcamp, Wikicamp, Proto.in and Rooftop Film Festival (that reminds me - I have to upload the RTFF photos tomorrow). Coming up are another edition of Proto, Mobile Monday and Shoot. Quite a few people were enthusiastic about having small gatherings on other such topics. It would be great to see more unconferences on a whole range of topics in various cities.

Silver Catalyst

I talked to a few thoughtworkers about Silver Catalyst - Amir, Sidu and Vivek. Amir had recorded my session and had also interviewed me at the second barcamp bangalore back in December. I also met Umesh and a couple of other guys who had also attended my session in the previous barcamp. It was interesting to see how the tool had progressed in the three months between the barcamps.

Sidu took some time out to point me towards Mingle, which is a tool developed by the new Studios division of Thoughtworks, scheduled to release in June. I wanted to discuss Silver Catalyst a bit more with Vivek, but I lost him in the crowd somewhere.

Organisational Culture

There were three sessions that I thought touched on the same topic of culture from different angles. The first was by Savita Kini on Products vs Services in Indian companies. The second was by Satish about whether soft skills training was required. The third was by Shourya on the Bambi code jam initiative at GE.

Savita pointed out a number of challenges and opportunities facing Indian companies. Shourya made a good point in the discussion asking why we think that products are superior to services. Some of the points raised were profit margin and scalability. I'm with Shourya here that I'm not convinced that products are always better than service. Profit margin depends on the product. The XBox for instance is sold at a loss. Mobile phones have very thin profit margins. Scalability is also not a great reason as it all depends on the market. If you put out a product in a small market, there may not be much scope for scalability.

In the end, I think the biggest reason that product companies are not as prevalent is because of the organisational culture. Large companies are by nature risk averse. Since there are a lot of profitable service companies, we tend to follow them and not take the risk of doing something new. One of the cool thinks about Barcamps is that it gets a lot of product startups together.

That ties in with the second talk about soft skills training in IT companies. Soft skills are certainly required. The problem arises when the training is imposed rather than sought. Sukumar made a good point that knowledge can never be given, it can only be taken. I dont think a one day training can build teams for instance.

Satish had a good point which, to me, got to the heart of the matter. Fishermen work in teams because they know that if they don't, all of them will have trouble with dinner. The result of success and failure is right in front of them. They don't need to take a training class to realise that. The problem in large organisations is that there is a disconnect between the work and the outcome. Developers who do the work cannot see the outcome of their work, either in the form of customer satisfaction or bottom line impact. Thus they associate with the closest outcome that they can see - the salary - and start working for the money instead of the company.

A startup culture is like the fisherman team. Everyone is aware of pretty much everything and they know the outcome of their work. The question is can we get a startup like culture in a medium or big company?

Which brings us to the third talk by Shourya on Bambi, a codejam initiative taken in GE. He explained how he got the idea from the Yahoo Hackday in the first Barcamp Bangalore, and implemented it in the form of a codejam at GE. He explained how there was trouble convincing the management about holding it. Finally when it was held, it produced one IP plus a number of other cool, but not so useful projects. The second time around, the management actually asked for it to be held and it was held with some changes in the format.

A codejam, 20% time to work on your own project and other such initiatives are 'cool' initiatives. The problem is that they do not have direct ROI consequences. However, the actual dynamics are a lot more complex. Such initiatives for instance may (or may not) help attract smarter programmers, or reduce attrition, or build up a specific company culture. It may fail if the current programmers are not the type to take part in such events. So the success depends on both the current culture as well as the future culture.

Another problem is that some companies don't really believe in it. Someone was talking about 20% time in their company and how you can work on what you want provided you finish the main work. Since work pretty much takes the whole time, plus a bit more, this is as good as maintaining the status quo. The point of 20% in Google is that you can spend that time even if you are working on something else. They are okay with the fact that the main work will take a bit more time. So that again points to a difference in company culture.

So how can these forces be resolved in managing the current culture or creating a new culture? Thats the big question, but the discussions were interesting.

Proto

There was a discussion on the venue for Proto. A lot of people from Bangalore wanted the next edition of Proto to be held in Bangalore. This was a completely ad-hoc session held under the trees (an ideal unconference session!). [Off-topic digression: When a projector is involved there is a clear speaker/audience divide. Get everyone sitting in a circle and it becomes a discussion with everyone pitching in their views.] Back on-topic, there was a good discussion where it was decided to have the next edition in Chennai with interested people from Bangalore joining the team, and the session after that will be in Bangalore.

Barcamp Bangalore 2 DVDs

Amir from Thoughtworks had recorded a lot of sessions from Barcamp Bangalore 2. It came to 8 DVDs in all after editing. Kesav brought his set of DVDs for us to make a copy. The problem was that it takes a long time to copy all eight. Finally, Kushal volunteered to make copies for us. Thanks Kushal!

Quake deathmatch

A few of us had a Quake deathmatch using OpenQuake on Saturday evening. Jace, Shreyas, myself and a couple of other guys were there. It's been many years since I played Quake so it was great fun. For some inexplicable reason OpenQuake would sometimes stop reading the horizontal axis of the mouse. The mouse would work fine in the menus, but in the game I could only look up and down and couldn't turn left and right. Quiting and rejoining the server fixed the problem, but the frag count went back to zero. It happened thrice in all. Highly frustrating. Absolutely no idea what caused it.

Dinners

A dinner was organised at the end of Saturday. I didn't attend it because I went to meet a few guys at the Forum. At the dinner were ex-thoughtworker and AI hacker :) Ravi Mohan, Bharat Guruprakash, Manoj Govindan, Rajesh Babu and myself. A number of interesting topics were discussed like machine learning, silver catalyst, the ongoing drama at the scrumdevelopment group that has been tragically comical, work at thoughtworks and subex, thoughts on the CSM, agile in India, applying agile to embedded systems, PhD, research, entrepreneurs, dungeons and dragons, board games and so on.

We had to catch our train on Sunday night. Ramesh Babu joined Sukumar and me for the evening. He had hosted Syed, Kiruba and Sukumar on Saturday night. Kiruba and Syed had a dinner appointment so we split up at the station. They returned 5 minutes before the train was scheduled to leave, running and sweating, and then the train left half an hour late :)

Photography

I met Mahesh Shantaram at lunch on Sunday. Mahesh is a documentary photographer based in Bangalore. He has some very nice photographs that you can see online here. Amogh had a session on conservation with stories based on his photographs. He goes into the forest every weekend to take photographs. We later caught up with him at Cafe Coffee Day after the event was over to have a discussion about the Shoot event thats coming up and also if we could join up with him on a trip sometime.

Sessions

A few quick words about the sessions that I attended.

Motvik showed a demo of their WWIGO product. They showed how to use a mobile phone as a webcam and transmit data via bluetooth to the laptop where it was streamed over skype to another person. Inactiv explained their Activ Mobs service, a group SMS service for groups of friends. There was a presentation about Bigswerve by Raj Bala. What they do is to go through the comments at various websites and generate a network based on comment patterns. Comment patterns an example of implicit networks where the network is not formally defined and a comment automatically implies a level of engagement with the site. A few thoughtworkers gave a presentation of Continuous Integration and Cruisecontrol.rb. Sowmya Karmali had a session explaining the Mifos project. Marek Tuszynski had an event on How to organise an open event. Marek was the organiser for AsiaSource and a few other events. I think it started early or I messed up the timing because I really wanted to attend but when I went there the session was just wrapping up. There was a nice session on usability by Hari and Ram.

Bangalore Barcamp 3

I'm off to Bangalore this evening to attend the third barcamp at bangalore. I'm hoping for some good discussion and meet up with interesting people. I'll post a roundup of the event when I get back on Monday.

Little Green Guys With Guns

I found out about Little Green Guys With Guns, a play by email game through Ian Hickson. I played through the tutorial and it looked pretty fun, so I've joined the newbie game thats running. If you're into this sort of thing, take a look at LGGWG and join in. It should be fun!

Python decorators and generators: An introduction

I gave an introduction to python decorators and generators at yesterdays ChennaiPy meet. I've uploaded the presentations to the files section of the yahoo group (OpenOffice 2 format).

More on python decorators

A few months ago, I wrote an article on Using python decorators to implement guards. When Parthan from ChennaiPy was preparing for talk on decorators due to be given at the FOSS event at NIT Calicut, he mailed me for a few more examples of decorators. He's put up the examples along with an explanation of each example over here. If you are looking to learn more about python decorators, check out the blog post.

The beauty of clean URLs

One of the objectives when I was setting up the Silver Stripe website was to have clean URLs. If you notice, almost none of the URLs on the Silver Stripe website contain the filename or file extension within them. I'm using Apache's mod_rewrite to point the clean URL to the actual file on the filesystem.

Today, I'm happy I did. I recently changed over a few pages from static html to php, and a corresponding change in the filename. However, since the URLs don't point directly to the file on the filesystem, I didn't have to change any URL.

If you are working on a website, I highly recommend using mod_rewrite or equivalent to use clean URLs that are separated from the actual implementation.

There is a body of information on how to design URLs. Check these out for a start

• Cool URIs don't change
• Slash Forward (Some URLs are Better Than Others)

Broken statistics: Normalising test scores

The state government recently announced that a common entrance test for admission into engineering courses will be abolished and replaced by normalisation. You see, there are many different boards, each with its own tests of differing difficulties. We need a method for comparing the scores from different boards for processing admission to colleges.

Lets say that we have two boards, Board A and Board B. The scores in the two boards are as follows

Board A	Board B
100	98
100	83
98	82
95	81
93	80

Looking at the above scores, it seems that the test paper for Board A was a lot easier. If we compare the scores directly, Board A students have an unfair advantage. That's why we need to normalise the scores in order to properly judge the relative merits of the students from different boards.

The proposed normalisation scheme (according to the newspaper) is as follows:

• The ratio between the highest marks constitutes a multiplication factor
• This multiplication factor is applied to all the scores

To get back to the example, the multiplication factor is 100/98. The table now looks like this

Board A	Board B	Normalised Board B
100	98	98x(100/98) = 100
100	83	83x(100/98) = 84.7
98	82	83.7
95	81	82.7
93	80	81.6

Now we can see why this method is so broken. Although a casual glance tells us that Board B's test paper was a lot harder, the scores after normalisation have hardly changed! This is because one person got a good score of 98. This single data point is an exception to the rule, yet it has influenced the process so much as to render the normalisation completely meaningless.

This is an example of broken statistics. The top mark is usually an outlier and its a bad idea to calculate statistics of some data based on the outlier values.

I'm pretty amazed that they adopted this method of normalisation. Surely some statistician must have brought up this issue??

So what can be done?

I'm not a statistician, but here are some ideas that come to mind.

Fitting to a normal curve

How this works is to take the top mark and map it to 100, take the bottom score and map it to 0, and then map the intermediate scores based on a normal distribution with mean 50 and some experimentally obtained standard deviation. The two distributions can then be compared.

Drawbacks: This only works if the score distribution is normal! Usually it is not. The graph is generally skewed towards higher marks, as there are a lot more people passing the test than failing it. A common mistake is taking a non-normal distribution and fitting it to a normal curve.

Percentiles

Another scheme that is used is percentiles. The percentile is the percent of people who scored below you. So a 95 percentile means that 95% of the population who took the test are below you. Or in other words, you are in the top 5% of the population. Then, instead of comparing the absolute marks, you compare the percentiles.

This is like comparing rank, except that it normalises the fact that different number of students might have taken the two tests.

Drawbacks: A big drawback with percentile is that it can break near areas of high density. Take the above example again

Score	Percentile
98	80
83	60
82	40
81	20
80	0

As you can see, only 4 points separates the 0 percentile with 60 percentile. Of course, the effect is pronounced in this example because the sample size is so small. The same thing happens to a lesser degree in larger samples if the data is very dense in certain parts of the distribution.

Conclusion

Neither of the above solutions are particularly satisfying. Both introduce distortions of mapping one distribution onto another. In one case we are mapping a non-normal distribution to a normal one, in the other we are mapping it to a linear distribution.

The ideal solution would be to find out the actual distribution for test scores. Once that is done, both sets of scores can be equalised using the parameters of that distribution and compared. Since the distribution will be the same for both sets of test scores, the mapping will not introduce any distortion and the comparision will be fair.

Self Portrait

Self Portrait
Originally uploaded by Siddhi.

A reflection off the window. Singapore, 2003

Do you have your OpenID yet? — An OpenID Primer

OpenID is a technology that allows you to login to multiple sites using a single user name and password. The website needs to support OpenID login for this to work, but recently a number of websites have announced support for OpenID. With OpenID starting to get some traction, I decided to stop sitting on the fence and to see what it was all about.

When I started out, I was terribly confused about OpenID. It's actually really simple and straightforward. Hopefully this guide helps.

Getting an OpenID username and password

The first step was to get myself an OpenID account. This part confused me to start with because I assumed that there would be one centralised OpenID provider. Not so! OpenID is completely decentralised which means that there are a number of OpenID providers. You can create an OpenID account with any of them. After some poking around, I decided to create an account with claimID. You'll be asked to set a username and password for your account, and in a couple of minutes the account will be ready. Along with the account, you'll get a page that you can fill with links. Mine is here. The important part though is in the page source. Do a View Source on that page, and you get these headers:



We'll see how these headers are used later on.

Logging into an OpenID enabled site

Right, so we've got an OpenID login. How do we use this to login to an OpenID enabled website? Lets go through an example. We are going to use OpenID to log into Ma.gnolia. First, go to the site and select the option to login with OpenID. You will be asked to enter your OpenID URL. Thats right, you don't enter your user name and password here. Instead you enter a URL. In this case, thats the claimID URL that we saw before. Enter that URL as shown below and click the Login button.



What Magnolia does is it retrieves the URL provided and reads the OpenID headers contained in it. Here it is again



The openid.server and openid.delegate fields tell the website which server to contact to handle the login. Magnolia then takes these bits of information and asks the server mentioned in the header to log in. You'll now be redirected to the login page on the OpenID server, where you'll be asked for a username and password. Enter the username and password that you set while creating your claimID account and click the Login button. The OpenID server will now verify your credentials and log you in. In this case, Mag.nolia is asking for some more information.



Fill up the form and you're done. There are two ways of logging in. "Log In" will just log you in normally, but will ask you for confirmation everytime you log in to the site. "Log In and Trust" will log you in and not ask for confirmation in the future. In any case, you can always change the settings later. Once you login, you'll be redirected back to the Ma.gnolia website to continue what you were doing. In the whole transaction, Ma.gnolia never finds out what your user name and password was. All they know is that the OpenID URL was validated correctly by the backend server.

Logging into another site

Let try it once more by logging into livejournal. First, select the option to login with OpenID.



Next, enter your OpenID URL as before.



As before, livejournal contacts the OpenID server, but as we are already logged into that server, it doesn't ask us for entering the username and password again (Had you logged out of the OpenID server, it would have asked for the user name and password). Instead it jumps straight to the Log In confirmation screen. Click Log In and you're in. No account creation required! Pretty cool. All you need to remember is one user name and password and the URL to the page containing the OpenID headers.

OpenID delegation

Right, so you've got OpenID accounts at a number of sites, and one day claimID goes out of business. Have you lost all your login details at every website? Thats a scary thought. Luckily its all been taken care of. Enter OpenID delegation. Remember, you log into websites by entering a URL. The only purpose of that URL is to retrieve the OpenID headers. So, if you can create a page with the relevant OpenID headers, you can use that page as your OpenID login URL. Now, what I've done is to modify the blog template to add the OpenID headers to this page. Do a View Source on this blog, and you'll find these lines



If you're using claimID, you dont even need to figure out the code because they conveniently provide you with the code snippet to paste into your own blog or website. I can now login using my blog address as the OpenID URL.



Lets say I create accounts on websites using this URL. As far as the website is concerned, it is the OpenID URL which is your identifier, not the backend details. So if claimID were to go out of business, or for any reason I wanted to switch, all I need to do is create an OpenID account at another provider and change the headers on this page to point to the new provider's details. When I login at a website, the website will now contact the new backend provider to log me in. My login URL at the website remains unchanged! Neat!

Security with OpenIDs

One of the big misconceptions that I had was that OpenID was a centralised system for authentication around the web (remember Microsoft Passport? Google is getting there too). It always scared me that if one account got compromised and someone figured out the username and password, it put all the other websites at risk. Nothing of the sort.

One thing to point out. Login details are stored on the backend OpenID server. The websites never get to know it. Even if a website gets compromised, cracked, whatever, and data gets stolen, your login details are safe. So OpenID is actually quite a bit safer in that respect.

Still, your login details may get compromised by any number of other means. Here are some things you can do to make it more secure.

First, OpenID delegation is a great thing, so set it up ASAP. If your user name and password were to get compromised, all you need to do is to create a new account with claimID (or any provider) and setup the delegation to the new account. The next time anyone uses your URL to log in, they will need to enter the NEW user name and password. Cool!

Second, nothing stops you from creating two or three OpenID accounts. You can then use one for say, all the social networking sites, another for your blogs and so on. If one were to get compromised, damage is only restricted to the sites that share the same OpenID account.

Conclusion

OpenID is pretty cool. It's becoming big. Get yourself an account and start playing around with it. Run a website? Think about supporting OpenID logins.

I've only been using OpenID for a couple of days, so if you have any feedback, comments or corrections, I would love to hear it.