Broken statistics: Normalising test scores

The state government recently announced that a common entrance test for admission into engineering courses will be abolished and replaced by normalisation. You see, there are many different boards, each with its own tests of differing difficulties. We need a method for comparing the scores from different boards for processing admission to colleges.

Lets say that we have two boards, Board A and Board B. The scores in the two boards are as follows

Board A	Board B
100	98
100	83
98	82
95	81
93	80

Looking at the above scores, it seems that the test paper for Board A was a lot easier. If we compare the scores directly, Board A students have an unfair advantage. That's why we need to normalise the scores in order to properly judge the relative merits of the students from different boards.

The proposed normalisation scheme (according to the newspaper) is as follows:

• The ratio between the highest marks constitutes a multiplication factor
• This multiplication factor is applied to all the scores

To get back to the example, the multiplication factor is 100/98. The table now looks like this

Board A	Board B	Normalised Board B
100	98	98x(100/98) = 100
100	83	83x(100/98) = 84.7
98	82	83.7
95	81	82.7
93	80	81.6

Now we can see why this method is so broken. Although a casual glance tells us that Board B's test paper was a lot harder, the scores after normalisation have hardly changed! This is because one person got a good score of 98. This single data point is an exception to the rule, yet it has influenced the process so much as to render the normalisation completely meaningless.

This is an example of broken statistics. The top mark is usually an outlier and its a bad idea to calculate statistics of some data based on the outlier values.

I'm pretty amazed that they adopted this method of normalisation. Surely some statistician must have brought up this issue??

So what can be done?

I'm not a statistician, but here are some ideas that come to mind.

Fitting to a normal curve

How this works is to take the top mark and map it to 100, take the bottom score and map it to 0, and then map the intermediate scores based on a normal distribution with mean 50 and some experimentally obtained standard deviation. The two distributions can then be compared.

Drawbacks: This only works if the score distribution is normal! Usually it is not. The graph is generally skewed towards higher marks, as there are a lot more people passing the test than failing it. A common mistake is taking a non-normal distribution and fitting it to a normal curve.

Percentiles

Another scheme that is used is percentiles. The percentile is the percent of people who scored below you. So a 95 percentile means that 95% of the population who took the test are below you. Or in other words, you are in the top 5% of the population. Then, instead of comparing the absolute marks, you compare the percentiles.

This is like comparing rank, except that it normalises the fact that different number of students might have taken the two tests.

Drawbacks: A big drawback with percentile is that it can break near areas of high density. Take the above example again

Score	Percentile
98	80
83	60
82	40
81	20
80	0

As you can see, only 4 points separates the 0 percentile with 60 percentile. Of course, the effect is pronounced in this example because the sample size is so small. The same thing happens to a lesser degree in larger samples if the data is very dense in certain parts of the distribution.

Conclusion

Neither of the above solutions are particularly satisfying. Both introduce distortions of mapping one distribution onto another. In one case we are mapping a non-normal distribution to a normal one, in the other we are mapping it to a linear distribution.

The ideal solution would be to find out the actual distribution for test scores. Once that is done, both sets of scores can be equalised using the parameters of that distribution and compared. Since the distribution will be the same for both sets of test scores, the mapping will not introduce any distortion and the comparision will be fair.

Self Portrait

Self Portrait
Originally uploaded by Siddhi.

A reflection off the window. Singapore, 2003

Do you have your OpenID yet? — An OpenID Primer

OpenID is a technology that allows you to login to multiple sites using a single user name and password. The website needs to support OpenID login for this to work, but recently a number of websites have announced support for OpenID. With OpenID starting to get some traction, I decided to stop sitting on the fence and to see what it was all about.

When I started out, I was terribly confused about OpenID. It's actually really simple and straightforward. Hopefully this guide helps.

Getting an OpenID username and password

The first step was to get myself an OpenID account. This part confused me to start with because I assumed that there would be one centralised OpenID provider. Not so! OpenID is completely decentralised which means that there are a number of OpenID providers. You can create an OpenID account with any of them. After some poking around, I decided to create an account with claimID. You'll be asked to set a username and password for your account, and in a couple of minutes the account will be ready. Along with the account, you'll get a page that you can fill with links. Mine is here. The important part though is in the page source. Do a View Source on that page, and you get these headers:



We'll see how these headers are used later on.

Logging into an OpenID enabled site

Right, so we've got an OpenID login. How do we use this to login to an OpenID enabled website? Lets go through an example. We are going to use OpenID to log into Ma.gnolia. First, go to the site and select the option to login with OpenID. You will be asked to enter your OpenID URL. Thats right, you don't enter your user name and password here. Instead you enter a URL. In this case, thats the claimID URL that we saw before. Enter that URL as shown below and click the Login button.



What Magnolia does is it retrieves the URL provided and reads the OpenID headers contained in it. Here it is again



The openid.server and openid.delegate fields tell the website which server to contact to handle the login. Magnolia then takes these bits of information and asks the server mentioned in the header to log in. You'll now be redirected to the login page on the OpenID server, where you'll be asked for a username and password. Enter the username and password that you set while creating your claimID account and click the Login button. The OpenID server will now verify your credentials and log you in. In this case, Mag.nolia is asking for some more information.



Fill up the form and you're done. There are two ways of logging in. "Log In" will just log you in normally, but will ask you for confirmation everytime you log in to the site. "Log In and Trust" will log you in and not ask for confirmation in the future. In any case, you can always change the settings later. Once you login, you'll be redirected back to the Ma.gnolia website to continue what you were doing. In the whole transaction, Ma.gnolia never finds out what your user name and password was. All they know is that the OpenID URL was validated correctly by the backend server.

Logging into another site

Let try it once more by logging into livejournal. First, select the option to login with OpenID.



Next, enter your OpenID URL as before.



As before, livejournal contacts the OpenID server, but as we are already logged into that server, it doesn't ask us for entering the username and password again (Had you logged out of the OpenID server, it would have asked for the user name and password). Instead it jumps straight to the Log In confirmation screen. Click Log In and you're in. No account creation required! Pretty cool. All you need to remember is one user name and password and the URL to the page containing the OpenID headers.

OpenID delegation

Right, so you've got OpenID accounts at a number of sites, and one day claimID goes out of business. Have you lost all your login details at every website? Thats a scary thought. Luckily its all been taken care of. Enter OpenID delegation. Remember, you log into websites by entering a URL. The only purpose of that URL is to retrieve the OpenID headers. So, if you can create a page with the relevant OpenID headers, you can use that page as your OpenID login URL. Now, what I've done is to modify the blog template to add the OpenID headers to this page. Do a View Source on this blog, and you'll find these lines



If you're using claimID, you dont even need to figure out the code because they conveniently provide you with the code snippet to paste into your own blog or website. I can now login using my blog address as the OpenID URL.



Lets say I create accounts on websites using this URL. As far as the website is concerned, it is the OpenID URL which is your identifier, not the backend details. So if claimID were to go out of business, or for any reason I wanted to switch, all I need to do is create an OpenID account at another provider and change the headers on this page to point to the new provider's details. When I login at a website, the website will now contact the new backend provider to log me in. My login URL at the website remains unchanged! Neat!

Security with OpenIDs

One of the big misconceptions that I had was that OpenID was a centralised system for authentication around the web (remember Microsoft Passport? Google is getting there too). It always scared me that if one account got compromised and someone figured out the username and password, it put all the other websites at risk. Nothing of the sort.

One thing to point out. Login details are stored on the backend OpenID server. The websites never get to know it. Even if a website gets compromised, cracked, whatever, and data gets stolen, your login details are safe. So OpenID is actually quite a bit safer in that respect.

Still, your login details may get compromised by any number of other means. Here are some things you can do to make it more secure.

First, OpenID delegation is a great thing, so set it up ASAP. If your user name and password were to get compromised, all you need to do is to create a new account with claimID (or any provider) and setup the delegation to the new account. The next time anyone uses your URL to log in, they will need to enter the NEW user name and password. Cool!

Second, nothing stops you from creating two or three OpenID accounts. You can then use one for say, all the social networking sites, another for your blogs and so on. If one were to get compromised, damage is only restricted to the sites that share the same OpenID account.

Conclusion

OpenID is pretty cool. It's becoming big. Get yourself an account and start playing around with it. Run a website? Think about supporting OpenID logins.

I've only been using OpenID for a couple of days, so if you have any feedback, comments or corrections, I would love to hear it.

Django screencast: Create a wiki in 15 minutes

I've uploaded a screencast on how to write a wiki in 15 mins with Django. Its available on the Silver Stripe Blog here.

Wikicamp round up

Yesterday was my decompression day after wikicamp on Sunday. I can now actually gather my thoughts and write out a round up of wikicamp.

Overall wikicamp went off pretty well. Jimmy Wales was totally awesome. He was pretty much mobbed by the press and audience throughout and was totally willing to mix in with everyone, answering questions all day long. He had a long talk + Q&A, followed by a press conference, then 1-on-1 interviews, posed for photos with everyone and still had the energy to participate in the sessions and with the audience. He was one of the highlights for his enthusiam.

I really liked how the booklet on wikis came out. Many may not know that the whole booklet was done in a few hours by collaborating over a wiki. It could have done with some final editing, but for just a few hours of work, it came out really well. Some more perspectives on this from Balaji who designed the booklet.

I missed most of the first session including Jimmy's talk because I was sitting at the registration counter. We closed the registration desk at lunch and I started attending the sessions. One of the problems was the distance between the rooms for the two tracks. For some reason, I never went upstairs, and apparantly neither did many of the audience...out of sight, out of mind I guess :( I heard from some audience members that the talks upstairs were really good.

A possible solution could be to have all sessions in different corners of the main hall itself. How do you prevent the sessions from interfering with each other? Get rid of the mic, speakers and projector :) Thats exactly what I posted to the bangalore barcamp 3 mailing list last week. Here is an example of how such a setup would look. Lots of small groups of under 10 people. A perfect example of this was when some of the guys from the organising group asked me about my session on writing a wiki in 15 mins. The session was removed because of a lack of speaking slots, so we decided to go through it on the spot, just the 7 of us.

One of the things that I noticed in Barcamp Bangalore 2 and happened here as well was that the energy levels after the lunch session seemed to drop. We were also running quite a bit late on the sessions in the main hall. Keeping things going after lunch is a big challenge. I've been wondering about group games. Many unconferences have a round of Werewolf, for example. I've never played any of these, and I'm not sure what the response would be like, but I've read a lot of good things about them.

The highlight for me was the BoF session. Unfortunately it was the last session and there was not much audience. Anyhow, I was there in the front row and there was a very nice discussion on a number of topics. Topics like getting started with editing, handling vandalism, discussions about the recent changes patrol, role of protection, how semi-protection works, policy of verifiability, delete vs speedy delete, how wikipedia handles referencing a newspaper article that is unavailable online, policy on original research, policy on biographical pages, and a whole lot more. If you are interested, just search wikipedia for the above words. Very entertaining and illuminating. Good job by the panel - Ganesh Krishnamurthy, BalaSundaraRaman, Arun Ramarathnam, Bhadani and Atul Chitnis.

I caught up with Ganesh Krishnamurthy and had a discussion about wiki bots (in Python!!). He was due to give a session on it, but unfortunately we ran out of time and he was unable to give it. Actually quite a few sessions got cut due to a lack of time. Anyway, I did catch up with him and we had a discussion about it. He runs a bot on wikipedia that reads a list of towns, then gathers information like population, latitude, longitude etc from some sources and automatically creates a wikipedia article stub on that town. Very cool.

Also met the guys Underdog Productions, who are making a documentary on Jimmy Wales. They are a group of six guys, three of whom had come to record footage for the documentary. Their previous movie was Piece by Piece, a documentary on San Frasisco's graffiti scene. Way to go guys.

So what was the benefit of wikicamp for me? Lot of knowledge gained from the BoF session. Really interested in wiki bots now and doing more investigation along those lines. Also, I've been having an account on Wikipedia since April, 2006, but I had never done any edits, except a few anonymous typo corrections. I figured out the password to my account and just started off with my first edit using the account. Plus getting to meet some interesting people.

All in all, wikicamp was really helpful.

Wikicamp Photos

I've uploaded my photos of Wikicamp here:

http://www.flickr.com/photos/siddhi/sets/72157594557105143/

You can find all the wikicamp photos by searching for the tag "wikicamp". That list on flickr is here

http://www.flickr.com/photos/tags/wikicamp/interesting/

Upcoming conferences

There are a lot of conferences (and unconferences) coming up in the next few months. The calendar is pretty packed. Here are some that I'm keeping track of, in order of the dates:

• wikicamp.in (25 Feb): For a start, wikicamp.in is happening this Sunday. It looks to be very interesting, with almost 300 participants signed up including the founder of Wikipedia, Jimmy Wales. Also check out the Speaker list, and add your name if you are interested in giving a talk. And don't forget to join the mailing list for announcements.
• FOSS @ NIT Calicut (2-4 Mar): There is a FOSS conference coming up at NIT Calicut on the 2nd, 3rd and 4th of March. More information on the FCI Wikia
• ChennaiPy workshop (17 Mar): The next gathering of the ChennaiPy group will be a workshop on the more advanced Python concepts. It will be in Chennai on the 17th of March (Sunday). Take a look at the ChennaiPy wiki to attend or give a talk. I'll be giving an introduction to decorators, and will be touching on my blog post, “Using Python decorators to implement guards.” If you are interested in Python and are in Chennai, do drop by. If you would like to talk about something in Python, go ahead and add the topic to the wiki.
• Bangalore Barcamp 3 (tentatively Mar): Planning has started for the third edition of the Bangalore Barcamp. If you are interested in organising or attending, I suggest joining the bangalore_barcamp mailing list. Take a look at the call for organisers here.
• Python Conference (tentatively April): There are plans for a first Python conference in India. Again, join the BangPypers list to keep track of this. If you would like to help out in organising, reply to this thread.
• Agile India 2007 (tentatively July): If you are interested in agile software development, then look out for Agile India 2007. Planning has started, check out the call for organisers on the mailing list.

Managers and developers: Are you two teams or one?

One of the reasons why I love Dilbert so much is because Scott Adams really feels the pulse of the typical IT organization. How many times have you seen the above situation? In one project, the whole world (including me, and I was not even a part of the project) knew that the project was burning, except the people who really needed to know — the managers and stakeholders — had no clue as to the real state of the project.

Why does this happen?

Most manager-developer problems in a software company are caused by one thing: trust, or rather, a lack of trust. Managers do not trust the developers to do any work. They believe that the developers have to be pushed, otherwise nothing will get done. Developers in turn do not trust the management to cover their back. They believe that the managers will screw them when things go wrong. As a result, instead of working together as one team, they become two teams, each defending their turf from outside attack.

Once that happens the project is as good as dead. Communication across the border will virtually stop.

Back to the above cartoon:

Dilbert is comfortable telling Wally the real situation of the project because he knows Wally will support him. After all they are in the same 'team'. When he tells the PHB that the project is fine, he doesn't really mean that its fine. He knows if he brings up any issue, the PHB will blame the team. What he is really communicating is "I don't want to talk to you. I'll say fine so that you go away quickly."

If you are a manager, be sure that you cultivate trust with the development team. What better way to do that then to show that you are on their side. When the team asks for more RAM for their computers, do you get it or do you say that its out of budget? When the team says that the schedule is too short, do you tell that to the customer and try to find a solution or do you ask the team to work weekends?

This is one reason why I like agile processes and the Scrummaster role in particular. Scrum tries hard to make the Scrummaster fit in with the team, to support the team, to enable the team to perform. In exchange, when things go wrong, the team is often more open in bringing up issues because they are all one team, working together in the same direction. They know that the Scrummaster will help in solving the problem rather than blaming the team.

Catalyst: Public beta available for download

Catalyst has entered public beta. The beta is available for download here. I would love all of you to try it out and give some feedback so that Catalyst can be made better. The goal is that it should be useful for you, the user, so take a look and let me know what you think.

Tester Tested!

If you want a good blog to read on exploratory software testing in India, check out the Tester Tester! blog. Most of the testers seem to be requirements based testers, so its nice to see some alternative approaches getting popular. For more on exploratory testing, also check out James Bach's blog.

The more you know, the more you realise how much you don't know

I've been programming since the mid-late eighties. I started out with BASIC on a Commodore 64. By 1992, I thought I knew everything there was to know about programming. Then I learnt C, and found out that I didn't know very much at all. I programmed for another ten years. By 2002, I had programmed a lot. Surely by now I'd figured out everything about programming. Then I saw Scheme and I realised that I didn't know very much at all.

When I started testing software, I thought I knew everything about testing. Just check the requirements, create some tests cases to validate the requirements, run the cases, and you're done, right? Then I ran into TDD, Cem Kaner, Brett Pettichord and James Bach. Now I know that I don't know very much at all.

When I started working with UI, it all seemed so easy. Create windows and buttons and hook up events. Then I started reading Alan Cooper, Bruce Tognazzini and Don Norman. Now I know that UI design is actually very complicated and I don't know any of it.

Peter Norvig says it takes 10 years to get good at a skill. If you think you know something, either you've been doing it for years, or you're still at the first step. Most likely the latter.

The more you know, the more you realise how much you don't know.

Can your programmers read code?

Surprising as it might seem, programmers spend a majority of the time reading code, not writing code. Take my case. Even though I am working alone, on a new project, I spend the majority of my time reading code. That's not just my code, but also the code in the various open source modules that I'm using. At some point or the other, I've had to delve headlong into the source of Django, Dojo, Mochikit and ToofPy, and try to make out what the code is doing by ignoring the docs (if there are any!) and reading the source. If you are working in a team, the effect is magnified even more.

This was no sudden revelation. When I was hiring, I had a standard question to see if candidates could read code. Here is the code snippet.

int function f(somestruct *a) {
    int x = 0;
    if (NULL == a) {
        return 0;
    }
    x += f(a->left);
    x += 1;
    x += f(a->right);
    return x;
}

Candidates had to try to figure out what function f was doing.

First, I expected the candidate to see the left and right fields and guess that "a" was a pointer to a binary tree of some sort. I then expected the candidate to see the pattern of the recursive calls and guess that it was a tree traversal (inorder in this case). Once that's done, its simple to try out the function with some data and figure out what it does (count the number of nodes).

Given that everyone learns about binary trees, and especially inorder traversal, you would have expected many to get it right. Surprise! Only a few have ever got it correct.

Most answers were something related to linked lists or arrays. It seemed that every programmer when confronted with a pointer in any form assumed it was related to linked lists!!

To be successful at code reading, it is the ability to execute the program in our mind that needs to be developed. This is a learned skill, developed by just reading lots of code.

One conclusion is that many programmers don't spend enough time reading code. It also explains why in many teams, most code becomes spaghetti — programmers can't read the original code, so they hack their fixes.

So, do you check to see if programmers can read code? As an aside, you can now find out how most programmers can't write code either.

wikicamp.in

Right after proto.in comes wikicamp.in.

Do you use wikis? Fascinated by them? Never heard of them? Want to know more? Then you need to attend wikicamp, India's first event dedicated to wikis. Think of it as blogcamp for wikis.

Another reason why you should attend — Jimmy Wales, founder of Wikipedia will be attending!! Yep, thats right.

Wikicamp.in is an unconference, which means that anyone can come and everyone can participate. Find out how wikis are used on the Internet, how they are changing knowledge management in enterprises and even how they can be used to organise your personal content. Find out more about user generated content and collaborative editing and how they are changing the way we use the Internet.

This event takes place on the 25th of Febuary, Sunday, at Tidel Park, Chennai (map). For more information, take a look at the event wiki. To register, just add your name to the participant page.

Eastern Winds blog

Vignesh over at the Eastern Winds blog is writing about his experiences starting a 1 person Micro-ISV in Hyderabad. Some interesting posts there about typical concerns of mISVs like experiences with AdWords, SEO, payment processors and so on. These are some things that I'll have to tackle when I start selling online, so its been very interesting reading. If you are interested in Micro-ISVs, especially from India, check out the blog.

White stallion

White stallion
Originally uploaded by Siddhi.

A white stallion from Camargue. The Camargue region is famous for its white horses. The horses are born dark, but get lighter with age, until they become pale gray or white.