Flexibility vs Efficiency

This post by Dmitri kicked off a mini-debate about agility and rigidity. The debate is about whether a developer should be interrupted for a day to perfrom some side job. Check out the posts linked above for the whole story.

This is a common enough issue that it requires further examination.

At the root of the issue is what I call the flexibility-efficiency dilemma. This tradeoff is, in fact, at the root of many issues of agile vs traditional processes. The essence is that if you want more flexibility, you trade in less efficiency and vice versa.

Example

You have ten packages that you want to send by post. Each package involves buying the cartons, collecting the items, packing the items, writing out the address, filling up forms, and finally going to the post office and delivering them. There are two ways to approach this problem.

Approach 1

1. Buy ten cartons
2. Collect all the items
3. Pack the ten packages
4. Write out ten addresses
5. Fill up the ten forms
6. Go to the post office and drop off all ten packages

Approach 2

1. Buy one carton
2. Collect items for one carton
3. Pack the carton
4. Write out the address
5. Fill up the form
6. Drop off carton at post office
7. Buy another carton, and repeat the whole process ten times

Which approach is better? Most people would say the first one. Hmm. First, we need to define what we mean by better. Let's rephrase that question. Which one is more efficient? I think we can all agree on the first one. Its a no brainer. Just the ten trips to the post office makes the second approach way more inefficient. Let us say that it took ten days to send the ten packages via Approach 1 and fifteen days via Approach 2.

Now, which one is more flexible? Ah. This time its the second one. Why? Because it is much better suited to handle unexpected surprises.

I'm going to add in a surprise now. Surprise: After five days, you are informed that you need to go overseas for an office trip right now. Packages sent using Approach 1: Zero. Packages sent using Approach 2: Three.

Lets try another surprise. Surprise: It seems that customs is not letting such large packages go through. You need to repack into smaller cartons. Approach 1 impact: All your ten packages come back in two days. You repack everything for another ten days. It takes twenty days to get everything through. Approach 2 impact: After two days, your first package returns. You repack and resend it. The unpacked items are packed into smaller cartons the first time around. It takes seventeen days to get everything through.

Apart from the above approaches, are intermediate approaches. You could make five trips to the post office, sending two packages each time. The net result of efficiency and flexibility will be somewhere between the two extremes.

Software processes

Surprisingly (if you think about it, not such a surprise), the two approaches have a direct correlation with software processes. Just a couple of quick definitions. In the next paragraph, when I say 'agile', I mean an Agile (captial A) process. When I say 'agility', I mean ability to handle unexpected requests. So high agility mean high capacity to take on unexpected process, not adherence to an agile process.

At one end is traditional waterfall, exemplified by an up-front plan and minimum adaptation to changing conditions. It can have high efficiency—provided the requirements never change and the software is understood perfectly.

At the other end is the ad-hoc process (bet you thought it would be agile at this end!). With ad-hoc process, everytime someone asks, you can just drop what you are doing and handle it. There is no process constraining you. However, efficiency is extremely low. You are multitasking so often that you never get anything done.

Somewhere in between, but more towards high agility, are agile processes. This is something like the five trip,two package per trip, example above. The idea is that you fix an iteration length. During the iteration, programmers are left alone to work. At the start of every iteration is a 'change point.' This is a point where stakeholders can come in and change the course of an iteration. By choosing shorter iterations, you create more 'change points' for more flexibility. Or you can choose a longer iteration for more efficiency. I'll write more about choosing iteration length in a future post.

FINALLY: The answer

With that under our belt, we can finally decide what to do about the problem. The situation is that one of your developers needs to be taken off for a day to implement a small feature for a sale. Should you do it? Scrum says no disturbance in the middle of the sprint. Joel says that doesn't sound agile to him.

What to do?

The simple answer: Ask the managers to decide if the sale is important enough to delay the other project and take a developer off if it is.

The complex answer: Your average manager is not Joel Spolsky. Most managers are under the mistaken impression that there will be no impact on the original project (hey, its only one developer for a day right?). Not all requests are really all that important. Be clear about the cost to the original project, because this is often hidden to them. Then go ahead and accept the request, but be _very_ careful that the process does not degenerate into a ad-hoc-handle-every-change-every-day process.

This post is a part of the selected archive.

Leaving today

I'll be leaving Singapore today at 2035 SGT, arriving in Chennai at 2200 IST. Only an hour to go before I leave for the airport.

Returning to Chennai

Last Friday was my last day of work here in Singapore. I will be returning to India sometime this week. I've been in Singapore for four and a half awesome years. So why am I returning? I just get this feeling in me that a lot of cool stuff is going to happen in India. So far, India has only been seen as a cheap destination for outsourcing, but that is starting to change. There are a lot of startups, interesting conferences and user groups that are being formed. I could feel the change in the environment when I was in Bangalore a year ago. Very different from when I was there four years ago.

Yesterday, I found that the second Barcamp Bangalore is going to be held on December 2-3. This is exactly what I mean. In the last year there have been Barcamps in virtually all the major cities. Add in stuff like the Agile India conferences, Proto.in, TieCon India.. the list goes on. Can you imagine that the Bangalore Python yahoo group has 230 members? A few years ago, I would have stuggled to find ten people interested in a language that is outside of what is used at work. Very cool. Its for stuff like this that I'm coming back.

Spree River

Spree River
Originally uploaded by Siddhi.

Berlin: Friederichstraße S-Bahn (Metro) Station over the Spree river

Do we need an Apgar score for software?

Daniel Read has a very thought provoking post on whether software needs an Apgar score.

First, I encourage you to read both the original New Yorker article and Daniel's post.

What is the Apgar score? If you don't know, you didn't read the article, so go back and read it. Okay, okay :) The Apgar score is a method used to determine the health of a baby at childbirth. The baby is scored between 0–2 on five factors: heart rate, breathing, muscle tone, skin irritability and skin colour, giving a total score between 0–10. The score is measured once at 1 minute after birth and again at 5 minutes after birth. A score of 7 or more is a healthy baby. Less than seven does not necessarily indicate an unhealthy baby, but doctors will look at the baby to be sure. Before the score was devised, determining the baby's state was a complex mass of subjective measures which often meant that the baby was not given treatment when required because some factor was overlooked.

The key idea behind the Apgar score is that it is simple. It is simple to calculate. It is simple in that it condenses a number of subjective factors into one number. It is simple in that an instant decision can be made as to whether the baby is healthy. The final Apgar score does not take into account the myriad complexities, but it is intentionally kept simple so that it can be used as a quick, shorthand method of assessing the baby's health. That's what it is—a quick, shorthand assesment.

Daniel asks a question—Do we need an Apgar score for software development? Something that can serve as a quick, shorthand assesment of the project's health. Current methods of determining a project's health are a complex mass of subjectivity. Maintainability? Security? Relaiability? Code coverage? Usability? Requirement satisfaction? Customer satisfaction? There is no effective means of quickly measuring the project quality in a way that would give a rough idea as to the quality of the software. An Apgar score could help a lot here. It need not be exact. It just needs to be somewhere in the ballpark so that a decision can be made as to whether the project is healthy or we need to investigate more.

I think this is an interesting idea. It may not be practical as yet for software, but is nevertheless thought provoking. Check out some of the comments on Daniel's post as well.

This post is a part of the selected archive.

Twenty five posts on agile

I was re-organising my bookmarks yesterday and I decided to collect some of them into a post. Here then, in no particular order, are twenty five posts on agile:

1. Ten-Minute Build
2. The Next Big Thing
3. Knowing when you're done
4. The Coad Letter: Strategy Edition, Issue 40, Working Together -- Lessons Learned From Fred
5. An XP Team Room
6. Using Continuous Integration? Better do the "Check In Dance"
7. The Satir Change Model
8. Being an effective Onsite Customer or Product Owner
9. Agile team room photos
10. Scrum Tools Roundup
11. Agile Adoption across the Industry
12. Testing: Coverage Reports Considered Dangerous
13. It's Not Just Standing Up: Patterns for Daily Stand-up Meetings
14. Risk To Pattern Table
15. Buildix: The Agile development platform on a disk.
16. Dysfunctional Agile, Agile-in-the-Large
17. Project risk reduction patterns
18. Sending People Home
19. Agile - Losing Sight of the Goal?
20. Agile, Top Down
21. No Best Practices
22. Baby steps for your project in the new year
23. A Pragmatic Project: Live In Concert
24. Badge of Merit
25. Unjust Deserts? [PDF]

This post is a part of the selected archive.

Proto.in, Chennai, January 20-21, 2007

What is Proto.in?

Proto.in is positioning to be something like the DEMO conference. From the Proto.in blog:

One question that came up over and over again, was the question of What this event is supposed to achieve and what is the goal that we are aiming for? It’s simple. We are not just an outsourcing destination. We have innovation happening and we just want to shout that out to the world, so that the investments are varied and will also take that into account.

The event is scheduled for the dates of January 20th and 21st. We are narrowing down on the venues and the final list of companies will be 30 of them, with ten minute slots to present and make their pitch. The three questions that basically need answering are:

1. Who are you?
2. What do you do?
3. Why do you want the money? or What can you do with more money?

This is a pretty exciting time to be in India. Lots of cool stuff happening.

Agile in the Communications of the ACM

I got my October 2006 issue of the Communications of the ACM in the post today. The featured topic of this issue is "Flexible and Distributed software processes", and a lot of articles feature agile software development, especially in the context of distributed software development. If you have ACM digital library access, you can read the articles online at the above link.

The leading article, Flexible and Distributed Software Processes: Old Petunias in New Bowls?" (ACM Digital Library membership required) has some interesting commentaries by well known experts. Although about global software development and distributed teams, much of it is also applicable to co-located teams. Here are some snippets

David Parnas: I have been hearing the term "software crisis" for more than 40 years. Clearly, it is not a crisis; it is a chronic problem. Each time someone uses the term "crisis," it is a preface to the announcement of a new miracle cure for the problem. Examinations of the cure usually reveal new words for ideas the have been tried before without much effect. This decade's crisis is global software development (GSD); this decade's miracle cure (for all the ills of the industry) seems to be a collection of methods known as "agile."

It is fashionable to speak of "grand challenges." The real grand challenge is not to find ways to to avoid producing documentation, but to find ways to produce useful documents—documents that take time but save more time. We will find that real agility comes from good design that is well documented in precise, lean documentation.

Barry Boehm: There are no one-size-fits-all solutions. The best way I have been able to find is to use risk as a way to determine where to go agile and where to go document-driven. Thus, for example, if you are developing a graphic user interface (GUI) for an unprecedented decision support system and want to document its requirements, the most frequent answer you will get from users is, "I can't tell you in advance, but I'll know it when I see it (IKIWISI)."

In such a case, it is a high risk to try to document the GUI in advance, and with a GUI builder tool, it is a low risk not to document it.

On the other hand, when you are outsourcing a relatively stable piece of business logic to a contractor 10 time zones away, it is a high risk not to invest in a significant amount of thorough documentation of the interfaces and protocols connecting the outsourced sofware to the rest of your software.

Giancarlo Succi: No one thinks that analysis and design are useless. But consider a system to dispatch tracing messages and other information to a group of trucks. This domain is likely to be alien to most software developers. In such a case, would it be better to first spend a lot of time doing the upfront design, and eventually writing the code, or to work incrementally, involving the end customer, interleaving some analysis, design, and even coding, so that the developers grow their knowledge of the system domain?

Matthew Simons: While the idea of story cards appeals to those with no desire for reading or writing technical documentation, the reality of the situation is that such scanty artifacts are rarely sufficient for development of anything beyond simple systems for highly accessible individuals or small groups of users. This scenario rarely applies to GSD. As the picture becomes more complex, we have found the story card-only approach quickly becomes inadequate.

While writing effective documentation is a good place to start, we have found it takes a lot more than that to deliver complex systems with distributed teams. Without the discipline that comes from the remaining agile practices (such as Test-Driven Development, Continuous Integration, Pair Programming, among others) good documents are nothing more than a step on the long and perilous path toward successful delivery.

Like I said, the quotes above are only snippets. The actual commentaries include a lot more context and information than I have provided here. Read the whole article if you can, it is well worth it.

This post is a part of the selected archive.

Agile is not a set of techniques

A few weeks ago, Steve Yegge had a post on Good Agile, Bad Agile which caused a huge flutter in the blogosphere. The reactions on the Extreme Programming list went from "this is the usual 'I don't understand agile, so I'll bash it' crap" to "listen to your enemies. They'll tell you things about yourself that your friends never will." with a whole bunch in between. The post even turned up on Slashdot and Joel. Wow. Steve then posted a followup titled Egomania itself.

I personally think that both the articles have some good parts and some bad parts. He correctly points out that agilists can sometimes be loud and dogmatic. This is true and I've written about it before: Agile is not XP.

Having said that, I really do think that agile principles from the Agile Manifesto have a lot of value. Most of the problems associated with agile come from the 5 pitfalls list. Just to summarize, here is the list in short:

1. Unfamiliarity
2. Top-down thinking
3. Culture change
4. Incomplete implementation
5. Silver bullet syndrome

When something starts to get popular, there are any number of managers who want to implement it to get immediate results. "This book says that if we write requirements on index cards instead of an excel sheet, we will be more successful." Obviously that is nonsense, but for a number of people, that is the take away message from agile. "Do X and you will be more successful," where X can be test first, pair programming, daily standups or what have you. You then have the manager forcing the team to have standups everyday, its 45 minutes long (Standups are supposed to be short, not more than 15 mins), and naturally everyone just hates it. The result? The engineers hate it, and the manager is disillusioned.

Don't obsess over techniques. If pair programming doesn't work for you, that's perfectly fine. Don't do it. Do what works. Maybe you are having a lot of success with daily standups. Continue to do it. A confession: I don't do pair programming either.

Techniques do not exist in isolation. There are a number of factors that contribute to the success of a technique: the people, the environment, the culture, the management. A technique might work great for Google, but be terrible in your environment. A technique that is terrible for Google might work great for NASA.

So if you are not going to focus on techniques, what do you focus on? I prefer to focus on principles. Being agile is not about writing requirements on index cards or doing test first development. To me, being agile is about valuing people, working collaboratively, improving continuously and delivering consistently.

Now, that is a lot easier said than done. A comment on Steve's post goes like this (the company mentioned is Google):

I worked on a team (at the same company as you) where we were essentially "forced" to use Scrum. I don't think a single engineer on our team found it useful, and I personally found it annoying, but we didn't really have a recourse for evaluation: I gave feedback that I didn't like it, but I'm not sure that everyone else was comfortable doing the same, and there weren't any of these wonderful double blind weekly evals. I also think its results were mis-used. The data that was derived from it (how much our team could accomplish in a week) was tainted and it shouldn't have been turned around and used to make decisions.

If the engineers didn't find it useful, the practices should have been changed. Continuous improvement of the process by the team is a cornerstone of agile. This is a perfect example of following the techniques and ignoring the principles.

Agile is not a set of techniques. It is the principles that matter more than the techniques. Focus on the principles, and do the techniques that work in your environment.

This post is a part of the selected archive.

Stairs

Stairs
Originally uploaded by Siddhi.

Near my office, late at night

Techcrunch on PayPerPost

A year and a half ago, I wrote a post, pointing to this cartoon on corporate blogging. I had said

While its meant to be a bit funny, I can see the day when something like this actually happens.

A couple of days ago, Techcrunch had a post about the controversial PayPerPost, where bloggers are paid to post product reviews and the company can mandate that they be positive. This kind of thing is nothing new, as it happens all the time in mainstream media (what, you though all those magazine articles were unbiased? See this), but it has generated a lot of controversy for 'tainting the unbiased nature of the blogosphere', whatever that means. Check out the comments on the Techcrunch page.

Bug Bash

Bug Bash is a techie comic strip that I've been following for a long time. It's absolutely hilarious, so if you haven't heard of it yet, take a look here.

Export django database to an xml file

Here is a simple python script to export your django database to an XML file. I haven't tested it out very thoroughly. It seems to work for the fields that I have in my model - CharField, TextField, DateField, IntegerField, PositiveIntegerField and ForeignKey. If you find any bugs, add a comment to this post.

# setup the environment
import os, sys
sys.path.append(os.pardir)
os.environ["DJANGO_SETTINGS_MODULE"] = "settings"

class XMLWriter:
    """Helper class to write out an xml file"""
    def __init__(self, pretty=True):
        """Set pretty to True if you want an indented XML file"""
        self.output = ""
        self.stack = []
        self.pretty = pretty
        
    def open(self, tag):
        """Add an open tag"""
        self.stack.append(tag)
        if self.pretty:
            self.output += "  "*(len(self.stack) - 1);
        self.output += "<" + tag + ">"
        if self.pretty:
            self.output += "\n"
        
    def close(self):
        """Close the innermost tag"""
        if self.pretty:
            self.output += "\n" + "  "*(len(self.stack) - 1);
        tag = self.stack.pop()
        self.output += "</" + tag + ">"
        if self.pretty:
            self.output += "\n"
        
    def closeAll(self):
        """Close all open tags"""
        while len(self.stack) > 0:
            self.close()
        
    def content(self, text):
        """Add some content"""
        if self.pretty:
            self.output += "  "*len(self.stack);
        self.output += str(text)

    def save(self, filename):
        """Save the data to a file"""
        self.closeAll()
        fp = open(filename, "w")
        fp.write(self.output)
        fp.close()

import django.db.models

writer = XMLWriter(pretty=False)
writer.open("djangoexport")
models = django.db.models.get_models()
for model in models:
    # model._meta.object_name holds the name of the model
    writer.open(model._meta.object_name + "s")
    for item in model.objects.all():
        writer.open(model._meta.object_name)
        for field in item._meta.fields:
            writer.open(field.name)
            value = getattr(item, field.name)
            if value != None:
                if isinstance(value, django.db.models.base.Model):
                    # This field is a foreign key, so save the primary key
                    # of the referring object
                    pk_name = value._meta.pk.name
                    pk_value = getattr(value, pk_name)
                    writer.content(pk_value)
                else:
                    writer.content(value)
            writer.close()
        writer.close()
    writer.close()
writer.close()
writer.save("export.xml")

Note to self: Find a better way to colorize the source code

This post is a part of the selected archive.

Handling conflict

Conflict. It's that word again. We have all faced it and most of us would rather not discuss it, but sooner or later you will be faced with such a situation and it's helpful to be able to understand it. In this article, I will discuss what I understand by conflict, and how we can handle it.

Conflict - good or bad?

Let's face it. People are different - that's what makes us people rather than machines. People with different viewpoints will naturally see things differently, inevitably leading to conflict.

A complete lack of conflict represents a dangerous situation. One possibility is that everyone thinks and behaves the same way. This represents a lack of diversity in the group that can easily lead to not exploring alternate viewpoints and 'groupthink'. Another possibility is even worse - that team members are afraid to speak up, due to some underlying fear. Most commonly, those who present alternate viewpoints are told that they are not 'team players', and the person in question stops raising objections from then on.

To a certain extent then, conflict is essential. It represents the heterogeneity of the group and leads to the exploration of alternate viewpoints.

It is all too easy however, to get caught up in a situation where you are forever arguing with someone with no end in sight. This happens when good conflict degenerates into dysfunctional conflict. After a point, the arguments are no longer on the merits of each case, but tied into the egos of the participants. A decision at that point takes political connotations and is perceived as an attack against the 'loser' of the conflict, which in turn leads into rapidly deteriorating relationships between all those involved.

Thus we see that conflict can be both good and bad. If you have no conflicts at all, then that is a sign that you need more diversity. In the rest of this article I will only talk about dysfunctional conflict.

Alignment

The first step in conflict resolution is alignment. The most obvious form of conflict is when two individuals/teams/groups are working in opposite directions. There are two possible reasons for this. One reason is that the groups are deliberately sabotaging each other due to political reasons. If this is case, the conflict is just a symptom or some other root cause, and you need to go fix that root cause. You have bigger problems.

The other reason is where both teams think that they are doing the right thing, but because there is no common vision, they are going in opposite directions. Both are convinced that they are doing the right thing and wondering what is going on with the other group. The result is a conflict where both sides are convinced about their righteousness. Resolving these conflicts is simply a matter of aligning both groups to a common vision, which of course is easier said than done and is left as an exercise to the reader :)

Handling conflict

Even when you have two people working to the same goals, it is likely that there will be conflicts in the various ways to get to that target. How can these conflicts be handled?

Acknowledge the conflict: The first step is to acknowledge the conflict. Acknowledge both sides in the conflict. Listen to both sides and make sure that they both get heard. The worst possible course of action is to pretend that a conflict does not exist or ignore it and hope that it goes away. First, it will not go away. Second, people want to be heard. It makes then feel empowered and feel that they are valued. Ignoring the individuals in the conflict sends a message that you don't care for their issues. Simply acknowledging the conflict says that you care about what they are feeling.

Separate the problem from the person: The next step is to separate the problem from the person. In any conflict, both sides have something invested in their position, which ties in their viewpoint with their ego. As long as that exists, it is impossible to solve the problem without making it seem personal. By separating the problem from the person, you can attack the problem without attacking the person.

Attack the problem together: Having got the problem in front of you, you can all discuss the problem together. By solving the problem together, you ensure that both parties will find the final solution acceptable. This is in contrast to mandating a solution, where one party might not like the solution but might accept it in order to close the issue. Remember, resolving a conflict is not about closing the issue, but about a solution that everyone feels good about. If one or both of the groups involved is not satisfied with the solution, you can be sure that the conflict is still simmering under the surface and will come out with more force the next time.

Example: How not to handle conflict

Amy: We need some way to bring new team members up to speed
Bob: We need some documentation
Chris: NO! We don't need more documentation

Amy: Chris, Bob has a lot of experience at this, maybe you should listen to him
Later...
Amy: Chris, you are always arguing with the team. You need to be a team player, not a disrupter
Chris: #$@!#$

The above is a case of attacking the person. Chris is likely to be extremely unhappy at the 'resolution'. Relationships are going to go downhill.

Example: A better way

Amy: We need some way to bring new team members up to speed
Bob: We need some documentation
Chris: NO! We don't need more documentation

Amy: Why do you say that?
Chris: Nobody reads documentation
Amy: Okay, that could be a problem. Bob, what do you think?
Bob: Yes, but the current situation is even worse
Amy: What other alternatives can we think of?
Chris: How about mentoring?
Bob: I'm not sure. We don't have the time to mentor each new person
Chris: Mentoring is the only way to teach the details that get lost in documents
Amy: Maybe we can have a small light document that covers the core areas, and we have mentoring to transfer knowledge of the details?
Chris: Yeah, we could do that
Bob: Good idea

In this case, Amy acknowledges both the team members positions, then attacks the problem, not the person. This allows all three parties to work together as a team to solve the problem.

Handling deep conflict

In some cases, the conflict has already been going on for a while. Both sides have already developed deep mistrust for each other and have their positions firmly entrenched. Solving such a conflict is much harder. The same principles apply as above, but because of the mistrust between parties, they do not want to talk to each other. More effort is expended towards preserving their position than towards resolving the conflict. In such cases, the you will probably need to work with the parties individually, as they will probably not see eye to eye together. The key goal is to change the mindset from defending a position to working towards a resolution. Once that happens, the above principles can be used again.

Most deep conflicts are rooted in certain past events. By understanding the history, it is possible to understand the viewpoints of the two sides and bring them to a resolution.

Example of deep conflict

Developer: The management is out to squeeze every ounce of work out of us. They make us work overtime for no particular reason. I refuse to work this weekend.

Manager: The developers are so unreliable. They keep giving excuses about why something cannot be done. We need to monitor them closely and keep pushing them or nothing would get done.

Another example

Widget team: The application team gets all the support. All the good programmers are there. We never get any support from them. It's better not to work with them if possible.

Application team: The widget teams keep asking us for support. How are we to support them and still get our work done? We need to minimize interaction with those teams.

Conclusion

The next time you face a conflict, keep in mind these three guidelines

• Acknowledge the conflict
• Separate the problem from the person
• Attack the problem together

This post is a part of the selected archive.

Testing is...

James Bach provides a great definition:

Testing is the infinite process
of comparing the invisible
to the ambiguous
so as to avoid the unthinkable
happening to the anonymous