Flickr Badge

Showing posts with label recursion. Show all posts
Showing posts with label recursion. Show all posts

Monday, May 22, 2006

Recursion Part 6: References and Further Information

The following is a part of a series of articles I had written on recursion.

This, the final part of the series contains sources for the articles and where to get further information.

All Parts:

Recursion Part 1: Introduction to recursion
Recursion Part 2: Tail recursion, Accumulators and Iteration
Recursion Part 3: Exercises in tail recursion
Recursion Part 4: Tree Recursion and Dynamic Programming
Recursion Part 5: Structural and Generative Recursion
Recursion Part 6: References and Further Information


The best way to learn about recursion is to learn a language that doesn't support iteration. Scheme is a good language to learn in this context. A Scheme interpreter can be got from here

Another good resource is the 6.001 course from MIT. The course material for this course is available for free.

Two highly recommended books:
1. Structure and Interpretation of Computer Programs (also called as The Wizard Book). There is an Indian edition, but not very easy to find.
2. How to design programs. This book is also available in an Indian Edition, but there is often not much stock.

Both books use the Scheme language, so they also serve the purpose of those trying to learn Scheme. Both books are also available for free on the Internet.

For more on dynamic programming, see Chapter 15 of the book 'Introduction to Algorithms' by Cormen, Leiserson and Rivest. This is a college textbook and is available in any bookstore. Most other books on algorithms also include a chapter on dynamic programming. Otherwise searching Google for "dynamic programming" will provide lots of articles on this topic.


Any questions? Comments? Please leave a comment using the comment form below.


This post is a part of the selected archive.

Recursion Part 5: Structural and Generative Recursion

The following is a part of a series of articles I had written on recursion.

This part deals with two areas for which recursion is commonly applied. The first, called structural recursion, is used to traverse through different parts of a data structure, processing each part in some way. The second, called generative recursion, is used when we want to divide a problem into smaller subproblems, which are then solved.

All Parts:

Recursion Part 1: Introduction to recursion
Recursion Part 2: Tail recursion, Accumulators and Iteration
Recursion Part 3: Exercises in tail recursion
Recursion Part 4: Tree Recursion and Dynamic Programming
Recursion Part 5: Structural and Generative Recursion
Recursion Part 6: References and Further Information


Let us start of with structural recursion. Here is an example, an inorder traversal of a binary tree:


void inorder(node element)
{
if (NULL == element) {
return
}
inorder(element->left);
process(element->data);
inorder(element->right);
}

This is a classic case of structural recursion. Each recursive call processes a part of the binary tree. Put together, we process all the elements in the tree. Structural recursion is often used when dealing with self-referential data structures like lists, trees and graphs. Functions to deal with these structures are hard to implement with iteration1, and even if we do manage to implement them with iteration, the resultant code is often very difficult to understand. Such code is best left as recursive.

In some cases, it is possible to modify the data structure itself so that common operations can be implemented iteratively. An example of this is the threaded tree data structure that modifies the classic binary tree to allow for iterative traversal.

In any case, except for exceptional cases, it is best to use recursive algorithms to implement structural recursion

Generative recursion is often used when we want to break up a problem into similar subproblems. The subproblems are solved and the results combined to get the final solution. Solving the subproblems in turn requires recursion to break the problem into smaller subproblems, and so on until we reach a trivial case. The recursive examples of factorial and fibonacci number calculations and the well known quicksort are examples of generative recursion.

Look at this fibonacci program again and convince yourself that it uses generative recursion:


int fibonacci(int n)
{
if (0 == n) {
return 0;
} else if (1 == n) {
return 1;
} else {
return fibonacci(n - 2) + fibonacci(n - 1);
}
}

Generative recursion that operates on a known finite set (eg: fibonacci(n) operates on integers between 0 and n) are good candidates for a dynamic programming approach. Generative recursion that operates on large or unknown sets (eg: functions that work with real numbers often fall into this category) are usually not condusive to dynamic programming. Some cases of generative recursion may have good iterative solutions, but this has to be considered on a case by case basis. In most cases, it is best to just leave them recursive.

Of course, all the above only applies to tree recursion. Tail recursion, whether structural or generative can always be converted into iteration.


1 The exception is linked lists. Linked lists usually result in tail recursion which can be converted to iteration.

Any questions? Comments? Please leave a comment using the comment form below.


This post is a part of the selected archive.

Thursday, April 27, 2006

Recursion Part 4: Tree Recursion and Dynamic Programming

The following is a part of a series of articles I had written on recursion.

This part introduces tree recursion, a form of recursion that occurs when a function calls itself more than once in a path. We then introduce dynamic programming, a class of algorithms to tackle certain types of tree recursive problems.

All Parts:

Recursion Part 1: Introduction to recursion
Recursion Part 2: Tail recursion, Accumulators and Iteration
Recursion Part 3: Exercises in tail recursion
Recursion Part 4: Tree Recursion and Dynamic Programming
Recursion Part 5: Structural and Generative Recursion
Recursion Part 6: References and Further Information


Some recursive functions call themselves more than once. Consider the function to calculate the nth term of a fibonacci series:


int fibonacci(int n)
{
if (0 == n) {
return 0;
} else if (1 == n) {
return 1;
} else {
return fibonacci(n - 2) + fibonacci(n - 1);
}
}

In order to calculate fibonacci(4), we need to calculate fibonacci(2) and fibonacci(3). For each call, there are two more recursive calls. This proceeds until n becomes 0 or 1, which are the terminating points of the recursion. We see that execution follows a tree form like this:


fibonacci(4)
+------------------------------
| |
fibonacci(2) fibonacci(3)
+----------------- +---------------
| | | |
fibonacci(0) fibonacci(1) fibonacci(1) fibonacci(2)
+-----------
| |
fibonacci(0) fibonacci(1)

Each node has two children, one for each recursive call. A function with 'n' recursive calls will have 'n' children at each node. Since program execution follows a tree like structure, we call this form of recursion tree recursion

Is there any way to convert tree recursion to iteration?

The short answer is no1. There is no general algorithm to convert tree recursion to iteration. However, specialised algorithms do exists for certain problems. These algorithms (if they exist) are usually specific to each problem.

And it just so happens that our fibonacci algorithm falls into a class of tree recursive problems that can be converted to iteration using a technique called dynamic programming!

Look at the execution pattern for fibonacci(4) again. Notice how fibonacci(2) is calculated twice: Once in the computation of fibonacci(4) and once again in the computation of fibonacci(3). fibonacci(1) is computed thrice!

What if we could save the values of previous fibonacci terms when they are first computed and reuse the values later on without having to compute them again? This optimisation is the cornerstone of dynamic programming.

Here is how we can proceed:
1. Store the initial values of fibonacci(0) and fibonacci(1)
2. Compute fibonacci(2) from fibonacci(0) and fibonacci(1). The saved value of fibonacci(0) is now no longer required, so we can discard it.
2a. In general, compute fibonacci(n + 1) from the saved values of fibonacci(n - 1) and fibonacci(n). Discard the value of fibonacci(n - 1).
4. Repeat step 2a until we have computed the desired term

Look at the execution tree again. Notice how we start the computation from the leaf nodes of the tree and work our way up the tree computing each node, until we reach the root, which is the value we desire.

Here is the implementation using iteration:


int fibonacci(int n)
{
int a = 0, b = 1, temp = 0;
int i = 0;

for (i=1; i<=n; i++) {
temp = a + b;
a = b;
b = temp;
}
return a;
}

We have just used dynamic programming to convert a tree recursive algorithm into an iterative algorithm. The basic idea of dynamic programming is to look at the execution tree, identify repeated computations and perform the calculations 'bottom-up' from leaf to root. At each step, the computed values of the subproblems are stored to be reused later.

Dynamic programming cannot be used on every problem. Some problems can be converted to iteration using other methods, and some problems cannot be converted to iteration at all. Nevertheless, dynamic programming works on a large class of problems, and it is a useful tool to have in the toolbox


1 You can write an iterative loop that manages it's own stack, but that is basically equivalent to recursion.

Any questions? Comments? Please leave a comment using the comment form below.


This post is a part of the selected archive.

Monday, April 17, 2006

Recursion Part 3: Exercises in tail recursion

The following is a part of a series of articles I had written on recursion.

This part provides some exercises related to the concepts introduced in the previous part.

All Parts:

Recursion Part 1: Introduction to recursion
Recursion Part 2: Tail recursion, Accumulators and Iteration
Recursion Part 3: Exercises in tail recursion
Recursion Part 4: Tree Recursion and Dynamic Programming
Recursion Part 5: Structural and Generative Recursion
Recursion Part 6: References and Further Information


Try out the following problems: Ask any questions/answers in the comments.

Is this tail recursion?

int fibonacci(int n)
{
if (n <= 2) {
return 1;
} else {
return fibonacci(n - 2) + fibonacci(n - 1);
}
}

What about this?

node findRoot(node child)
{
if (NULL == child->parent) {
return child;
} else {
findRoot(child->parent);
}
}

And this?

node findNodeInList(node head, int dataToFind)
{
if (NULL == head) {
return NULL;
} else if (head->data == dataToFind) {
return head;
} else {
return findNodeInList(head->next);
}
}

Try converting the above three programs to iterative versions. Do you find some harder than the others?


Convert this iterative program into a tail recursive version. First convert it to an accumulator based function and then convert that to an ordinary tail recursive version. Hint: First convert the for loop into a while loop.

int sumOfFirstNnumbers(int n)
{
int i = 0, sum = 0;

for (i=1; i<=n; i++) {
sum += i;
}
return sum;
}

How about this one? Assume all numbers are positive (> 0). Hint: The accumulator does not always have to perform some mathematical operation. It can also be used to keep track of the state so far.

int findMax(int *numberArray, int arrayLength)
{
int i = 0, max = 0;

for (i=0; i<arrayLength; i++) {
if (max < numberArray[i]) {
max = numberArray[i];
}
}
return max;
}

Any questions? Comments? Please leave a comment using the comment form below.


This post is a part of the selected archive.

Thursday, April 13, 2006

Recursion Part 2: Tail recursion, Accumulators and Iteration

The following is a part of a series of articles I had written on recursion.

This article introduces an important concept in recursion: the tail recursion. We see what tail recursion is, and where it is used. We end the article with the relationship between tail recursion and iteration.

All Parts:

Recursion Part 1: Introduction to recursion
Recursion Part 2: Tail recursion, Accumulators and Iteration
Recursion Part 3: Exercises in tail recursion
Recursion Part 4: Tree Recursion and Dynamic Programming
Recursion Part 5: Structural and Generative Recursion
Recursion Part 6: References and Further Information


Let us get back to the factorial program:


int factorial(int n)
{
if (0 == n) {
return 1;
} else {
return n * factorial(n - 1);
}
}

As we saw in Part 1, the execution of this program goes like this:


factorial(4)
4 * factorial(3)
4 * 3 * factorial(2)
4 * 3 * 2 * factorial(1)
4 * 3 * 2 * 1 * factorial(0)
4 * 3 * 2 * 1 * 1
4 * 3 * 2 * 1
4 * 3 * 2
4 * 6
return 24

In this program, the multiplication with 'n' is the deferred operation (See Part 1 of this series for an introduction on deferred operations). Apart from the deferred operation, there is the recursive call to calculate factorial(n - 1).

Since the recursive call is the last step of the function, this type of recursion is called tail recursion. The name derives from the fact that the recursive call occurs at the end of the function. Oridinary tail recursion has a deferred operation and a recursive call. Pure tail recursion has no deferred operation, it only has a recursive call. The above implementation of the factorial is ordinary tail recursion because it has a deferred operation.

As we saw in Part 1, the deferred operation is carried out when the stack gets popped. Is there any way we can carry out the operation immediately and pass in the value to the function? Take a look at this implementation of factorial:


int factorial(int n)
{
return factorial_acc(1, n);
}

int factorial_acc(int acc, int n)
{
if (0 == n) {
return acc;
} else {
return factorial_acc(n * acc, n - 1);
}
}

Let us trace the execution of this program to calculate factorial(4):


factorial(4)
factorial_acc(1, 4)
factorial_acc(4, 3)
factorial_acc(12, 2)
factorial_acc(24, 1)
factorial_acc(24, 0)
return 24

Notice the difference with the normal factorial implementation? In this version, there are no deferred operations. Instead, the value of the deferred operation is calculated immediately and passed as the first parameter to the recursive function. This parameter is known as the accumulator parameter (in case you were wondering, that is why it is called acc, and the function is called factorial_acc). The role of the accumulator parameter is to accumulate the result of the operations at each step. Note that this version computes the multiplication during a stack push rather than a stack pop. Nothing is done during stack pop, and so this version is pure tail recursion.

Take a look at this iterative implementation of a factorial:


int factorial_iter(int n)
{
int fact = 1, i = n;

while (i > 0) {
fact = fact * i;
i--;
}
return fact;
}

Notice any similarities with the accumulator version? Look at how the variables change with each iteration while calculating factorial(4):


factorial_iter(4)
fact = 1, i = 4
fact = 4, i = 3
fact = 12, i = 2
fact = 24, i = 1
fact = 24, i = 0
return 24

Compare the fact and i variables in the iterative version with the acc and n parameters in the accumulator version. They are identical! In other words, the accumulator version simply implements a while loop using recursion!

So here is the big result: Any ordinary tail recursive program can be converted to a pure tail recursive program with accumulator, which in turn can be converted to a while loop. Thus, ordinary tail recursion and iteration are equivalent, and the above steps give us a way to convert between the two forms!

Because iteration is nothing but a special case of recursion, some languages like Lisp, Scheme and others do not have any iteration methods. There are no for loops, while loops or do-while loops in these languages. All looping is accomplished by using either ordinary or pure tail recursion! These languages have special mechanisms for optimising tail recursion so that they do not increase the stack size.

Languages like C include iterative methods and offer no special support for tail recursion. Since tail recursion is exactly equivalent to iteration, it makes sense to convert all tail recursion into iteration. However, beware of trying to convert non-tail recursion into iteration. As we shall see in later articles in this series, non-tail recursion is a very different beast altogether.


Any questions? Comments? Please leave a comment using the comment form below.


This post is a part of the selected archive.

Wednesday, April 12, 2006

Recursion Part 1: Introduction to Recursion

The following is a part of a series of articles I had written on recursion for a newsletter.

This is the first part of a mini-series of full length articles dealing with recursion. In this, the first part, we talk about some theory behind recursion. This theory will form the foundation and help in understanding the remaining parts of the series. Most of this theory will already be known and it is just a quick overview to refresh your memory.

All Parts:

Recursion Part 1: Introduction to recursion
Recursion Part 2: Tail recursion, Accumulators and Iteration
Recursion Part 3: Exercises in tail recursion
Recursion Part 4: Tree Recursion and Dynamic Programming
Recursion Part 5: Structural and Generative Recursion
Recursion Part 6: References and Further Information


Let us start with the most common example of recursion, calculating the factorial of a number.Here is the C code:


int factorial(int n)
{
if (0 == n) {
return 1;
} else {
return n * factorial(n - 1);
}
}

How does this work? Say we want to calculate factorial(4). Since 4 is not equal to 0, the expression evaluates to 4 * factorial(3). factorial(3) in turn is expanded to 3 * factorial(2). At each point, the previous state of execution is pushed on to the stack. Continuing this way, the expression evaluates to:


4 * 3 * 2 * 1 * factorial(0)

factorial(0) is 1, so we get:


4 * 3 * 2 * 1 * 1
4 * 3 * 2 * 1
4 * 3 * 2
4 * 6
24

The final answer is 24, which is indeed the factorial of 4. Let us summarize the state of execution at every step:


factorial(4)
4 * factorial(3)
4 * 3 * factorial(2)
4 * 3 * 2 * factorial(1)
4 * 3 * 2 * 1 * factorial(0)
4 * 3 * 2 * 1 * 1
4 * 3 * 2 * 1
4 * 3 * 2
4 * 6
24

Note an important point: No multiplications are performed until the stack starts to get popped. Thus, the execution of this function consists of a series of "deferred operations" which are stored in the stack state during the push operation, and which are executed after the pop operation.

Understanding the above point is the key to understanding many concepts in recursion. We will be using it in the other articles in this series.


Any questions? Comments? Please leave a comment using the comment form below.


This post is a part of the selected archive.