Due Date: Wednesday, February 15, 2012.

The goal of this assignment is to implement a version of augmented reality, in which a planar region is recognized and replaced with an accurately aligned new image. The assignment gives experience with robust least-squares as well as image resampling.

The assignment

This assignment will build on what was done for Homework 1. In that homework, two images were matched using SIFT features, and the correct matches were selected using the method of semi-local constraints.

For this assignment, the task is to select the largest consistent set of matches and solve for an affine transformation that maps the second image into the coordinates of the first. Then pixels from the third image will replace those within the matched region of the first image. This is a form of augmented reality in which one image region is substituted with another.

Some sample triplets of images are given the following zip file: hw-2.zip. SIFT features have been computed for the first two images in each triplet, as in Homework 1. There is no need for SIFT features from the third image, as it is used just to replace the matched region from the first image. You can assume the the sizes of the second and third images are the same.

Following are hints for solving this problem:

1. The first step is to find the largest cluster of consistent features between the first and second images. You can use the results from Homework 1, then select the seed feature with the largest number of neighbors in agreement, and test all other image features for consistency with this seed feature to create a large consistent cluster. Another similar approach would be to use RANSAC, in which you test random seed features and check how many others are consistent with it, until finally selecting the largest consistent cluster.

2. Next, create a least-squares solution for the affine transform mapping the features from the second image to the first. This requires setting up the linear system from the matching feature locations as described in Section 7.4 of the paper "Distinctive image features from scale-invariant keypoints" by D. Lowe (in the reading list). In Matlab, you can find the least-squares solution of the overdetermined system Ax = b using x = A \ b.

3. Compute the average residual error of your solution. Use a robust method, such as iteratively discarding 10% of the points with the highest residuals as long as their residual is greater than 2 pixels, and re-solving after each iteration. Give the initial and final residual errors in your report, along with the number of discarded matches and a description of the robust method that you use.

4. To visualize your affine transform, display 4 lines corresponding to the locations of the boundaries of the second image superimposed on the first image under the computed affine transform.

5. Finally, compute the inverse of the affine transform so that we can map the locations of pixels in the first image into their corresponding location in the second image. This allows us to resample the third image under the same transform, so that we can replace pixels in the first image with a sample from the transformed location in the third image. You can use nearest-neighbour resampling (sampling the closest pixel), but better results can be obtained with bilinear resampling.

Deliverables

Test your results on the two given sets of images, and show both the projected boundaries of the second image on the first, as well as the third image inserted at the correct location. Write a short report of about 1 page that describes your approach and any issues that arose. Report on the average image residual error both before and after applying the robust method to the least-squares solution. Include a commented copy of your source code.