Due Date: Tuesday, January 28, 2014.

The purpose of this assignment is to learn how to implement some basic methods for correlation-based stereo matching and to gain experience with optimizing system parameters using real world data.

The assignment

1. We will assume that our stereo images have been rectified so that corresponding epipolar points are along the same scan lines (image rows). The first step will be to implement some Matlab (or Python) code that performs normalized cross correlation to compare a square window around each pixel in the left image against windows at a range of horizontal disparities in the right image. See the paper by Pascal Fua for a description of this approach. This should generate a disparity image that can be displayed and used for further stages. Try window sizes of 3x3, 5x5, and 7x7, and include sample disparity images for these in your report.

   We will use the following pair of stereo images from Tsukuba University: imL.jpg and imR.jpg. You can download these by right-clicking in your browser. These images are in a lightly compressed JPEG format, which will load easily into Matlab. In addition, there is a ground truth disparity image in which the correct disparities have been determined by hand for the left image: trueL.jpg. For this pair, you should search over a disparity range of [0,15]. (HINT: its easy to get the ordering wrong, but you can debug this by switching the images and see which order gives the best result). For the ground truth disparity image, the disparities have been multiplied by 16 to make them suitable for display, so you should do the same.

2. As a second step, write a scoring function that computes the fraction of correct pixels in your result that are within 1 disparity value (equivalent to 16 values in the scaled image) of ground truth. Which window size produces the largest fraction of correct pixels?

3. Try at least one additional method to improve the accuracy of your stereo system. For example, you can try the validity check in the Fua paper, in which images are matched in both directions, and pixels are marked invalid when the directions disagree. You can also mark pixels invalid if the NCC score is similar for multiple disparities. Then you can fill invalid pixels with estimated values by setting them to the disparity of the nearest valid pixel. Other ideas include using color images (the color versions are available on the Scharstein & Szeliski web site) or using multi-scale windows such as described in the Fua paper. Clearly explain your method in your report and describe its effect on the final score.

Matlab resources

Brief Introduction to Matlab for Computer Vision.

Ian Mitchel's Matlab Resources.

Deliverables

Hand in a copy of your commented code, disparity images showing results at various stages of computation, and a written description of what you did and the conclusions you reach (no more than 2 pages of written description).