Problem with image rectification and stereo disparity.

I am working on stereo vision for my project in robotics.I have tried to use my own images in the code given in http://www.mathworks.in/help/vision/examples/stereo-calibration-and-scene-reconstruction.html. The both cameras i use are same and having exactly same camera parameters.The relative distance of camera 2 compared to 1st are 100mm and 50 mm in x and y axis respectively.When rectifying the images,while both objects seems to be parallel,the whole object is not getting rectified.

The images i have used are http://i.imgur.com/o31C5cP.jpg and http://i.imgur.com/FrPM4R4.jpg.

The images before and after rectification are http://i.imgur.com/ZQj4bkc.jpg and http://i.imgur.com/ZFZBjx2.jpg. As you can see in the second image, the whole object is not rectified.

Stereo disparity and pointcloud,i presume,are also wayward as a result of this.http://i.imgur.com/KVLxzJD.jpg and http://i.imgur.com/as5ALtf.jpg.

Since i have used Camera calibration toolbox,i have found out stereo parameters by the following code:

I1 = imread('lftjb2.jpg');

I2 = imread('rightjb2.jpg');

stereoParams=stereoParameters(cameraParams,cameraParams,[0 0 0;0 0 0;0 0 0],[100,50,0])

% Rectify the images.

[J1, J2] = rectifyStereoImages(I1, I2, stereoParams);

% Display the images before rectification.

figure; imshow(cat(3, I1(:,:,1), I2(:,:,2:3)), 'InitialMagnification', 50);

title('Before Rectification');

% Display the images after rectification.

figure; imshow(cat(3, J1(:,:,1), J2(:,:,2:3)),'InitialMagnification', 50);

title('After Rectification');

disparityMap = disparity(rgb2gray(J1), rgb2gray(J2));

figure; imshow(disparityMap, [0, 64], 'InitialMagnification', 50);

colormap('jet');

colorbar;

title('Disparity Map');

pointCloud = reconstructScene(disparityMap, stereoParams);

Could anyone kindly help me to fix this problem? Thanks.

Answer 1

[Dima Lisin]

The biggest problem here is that [0 0 0;0 0 0;0 0 0], which you are passing into the constructor of stereoParameters, is not a valid rotation matrix. If there is no rotation between the cameras, i.e. if they are exactly parallel, then the matrix should be identity, eye(3). However, from looking at your images, there is definitely 3D rotation between your cameras, so the identity matrix will not work either.

I would highly recommend using a pair of cameras fixed relative to each other, and calibrating them using the Stereo Camera Calibrator app, or the estimateCameraParametersFunction.

If you absolutely must use a single camera, then you need some reference object in the field of view to give you a world coordinate system, to be able to compute R and t between the cameras. See Sparse 3-D Reconstruction From Two Views.