Question Regarding 'pcfromdepth' Function Example

Hello,

I have been working with the example code provided in the 'doc pcfromdepth' documentation in MATLAB. I used the example color image 'sampleImage.png' along with depth images extracted using the 'MiDaS' and 'Depthanything v2' models to generate the point cloud. However, the point cloud I obtained was inaccurate compared to the result shown in the example.

I am curious about how the DepthImage used in the MATLAB example was extracted, and how the camera parameters (focalLength, principalPoint) were determined. Could you please provide more information on this?

Thank you for your assistance.

https://www.mathworks.com/help/vision/ref/pcfromdepth.html

[Steven Lord]

"inaccurate" is a very general term. If you provide more information about what you receive (so we could compare it with the results shown on that documentation page) Answers readers may be better able to offer suggestions for ways to improve the agreement of your modification of the example with what's in the doc.

[Seungju]

I am attaching the results of my three experiments.

The attachments can be found at the following Google Drive link.

https://drive.google.com/drive/folders/1myIaU7lB7tRb5xvgr0s-sQy3ca9R60AX?usp=drive_link

First Experiment - Using SampleDepth Image from the 'doc pcfromdepth' Page

The code used in the first experiment is as follows:

depthImage = imread("sampleDepth_matlab.png");
imshow(depthImage)
colorImage = imread("sampleImage.png");
imshow("sampleImage.png")

focalLength      = [535.4, 539.2];
principalPoint   = [320.1, 247.6];
imageSize        = size(depthImage,[1,2]);
intrinsics       = cameraIntrinsics(focalLength,principalPoint,imageSize);
depthScaleFactor = 5e3;
maxCameraDepth   = 5;

ptCloud = pcfromdepth(depthImage,depthScaleFactor, intrinsics, ...
                      ColorImage=colorImage, ...
                      DepthRange=[0 maxCameraDepth]);

pcshow(ptCloud, VerticalAxis="Y", VerticalAxisDir="Up", ViewPlane="YX");

I used the 'sampleDepth_matlab.png' and 'sampleImage.png' images as provided in the documentation. The resulting point cloud, which matched the documentation, is attached as '1st_matlabdoc_result.fig'.

Second Experiment - Using DepthImage Extracted by MiDaSV3.1 Model

In the second experiment, I extracted the DepthImage from 'sampleImage.png' using the MiDaSV3.1 model, which I have attached as 'sampleDepth_midasv3.png'. The code used was identical to that of the first experiment, except for the image filename. The result using this DepthImage is attached as '2nd_midasv3_result.fig'.

Third Experiment - Using DepthImage Extracted by Depthanything v2

In the third experiment, I extracted the DepthImage from 'sampleImage.png' using the Depthanything v2 model from the following link: https://huggingface.co/spaces/depth-anything/Depth-Anything-V2. The extracted DepthImage, attached as 'sampleDepth_depthanythingv2.png', was originally in color, so I converted it to grayscale before using it in the pcfromdepth function. The code used is as follows:

depthcolorImage = imread("sampleDepth_depthanythingv2.png");
depthImage = im2gray(depthcolorImage);
imshow(depthImage)
colorImage = imread("sampleImage.png");
imshow("sampleImage.png")

focalLength      = [535.4, 539.2];
principalPoint   = [320.1, 247.6];
imageSize        = size(depthImage,[1,2]);
intrinsics       = cameraIntrinsics(focalLength,principalPoint,imageSize);
depthScaleFactor = 5e3;
maxCameraDepth   = 5;

ptCloud = pcfromdepth(depthImage,depthScaleFactor, intrinsics, ...
                      ColorImage=colorImage, ...
                      DepthRange=[0 maxCameraDepth]);

pcshow(ptCloud, VerticalAxis="Y", VerticalAxisDir="Up", ViewPlane="YX");

The resulting point cloud from this experiment is attached as '3rd_depthanythingv2_result.fig'.

Summary

Except for the first experiment (using the DepthImage provided in the MATLAB documentation), I was unable to achieve the same results with the other depth estimation models. Therefore, I am curious about the process used to extract the DepthImage provided in the MATLAB documentation. Additionally, I would like to know how the camera parameters were determined.

Thank you for your help.

Answer 1

[Gayathri]

Hi @Seungju,

I understand that you want to know the source of the Depth Image and the intrinsic camera parameters used in the “pcfromdepth” documentation example.

The image and the depth image were captured using a RGB-D camera. This is mentioned in the “pcfromdepth” documentation link given below.

https://www.mathworks.com/help/releases/R2024a/vision/ref/pcfromdepth.html

Also to know more about the dataset from which the image was taken, please refer to the example given in “Visual SLAM with RGB-D Camera” documentation. The link for the same is attached below for your reference.

https://www.mathworks.com/help/releases/R2024a/vision/ug/visual-slam-with-an-rgbd-camera.html

The image is taken from the TUM RGB-D benchmark dataset. The dataset can be accessed using the below mentioned link.

https://cvg.cit.tum.de/data/datasets/rgbd-dataset

The intrinsics of the dataset such as “focalLength”, “principalPoint” can be found in the below mentioned link.

https://vision.in.tum.de/data/datasets/rgbd-dataset/file_formats

Hope you find this information helpful.

[Seungju]

Hi @Gayathri,

I truly appreciate your quick and helpful response to my question.

Your explanation has greatly clarified the issue I was facing, and it will be very beneficial as I move forward.

Thank you for taking the time to assist me.