Lidar Camera Calibration Issue

Hi,

I used the Lidar Camera Calibration app from matlab to find the transformation between the two sensors. However, after projecting lidar points into the image, there is a clear shift in the projection. Sample images are as below; (Os: Ubuntu 20.04, MATLAB R2021b)

checkerSize = 110; % millimeters
padding = [44 35 43 35]; % millimeters
imageFilesPath = fullfile('camera');
pcFilesPath = fullfile('velo_2');
lidarCameraCalibrator(imageFilesPath,pcFilesPath,checkerSize,padding)

When projecting data, I used the below code;

J = undistortImage(image, camIntrincis.cameraParams.Intrinsics);
indices = findPointsInROI(ptCloud,[0 50 -15 15 -3 3]);
ptCloudProcessed = select(ptCloud,indices);
[imPts, imPtIdx]  = projectLidarPointsOnImage(ptCloudProcessed, ...
    camIntrincis.cameraParams.Intrinsics, veloToCam.tform);
clr = ptCloudProcessed.Location(imPtIdx, 1);

figure;
pcshow(ptCloudProcessed);
figure(f1);
imshow(J)
hold on
scatter(imPts(:,1),imPts(:,2), 2, clr, 'filled');
colormap hsv;
hold off
pause

My data has Lidar point clouds are from Velodyne HDL 64 with 10 fps and camera is 20 fps. However, images does not have a timestamp on it. Both lidar data is are stored as vpcap files for 60 seconds durations. But I manually achieve the synchronization between the two sensors. There two possible scenarios for the probelm, that I can think of;

1. Two sensors are not properly synchronized. But when you look at the data this may not be the case. Lidar projectin seems always shifted higher than that of image position. Also always some of the lidar points on the road are projected on to the vehicle bonnet.
2. I am doing something wrong or I need better data ?

Calibration results are also attached herewith. I have used about 30 image-lidar pairs for calibration.

Is there a way to rectify this error?

Appreciate any suggestions or inputs.

[Kritika Bansal]

Hi,

As you mentioned- the images do not have timestamps, could you help us understand how are you achieving the synchronization between the camera and the lidar?

Also, would it be possible for you to share the data you are working with? You can also contact the technical support team to help you understand the results better.

-Kritika

[Hasith Karunasekera]

Data is saved to disk each 60 seconds duration. The code how it is achieved is as below with explanation.

Also I am planning to collect data keeping the environment static, i.e. with calibration board attached to a tripod or hanging (without human intervention) and collect data at different positions to mitigate any error that may be coming due to the synchronization issue (without using the code below). Do you think it's good way to get rid of th sync issue ?

Thank you

% Video processing delay is decided by checking (trial & error manner) which image frame
% corresponds to which velodyne frame, best to the naked eye.
% More specifically it is set observing person jumping infront of the sensors and observing 
% both velo and image frame that captures the same motion and/or by observing clockwise and 
% anticlockwise movement of the checkerboard and the point the direction changes.
VIDEO_PROCESSING_DELAY = 0.05; % seconds

% Reading the timestamp for each velo frame
veloTime = veloData.timeStruct.ScanTimestamps.veloMicroSecSinceStartGPSWithoutLeapSecs ...
    /1000000;
veloTime = veloTime + veloData.timeStruct.ScanTimestamps.leapSeconds;

% Reading the timestamp velodyne start the first scanline of the first frame
timeStart = veloData.timeStruct.PacketTimestamps.leapSeconds(1) + ...
    veloData.timeStruct.PacketTimestamps.veloMicroSecSinceStartGPSWithoutLeapSecs(1)...
    /1000000;
% Reading the timestamp velodyne ends the last scanline of the last frame
timeEnd = veloData.timeStruct.PacketTimestamps.leapSeconds(end) + ...
    veloData.timeStruct.PacketTimestamps.veloMicroSecSinceStartGPSWithoutLeapSecs(end)...
    /1000000;
% Creating the timestamp for images, assuming all images are equally spaced between timeStart and timeEnd
imgTimeStamp = linspace(timeStart, timeEnd, v.NumFrames+1)';
imgTimeStamp = imgTimeStamp(1:end-1);
imgTimeStamp = imgTimeStamp - VIDEO_PROCESSING_DELAY;

% Generating an array 'imageIdx', which gives the frame number of the 60 secnds video that corresponds 
% to each velo frame
for frameIter = 1:length(pcFrames)-1
    matchIdx = find(imgTimeStamp(:) >= veloTime(frameIter) ...
        & imgTimeStamp(:) < veloTime(frameIter+1));
    if ~isempty(matchIdx)
        selectedIdx = round(median(matchIdx));
        imageIdx(frameIter) = selectedIdx;
    end
end

lastMatchIdx = find(imgTimeStamp(:) >= veloTime(end));
if ~isempty(lastMatchIdx)
    imageIdx(end) = round(median(lastMatchIdx));        
end