How to preset the intrinsic camera parameters in stereo calibration?

Hi everyone, I was trying some mapping between different cameras which failed due to the fact that Matlab estimates quite different intrinsics for one camera (using the very same calibration images) when the camera is calibrated individually than if it is calibrated in a stereo. Also if you have for example 3 cameras and you calibrate cam1 once with cam2 and once with cam3, the intrinsic parameters will be very different. I tried to feed the intrinsic parameters to the estimateCameraParameters function using: 'InitialIntrinsicMatrix' and 'InitialRadialDistortion'. But still the values are different:

    My preset: 
    1.0e+04 *
        1.4431         0         0
       -0.0057    1.4379         0
        0.3141    0.1477    0.0001

    And the final values calculated by Matlab:

    1.0e+04 *
        1.4172         0         0
       -0.0061    1.4114         0
        0.3231    0.1433    0.0001

    If I do not preset the initial values I would get:
    1.0e+04 *
        1.4173         0         0
       -0.0061    1.4115         0
        0.3231    0.1433    0.0001

So basically giving the initial values did not do much. I also get very different values for distortion.

Of course I understand that there's always some error in calibration, but I was hopping that I don't get such big difference. Here for example the difference between the two principal points is 100. That's a lot.

I appreciate it if some one would kindly help find a solution without telling me I shouldn't try to calibrate these three cameras. Of course it would be good if there was a way to calibrate all three cameras together.