Although you can probably get interp3 to work in this case, I think it is preferable to to use the function TriScatteredInterp for this. It is a two-step process, in which you first create the interpolant from the calibration file, then use that interpolant on the data. You also don't need to use the for loops, because the whole thing can be vectorized. I believe that the code below does what you want, but of course you should check:
temperatureInterpolant = TriScatteredInterp(calRGB(:,1), calRGB(:,2),calRGB(:,3),calRGB(:,4));
output = [x,y,temperatureInterpolant(v,w,z)]
Note that there is also a syntax for calling TriScatteredInterp that does not require you to split up your RGB data into three separate vectors, but I did not want to stray too far from the syntax of your original code.
Hope that helps. [Note that I included (commented out) a very simple example of RGB arrays that worked for me.]