I'm not 100% sure what you are asking, but if I understand your question correctly I don't think what you are asking can be done easily with pcolor alone.
I think you want to use imagesc instead. The problem is that you cannot set the color of individual faces in the surface object created by the pcolor command, you have to either set all the faces to the same color, or rely on the data, but if you set all the faces to the same color the AlphaData used to control transparency is ignored. You can hack around this by appending a color to the colormap, but it is easier to just use the image object, which doesn't have those restrictions.
You can use imagesc instead of pcolor for both the data and the mask, or use pcolor for the data and imagesc for the mask. The downside is that it will be harder to align the two if you do that.
maskmatrix = ones(size(peaks));
maskmatrix(5:20,15:35) = NaN;
maskmatrix(30:45,15:35) = NaN;
maskim = repmat(maskmatrix~=1, 1, 1, 3);
h.AlphaData = ~isnan(maskmatrix);
h.AlphaData = ~isnan(maskmatrix)*0.6;