The purpose of Simulink's Reshape block and MATLAB's reshape command is only to "reinterpret" the dimensions (aka size) of the signal/variable. The number of elements is not changed. The values as they would appear in the "flat memory" of the computer would also be unchanged.
Your usage doesn't fit the reshape behavior. Your input to the first reshape block has this many elements.
inputNumel = prod([512,512,3])
Your desired output has this 1/3 as many elements.
outputNumel = prod([512,512])
Since you are trying to change then number of elements, the reshape block is throwing an error.
One approach to solve this is to put the following code into a MATLAB Function block
function [Y,Cb,Cr] = splitData(u)
Y = u(:,:,1);
Cb = u(:,:,2);
Cf = u(:,:,3);
But there is probably a simpler way to achieve your high level goal using capabilities of Image Processing Toolbox.