How to reconstruct a silhouette after shadow removal to regain legit objects of human figure?

I have segmented the foreground using H,S channels of HSV color space with foreground detector. Shadow has been eliminated as expected but it also removes the non shadow parts of the human too. How to regain them? I tried this as of now but not very satisfying,

Res = imdilate (HsfilteredForeground,ones(5));
re =  imfill(Res,'holes');

.. Full code:

foregroundDetector = vision.ForegroundDetector('NumGaussians',3,'NumTrainingFrames',10,'LearningRate',0.000001); 
foregroundDetector1 = vision.ForegroundDetector('NumGaussians',3,'NumTrainingFrames',10,'LearningRate',0.000001,'InitialVariance',  (30/255)^2); 

vid = videoinput('winvideo', 1); %select input device 
i=1;
DlgH = figure('units','normalized','outerposition',[0 0 1 1]);
H = uicontrol('Style', 'PushButton', ...
                    'String', 'Close', ...
                    'Callback', 'delete(gcbf)');

while (ishandle(H))

      frame = getsnapshot(vid);  
       subplot(1,3,1), imshow(frame);
      title('frame');
      foreground = step(foregroundDetector, frame);
      %imshow(foreground);    
      hsv = rgb2hsv(frame);
      hsfore = step(foregroundDetector1, hsv(:,:,1:2));

      se = strel('square', 3);

      filteredForeground = imopen(foreground, se);
      HsfilteredForeground = imopen(hsfore, se);
      Res = imdilate (HsfilteredForeground,ones(5));
      re =  imfill(Res,'holes');
     subplot(1,3,3), imshow(re);
     title('HS foreground Without shadow');

     subplot(1,3,2), imshow(filteredForeground); 
     title('RGB Foreground');
     drawnow 

end
close