Clear multiple layers from a plot

I'm using bounding boxes plotted on top of text in an image as a sort of heat map. I want to be able to show all the bounding boxes for one condition, then clear them and show a different set. I read the post about getting the plot handle to clear the most recent one (<http://www.mathworks.com/matlabcentral/answers/1444-clear-one-plot-in-multiple-hold-figure#)>, but because the rectangle plotting function can only plot one box at a time, it doesn't seem feasible to clear all the boxes individually.

Example; the comment is what I'm trying to figure out how to do without just clearing everything including the background image:

    imshow(I)
    for i = 1:x
        % Plot the bounding boxes
        for j = 1:y
            rectangle('Position', pos(i, :), 'EdgeColor', c(j, :));
        end
        % Clear the bounding boxes, but leave I
    end