I am thinking maybe I can put some data in the axis "user data" property. Such as the handles to the patches.
set(gca,'UserData',hp)
And then set up a call back function to update the plot
set(zoom(gca),'ActionPostCallback',@(src,event) ZoomCallback(src,event))
since the WindowSize and AxesSize don't change. I imagine I would need to scale acording to the level of x and y zoom.
Is there a property for the current level of zoom? Or would I need to find the current XLim and Ylim and work out how much they changed?
I think then I can just edit the Vertices property of the patch handle.