Intercepting Clicking action within an image figure, and user-driven change in figure

Goal:

The function should enable the capture an unbounded number of data points on an image,

when the user 'clicks' on the image; clicking n times for any number n of data points that the

user decides on the spot to collect. 'Clicking' may be achieved either by 'mouse click's or

'keyboard clicks' (= key pressing on keyboard while the mouse is hovering the desired point);

this enable the capture of different classes of data points;

in the example below only one keyboard key ('f12') is designed active for 'clicking');

Also

- some zoom control should be availlable using pressed keyboard keys.

- some (future) pan control via the keyboard will be incorporated

- possibility to delete the last collected data point(s)

- possibility to delete a collected data point by clicking again on that point

(yet to be usable when a visual feedback of the clicked point will be incorporated in the code)

- function exit is triggered by pressing the 'esc' key

I acknowledge getting inspiration from from an old posted example dealing with line plots or scatter plots:

https://stackoverflow.com/questions/23078842/get-coordinates-of-clicks-in-figure-and-specific-mouse-button-clicked-for-each-p

Below is some experimental code (yet not fully properly working) for collecting user data

on an image, without using a GUI, without using ginput, nor DataTips.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

% Testing the code below

load('PEPPERS.mat');  % Image= imread('peppers.png'); 
imshow(Image);             %  (any other image will do)
set (gcf, 'Position', get(0, 'Screensize'));      % figure on full screen
pause(0.1);
[ CollectedUserData] =  ginput_mod3b(gca, Inf)

% the 'CollectedUserData' output is a n-by-6 array in the case of an RGB image.
% with each line containing:
% [ ClickCode, X, Y, [ RGB color triplet ] ]   
% the ClickCode is 1 for mouse left click, 3 for mouse middle click, and 112 for keyboard 'f12' key "click"

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

MIXED RESULTS:

- Before launching the function:

the 'set (gcf, 'Position', get(0, 'Screensize')); ' maximize the figure to full screen,

but does *not* properly maximize the image within the figure. How to correct for that ?

Also, if the figure is not fully expanded to full screen before launching ginput_mod,

boundary problems are encountered if the user then sets the figure to full screen.

How to intercept the user-driven event that the figure window has been repositioned/ has changed size?

- At startup, the mouse cursor changes shape as the mouse is hovering the figure window or outside.

That change is positioned correctly for the Y-displacement (corresponding exactly to hovering the image or not)

However, the change is not correct in X (the cursor shape is preserved in some of

the left region and the right region surrounding the image. How to correct for that ?

pressing 'z' or 'u' is expected to zoom/unzoom with respect to the current mouse position;

How to expand the visible part of the image to the *full surface of the figure*, e.g. when the zooming is high ?

pressing 'f12' while hovering a region of the image captures the corresponding point of the image

(X,Y coordinates, color value) -> OK

Clicking (left or right button of the mouse) to capture data points -> OK

[ I tried attaching the 'mouse-click function' to 'ButtonDownFcn', either to the axis, or to the figure;

I encounter problems when attaching the function to the axis because then mouse clicks are not intercepted;

How to prevent another object (the figure?) to capture mouse clicks ?

I found the empirical solution of attaching the 'mouse-click function' to 'ButtonDownFcn' for the figure

instead of the axes; the mouse clicks are then intercepted ].

Current mouse position:

if using ' get(src,'CurrentPoint');' in the function getpoints(src,evnt) :

the X or Y mouse position is typically out of range with respect to the size of the image;

I use instead 'get (AxisHandle, 'CurrentPoint')' to get the appropriate XY coordinates

of the current mouse position on the image. -> OK

-pressing the 'del' key on the keyboard deletes the last data point(s) that was collected. -> OK

- pressing the 'escape' key on the keyboard induces the termination of the function. -> OK

- how to intercept the user-driven event that the user has selected some standard MatLab

tool attached to the figure, such as zoom; pan, etc..., and suspend the action of the ginput_mod function

until such standard tool has been deselected ?

% * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *

function varargout = ginput_mod3b(ha,n)
if nargin<2
    n=1;
end

VERBOSE_DEBUG =0;  % set to 1  for getting debug messages <<<
k = 0;
Zoom= 20 ; % Percent zoom amount upon pressing key 'z' / unzoom upon key 'u'

%%// Placeholders 

MouseHov= []; %
Current_X_limits= [];
Current_Y_limits= []; 
Dx =0.;
Dy = 0.; 
X_Data= [];
Y_Data= [];
CollectedUserData=[];
UserClicks=[];
xy = [];

% handles and function attachment:
hf = get(ha,'parent');
figure(hf);
set(hf,'WindowButtonMotionFcn',@changepointer)
%set(ha,'ButtonDownFcn',@getpoints) % <<<<<     attach to axis
set(hf,'ButtonDownFcn',@getpoints)  % <<<<    attach to figure itself  
set(hf,'KeyPressFcn',@getKey) % keypress detection

hp = get(ha,'children');
FirstLayerPlot = hp(end);
ThisIsAnImage = isprop(FirstLayerPlot, 'CData');

if (ThisIsAnImage==0)
    Problemo = 1;
    disp('no image data !')
else
    Problemo = 0;		
    ImX_Size = FirstLayerPlot.XData	;			
    ImY_Size = FirstLayerPlot.YData ;		
    X_Data = ImX_Size(1): ImX_Size(2); 	
    Y_Data = ImY_Size(1): ImY_Size(2) ; 	
    TheImage = FirstLayerPlot.CData; 	
    
end
ht = get(hp,'hittest');
set(hp,'hittest','off')
axlim = get(ha,'Position');
fglim = get(hf,'Position');
x1 = axlim(1)*fglim(3) + fglim(1);
x2 = (axlim(1)+axlim(3))*fglim(3) + fglim(1);
y1 = axlim(2)*fglim(4) + fglim(2);
y2 = (axlim(2)+axlim(4))*fglim(4) + fglim(2);
waitfor(hf,'WindowButtonMotionFcn',[])
if iscell(ht)
    for jj=1:length(ht)
        set(hp(jj),'hittest',ht{jj})
    end
else
    set(hp,'hittest',ht)
end


if nargout==3
    varargout{1} = CollectedUserData;
    varargout{2} = UserClicks;
    varargout{3} = xy;
elseif nargout==2
    varargout{1} = CollectedUserData;
    varargout{2} = UserClicks;
else
    varargout{1} = CollectedUserData;
end

if Problemo ==1
    set(hf,'WindowButtonMotionFcn',[]);
    %set(ha,'ButtonDownFcn',[]) ;
    set(hf,'ButtonDownFcn',[]);  
    set(hf,'KeyPressFcn',[]) ;
    return
end
%  * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *

    function changepointer(~,~)
        pntr = get(0,'PointerLocation');
        if pntr(1)>x1 && pntr(1)<x2 && pntr(2)>y1 && pntr(2)<y2
            set(hf,'Pointer','crosshair')
        else
            set(hf,'Pointer','arrow')
        end
    end

%  * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * 
    function getpoints(src,evnt) % mouse click
        cp = get(src,'CurrentPoint');
    	% MouseHov = cp(1,1:2); %   this does NOT give appropriate coordinates 
    	MouseHov = get (ha, 'CurrentPoint'); % current position within data axes
    	MouseHov = MouseHov(1,1:2);
    	set(hf,'Pointer','circle'); % transient mouse cursor shape change
    	Current_X_limits= get(ha, 'XLim');  
    	Current_Y_limits= get(ha, 'YLim'); 
    	Dx = (Current_X_limits(1,2)-Current_X_limits(1,1));
    	Dy = (Current_Y_limits(1,2)- Current_Y_limits(1,1));
    	
  	  % Mouse-Button recognition:
  	button = get(hf, 'SelectionType');
    
    if VERBOSE_DEBUG
        {'Mouse Click!' ; 'BadCurrentPoint (src):'; cp; 'GoodCurrPoint'; MouseHov; ...
            'Current X-limits'; Current_X_limits; ...
            'Current Y-limits'; Current_Y_limits; 'ButtonClicked'; button} 
    end
    if(strcmp(button, 'normal'))
        TheButton = 1; % left
        ClickName = sprintf('Mouse-Click %s',button); 
    elseif(strcmp(button, 'extend'))
        TheButton = 2; % right
        ClickName = sprintf('Mouse-Click %s',button); 
    elseif(strcmp(button, 'alt'))
        TheButton = 3; % middle
        ClickName = sprintf('Mouse-Click %s',button); 
    else
        Thebutton = 4; % double click any mousebutton ?
        ClickName = sprintf('Mouse-Click %s',button); 
    end
    
    [Xp, Yp, Err] =  FindProximal_ImageXY(MouseHov(1), MouseHov(2));
    if Err==0
        if k==0
            AddToDataCollection(TheButton, Xp, Yp)
            UserClicks = [UserClicks; {ClickName}]; 
            xy = [xy ; MouseHov];
            k = 1;
        else 
            Bingo = EventuallyDeleteOldData( Xp, Yp);
            if Bingo == 0
                AddToDataCollection(TheButton, Xp, Yp)
                UserClicks = [UserClicks; {ClickName}];
                xy= [xy; MouseHov];
    			
            end
    		
        end
    	
        if  (k==n)   % exit function
            set(hf,'Pointer','arrow')
            set(hf,'WindowButtonMotionFcn',[])
            set(hf,'KeyPressFcn',[]) %  
            %set(ha,'ButtonDownFcn',[]) % or next line, see below
            set(hf,'ButtonDownFcn',[])  %<<<<<
            return;
        end
    end
    
    end

%  * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
    function getKey(src, event) %  Upon key-press detection
    	TheKey = event.Key;
    	hz= zoom(ha);
    	hz.Enable = 'off';  % turn zoom off
    	Current_X_limits= get(ha, 'XLim'); 
    	Current_Y_limits= get(ha, 'YLim'); 
    	Dx= (Current_X_limits(2)-Current_X_limits(1));
    	Dy= (Current_Y_limits(2)- Current_Y_limits(1));
    	MouseHov = get (ha, 'CurrentPoint'); % current position within data axes
    	MouseHov = MouseHov(1,1:2);
    	if VERBOSE_DEBUG
    		{'Keyboard!;'; 'CurrMouseXY:'; MouseHov; ...
        		'Curr X-limits'; Current_X_limits; 'Curr Y-limits'; ...
    			Current_Y_limits; 'Key pressed'; TheKey; 'ENDKeyboard!'}% DEBUG
    	end		
    	if strcmp(TheKey, 'delete')  % Delete last acquired data point
            if k>=1
                k= k-1; 
                CollectedUserData(end, :) = [];
                xy(end,:) = [];	
                if k~=1
                    UserClicks = UserClicks(1:(end-1)); 
                else
                    UserClicks = [];
                end
            end
    	    
    	elseif strcmp(TheKey, 'f12')  
    		%  "keyboard click" to acquire (without mouse button clicking) 
    		%       the data point hovered by the mouse
    		set(hf,'Pointer','circle'); % transient pointer change upon click
    		[Xp, Yp, Err] =...
                FindProximal_ImageXY(MouseHov(1), MouseHov(2));
    		if Err == 0
        		
    			if (k~=0)
                    Bingo = EventuallyDeleteOldData( Xp, Yp);
                    
    			else
    				k= 1;
    				Bingo = 0;
    			end
    			
    			if Bingo == 0
    				ClickCodeNumber = 112.;
    				ClickName = strcat('Click_Keybd_', 'f12');
    				AddToDataCollection(ClickCodeNumber, Xp, Yp)
    				UserClicks = [UserClicks; {ClickName}]; 
    				xy= [xy; [MouseHov(1), MouseHov(2)]];
                    
    			end
    		end
            
    	elseif strcmp (TheKey,  'z') 
    		% Homogeneous zooming Up
    		set(hf,'Pointer','circle'); % transient pointer change upon key press
    		TheZoom = 1/(1+ Zoom/100.); 
    		xx1 =  MouseHov(1)  -  TheZoom * Dx/2.;
    		xx2 =  MouseHov(1) +  TheZoom * Dx/2.;
    		yy1 = MouseHov(2) -  TheZoom * Dy/2.;
    		yy2 = MouseHov(2) +  TheZoom* Dy/2.;
    		set(ha, 'XLim', [xx1, xx2],'YLim', [yy1,yy2]); 
    		UpdateBoundaries();
    		
    	elseif strcmp (TheKey,  'u') 
    		% homogeneous un-zooming
    		set(hf,'Pointer','circle'); % transient pointer change upon key press
    		TheZoom = (1+ Zoom/100.); 
    		xx1 =   MouseHov(1)  -  TheZoom * Dx/2.;
    		xx2 =   MouseHov(1) +  TheZoom * Dx/2.;
    		yy1 =  MouseHov(2) -  TheZoom * Dy/2.;
    		yy2 =  MouseHov(2) +  TheZoom* Dy/2.;
    		set(ha, 'XLim', [xx1, xx2],'YLim', [yy1,yy2]); 
    		UpdateBoundaries();
            
    		
    	elseif strcmp(TheKey, 'escape')  % Quit upon user pressing 'escape' 
        	
            set(hf,'Pointer','arrow')
            set(hf,'WindowButtonMotionFcn',[])
    	    set(hf,'KeyPressFcn',[]) %  
            %set(ha,'ButtonDownFcn',[]) % <<<<<<<      or see below:
    	    set(hf,'ButtonDownFcn',[])
    		return;
    		
    	end
        %{
  <-----------               + add panning keys                   -------------->
        %}
        
    end
%  * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
    function [Xp, Yp, Err] = FindProximal_ImageXY( Px, Py)
        Err=0;
        
        if (round(Px) < ImX_Size(1)) || (round(Px) > ImX_Size(2))
            Err =1;
        else
            Xp = round(Px) ;
        end
        
        if (round(Py) < ImY_Size(1)) || (round(Px) > ImY_Size(2))
            Err= 1;
        else
            Yp = round(Py) ;
        end				
        
        if Err ==1
            Xp=[]; Yp= [];
        end
        if VERBOSE_DEBUG	
            {'FindProx'; 'Image_XY'; 'InputData:'; [Px Py]; 'OutputData:' ; ...
                [Xp Yp]; 'Error:'; Err; 'END of';  'FindProx_Im_XY' }   
        end
    end

%  * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
    function Bingo = EventuallyDeleteOldData( Px, Py)
        
        Dist1 = abs(CollectedUserData(:, 2) - Px)+ abs(CollectedUserData(:, 3) - Py);
        [minDist1, minIdx1] = min(Dist1);
        minIdx1= minIdx1(1); minDist1 = minDist1(1);
        
        LocalIdxX = find( (X_Data <= ImX_Size(2)) &...
            (X_Data >=  Current_X_limits(1)) &  (X_Data <=  Current_X_limits(2)) ); 
        
        LocalIdxY = find( (Y_Data >= ImY_Size(1)) & ...
            (Y_Data >=  Current_Y_limits(1)) &  (Y_Data <=  Current_Y_limits(2)) );
        
        Dist2 = abs((X_Data(LocalIdxX)) - Px) + abs(Y_Data(LocalIdxY) - Py)';
        [minDist2, minIdx2] = min(Dist2,[], 'all', 'linear'); 
        minDist2 = minDist2(1);
        
        if minDist1 <= minDist2 
            CollectedUserData(minIdx1,:) =[];
            xy(minIdx1,:) = [];
            UserClicks(minIdx1) = [];
            k= k-1; 
            Bingo = 1.;
            
        else
            Bingo = 0.;
            k= k+1;
        end
        if VERBOSE_DEBUG
            {'Eventually';'DeleteOldData'; 'InputData:'; [Px, Py]; 'MinDist:'; [minDist1, minDist2]; ...
                'Bingo Delete ?'; Bingo; 'Eventual index'; 'of deleted'; 'CollectedUserData:'; minIdx1;...
                'END of'; 'Eventually'; 'DeleteOldData'}   
        end			
    end	
%  * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
    function AddToDataCollection(ClickCode, Xa, Ya)
        Z_Data = double( reshape((TheImage(floor(Ya), floor(Xa), :)), 1, size(TheImage,3))); 
        CollectedUserData = [CollectedUserData; ClickCode, Xa, Ya, Z_Data ];
        if VERBOSE_DEBUG
            {'AddToData'; 'ClickCode:'; ClickCode; 'X-Y:'; [Xa, Ya]; 'Z-value';Z_Data;  ...
                'END of';'AddToData'}  
        end
    	
    end
%  * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
    function UpdateBoundaries()
        % update boundaries of figure window: (in pixel units)
        
        hp = get(ha,'children'); % update children 
        set(hp,'hittest','off')
        axlim = get(ha,'Position');
        fglim = get(hf,'Position');
        x1 = axlim(1)*fglim(3) + fglim(1);
        x2 = (axlim(1)+axlim(3))*fglim(3) + fglim(1);
        y1 = axlim(2)*fglim(4) + fglim(2);
        y2 = (axlim(2)+axlim(4))*fglim(4) + fglim(2);
        
        if iscell(ht)
            for jj=1:length(ht)
                set(hp(jj),'hittest',ht{jj})
            end
        else
            set(hp,'hittest',ht)
        end
        %DEBUG=  {'UpdateBounds'; 'x1, x2, y1, y2:'; [x1, x2, y1, y2]; 'END of '; 'UpdateBounds'}   
    end
%  * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * 


end  % <--  end of  the "function varargout = ginput_mod3b(ha,n)"
%  * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *