How to determine the coordinates of mouse pointer after a mouse click event.

I have written a rather rudimentary function to automate an external software GUI by use of the java robot

Example:

robot.mouseMove(coordinates(1,1),coordinates(1,2));
robot.mousePress(java.awt.event.InputEvent.BUTTON1_MASK);
robot.mouseRelease(java.awt.event.InputEvent.BUTTON1_MASK);

The software that I am using this function to automate is antiquated and requires the user to click through several settings and tabs within the GUI each time he/she wants to run the software. The function works wonderfully on my computer without any hiccups but I know that my mouseMove locations are unique to my monitor and will differ for another computer.

I am looking for a way to capture the location of the mouse pointer by using mouse clicks as an event (i.e., user clicks the mouse and Matlab retains the pixel coordinates). This way I can create a "setup file" for each user to run the first time they run the function. I currently am using coordinates(n,1:2)=get(0,’PointerLocation’) but I need to know if there is a way to tie this to a mouse click event.

An idea of what I am looking for:

n=0

MouseClickEvent toggles below code:

    n=n+1;
    coordinates(n,1:2)=get(0,’PointerLocation’)

Any advice is appreciated and I thank you in advance!

Answer 1

[Jan]

You can use FEX: WindowAPI to create a visible or almost transparent full-screen figure. Using an invisible or completely transparent window will most likely not work, because it does not catch mouse events.

Then the WindowButtonDownFcn can trigger the recording of the mouse position.

A more reliable method would be to get the screen position of the foreign GUI and set the mouse position relatively. This will work in a multi-monitor setup also, and the GUI can be moved freely by the user. Depending on the operating system there are different method to get the screen position of windows of other programs.

[EDITED] An example code taken from the source of WindowAPI.c:

void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[]) {
  HWND hWnd; 
  RECT Win_Rect, Mon_Rect;
  HMONITOR hMonitor;
  MONITORINFO mInfo;
  double *p;
  // Set name of the window accordingly:
  hWnd = FindWindow(NULL, "Name of the Window");
  //
  GetWindowRect(hWnd, &Win_Rect);  // Window dimensions
  hMonitor = MonitorFromWindow(hWnd, MONITOR_DEFAULTTONEAREST);
  mInfo.cbSize = sizeof(mInfo);
  GetMonitorInfo(hMonitor, &mInfo);
  Mon_Rect = mInfo.rcMonitor;  // Monitor dimensions
  //
  // Create output:
  plhs[0] = mxCreateDoubleMatrix(1, 4, mxREAL);
  p = mxGetPr(plhs[0]);
  p[0] = Win_Rect.left   - Mon_Rect.left + 1;    // X from the left
  p[1] = Mon_Rect.bottom - Win_Rect.bottom + 1;  // Y measured from bottom
  p[2] = Win_Rect.right  - Win_Rect.left;        // Width
  p[3] = Win_Rect.bottom - Win_Rect.top;         // Height
  //
  return;
}

UNTESTED: I do not have a Matlab or compiler currently, such that this is written from the scratch only.

[Shane]

I did think about the full screen figure but was trying not to go that route because I believe it is even more cumbersome than my original code. A relative positioning would work wonderfully if you could tell me how to get the position of the external windows. We are currently on Windows XP but will be upgrading to 7 soon (I think). The only thing I could find via google and mathworks searching is the psychtoolbox that appears to have some screen positioning items. But, knowing me, there is probably a "get()" callout that I am unfamiliar with that could handle it.

Thanks for your help Jan!

[Matt Kindig]

I'm not sure I fully understand your question, but

scr = get(0,'ScreenSize')

returns the dimensions of the monitor (in pixels). If that helps.

[Shane]

Thank you Matt. I am aware of the ScreenSize function and have used it to set a relative position for some external GUI items that I know appear at the center of the screen. However, the primary GUI can be moved around and I was trying to find a location of the GUI window on the screen to use as a relative position for mouse cursor locations.

@Jan , I will look into the WindowAPI functionality but I have a feeling that my current code will end up being simpler, if a bit tedious.

Thank you all for the help. = )

[Jan]

Ok, but how do you catch the mouse clicks outside a Matlab window?

[Shane]

Well, I would like to reiterate that I stated that this solution was tedious and cumbersome. That being said, the only way I found to create an event driven capture of the mouse pointer is to use a keyboard function break from the code to allow the user to position the mouse at the critcal points (there are 4 points of interest). I essentually use the matlab command window as a temporary instruction window with a "workaround" type of coding for the setup as follows:

function getScreenLocations
clc
disp(' ')
disp(' ')
disp('Position the Mouse pointer over the [position #1]')
disp(' ')
disp('And then type "return" in the line below and hit enter.')
disp(' ')
keyboard
coordinates(1,1:2)=get(0,'Pointerlocation');
clc

%repeat for the other locations

This obviously requires the user to manually re-initiate the code at each location for the setup. However, as I know that only a handful of people will be using this file and I work closely with all of them, I find this acceptable as I will be able to walk them through the setup.

Not the best solution, but the best I could come up with off the top of my head without mouse click events or excessive amounts of programming for a one time use function.

Again, thanks for your help :).

Answer 2

[William Leath]

Here is an alternate way to get mouse click coordinates outside of figure or frame constraints:

build an executable (I used MinGW g++):

C:\"Program Files (x86)"\CodeBlocks\MinGW\bin\g++.exe -c mouse_loc.cpp

C:\"Program Files (x86)"\CodeBlocks\MinGW\bin\g++.exe -o mouse_loc mouse_loc.o

from the following (mouse_loc.cpp) source code:

----------------------------------------------------------------------------------------------------

#define WINVER 0x500

#include<windows.h>

#include<stdio.h>

int main()

{

   GetAsyncKeyState(VK_LBUTTON); /** clears buffer **/
    POINT pos;
    while (GetAsyncKeyState(VK_LBUTTON)==0) {Sleep(15);}        
      GetCursorPos(&pos);
      printf("%i, %i", pos.x, pos.y);
  return 0;
}
--------------------------------------------------------------------------------------------------
call it from Matlab using either a DOS or system call:

        [status,mloc]=dos('mouse_loc');
        pause(0.05);
        loc=str2num(mloc);