MATLAB freezes the OS when using java environments: a Fatal Java Exception occurs (Resource temporarily unavailable)

Hi all,

When I close MATLAB after having run a code using java environments (java.awt.Robot & java.awt.event.*), the entire OS (Linux Ubuntu 18.04) freezes and becomes unresponsive. The problem also occurs without closing MATLAB after some time with the code running. It also occurs when un run the code in the terminal with matlab -bash "my_script". MATLAB works perfectly when I don't import java environments.

After reboot, I have the two following log files (matlab_crash_dump.10824-1 & java.log.10824) with a fatal java exception (FATAL ERROR in native method: Nativewindow X11 IOError: Display 0x7ff5114cac10 (:0): Resource temporarily unavailable). It seems to be related to the use of the getPixelColor of the java robot class. You'll find the logs below.

In a simplified version, my code import java classes in a script and then uses them in functions:

%IMPORT JAVA CLASSES
import('java.awt.Robot');
import('java.awt.event.*');
r=Robot();

%FUNCTION
function [status]=my_fun(r)
%Pixels coordinates
x=[300 600 900 1200 1500];
y=[500 500 500 500 500];

%Get pixels color until they are all black (or timeout at 20 s)
t=tic;
status=false;
while toc(t) < 20
    RGB=zeros(numel(x),3);
    for i=1:numel(x)
        RGB(i,:)=[r.getPixelColor(x(i),y(i)).getRed ...
            r.getPixelColor(x(i),y(i)).getGreen ...
            r.getPixelColor(x(i),y(i)).getBlue];
    end
    if  isequal(RGB,zeros(numel(x),3))
        status=true;
        break;
    end
end

%Add a delay (2 s)
if isequal(status,true)
    r.delay(2000);
    r.waitForIdle;
end
end

matlab_crash_dump.10824-1

--------------------------------------------------------------------------------
            Assertion detected at dim. févr. 21 20:20:38 2021 +0100
--------------------------------------------------------------------------------

Configuration:
  Crash Decoding           : Disabled - No sandbox or build area path
  Crash Mode               : continue (default)
  Default Encoding         : UTF-8
  Deployed                 : false
  Desktop Environment      : ubuntu:GNOME
  GNU C Library            : 2.27 stable
  Graphics Driver          : Unknown hardware 
  Graphics card 1          : 0x8086 ( 0x8086 ) 0x5917 Version 0.0.0.0 (0-0-0)
  Java Version             : Java 1.8.0_202-b08 with Oracle Corporation Java HotSpot(TM) 64-Bit Server VM mixed mode
  MATLAB Architecture      : glnxa64
  MATLAB Entitlement ID    : 2127504
  MATLAB Root              : /usr/local/MATLAB/R2020a
  MATLAB Version           : 9.8.0.1417392 (R2020a) Update 4
  OpenGL                   : hardware
  Operating System         : Ubuntu 18.04.5 LTS
  Process ID               : 10824
  Processor ID             : x86 Family 6 Model 142 Stepping 10, GenuineIntel
  Session Key              : ea17b098-38b5-4992-b46e-a7ce4e5c8222
  Static TLS mitigation    : Enabled: Full
  Window System            : The X.Org Foundation (11906000), display :0

Fault Count: 1


Additional Log Files:
  Java Crash Report        : /home/victor/hs_error_pid10824.log
  Java Log                 : /home/victor/java.log.10824


Assertion in void {anonymous}::mwJavaAbort() at javainit.cpp line 1395:
Fatal Java Exception. See this file for details: /home/victor/hs_error_pid10824.log

Register State (captured):
  RAX = 00007ff54b3df0ec  RBX = 00007ff5da60b088
  RCX = 000000000000000a  RDX = 00007ff5da5ea6a8
  RSP = 00007ff5c93361b0  RBP = 00007ff5c9336570
  RSI = 00007ff5da5b0bee  RDI = 00007ff5c93361c0

   R8 = 00007ff5c9336484   R9 = 00007ff5c9336484
  R10 = 0000000000000000  R11 = 0000000000000000
  R12 = 00007ff5da5fb008  R13 = 00007ff5c9336df0
  R14 = 00007ff5b6c94583  R15 = 00007ff5b6c9f4c0

  RIP = 00007ff5da50184e  EFL = 0000000000000000

   CS = 0000   FS = 0000   GS = ff00

Stack Trace (captured):
[  0] 0x00007ff5da4f8553    /usr/local/MATLAB/R2020a/bin/glnxa64/libmwfl.so+00853331
[  1] 0x00007ff5da4f874c    /usr/local/MATLAB/R2020a/bin/glnxa64/libmwfl.so+00853836 _ZN10foundation4core4diag15stacktrace_base7captureEm+00000028
[  2] 0x00007ff5da4fd989    /usr/local/MATLAB/R2020a/bin/glnxa64/libmwfl.so+00874889
[  3] 0x00007ff5da4fdaf0    /usr/local/MATLAB/R2020a/bin/glnxa64/libmwfl.so+00875248
[  4] 0x00007ff5b6c407ee /usr/local/MATLAB/R2020a/bin/glnxa64/matlab_startup_plugins/jmi/../../../../bin/glnxa64/libmwjmi.so+01415150
[  5] 0x00007ff5aee8b700 /usr/local/MATLAB/R2020a/sys/java/jre/glnxa64/jre/lib/amd64/server/libjvm.so+09496320
[  6] 0x00007ff5aec4273b /usr/local/MATLAB/R2020a/sys/java/jre/glnxa64/jre/lib/amd64/server/libjvm.so+07100219
[  7] 0x00007ff44d5b5c42 /usr/local/MATLAB/R2020a/bin/glnxa64/libnativewindow_x11.so+00011330 NativewindowCommon_FatalError+00000268
[  8] 0x00007ff44d5b6988 /usr/local/MATLAB/R2020a/bin/glnxa64/libnativewindow_x11.so+00014728
[  9] 0x00007ff5d24dc95e              /usr/lib/x86_64-linux-gnu/libX11.so.6+00264542 _XIOError+00000078
[ 10] 0x00007ff5d24daa41              /usr/lib/x86_64-linux-gnu/libX11.so.6+00256577 _XReply+00001169
[ 11] 0x00007ff4b2b0756d       /usr/lib/x86_64-linux-gnu/libXcomposite.so.1+00005485 XCompositeGetOverlayWindow+00000125
[ 12] 0x00007ff54b3c91bb /usr/local/MATLAB/R2020a/sys/java/jre/glnxa64/jre/lib/amd64/libawt_xawt.so+00229819 Java_sun_awt_X11_XRobotPeer_getRGBPixelsImpl+00000411
[ 13] 0x00007ff562c38920                                   <unknown-module>+00000000
[ 14] 0x00007ff56227de04                                   <unknown-module>+00000000

java.log.10824

  Operating System: Linux 4.15.0-135-generic #139-Ubuntu SMP Mon Jan 18 17:38:24 UTC 2021 x86_64
  Processor ID:     x86 Family 6 Model 142 Stepping 10, GenuineIntel
  Host Name:        katla
FATAL ERROR in native method: Nativewindow X11 IOError: Display 0x7ff5114cac10 (:0): Ressource temporairement non disponible
	at sun.awt.X11.XRobotPeer.getRGBPixelsImpl(Native Method)
	- locked <0x00000000d3fd6a48> (a java.lang.Class for sun.awt.X11.XRobotPeer)
	at sun.awt.X11.XRobotPeer.getRGBPixel(XRobotPeer.java:84)
	at java.awt.Robot.getPixelColor(Robot.java:395)
	- locked <0x00000000d419a130> (a java.awt.Robot)

Answer 1

[Cam Salzberger]

I assume that you are using SSH with X11 forwarding to work with MATLAB on a remote machine? I would recommend against doing that at all. X11 forwarding tends to have deletrious effects on MATLAB. Using VNC to connect to the machine and view the display tends to be more stable.

Also, directly accessing Java like this isn't really a recommended workflow either. Maybe try to find workarounds that use documented MATLAB tools?

-Cam

[Totor]

I'm not using SSH, I work only on my local machine. I'm trying to use the java robot class in matlab to interact (e.g. move the mouse, send clicks, get pixels color) with another application on the same machine. Do you have any suggestion regarding a proper workflow to do so?

Totor

[Cam Salzberger]

I'm afraid I am not aware of any alternative for programmatically simulating mouse interaction. What is the other application that you are trying to manipulate though? There may be alternatives for interacting with it (e.g. external language interfaces, hardware connectivity, web access, serial communication, etc.).

-Cam