Why is rand() causing a crash in parallel batch jobs?

Hi,

We have a user on our cluster that keeps getting crashes when he attempts to run his matlab job as a batch parallel job. I've narrowed the problem down to the rand() function.

If I do the following interactively from the GUI, all is well:

n = 10000

A = rand(n,n)

This should yield a 100 million element matrix. Now, if I add those two lines to 'test.m' and then pass it to a batch job like so:

job = batch('test')

the job runs successfully, but if I attempt to run a parallel batch job (with the Parallel Computing Toolbox) then I get the following:

*****

>> job = batch('test', 'matlabpool', 2)

...

>> wait(job)

>> load(job)

Warning: Variable 'errormessage' not found. 
> In distcomp.fileserializer.getFields>iLoadMat at 92
  In distcomp.fileserializer.getFields at 70
  In distcomp.fileserializer.getField at 11
  In distcomp.abstracttask.pGetErrorMessage at 15
  In distcomp.abstractjob.load at 51 
Warning: Variable 'errormessage' not found. 
> In distcomp.fileserializer.getFields>iLoadMat at 92
  In distcomp.fileserializer.getFields at 70
  In distcomp.fileserializer.getField at 11
  In distcomp.abstracttask.pGetErrorMessage at 15
  In distcomp.abstractjob.load at 51 
Error using distcomp.abstractjob/load (line 54)
The job failed to run correctly but no error message was returned.
This could be because the scheduler failed to start MATLAB correctly on
the cluster, or because the files needed to run the batch job were
unavailable to the MATLAB on the cluster. You may find more information
in the debug log for this job. To find out more about debug logs look at
the getDebugLog function in the documentation.
*****

I hope you'll forgive the example. It's obviously not useful to run the program above on several processes in parallel; it's just to test.

The job obviously crashes, but Matlab doesn't have anything useful to say, as if it wasn't able to handle the error itself. The output from using getDebugLog as it suggested didn't yield anything useful either. This lead me to believe that perhaps a limitation was hit in the JVM, thus the crash is happening outside of Matlab

I read in the troubleshooting section of the manual that the maximum object size that the JVM allows is 2GB, so I connected to one of the processes via VisualVM to watch the heap, etc, and it doesn't seem to be getting over 100MB. It's possible I'm looking in the wrong place, or at the wrong time.

So I'm at a loss of how to troubleshoot this further, or at least understand why it's happening.

Error, or memory limit?? Can anyone shed light on why this is happening, how to solve it, or possibly give an explanation of what limit is being hit?

Thanks in advance for any help.

Answer 1

[Jason Ross]

1. Are there any dependent files that you didn't attach? From your example, it looks like the answer is "no", but your real code may depend on a function that isn't included or can't be found -- even if it's a trivial function.
2. Is it possible you are shadowing something with your function names?

[Daniel]

Hi Jason,

Thanks for your swift reply. As I pointed out, the application is dead simple; just these two lines:

n = 10000

A = rand(n,n)

This is the entire application, so there are no other files, and the only function I'm calling is rand() which is a built in function call, and unless 'n' and 'A' are reserved variable names, they shouldn't be a problem either.

[Jason Ross]

I agree.

What release, OS, and cluster type are you using? Where are the cluster MATLABs installed?

FWIW there is a "test" function in the stats toolbox. If you rename "test" to "test1", do you get the same result?

Do you get the same result with the local cluster?

[Jason Ross]

FWIW I did try this on my local setup and it did work properly.

[Daniel]

Hi Jason, It's 2011b, we're running Ubuntu 10.04LTS 64bit on our computing nodes.

We're not using Cluster MATLAB, just the Parallel Computing Toolbox for local parallel jobs. I think that's what you're referring to when you say 'local cluster', right?

I'll double check the file name just in case, though I doubt that's the problem since I think I had initially tried with a filename 't1.m' and still had the same problem.

off topic, but why are my newline characters being swallowed in my text on this page? I have to hit enter twice for it to show up...annoying.

[Jason Ross]

Yes, the "local" cluster is what I was referring to.

I looked further into it. I found the following:

• In 2011b, I can reproduce the issue using your example. Filename does not matter at all.
• In 2012a and 2012b, it works as you would expect. The job runs and you can load the results without incident.

I was able to get a bit more information by using getAllOutputArguments(job), but it didn't provide help in solving the issue. It looks like the problem you were seeing was something that was fixed in 2012a, though.

I don't know about your newline problem. Perhaps clearing a browser cache would help?

[Jan]

In fact, you need two newlines to start a new paragraph. A single newline does not wrap the text.