How to solve "out of memory error" caused by bsxfun?

If I try to enlarge the base sample size of 10000 by 500 samples (in the context wie a Sensitivity Analysis) the following error message occurs:

Error using bsxfun
Out of memory. Type HELP MEMORY for your options.

Error in ipdm>distcomp (line 1008)
      d = bsxfun(@minus,set1(:,1),set2(:,1)').^2;

Error in ipdm (line 364)
      d = distcomp(data1,data1,params);

Error in lhcube_extend (line 89)
    dd=(ipdm(Xi,'metric',2)); dd=min(dd(dd>0));

Error in AAT_sampling_extend (line 125)
Uext = lhcube_extend(U,Next,nrep) ;

The script AAT_sampling_extend produces the error. AAT_sampling_extend is integrated in the SAFE-Toolbox (and has already been used several times).

I started this with the following command:

Xext = AAT_sampling_extend(X,DistrFun,DistrPar,2*(N+N2))

with

• N = base sample size (N = 10.000)
• N2 = extended sample size (N2 = 500)
• X = Matrix of size 10.000 * 31
• DistrFun = Latin hyper cube sampling
• DistrPar = Parameters and their range

How can you solve the problem (without changing the source code too much)?

[Glazio]

I have upgraded the RAM (from 4 GB to 12 GB) and the same error message appears.

I have increased the virtual memory to the recommended value (18430 MB).

I have closed other applications (processes), cleaned unused large variables and pre-allocated arrays.

At the time of program-crash, more than 4 GB of RAM were still free. Currently I have no idea what the problem could be.

Does anyone have an idea (RAM, CPU aren't the problem)?

Answer 1

[Jan]

A 10000 by 500 array of type double requires 40MB only. If your computer does not have enough memory to allocate such a small array, then the best idea is to install more RAM. Other ideas:

• Close other applications
• Increase the virtual memory
• Use the type single if precision is sufficient
• Clean unused (large!) variables
• Pre-allocate arrays (this might not necessarily help, but is a good programming practice in every case)
• But nothing will beat real hardware RAM, so install more.

[Glazio]

@ Jan Simon: Thanks for your answer.

I've tried the following:

• closed other applications
• increased the virtual memory (inreases swap-memory in windows7)
• cleaned unused variables
• pre allocated arrays
• the RAM has 4 GB (and according to taskmanager more than 2.5 GB are freely available)

Then I will expand the RAM to 16GB (and then hopefully the execution of the function will work).

[Jan]

Try

feature memstats

to check the memory. It is surprising, that there is no contiguous block of 40MB left. Perhaps the array is larger than 1e4x500 in opposite to your expectations?

[Glazio]

@ Jan Simon:

feature memstats:

    Physical Memory (RAM):
        In Use:                             2170 MB (87a10000)
        Free:                               1925 MB (78539000)
        Total:                              4095 MB (fff49000)
    Page File (Swap space):
        In Use:                             2290 MB (8f271000)
        Free:                              10803 MB (2a3307000)
        Total:                             13093 MB (332578000)
    Virtual Memory (Address Space):
        In Use:                              718 MB (2ce73000)
        Free:                               3377 MB (d316d000)
        Total:                              4095 MB (fffe0000)
    Largest Contiguous Free Blocks:
         1. [at 7fff0000]                   2045 MB (7fdf2000)
         2. [at 2c710000]                    953 MB (3b9b0000)
         3. [at 683dd000]                    100 MB ( 64d3000)
         4. [at 770f0000]                     59 MB ( 3bb0000)
         5. [at 6ebdb000]                     20 MB ( 1445000)
         6. [at 7c427000]                     16 MB ( 1089000)
         7. [at 7b58e000]                     14 MB (  e02000)
         8. [at 71ea2000]                     11 MB (  b4e000)
         9. [at 26f70000]                      8 MB (  850000)
        10. [at 7e1ea000]                      8 MB (  826000)
                                            ======= ==========
                                            3238 MB (ca6b9000)

I have tried now to run the function with different sample sizes. In addition, I have increased the swap-memory even further. Now the function runs up to a sample size of 500, without any problems.

However, about 9000 additional samples would be required.

So I'll have to upgrade the RAM.

How big should be the RAM for such a sample?

[Jan]

 How big should be the RAM for such a sample?

More. Buy more RAM.

It is surprising, that you cannot allocate a 40MB array, if you have a free block of 2045 MB. There must be another problem. Therefore I cannot extrapolate how much RAM you need and what "additional 9000 samples" require. Some programming techniques help to save memory, see e.g. Loren's Blog: In-place operations and Loren's Blog: Memory Management .

Most of all find out, what exactly has exhausted you memory. Perhaps it is an orphaned zeros(1e6) if zeros(1,1e6) was meant.

[Glazio]

@ Jan Simon:

Thanks for your answer and the helpful links to programming techniques to save memory.

So I'll upgrade the RAM and try to find out what has exhausted my memory.

[Image Analyst]

Did you run memstats in your code right before you call bsxfun, or did you call it on the command line without your program running?

[Glazio]

@ Image Analyst: I called it for both situations.

And as mentioned above: I have tried now to run the function with different sample sizes. In addition, I have increased the swap-memory even further. Now the function runs up to a sample size of 500, without any problems.

So I think the main problem is the RAM (with 2 x 2 GB -> in sum 4 GB).

I can expand the RAM to max. 16 GB (more is not possible with this PC).

So I hope that in the following several runs will be possible.

Feature memstats gives the following result during the run for 501 samples (and breaks):

 Physical Memory (RAM):
        In Use:                             2111 MB (83f1a000)
        Free:                               1984 MB (7c02f000)
        Total:                              4095 MB (fff49000)
    Page File (Swap space):
        In Use:                             2410 MB (96ad9000)
        Free:                              10682 MB (29ba9f000)
        Total:                             13093 MB (332578000)
    Virtual Memory (Address Space):
        In Use:                              761 MB (2f9e3000)
        Free:                               3333 MB (d05fd000)
        Total:                              4095 MB (fffe0000)
    Largest Contiguous Free Blocks:
         1. [at 7fff0000]                   2045 MB (7fde3000)
         2. [at 2e0c0000]                    919 MB (39780000)
         3. [at 6bd0d000]                     71 MB ( 47f3000)
         4. [at 67871000]                     50 MB ( 320f000)
         5. [at 77e30000]                     46 MB ( 2e70000)
         6. [at 7c427000]                     16 MB ( 1089000)
         7. [at 6ab11000]                     15 MB (  fbf000)
         8. [at 7b58e000]                     14 MB (  e02000)
         9. [at 74057000]                     10 MB (  ad9000)
        10. [at 7e1ea000]                      8 MB (  826000)
                                            ======= ==========
                                            3199 MB (c7f1e000)

[Glazio]

I have upgraded the RAM (from 4 GB to 12 GB) and the same error message appears.

I have increased the virtual memory to the recommended value (18430 MB).

At the time of program-crash, more than 4 GB of RAM were still free. Currently I have no idea what the problem could be.

Does anyone have an idea (RAM, CPU aren't the problem)?

[Jan]

Sorry, the question is still not clear. Please stop Matlab when the problem occurs:

dbstop if error

Then tell us the dimensions of the variables "set1" and "set2". Formerly you mentioned a 10000 by 500 matrix, which needs 40 MB only. If you have 4GB of free memory, this cannot be the problem. If bsxfun tries to create a 10'000 x 10'000 matrix, this requires 800MB of RAM.

I hesitate to install the mentioned toolbox only to reproduce the problem, if you can find out the source of the problem so fast. Please check how much memory is required in the failing command.

Finally remember, that some problems demand for a huge RAM. If a code works with 500 samples, the required computation time can explode to millions of years for 1000 samples.

[Glazio]

@ Jan Simon: Thanks for your answer,

the input-matrix has a dimension of 10000 x 31.

If I extend it by 10000 samples there are generated three matrices:

XA with 5000 x 31, XB with 5000 x 31 and XC with 155000 x 31.

If I extend it by 500 samples there are generated three matrices:

XA with 200 x 31, XB with 200 x 31, XC with 620 x 31

dbstop if error gives the following result:

Error using  .^ 
Out of memory. Type HELP MEMORY for your options.

Error in ipdm>distcomp (line 1015)
          d = d + bsxfun(@minus,set1(:,i),set2(:,i)').^2;

Error in ipdm (line 364)
      d = distcomp(data1,data1,params);

Error in lhcube (line 88)
    dk=(ipdm(Xk,'metric',2)); dk=min(dk(dk>0));

Error in lhcube_extend (line 86)
    Xext=lhcube(N_new-N,M);

Error in AAT_sampling_extend (line 125)
Uext = lhcube_extend(U,Next,nrep) ;

Error in extended_sample (line 76)
Xext = AAT_sampling_extend(X,DistrFun,DistrPar,(N+N2)) ; % extended sample

Answer 2

[Steven Lord]

Are you using a 32-bit or 64-bit version of MATLAB and version of your operating system? If you're using a 32-bit system, upgrade to a 64-bit system.

It seems like the function you're using calls this File Exchange submission or something similar and creates a 10000-by-10000 inter-point distance matrix during the course of its execution. That would require a contiguous block of memory about 750 MB in size. Anything you do that requires a new copy of that matrix may require another contiguous block of that same size.

[Glazio]

@ Steven Lord:

Thanks for your answer. I use a Matlab Student Version (R2012a - 32 bit version). The operating system of my desktop-pc is Windows 7 (64 bit-version).

How can I upgrade my student license (from 32 bit to 64 bit)? Does a new license need to be purchased?

[Jan]

+1. A good idea to ask for a 32 bit version.

Did you associate your Matlab account with the student license? I'm not sure, if this is possible. But if it is, I'd expect that you find the downloads at https://www.mathworks.com/downloads/web_downloads/select_release?mode=gwylf. Can you open this page?

If not, does https://www.mathworks.com/licensecenter/licenses help to associate with your license?

[Glazio]

@Jan Simon: Thanks for your answer.

Yes my Matlab account is associated with the student license and I can open the first web link.

The only problem is that no 64-bit version is offered as a download for Windows.

The following download versions are offered:

• Windows (32bit)
• Mac OS X (Intel 64bit)
• Linux (32bit)

[Jan]

Then buy an Apple computer. Sorry, just kidding.

Perhaps the student versions have been limited to 32 bit. Contacting the support directly can clear this directly. Simply use the "Contact us" button on this page.

You can upgrade a student version, see https://de.mathworks.com/matlabcentral/answers/242937-how-can-i-update-my-matlab-student-license-to-a-newer-version.

[Glazio]

@ Jan Simon: Thanks for your answer. I would prefer Linux as a operating system ;-)

2012a is not available for 64-bit Windows system. This means that a new license must be purchased.