Out of memory and new computer

Hello forum, I have to re-run a simulation that requires to create very large arrays (up to 160.000.000 elements) but with my actual computer I got the error "out of memory". Of course, I think it is due to the CPU (AMS A4) and the ram (4 gb). I run this simulation, the first time, on a very powerful computer (i7 and 64gb of ram) but it is something I can not afford. So I am thinking to buy (or rent) a new computer, but I would know which technical specifications are required, in order to not buy something useless. Can you help me to estimate which CPU and how much ram I need in order to do this? Sorry for my bad language and thanks in advance.

[Stephan]

I dont think that it is a good idea to buy a new computer. If it runs bad for you, you get the same error on the new machine.

Matlab has

tall

arrays for problems that are to big to fit in memory.

Since we dont know what exactly you try to do, this is the best hint i can give you.

Best regards

Stephan

[Alessandro Longo]

Basically, I need to calculate the response of a linear system with the command lsim. I create the vector time (t) and input (u1,u2) to calculate the output (y1,y2). These vectors are very large, about 160000000x1. How can I use the tall command? I tried to type it in the first line of my script but it doesn't seem to work.

[Walter Roberson]

Alessandro,

I see hints you might be in Italy? In particular not USA or Canada?

You might be able to use AWS for this purpose; see https://www.mathworks.com/products/parallel-computing/parallel-computing-on-the-cloud.html

[Aakash Deep]

MATLAB has a command called,

bench

which gives you the ranking of your system. This can give you an estimate of the configuration you should prefer.

[Alessandro Longo]

@Walter Robertson, unfortunately I have a student (educational) version, so I cannot run Matlab on parallel machines. Do you know any alternative solution?

@Aakash Deep, the problem is that I know the bench of this actual machine but here the script doesn't work.

[Walter Roberson]

I wonder what resources are available for MATLAB Online?

Unfortunately my license type is not eligible for MATLAB Online so I cannot look.

[David Goodmanson]

Hi Allesandro,

It's not hard to figure out how much memory is required just for the variables. Assuming t,u1,u2,y1,y2 are all the same size and you are storing double precision 64 bit numbers, 5 * 1.6e8 * 8 = 6.4 gB. Of course you have to have more than that to give Matlab space to work. Add on system requirements and that puts you up in the 10 gB range. If possible, the most direct and probably cheapest way to go would be to add more memory to the computer you have. New vs. old CPU, that is not much of a factor.

[Greg]

MATLAB Online crashes in spectacular form when I try:

a = randi(10,1.6e8,2); % So not enough for Alessandro

Simply calculating variable size is not a good answer, especially when talking about spending money on a tight budget. Run the profiler in memory mode to see how much RAM the whole codebase actually uses:

profile on -memory
% Run your code here
profile off
profile viewer

However, I still don't recommend making a purchase based on one codebase. What happens when the requirements go up to 1.8e8 elements?

Do more research into tall arrays as initially recommended. These are designed for exactly what you're experiencing. It is not a simple command you just put at the top of existing code; you will have to learn to modify the code to properly utilize tall.

Finally, since tall essentially just uses hard drive storage as RAM, there would be little difference from pagefile in your case. Try going into system settings and cranking up your pagefile size. Note, things will run much slower on HDD (or even SSD) than RAM.