memory leaks (simple coding to complex ones)

Hi all,

I am desperate with memory leaks on Matlab coder.

A simple code as this would increase the ram usage of Matlab over time until I’ll have to kill the process or it will simply end itself the process.

close all

clear all

for ii=1:15

eval(strcat('A_',num2str(ii),'=zeros(1,28800);'));

end

i_count=1;

while true

for ii=1:15

eval(strcat('A_',num2str(ii),'=[A_',num2str(ii),'(1,2:end) rand];'))

end

pause (0.001)

i_count=i_count+1;

display(i_count)

end

This is just an example of the main code. My problem isn’t this code, but one build of multiple functions, multiple GUIs, but the idea is the same: a function working over a very very long time (several months). A very long term perspective where only after several hours Matlab’s RAM is over 2 Gb. In the actual code I defined bigger matrix, vector/cells and variables before the smaller ones. Variables NEVER change size during the endless loop. The code is clean (as much as I can think of). Actual sum of size of variables is 43 Mb constant (a file is saved every second with the Whos results), I used sparse matrix I clean and reallocate memory of the exact same size (like figure images of same resolutuion, plot axes and 3d objects, everything is the same size after millions of loops….

But I’m not asking what is wrong with that main code. I’m asking what is wrong with this simple one? A stop of the process will show same size of the files and a clear all will not make the ram go down. I read a lot on the net about this saying you can only restart Matlab (so I have a backup solution to restart Matlab at midnight everyday… but this is patching instead of solving the main problem). Please, any idea is welcome.

(do note that in order for this to work and not have an out of memory problem from java you have to adjust the java heap memory allocation to a bigger value).

Answer 1

[Jan]

"The code is clean" - no, it is not clean. You are using the evil eval, which is a desaster for efficient code. Creating variables dynamically by eval is an really bad idea, see TUTORIAL: Why and how to avoid EVAL

Do you really get problems with Java's heap memory with this example? Note that this code does not use any Java resources at all, so the cause of the problem must be somewhere else.

The dynamic allocation of RAM requests memory from the operating system. Overwriting the formerly created arrays gives the memory back to the OS, but it is free'd, when the OS finds the time to do this. Therefore it matters, how you measure the amount of used memory. As long as no other program requests the memory, the OS might mark it as "not free", but releases it as soon, as it is needed elsewhere. This does not mean, that there is a memory leak.

So please clarify, what exactly you call "ram usage of Matlab". This is not uniquely defined.

Answer 2

[Cristian JECU]

Hi, thank you for the quick answer.

I have cleaned all the evil evals in the main code (it has some thousands lines so it took me some time).

After this modification my main code has a usage like this: (see fig from the memory command of matlab regarding the MemUsedMatlab field) .This is somehow proportional to the ram usage that the task manager can show.

Do note that the code has a main while loop that cycles at ~1.5 seconds roughly with 3 different pause cmds of a couple of ms.

I can not share the entire code as it is very long/ several files/figures/ folders and also for confidential reasons. I can share some of the functions monitored with the matlab profiler if needed. One aspect of this test for memory is that the code worked in supervision mode (no arbitrary input from user, hence the code enters in a routine functioning updating values to a main GUI by reading them from different devices (some real some dummy PCs).

Updating is done mainly using set/get for values of type string and for axes is done by a set(figure_element,'Xdata',Xdata_new,'Ydata','Ydata_new), where Xdata_new = [Xdata_old(2:end) new_time]; and Ydata_new = [Ydata_old(2:end) new_value]. But this can't be the source of the problem as they run every time on the loop and the memory usage doesn't change for several tens of cycles. Big spikes on memory occurs once every 40 minutes so every more than a couple thousand cycles. Two different runs will not behave identically regarding the memory value over time.

[Jan]

The diagram has the problem, that the scale of the X axis is not clear: Which units do it use? Why is the time negative? What exactly is the problem you want to solve? Is it shown in the diagram of the memory usage?

[Cristian JECU]

The X is the time in Hours and represents a sliding window what happened in the past: "0" =now, "-1" represents 1 hour ago.

The problem I want to solve is to understand why matlab's (ram) memory randomly grows by very large values (100 to 200 Mb). As mentioned, the loop that cycles in this specific test (graph in previous message) has nothing random that might happen once in ~40 minutes, where the average cycle time is ~1.5 seconds. Every loop goes by the same routine (not one if changes output, not one switch changes case, it goes over and over again by the exact same functions).

I've recorded (using whos) the bytes and size of every variable in the workspace at the start of the program and after the stop of the coed (1.2 hours later in this case - see graph). all variables occupy the exact same memory and have the same size type and structure. In this case the workspace has a total of 43 Mb.

I can detect when the memory jumped...I could stop the code at the moment it jumped...but I don't know what to look for except a new variable (not predefined) or a sudden change in size of variables (it is not the case, i've cheked this). Axes never change size of children's Xdata and Ydata...this is guaranteed by the fact the future Xdata and Ydata contains the [2 end] of the previous ones (so it inherits old size).

No persistent used. Every variable bigger than 8 bytes is predefined at the start of the main function. I use alot of tcpip dec2hex and str2mat, but nothing of these do a substantial memory grow as they run several thousands time per cycle ...and the memory grow is not linked to a cycle but to an event that occurs once every ~2400 cycles... I am unable to discriminate what special happends that rare demanding up to 200 Mb memory.

I am probably looking for random bad management of functions. Something like you said on the dynamic eval..something that is out of an obvoius reach...and happends randomly

Thank you for trying to help.

[Jan]

Without seeing the code, it is impossible to get an educated guess of the cause of your observations.

eval creates variables in an internal look-up table such that the contents can be found dynamically. Variables created by standard code are caught by Matlab's JIT acceleration and the access is much faster. (By the way, variables changing their type are affected also, so don't do this for efficiency.) It is bold guessing, that this look-up table must be reconfigured from time to time or if it exceeds a certain limit. Note that for small look-up tables a linear search is optimal, for larger a binary search, and for tables exceeding the cache size a Bayer-tree might be better. The memory manager is not documented, but it is a well known fact that eval's do not use it efficiently.

Xdata_new = [Xdata_old(2:end) new_time];

I do not know, if this is done effiiciently inplace or if it reserves a new memory block and copies the contents of the old one. In the 2nd case the formerly used memory is given back to the operating system, but the OS decides, when to overwrite it by zeros and release it. So this can be a timing problem.