Slow down of save(..., '-append')

Hi there, I am currently using the v7.3 mat file to store hourly hydrologic simulation results. In the simulation we generate a few million variables so that I have to write them to monthly file to maintain a reasonable number files. In each monthly file, I have 24*30*10 variables. I am using

if exist(fileName,'file')~=7
   save (fileName,varName,'-v7.3');
else
   save (fileName,varName,'-append'); 
end

to save the multi-variable file because I figure out it is faster than

S=matfile(fileName,'Writable',true);
cmd=['S.',varName,'=',varName];
eval(cmd);

With the number of variables increasing, the writing speed slows down greatly. I understand that matlab needs to check the existence of a variable in the file before writing and the checking time is longer when the number of variables is larger. However, in my case, this checking is redundant because my model can ensure that each variable I am writing does not exist in the file. I am wondering if there is a way to avoid this time consuming checking when I use the save command with '-append' option.

Thanks!

[Adam]

S=matfile(fileName,'Writable',true);
S( 'varName' ) = varName;

would seem to be a better way of doing your second option.

How have you ascertained that it is the checking if your variable exists or not that is taking the extra time rather than just the fact that your file is getting bigger and bigger therefore appending to it will get slower and slower?

[dpb]

"generate a few million variables"

I'd suggest revamping the simulation to not create dynamic variables but use a structure or other data organization instead.

[Walter Roberson]

Xinyi commented to Adam:

Thanks. The reason I did not use the second option because I figured out that the statement S=matfile takes most of the time because it need to load all variable names into the structure S. I am not 100% sure that for my first method, the major time is spent in checking the variable list. But I read from other answers to other questions about the overwriting of existing variables. It is answered that the overwriting is time consuming because it needs to check the variable list. But if there is another way to preallocate the file size, it is more than welcomed because I can roughly estimate the file size.

[Walter Roberson]

Xinyi commented to dpb:

I am not generating dynamic variables but dynamic files. Each variable remains unchanged in the file. It is impossible for me to put them all in memory and write them to the disk once for all because the RAM is also limited.

[jgg]

Why are you creating so many variables? Why not create a single matrix to store them instead? This would completely avoid your problem. if it's a RAM issue, use several matrices or a struct instead which doesn't use continguous memory.

Answer 1

[Walter Roberson]

Do not use eval() . At most,

S.(varName_as_string) = varName_not_as_string

Secondly: save -append does not remove any content from the file, except possibly if the variable happens to be the very last thing in the file. Instead, it just marks any existing variable of that name as being unused, leaving it in the file. This might not matter to you if you are never replacing variables. Using -append to add new variables might result in a file that is not as efficient to access as if it had all been written at one time. MATLAB does need to examine the entire file to determine whether there is a variable of that name; it will not "take your word for it"

Third: matFile is the proper way to do these kinds of updates of a .mat file . I would recommend you matFile only once and leave it existing until you no longer need it, rather than calling it each time you want to add a variable.

Fourth: If your variables are a consistent size, consider using binary files, possibly together with memmapfile. If you need .mat files specifically for later use, it can still make sense from an efficiency standpoint to write binary files during the processing stage, and then later go back and convert the binary files into .mat files... at which time you already have all the data available and so can use a single save() to put it into the file.