Write data disc periodically: how to organize this?

Introduction

I would like to run simulations for an unknown number of time increments until a certain condition is reached. Typically, I would have several matrices of e.g. 50.000x50 big and some matrices of e.g. 50.000x17x50. After each time increment, the results are written to these big matlab files for storage.

Question

I would like to store the matlab files in a more efficient way, write the results to disk periodically. What is the best way to organize this?

• Should I have a 50.000*50 matrix in xxx.mat, and save to entries (3001:4000,1:50) after every 1000 increments for example?
• Or should I save the matrix into smaller parts of 1000x50 big? If so, what kind of structure/cell/array is good to use for that? And what name giving is handy in that case?

Thank you very much in advance!

Amy

Answer 1

[Walter Roberson]

Perhaps you should consider using matFile

[Amy]

Thank you! I hadn't heard of that before. It looks promising.

[Amy]

I implemented matfile and I run a profiling test for about 40 minutes. It seems that this method works quite slow, the way I implemented it.

In this time I ran my script for 398 increments, and after each increment I wrote data to the matfile for 80 different variables. This resulted in 47.6 MB of data.

The profiler took 424 seconds (7 minutes) for Matfile, see picture.

What can I do to make it faster? Is matfile the solution for me?

My implementation:

m=matfile('newtest1.mat','Writable',true);
m.t(10000,1)=0;
m.saveddv(10000,20)=0;
etc. for all 80 variables

And:

m.t(i,1)=t_now;
m.saveddv(i,20)=Vsave;

etc.

where i is the number of the increment.