What's the upper limit for how much data can be stored in a structure?

I am working with about 20 datasets, each with a size of about 2500x300. Is it possible to open these datasets, store them, and save off the resulting structure as a mat file without running into memory issues? This is my current code, and I run into an "Out of memory" error when I run it.

for i=1:length(fieldnames(dat))
    a=fieldnames(dat); %files to work with
    a=a{i}; %selects upper structure
    a2=fieldnames(foldername); %foldername where .dat files are held
    a2=a2{i}; %select which folder
    for j=1:length(fieldnames(dat.(strcat(a))))
        b=fieldnames(dat.(strcat(a))); %list of filenames in upper structure
        b=b{j}; %select file to open
        for k=1:length(dat.(strcat(a)).(strcat(b)))        
            filepath = strcat(foldername.(strcat(a2)),'\', dat.(strcat(a)).(strcat(b))); %sets file path
            fid = fopen(filepath{k}); %opens file
            fmt = repmat('%s', 1, 360); %reads file, text string delimter 
            C= textscan(fid,fmt,'Delimiter','\t','CollectOutput',true); %records csv data to cell array C
            fclose(fid); %closes file
            DLG.(strcat(a)).(strcat(b)){k}=C{1}; %writes data to structure
        end
    end
end

[Guillaume]

What is the purpose of all these strcat(a)? If it's just to use the side effect that strcat removes trailing whitespace, then

deblank(a)

would be a lot clearer.

[Ibro Tutic]

I can't call Dat.a. I need to turn the character array into a string before I can have the program be pointed to the correct file within the structure.

[Guillaume]

"I can't call Dat.a"

Indeed, the proper syntax is simply:

dat.(a)

No need for the strcat that does absolutely nothing.

[Ibro Tutic]

Thanks for that, works well.

Answer 1

[Guillaume]

Your code is a bit confusing with all these apparently useless strcat (see comment to the question).

Anyway, to answer your question, you at least need 2500*300*20*8 bytes ~= 120 MB to store your datasets. Add to that the overhead of the structure fields (around 20*(numberofcharactersinfieldname + constant) and of the structure itself (not much).

Unless you've got very little memory on your computer or the data size is vastly different from what you've stated, it should all fit in memory.

[Ibro Tutic]

Just found a way to shrink the data sets even further, thanks for the reply, it was helpful.

Answer 2

[Jan]

A cleaner version of the code:

fmt          = repmat('%s', 1, 360); %reads file, text string delimter
datFields    = fieldnames(dat);
folderFields = fieldnames(foldername);  % "foldername" is a bad name for a struct
for i = 1:length(datFields)
  a  = datFields{i}; %selects upper structure
  a2 = folderFields{i}; %select which folder
  aFields = fieldnames(dat.(a));
  for j = 1:length(aFields)
      b = aFields{j}; %select file to open
      filepath = fullfile(foldername.(a2), dat.(a).(b)); %sets file path
      DLG.(a).(b) = cell(1, length(dat.(a).(b)));   % Pre-allocate
      for k = 1:length(dat.(a).(b))
          fid = fopen(filepath{k}); %opens file
          C   = textscan(fid,fmt,'Delimiter','\t','CollectOutput',true); %records csv data to cell array C
          fclose(fid); %closes file
          DLG.(a).(b){k} = C{1}; %writes data to structure
      end
  end
end

I tried to move all repeated work before the loops, removed the useless strcat and there is still much potential to improve the readability. Fieldnames like "a" and "b" are not clear.