load multiple .mat files

I am trying to load multiple .mat files through a script. Each file is a recorded channel, that contains data logged at high frequency (so some files are large) with a time stamp. The file name corresponds to the channel name allocated in a logging device. Ideally, I would like to end up with a table with all data combined from all .mat files.

I tried the following:

    clear

    PathName        = uigetdir;
    file_all        = dir(fullfile(PathName,'*.mat'));
    matfile        = file_all([file_all.isdir] == 0); 
    clear file_all PathName

    %d=dir('*.mat');  % get the list of files
    x=[];            % start w/ an empty array
    for i=1:length(matfile)
    x=[x; load(matfile(i).name)];   % read/concatenate into x

    if true
      % code
    end

but get the error: Error using vertcat Names of fields in structure arrays being concatenated do not match. Concatenation of structure arrays requires that these arrays have the same set of fields.

file names are different sizes, I don't know if this is the issue? Is there a simple way to load multiple files of numerical data? many thanks

[KSSV]

   x=[x; load(matfile(i).name)];

This is wrong. Simply use

load(matfile(i).name) ;

And initialize the table and put your variables in that table.

[Stephen23]

@KSSV: it is recommended to always load into a structure:

S = load(...)

this avoids many of the problems related to creating variables in workspaces, and makes debugging simpler.

"This is wrong"

Why? As long as the variables in the .mat have the same names, concatenating the load-ed structures will work perfectly, and is in fact preferable to loading directly into the workspace (as you showed). Can you please explain why you believe that "This is wrong".

Answer 1

[Guillaume]

Yes, you can't concatenate structures (the output of load) with different field names, which will always be the case with load. You can however easily concatenate tables with different columns names, so converting that structure in the loop to a table would work:

    pathname        = uigetdir;
    allfiles        = dir(fullfile(PathName,'*.mat'));
    matfiles        = allfiles([allfiles.isdir] == 0);   %probably not needed since it's unlikely that a xxx.mat is a directory

    fulltable = table(); 
    for fileid = 1:numel(matfiles)
       [~, filename] = fileparts(matfiles(fileid).name);
       filetable = struct2table(load(fullfile(pathname, matfiles(fileid).name))); 
       filetable.Properties.VariableNames{2} = filename;  %rename 2nd variable
       fulltable = [fulltable, filetable(:, 2)];  %only copy second variable
    end

[woody1983]

Thanks Guillaume, There are two variables in the .mat file. Column 1 is always the time stamp named for example variable1_time, the data I require is always in column 2 named variable1 for example. Is there any way to just pick the second column? many thanks

[Stephen23]

@woody1983: the simplest solution would be to simply always use the same variable names. The your code would be simple, and you could trivially concatenate them in a loop (or with preallocation).

By choosing to put meta-data into the variable names you just make your code pointlessly complicated. Much simpler is to store any meta-data as data in its own right:

time = [...];
data = [...];
test = 'opel vectra';
save('file1.mat','time','data','test');

etc for more files. The to load the files is trivially easy because the names are always the same:

D = dir('*.mat');
S = struct();
for k = 1:numel(D)
    S(k) = load(D(k).name);
end

Read more about why putting meta-data into variable names and why this is not a good code practice:

https://www.mathworks.com/matlabcentral/answers/304528-tutorial-why-variables-should-not-be-named-dynamically-eval

[Guillaume]

@woody1983,

I've edited my answer to only rename and copy the second variable.

[Callum Gannon]

This for loop seems to overwrite the previous .mat file data on every iteration. I have 36 files and each file contains 4 different arrays of different sizes. How can I use this for loop to place all the data in the same table?

Answer 2

[woody1983]

That works well, thank you very much