Looping through several files with the same name

Hello,

I have the following code to load a number of files with the same name

for i = 1:20
  fileName = ['flow' num2str(i)]; 
  load(fileName);  
end

All files are the same size (100 * 48). For each file I would like to do the following (i've used {filename} to indicate where the file is being used)

{filename} = rad2deg({filename}); % convert all data to degrees

data = zeros(100, 48); % create an empty array to store data

for g = 1:100
  for t = 1:48
      data(g, t) = ({filename}(g, t)-nanmean({filename}...(g,:)))/nanstd({filename}(g, :)); % convert all data to Z score
  end;
end;

I've tried to use

file = (flow(num2str(i)));

but, for some reason this returns a weird array of numbers

I'm pretty new to MatLab so any help will be greatly appreciated!

Many thanks

[Matt J]

When you do load(fileName), what is the name of the variable that you are loading? What is the name of the thing contained inside the file?

[Walter Roberson]

http://matlab.wikia.com/wiki/FAQ#How_can_I_process_a_sequence_of_files.3F

http://matlab.wikia.com/wiki/FAQ#How_can_I_create_variables_A1.2C_A2.2C....2CA10_in_a_loop.3F

[Tracey]

Hi Matt,

The files are named flow1, flow2, flow3, flow4 and so on until flow 20 They are basically arrays of numbers

Thanks

[Matt J]

I didn't ask what the names of the files were. I asked for the names of the matrices that the files contain. For example, the following saves an array called 'A' to the file flow1.mat and then displays the contents of that file.

>> A=rand(5); save flow1 A;

>> whos -file flow1
  Name      Size            Bytes  Class     Attributes

    A         5x5               200  double

In this example, if someone asked you for the name of the variable inside flow1.mat, your answer would be 'A'. Or, do you mean that your files are just ASCII text files?

[Tracey]

Hi Matt,

Apologies for my ignorance, but I'm not quite sure what you mean. I have 20 .mat files that I load. The files contain numerical data <100x48double>

I would like to process each file in the same way using something like

for i = 1:20
   flow(i) % do soemthing to flow file here
end

I have found that this doesn't work and neither does flow(num2str(i)). Is this possible?

[Matt J]

Please type the following commands and tell me what the output is

 >> S1=load('flow1.mat');
 >> whos S1,
 >> display(S1),

[Tracey]

S1 = load('flow1.mat')

S1 =

    flow1: [48x100 double]

>> whos S1

Name      Size            Bytes  Class     Attributes

S1        1x1             38576  struct

>> display S1

S1

>>

[Matt J]

OK. Well just so you know, the steps you have just done teach us 2 things

1. The file flow1.mat does NOT contain pure numerical data. It contains a MATLAB matrix variable called 'flow1'. Presumably flow2.mat contains a variable called flow2 and so forth.
2. The matrices inside your .mat files are not 100x48. They are 48x100

[Tracey]

1. I'm not quite sure what you mean by a "MATLAB variable" To me, flow1 is exactly the same as an excel spreadsheet - each cell in the 48x100 contains a number. In my workspace S1 is a struct whereas flow1 is a double

2. Apologies for mixing up my 48x100 with my 100x48 I usually transpose the data using ' I didn't in this instance.

[Matt J]

MATLAB variables are the named arrays and other quantities that you create in the MATLAB workspace, e.g.

>> a=1; b=[1,2 3]; s='cats and dogs';   %some variables

A .mat file is a file to which you can save collections of variables. It is very different from an Excel spreadsheet, since it may contain different variables of different shapes, sizes and types. The file itself can't be considered a rectangular array of "cells" at all.

The following is a good exercise to get a clearer picture of how .mat files work

>> save myfile.mat a b s

>> whos -file myfile.mat  %Look inside the file

    Name      Size            Bytes  Class     Attributes

    a         1x1                 8  double              
    b         1x3                24  double              
    s         1x13               26  char

As you can see, the above creates a file called myfile.mat which holds not one, but three variables of very different types that were previously created. Furthermore, the variables contained in this file all have their own names, completely independent of the name of the file itself.

When you tell people that you want to load a .mat file, people have to know which of the possibly many variables inside the file you want to load and what their names are. The following will load back in just the variables 'a' and 's' for example

>>load myfile a s

Answer 1

[Matt J]

So here's a solution to your original question

data=zeros(100,48,20);  %data from all 20 files

for i = 1:20

    fileName = ['flow' num2str(i)]; 
    S=load(fileName);  

    thisslice = S.(filename).';

    thisslice=thisslice-nanmean(thisslice(:));
    thisslice=bsxfun(@rdivide,thisslice,nanstd(thisslice,0,2));

    data(:,:,i)=thisslice; 

end

[Tracey]

Hi Matt,

Thanks so much for persevering with me. I tried your code and it does seem to do something along the lines of what I want, but I'm not quite sure what it's doing.

I have gone into my code and tried to write a full example of what I want. This is what I have come up with (this is assuming all files have been loaded separately i.e. one matrix for flow 1, a separate matrix for flow 2 and so on)

 for i = 1:20

    flow1 = flow1(1:94, :); 
    data1 = zeros(94:48); 
    flow_clean1 = zeros(94, 48); 

    for g = 1:94
        for t = 1:48
            data1(g, t) = (flow1(g, t)-nanmean(flow1(g, :)))/nanstd(flow1(g, :));  
            if data1(g, t) > 3 || data1(g, t) < -3
                flow_clean1(g, t) = NaN;
            else
                flow_clean1(g, t) = flow1(g, t);
            end; 
        end;
    end;   

 end;

The i is to represent the flow files 1 to 20 (and currently doesn't do anything) for each file labelled with a 1 (i.e. flow1, data1, flow1_clean) I would like i to replace the number.

I'm thinking I'm going to have to use some sort of structured array, is this correct?

Thanks for your help.

[Matt J]

I'm not sure what your terminology "structured array" means. Because all of your flow_i matrices are the same size and are pretty small, it was never a good idea to have them in separate files in the first place. That makes them a lot more difficult to simultaneously manipulate. We will undo this first. What you really want to do is stack the matrices in a 3D array and save that stack to one file, which is what the next code segment shows you how to do.

    flow=zeros(100,48,20);  %flow matrices from all 20 files

    for i = 1:20
        fileName = ['flow' num2str(i)]; 
        S=load(fileName);  
        flow(:,:,i) = S.(filename).';
    end

    save AllMyFlows flow

Now we can begin to do the manipulations you really want. The result of the code below should be a 94x48x20 array 'data' and another array 'flow_clean' of the same dimensions. flow_clean(:,:,i) and data(:,:,i) correspond to the i-th set of flow data.

    S=load('AllMyFlows'); %A safer way to load variables

    flow=S.flow(1:94,:,:);

    rowmeans=nanmean(flow,2);
    rowstds=nanstd(flow,0,2);

    data=bsxfun(@minus,flow,rowmeans);
    data=bsxfun(@rdivide,data,rowstds);

    flow_clean=nan(size(data));
    flow_clean(abs(data)<=3)=0;

[Tracey]

Hi Matt,

Thanks so much for your help.

I'm now able to process my data using very few lines of code! Much appreciated!

Tracey

Answer 2

[George]

Tracey,

'flow' is a built-in matlab function for visualizing 3-D data. See >>help flow. Therefore you should not use "file = (flow(num2str(i)));"

Similar to what Matt mentioned, if you load a mat file, using e.g. load('flow15'), then to process it as your code tries, then the array found within each file must have the same name as your filename.

From your posting, I do not really understand what is failing. What, specifically is the problem?

Also, unless you have written a function 'rad2deg', I do not think that Matlab has a built-in rad2deg.

[Tracey]

Hi George,

I didn't realise flow was already a built in function. I will rename my files.

I have 20 files, they are all .mat files and contain numerical data (100 x 48 arrays)

I would like to process each file individually, using a loop

For example, (I've given the files the name "file" for now) I would like to run the following loop on all 20 files without having to copy and paste the loop twenty times.

   file1_clean = zeros (100, 48); 

   for g = 1:100
       for t = 1:48
           if data(g, t) > 2 || data(g, t) < -2
               file1_clean(g, t) = NaN;
           else
               file1_clean(g, t) = file1(g, t);
           end; 
       end;
   end;

I already have rad2deg so that's not a problem.

Thanks for your help

[George]

After you load one of the files, e.g. 'file1', into an empty workspace, what variable name(s) do you see in the workspace, and what are the variable's dimensions? [100 x 48]? Alternatively, type 'whos' in the command window to get var name and size.

[Tracey]

Name Size Bytes Class Attributes

flow1      100x48            38400  double

[Tracey]

If I type in whos flow1 I get

Name         Size            Bytes  Class     Attributes

flow1      100x48            38400  double