How can I import multiple .CSV files in MATLAB and SISTEMATICALLY process the data of each file?

Hi all. I have a couple of csv files I need to analyse (1488 files, each having 3 columns and 2304 rows containing only numerical values). I would like my script to open these csv files one by one and extrapolate the informations I need. For example, the script opens the first file, does its calculations, stores these 2 relevant values (that are unique to that file) and then moves on to the next file repeating the same calculations. The script I have right now does this but just for one file. How do I change the script so that it proceeds file by file and stores the obtained values?

1:All the files are contained in the same foulder

2: The first row of the file is Text, this is why i use :

vector=csvread(filename,1);

3: The relevant values I want the script to save before it moves to the next file are: Hav and Tz (at the end of the script).

4: What I'm struggling with here is to give Matlab the right order to do things: ie: open file-->do the calculations--->store the values----> open new file-->repeat the same calculations etc.

Thanks.

filename='2016\May\2016-05-31T23h36.csv';
vector=csvread(filename,1);

eta = vector(:,1)/100; % heave disp in metres
north = vector(:,2)/100; % northing in metres
west = vector(:,3)/100; % westing in metres

%------ TIME-SERIES ANALYSIS ----------------------------------------------------
% de-mean the time series
eta = eta - mean(eta);

sr=length(eta)/(30*60); %sampling frequency estimate based on a full 30minute record
dt=1/sr;
for i=1:length(eta) % time vector for data
    t(i)=(i-1)*dt;
end

% find the zero up-crossings
[ind,t0]=crossing(eta,t); % this finds all crossings (up and down)
n_up = 0;
for i = 1:length(ind)
    if eta(ind(i)+1) - eta(ind(i)) > 0
        n_up = n_up+1;
        ind_up(n_up) = ind(i);
        t0_up(n_up) = t0(i);
    end
end
fprintf(1,'no. of crossings = %g\n',length(ind))
fprintf(1,'no. of up_crossings = %g\n',n_up)
nwaves = n_up-1;

for i = 1:nwaves
    wv = eta(ind_up(i):ind_up(i+1));
    crest(i)=max(wv);             % determine crest of that wave
    trough(i)=abs(min(wv));     % determine trough of that wave
    ht(i)=crest(i)+trough(i);  % determine height of that wave
end

Hav=mean(ht)
Tz=mean(diff(t0_up)) 

Answer 1

[dpb]

See <Importing-all-files-from-a-specific-folder?> from yesterday. There's a section in the doc with examples on processing multiple files as well.

In addition, at a higher level there's a function FILEFUNCTION on the File Exchange that lets you simply use your function as an argument and will call it with all files in the argument list that can let you avoid actually writing the looping construct yourself.

I forget the author but I've used it successfully several times to save just such effort.

[Christian Scalia]

I must say I am a bit confused on how to make this loop. I.e how do I tell matlab to open the file, do the calcs, store the numbers, open the next file and repeat the same calculations

[dpb]

datapath=uigetdir([],'Select Data Directory'); 
d=dir(fullfile(datapath,'*.txt');
for i=1:numel(d)
  txt_file = fullfile(datapath,d(i).name);
  ...
end

is the code in the Answer linked to above...just insert your code inside the loop changing to use your variable instead of that poster's:

datapath=uigetdir([],'Select Data Directory'); 
d=dir(fullfile(datapath,'*.txt');
for i=1:numel(d)
  % txt_file = fullfile(datapath,d(i).name);  % Other poster's variable
  vector=csvread(datapath,d(i).name,1);       % Your variable instead...
  ...
  % rest of your code goes here...
end

If you already know the directory containing the files you can dispense with retrieving it if want, of course.

Answer 2

[Jeremy Hughes]

You could also look at tabularTextDatastore with transform, and writeall functions. This is the basic way that's done, but you may need to tweak the inputs for these depending on what you need to do

ttds = tabularTextDatastore(fileOrFolderName)
tds = transform(ttds,@myProcessFunction)
writeall(tds,outputLocation)

where

myProcessFunction takes the output of tabularTextDatastore's read and outputs the data you want to write.