Elements of table equal to elements of another table

Hi all!

I have a list of stations (see .txt) and a .csv (see attached) with several data. What I need to do is scan the entire .txt and for every station, eg. Airport, to go and scan column 14 of the .csv. If the station is the same, I want to do several things, eg. extract the date (column 2) and Temperature (column 15) every time Airport appears.

I tried with eq but it doesn't work.

list1 = rdir ('C:\Projects\LIFE ASTI\C.3\C.3\Weather station data\from desktop\Obs. data 4 Keppas\'); 
wrfdir  = 'C:\Projects\LIFE ASTI\C.3\C.3\Weather station data\from desktop\2019 run\Thess\*.csv';
stations = readtable('C:\Projects\LIFE ASTI\C.3\C.3\Weather station data\from desktop\Stations coordinates.txt'); 
output_path='C:\Projects\LIFE ASTI\C.3\C.3\Weather station data\from desktop\OutputSimulations';

WRF_Data = readtable('C:\Projects\LIFE ASTI\C.3\C.3\Weather station data\from desktop\2019 run\Thess\THESS_Temp.csv');

% Variables
vars={'Temperature'; 'Relative humidity'};
years = {'2015'; '2019'};
varunits={'Celsius'; '%' };

%Starting loop for all files
for i=1:size(stations,1)
 
 for k = 1:size(WRF_Data.Station,1)
    if  eq(stations(i,1) , WRF_Data.Station(k,1))

              KEEP TEMPERATURE OR WHATEVER...
    end
   end
end

Answer 1

[Guillaume]

I'm not entirely clear what the ultimate goal is but one thing for sure, you don't need a loop at all.

To keep only the rows of WRF_Data that have a station in stations AND append the rest of the station variables to these rows, it's simply:

merged = innerjoin(WRF_Data, stations, 'Keys', 'Station');  %And you don't even need the 'Keys', 'stations' although it makes the code clearer. 

If you don't want all the variables in the output, you can tell innerjoin which one you want:

merged = innerjoin(WRF_Data, stations, 'LeftVariables', {'Station', 'Var1', 'Temp'}, 'RightVariables', {'Lat', 'Lon'}) %for example

[Daphne PARLIARI]

Thank you for your answer!

What I need to do in the end, is end up with one .csv (or whatever) for every station that contains the respective data for that station only. In Stations Coordinates.txt you can see that I have 16 stations, therefore I must end up with 16 files which will have hourly data (temperature or whatever) plus everything else, for each station.

I tried merged as you suggested but I get

Error using tabular/innerjoin (line 98)

Left and right key variables 'Station' and 'Station' are not comparable because one is a non-cell.

Error in WRF_Simulations (line 17)

merged = innerjoin(WRF_Data, stations, 'Keys', 'Station')

[Guillaume]

I have no idea how you imported your data, the following works fine (R2019b) for me with your input files

stations = readtable('Stations coordinates.txt');
%use detectImportOptions to fix wonky date import of the csv file. Encoding of the dates in that csv file is ... less than ideal!
opts = detectImportOptions('THESS_Temp.csv');
opts = opts.setvaropts('Date', 'InputFormat', 'dd-MM-yy H:mm', 'DatetimeFormat', 'dd-MMM-uuuu HH:mm');
WRF_Data = readtable('THESS_Temp.csv', opts);

%then
merged = innerjoin(WRF_Data, stations, 'Keys', 'Station');

Again, the function you want is innerjoin, not ismember in a loop which just complicates things (and is most likely slower)

Note that your demo files have only one station in common 'Airport'

To then split and save merged per station:

[group, stations] = findgroups(merged.Station);
splitapply(@(rows, statidx) writetable(merged(rows, :), sprintf('%s.csv', stations{statidx(1)})), (1:height(merged))', group, group);

[Daphne PARLIARI]

I am running on 2019a, I don't know if the problem lies there....

[Guillaume]

Doesn't the above work exactly as written?

I suspect you're getting your innerjoin error because in one table Station is a string array and in the other a cell array of char vectors. Converting one to the other would fix the problem.

[Daphne PARLIARI]

I used

WRF_Data.Station = cellstr(WRF_Data.Station);

and indeed it fixed the compatibility problem.

It works like a doll, million thanks!

The only thing remaing now, is that the split files must go to respective folders so I can detect them easier. With the code I have so far, they are saved to the current dir...

[Guillaume]

I recommend you use the import code I wrote.

You can modify the anonymous function in the splitapply code to do whatever you want, eg. to save into individual directories named after the station:

rootpath = 'C:\somewhere\somefolder';
[group, stations] = findgroups(merged.Station);
splitapply(@(rows, statidx) writetable(merged(rows, :), fullfile(rootpath, stations{statidx(1)}, 'station.csv')), (1:height(merged))', group, group);

Or if it's too complex, use a loop:

rootpath = 'C:\somewhere\somefolder';
[group, stations] = findgroups(merged.Station);
for gidx = 1:numel(stations)  %iterate over each unique station group
    writetable(merged(group = gidx, :), fullfile(rootpath, stations{gidx}, 'station.csv'));
end

[Daphne PARLIARI]

Great, thank you!

If you have time, could you please explain to me what each of line

splitapply(@(rows, statidx) writetable(merged(rows, :), sprintf('%s.csv', stations{statidx(1)})), (1:height(merged))', group, group);

does?

[Guillaume]

splitapply split the input arrays into groups and apply the same function to each group, that is it takes the rows of each input array corresponding to a group and pass that to the function, then it passes the rows corresponding to the next group, etc.

The groups are defined by the last input to splitapply, in this case group. The two arrays that I've decided would be split are (1:height(merged))' which is basically the row indices of the table and group as I actually need the group index to be passed to the processing function. The processing function is the anonymous function: @(rows, statidx) writetable(merged(rows, :), sprintf('%s.csv', stations{statidx(1)})). This is a function with two inputs rows and statidx. rows receives in turn the elements of the first input of splitapply: (1:height(merged))', that is it receives the row indices of the table corresponding to a group (a station), statidx receives in turn the elements of the 2nd input, group corresponding to a group, so it's a vector of identical numbers, hence why I use statidx(1) in the function body.

The function body writes the sekect rows of the table to a file whose name is the station. If we were to rewrite the anonymous function, it would be more or less

function anonfun(rows, statidx)
    %rows: row indices of the table corresponding to a single station. Column vector
    %statidx: column vector of identical values containing the station/group number
    %Unlike normal function, anonymous functions can capture variable from the enclosing workspace
    %Here, it also captures
    %merged: the table
    %stations: list of unique station names, ordered by group number
    subtable = merged(rows, :);  %extract portion of the table corresponding to station
    currentstation = stations{statidx(1)};
    filename = sprintf('%s.csv', currentstation);
    writetable(subtable, filename);
end

And if you were to write what splitapply does as a loop, it would be (in this case):

input1 = (1:height(merged))';  %row indices of the table
input2 = group;
for g = 1:max(group)
    thisgroup_input1 = input1(group == g);
    thisgroup_input2 = input2(group == g);
    anonfun(thisgroup_input1, thisgroup_input2);  %anonfun also captures merged and stations without needing to pass them as arguments
end

Answer 2

[Philippe Lebel]

Here you go:

I loop over all station names and check where in the data table i find these names. I generate a boolean index in order to fetch data from the table and store them in a structure.

clear
stations = readtable('\Stations coordinates.txt'); 
WRF_Data = readtable('\THESS_Temp.csv');
%Starting loop for all files

for i=1:size(stations,1)
    WRF_Data_station_mask = ismember(WRF_Data(:,14),stations(i,1));
    sorted_values(i).name = char(stations{i,1});
    sorted_values(i).temps = WRF_Data(WRF_Data_station_mask,15);
end