How to recursively read in .TAB files that are located in a hierarchy, then save as one matlab table?
2 visualizzazioni (ultimi 30 giorni)
Mostra commenti meno recenti
Hello, I want to create one GIANT matlab table (if possible) that contains 19 columns of chronological data:
I have the data sorted like this:
![](https://www.mathworks.com/matlabcentral/answers/uploaded_files/200140/image.png)
where each folder 001_031_JAN, 032_059_FEB, etc contains daily files:
![](https://www.mathworks.com/matlabcentral/answers/uploaded_files/200141/image.png)
The .LBL files are of no importance to me. All the data I want is located in the .TAB files.
I want to read in ALL of the .TAB files from each month and each year. There is ~25 GB worth of data.
Each of the .TAB files are organized into 19 tab delimited columns which I have specified as varNames below.
I want to create one GIANT matlab table (if possible) that contains 19 columns of data from the time period 2011 to 2015.
This is my attempt to go through the MAG_tab_and_lbl_files folder and subfolders to load all the data into one table:
Loc = dir('/Users/chle4971/Google Drive/LASP/MERCURY/MESSENGER_instruments/MAG_tab_and_lbl_files/')'
varNames = {'year','DOY', 'hour', 'min','sec','time_tag','navg', ...
'Bx_sensor','By_sensor','Bz_sensor','DBx_sensor','DBy_sensor',...
'DBz_sensor','Bx_spacecraft','By_spacecraft','Bz_spacecraft',...
'DBx_spacecraft','DBy_spacecraft','DBz_spacecraft'};
varTypes = {'int64','int64','double','double','double','double',...
'double','double','double','double','double','double','double',...
'double','double','double','double','double','double'};
delimiter='\t';
datastartline=1;
opts = delimitedTextImportOptions('VariableNames',varNames,'VariableTypes',...
varTypes,'Delimiter',delimiter,'DataLines',datastartline);
ttds = tabularTextDatastore(Loc,opts,'IncludeSubfolders','0','FileExtensions','.TAB')
That's all I got. I don't know how to proceed. Also any suggestions on a better way to store such a large dataset is much appreciated.
Thanks,
-Christy
1 Commento
Siddharth Bhutiya
il 7 Gen 2019
For such large datasets, datastore would be the way to go. You could use Tall arrays with it as well.
When you tried using it, were you getting any errors ?
Risposte (0)
Vedere anche
Categorie
Scopri di più su Tables in Help Center e File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!