How to import and read several large csv in matlab
7 visualizzazioni (ultimi 30 giorni)
Mostra commenti meno recenti
Ricardo Duarte
il 21 Giu 2022
Commentato: Sulaymon Eshkabilov
il 22 Giu 2022
Dear all,
I have a huge amont of large csv files (about 170Mb each).
I need to open them and use the information in two of the columns to plot, let's say collumn D and E (see the attachment).
This is what I did so far, however I found that it is not very efficient.
%Load directory
directorio=uigetdir;
x=dir([directorio,'/','*.csv']);
K=length(x);
Atotal=[];
tabletotal=[];
fprintf('Leitura: \n')
tic;
for k=1:K
FILENAME = strcat(x(k).folder,'/', x(k).name);
TotalData=readtable(FILENAME);
table=TotalData(:,[4 7 18 19 27]); %4-vessel name; %7-vessel type; 18-longitude; 19-latitude
fprintf('File_%s: %s \n', num2str(k), FILENAME);
tabletotal=[table; tabletotal];
%Coluna 2= longitude
%Coluna 3= latitude
end
loadAIS=toc;
fprintf('Tempo de carregamento do AIS: %s \n', num2str(loadAIS) );
clear table;
clear TotalData;
A = table2array(tabletotal(:,2:5));
Thank you in advance
1 Commento
Risposta accettata
Sulaymon Eshkabilov
il 21 Giu 2022
If your data file names are sequential, then you may consider using this type of approach here:
FILE = fullfile('C:\Users\...', '*.csv'); % Directory where the files are residing.
LIST = dir(FILE);
N = length(LIST); % N = number of files
D = zeros(???, ???, N); % Memory allocation to speed up the process of data storing
for ii = 1 : N
FFName = fullfile(LIST(ii).folder, LIST(ii).name);
DATA = readmatrix(FFName);
D(:, :, ii) = DATA; % NOW D contains all data from 10000 files
end
... % Select and process the part of D that is necessary
An alternative way might be this one:
P=pwd; % If your files are located in the current directory
P = fullfile('C:\...\...', '*.csv'); % Specify: directory where the files are residing.
S = dir(fullfile(P, '*.csv')); % Select the file extension to suit your data files
M = 0;
for k = 1:numel(S)
F = fullfile(P, S(k).name);
D = D + readmatrix(F);
end
...
2 Commenti
Sulaymon Eshkabilov
il 22 Giu 2022
readmatrix() and readtable() are recommended ones due to their efficiency instead of xlsread() or csvread(), etc.
Più risposte (0)
Vedere anche
Categorie
Scopri di più su Dialog Boxes in Help Center e File Exchange
Prodotti
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!