How would I manipulate the following functions to use a tall array?
5 visualizzazioni (ultimi 30 giorni)
Mostra commenti meno recenti
Hello,
I'm having difficulty figuring out how to turn these functions to utilize tall arrays. The only examples online about using tall arrays I've found so far have been using NYC taxi data as examples, which doesn't help me at all. I can't find examples that go beyond multiplying two columns togethor, or getting the mean of a column. I'm working with a single column, no header array of data. I essentially need to apply of series of transformations on the data.
Any help is appreciated, thanks.
% Say both files are 6,000,000,000 (6e9) data points long, representing a wave.
% I need to flatten with a linear fit every 400,000, then gather variance every 100,000
Dir = 'C:/';
Data{1} = 'File1';
Data{2} = 'File2';
for filestoread = 1:length(Data);
PathFile = fullfile(Dir, Data{filestoread};
fidi = fopen(PathFile);
while ~feof(fidi)
yraw = fscanf(fidi, '%f\n', 1200000);
yflat = detrend(reshape(yraw, 400000, []));
yflat = yflat(:);
variance = var(reshape(yflat, 100000, []));
dlmwrite(varASCII, transpose(variance), 'delimiter', '\t', '-append', 'precision', 16);
end
end
0 Commenti
Risposte (1)
Josh Meyer
il 3 Apr 2020
First you need to make a datastore that references the underlying files for each array (one datastore per array). Then you can convert the datastores into tall arrays with T = tall(ds).
While the movvar function (moving variance) supports tall arrays, the detrend function does not (the bottom of each doc page has an Extended Capabilities section that says whether it supports tall arrays). So to detrend the data, you would probably have to use the package function matlab.tall.movingWindow to operate on the tall array with a window function you write that does the detrending.
0 Commenti
Vedere anche
Categorie
Scopri di più su Tall Arrays in Help Center e File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!