Out of memory error for big data

3 visualizzazioni (ultimi 30 giorni)
Sharif Khalil
Sharif Khalil il 29 Dic 2020
Modificato: Sharif Khalil il 31 Dic 2020
The masterDataMatrxOffset, contains 464780 rows and 256 columns of complex numbers. I use meshgrid to get 256X256 matrix of the real and imaginary and store it in a cell, but I get this error: Is there a way to overcome the out of memory?
Out of memory.
Error in meshgrid (line 62)
yy = repmat(ycol,size(xrow));
Error in auto_encoder (line 34)
[Po,Ph] = meshgrid(power(obsrv,:),phase(obsrv,:));
More information
tic;
clear; clc; close all;
%% Load Files
load masterDataMatrxOffset;
%% Extract Parameters
iAntenna = size(masterData,2) - 16;
data = masterData(:,1:iAntenna);
power = normalize(real(data),'range');
phase = normalize(imag(data),'range');
Region = masterData(:,261);
iReceiver = length(data);
iRegion = max(Region);
%% Initializations
xTrn = cell(1,max(Region));
xTst = cell(1,max(Region));
datamat = cell(1,size(data,1));
%% Split into training and testing
for idx = 1:iRegion
IDX = find(Region==idx);
train = sort(randperm(size(IDX,1),ceil(0.8*size(IDX,1))));
trnidx = IDX(train);
tstidx = setdiff(IDX,trnidx);
xTrn{idx} = trnidx;
xTst{idx} = tstidx;
end
xTrnidx = cell2mat(xTrn');
xTstidx = cell2mat(xTst');
Trainreg = zeros(numel(Region), max(Region));
Trainreg(sub2ind(size(Trainreg), 1:numel(Region), Region')) = 1;
tTrain = Trainreg';
for obsrv = 1:size(power,1)
sprintf('observation = %d',obsrv);
[Po,Ph] = meshgrid(power(obsrv,:),phase(obsrv,:));
datamat{obsrv} = Po.^2 + Ph.^2;
end
toc;
  4 Commenti
Ive J
Ive J il 29 Dic 2020
Modificato: Ive J il 29 Dic 2020
You still can store them in a cell array
myCell = {};
myCell{1} = tall(1:100); myCell{2} = tall(101:200);
myCell{3} = myCell{1}.^myCell{2};
myCell =
1×3 cell array
{1×100 tall} {1×100 tall} {1×100 tall}
Sharif Khalil
Sharif Khalil il 31 Dic 2020
Modificato: Sharif Khalil il 31 Dic 2020
It still shows out of memory. The cell is 1X464780, but the matrix inside each cell is 256X256.
for obsrv = 1:size(power,1)
[Po,Ph] = meshgrid(power(obsrv,:),phase(obsrv,:));
Talldata = tall(Po.^2 + Ph.^2);
datamat{obsrv} = Talldata;
end

Accedi per commentare.

Risposte (0)

Categorie

Scopri di più su Large Files and Big Data in Help Center e File Exchange

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by