Computation of the distance of the points from the interpolation curve?

Hi everybody..

I am trying to find the distance of points from interpolation curve

dist = pdist([flowrate y], 'euclidean');

but I have the following error:

Error using internal.stats.pdistmex

Requested 1x123420272865 (919.6GB) array exceeds maximum array

size preference (15.9GB). This might cause MATLAB to become

unresponsive.

The function is trying to create an array that is larger than the maximum array size preference set in MATLAB. The maximum array size preference is typically set to 15.9GB in MATLAB. When an array exceeds this limit, MATLAB may run out of memory and become unresponsive.

What can I do to fix this error?

For completeness I put the code who gave me kindly as hint @Mathieu NOE whom I am really grateful.

flowdata = readtable("flodata.txt");
% column 1: pressure at pipe start [bars]
% column 2: acoustic noise  at start [standard deviation]
% column 3: pressure at pipe end (10km from start) [bars]
% column 4: acoustic noise  at pipe end[standard deviation]
% column 5: flow rate [m3/hour]
load('flowdata.mat');

% summary(flowdata)
% Vectors of Pressures
pressureIN = flowdata.pressureAtPipeStart_bars_;
pressureOUT= flowdata.pressureAtPipeEnd_10kmFromStart__bars_;
% Vectors of Acoustic Noises at IN and at OUT
aNoiseIN=flowdata.acousticNoiseAtStart_standardDeviation_;
aNoiseOUT=flowdata.acousticNoiseAtPipeEnd_standardDeviation_;
% Vector of flowrate
flowrate = flowdata.flowRate_m3_hour_;
% % Table of flowRate
% flowRate = table(flowrate);
% rawData = [pressureIN aNoiseIN pressureOUT aNoiseOUT flowrate];
% t = (linspace(0, length(cleanData)*10, length(cleanData)))/86400;
sample_distance = 10;
%t = (linspace(0, 7948790, 794879))/86400;
[samples,channels] = size(flowdata);

t = linspace(0, samples*sample_distance, samples)/86400;

% all main curves overlaid 
figure()
plot(t, pressureIN, '-g',t, pressureOUT, '-r',t, flowrate, '-b');
axis tight;
title(' Input , Output Pressure , FR vs time');
legend('Input Pressure','Output Pressure','Flow Rate');
xlabel('Tempo [Days]');
ylabel('all');

PHeadLoss = pressureIN-pressureOUT;
densityplot(flowrate, PHeadLoss, [50,50])

% Prima considerazione
% large variations on the output pressure can be seen at days 1 to 15 , 
% not related to input pression
% we should not use that portion of the data 
% It's a brutal cleaning but... 
% I am sceptical about what happens during the first 15 days : 
% lots of peaks are seen on the output pressure side that are 
% not reflected / correlated to the input side , 
% so I prefer to discard the first 20 days (yes it's brutal !)
ind = t>20;
t = t(ind);
pressureIN = pressureIN(ind);
pressureOUT = pressureOUT(ind);
flowrate = flowrate(ind);

figure()
plot(t, pressureIN, '-g',t, pressureOUT, '-r',t, flowrate, '-b');
axis tight;
title(' Input , Output Pressure , FR vs time');
legend('Input Pressure','Output Pressure','Flow Rate');
xlabel('Tempo [Days]');
ylabel('all');

% seconda considerazione
% keep data (valid) for gradient below a certain threshold (and non zero
% FR too btw)
% i also don't like those sudden drops to zero , 
% because to me its like dividing zero by zero , 
% when you want to make a relationship between two variables ,
% I avoid using background noise data , it will blurr the result.
% This function gradient calculates the gradient of a signal, 
% which is the rate of change of the signal over time.
pressureIN_grad = abs(gradient(pressureIN));
pressureOUT_grad = abs(gradient(pressureOUT));
flowrate_grad = abs(gradient(flowrate));

threshold = 0.001; 
ind1 = pressureIN_grad<threshold*max(pressureIN_grad);
ind2 = pressureOUT_grad<threshold*max(pressureOUT_grad);
ind3 = flowrate_grad<threshold*max(flowrate_grad);
ind4 =  flowrate> 0; 

ind = ind1 & ind2 & ind3  & ind4;
t = t(ind);
pressureIN = pressureIN(ind);
pressureOUT = pressureOUT(ind);
flowrate = flowrate(ind);

figure()
plot(t, pressureIN, '.g',t, pressureOUT, '.r',t, flowrate, '.b');
axis tight;
title(' Input , Output Pressure , FR vs time');
legend('Input Pressure','Output Pressure','Flow Rate');
xlabel('Tempo [Days]');
ylabel('all');
PHeadLoss = pressureIN - pressureOUT;

densityplot(flowrate, PHeadLoss, [50,50])

% terza considerazione
% PRESSURE HEAD LOSS (APPROX _no fattore di attrito, flusso del fluido e
% densità del fluido)
PHeadLoss = pressureIN - pressureOUT;

% consider only positive values 
ind =  PHeadLoss> 0; 
t = t(ind);
pressureIN = pressureIN(ind);
pressureOUT = pressureOUT(ind);
flowrate = flowrate(ind);
PHeadLoss = pressureIN - pressureOUT;

densityplot(flowrate, PHeadLoss, [50,50])


figure('Name',' PRESSURE HEAD LOSS vs FLOWRATE')
plot(PHeadLoss, flowrate, '*k');
title('Pressure Head Loss vs FLOW RATE');
xlabel('PHeadLoss');
ylabel('flow rate');

%DATA
figure('Name',' PRESSURE HEAD LOSS vs FLOWRATE')
plot(flowrate, PHeadLoss, '*k');
title('Pressure Head Loss vs FLOW RATE');
xlabel('flow rate');
ylabel('P Head Loss');
hold on
%densityplot(flowrate, PHeadLoss, [50,50]);

[p, S, mu] = polyfit(flowrate, PHeadLoss, 2)

[y,delta] = polyval(p,flowrate,S,mu);

%LINEAR FIT
plot(flowrate, y, 'r-');
%hold on

% Linear Fit of Data with 95% Prediction Interval
%plot(flowrate,y+2*delta,'m--',flowrate,y-2*delta,'m--')

title('Pressure Head Loss vs FLOW RATE');
xlabel('flow rate');
ylabel('P Head Loss');

densityplot(flowrate, y, [50,50]);

dist = pdist([flowrate y], 'euclidean');

Answer 1

[Matt J]

dist = pdist([flowrate y], 'euclidean','Smallest',1);

[Giuseppe Zumbo]

@Matt J Hi, thanks for the answer.

Unluckily also adding this parameter I get the same error:

Error using internal.stats.pdistmex

Requested 1x123420272865 (919.6GB) array exceeds maximum array

size preference (15.9GB). This might cause MATLAB to become

unresponsive.

Any ideas to fix it?

[Matt J]

We do not have the input data files referenced by your code (so we cannot run it).