Do the boxplot stats without boxplot

Dear all, I have an enormous number of numbers and I want to plot the box plot

Actually I have 7 sets of 125.000.000 numbers (luckily I am running this in a system with huge ram). As boxplot with 7 inputs takes so much time I was thinking if there might be a way to calculate for every set all the statistics boxplot calculates.. and then feed them in a boxplot_type function to do the stupid plotting.

Is there something like that in matlab?

I would like to thank youf or your help

Best Regards Alex

Answer 1

[Oleg Komarov]

Time boxplot:

tic 
boxplot(...)
toc

against:

tic
prctile(...)
max(...)
min(...)
toc

and also against:

tic
sort()
toc

for one of your datasets, if there's significant gain for one of the alternatives then yes you could improve boxplotting...but I doubt it.

EDIT

Avoiding to call boxplot on the big database I create a fake boxplot and adjust it with the stats calculated from the real database:

% Suppose truedata is your dataset A

truedata = rand(1e6 + 123423,7);
sz       = size(truedata);

% Create fakedata and boxplot it

fakedata = rand(10,sz(2));
h = boxplot(fakedata,'labels',10:10:10*sz(2));

% Now sort your truedata and calculate min,max,25,50,75 percentile

truedata = sort(truedata);
s.mins   = truedata(1,:);
s.maxs   = truedata(end,:);
xi       = bsxfun(@plus,sz(1).*[0.25; 0.5; 0.75], sz(1) * (0:sz(2)-1));
x        = [floor(xi(:)); ceil(xi(:))];
s.ptiles = reshape(interp1(x,truedata(x),xi(:)),3,sz(2));

% Readapt the fake boxplot:

% 1.Adjust upper whisker

set(h(1,:),{'Ydata'}, num2cell([s.ptiles(3,:); s.maxs].',2));
set(h(3,:),{'Ydata'}, num2cell(repmat(s.maxs.',1,2),2));

% 2. Adjust lower whisker

set(h(2,:),{'Ydata'}, num2cell([s.mins; s.ptiles(1,:);].',2));
set(h(4,:),{'Ydata'}, num2cell(repmat(s.mins.',1,2),2));

% 3. Adjust body and median

set(h(5,:),{'Ydata'}, num2cell(s.ptiles([1 3 3 1 1],:).',2));
set(h(6,:),{'Ydata'}, num2cell(repmat(s.ptiles(2,:).',1,2),2));

% 4. Delete outlier marking

delete(h(end,:))

[Alex]

Thanks a lot I will check and I will report back

[Alex]

Thanks a lot. I have notirced something really strange.

After doing sort in the A matrix... boxplot returns much faster back.

Could you please explain why is this behaviour? For me it is something so strange..

[Oleg Komarov]

I don't observe the same behaviour. What code did you call?

[Alex]

size(A)

ans =

131072000 7

A=sort(A);

boxplot(A,'symbol','','labels',[1.14 3.43 5.72 8.01 10.3 12.6 14.9]);

set(gca,'YScale','log');

this takes 30 mins and plots. If I do not sort the data I do not get results

[Oleg Komarov]

It's possible that the internal sort of boxplot "recognizes" that the array is already sorted and do not call othe internal routines or it may be that the subfunction don't end modifying A and copy is created keeping memory consumpution at a "reasonable" level.

Anyways, the fastest method should be the one I indicated in my edit. Consider that you can create the fake boxplot with all the features you want and adjust it later keeping all of those. Right now I am eliminating the red outliers but it's very simple to keep them and update.

[Alex]

just to make it more "formal"

tic;

A=sort(A);

boxplot(A,'symbol','','labels',[1.14 3.43 5.72 8.01 10.3 12.6 14.9]);

set(gca,'YScale','log');

toc;

Elapsed time is 1123.973379 seconds.

I comment out the sort

tic;

%A=sort(A);

boxplot(A,'symbol','','labels',[1.14 3.43 5.72 8.01 10.3 12.6 14.9]);

set(gca,'YScale','log');

toc;

Elapsed time is 1502.582405 seconds.

why it does more time in the second casE?

[Oleg Komarov]

Use the profiler to check the differences.

Have you tried my other solution?

Answer 2

[Alex]

Hmm... what I tried is the following: I only did boxplot for the first set only 1*125.000.000 sets and matlab replied after 15 minutes. Then I tried boxplot with the 7 sets (7*125.000.000 sets) which unfortunately has not retuned after 3 hours..

So far matlab consumed 110 Gb of memory and keeps going on....

If there was a way to do 7*boxplots(for only one set) and then put together I think I will be finished by now. Is there any way to do that?

B.R Alex

[Oleg Komarov]

You said 1*125.000.000 sets, what do you mean?

Sorting 1e8 elements takes 11 seconds on my laptop, after that most of the calculations can be done in instants. I really don't see how boxplot is taking som much time.

Please post the code you're using.

Answer 3

[Alex]

I wouldl ike to thank you for your answer

 size(A)

ans =

   131072000           7

so I have a matrix of 131072000*7 and I want a box plot

this boxplot(A','symbol','');

will crash after 6 hours after consumerd 300GB (three hundred gigabytes of ram!)

[Oleg Komarov]

Please don't create additional answers but use the comment facility (I know it doesn't allow code formatting but it keeps the logical track of question-answer).

Have you tried to

A = sort(A);

If you can sort it in a reasonable amount of time then we could find a workaround. You can also try to sort column by column (to break the memory requirements:

for n = 1:size(A,2)

A(:,n) = sort(A(:,n));

end

It takes so much time because boxplot uses uniquerows and unique. What happens is that it fills up all of your memory (300 GB who do you work for?) and starts swapping on hard disk.

[Alex]

(I hope this time I did reply correct!)

tic; test=sort(A); toc

Elapsed time is 35.341047 seconds.

As you can see that was extremely fast. There is no more swap space and the system administrator is not here to ask him to add more!

One solution is to try to do 7*different Box plots and then with some magic way to make them one. As you can read below , yesterday I have explained what I have tried but all the boxplots get stuck the one over it.

B.R

Alex

[Oleg Komarov]

Give a try to my EDIT in my answer.

Answer 4

[Alex]

I then tried the following I wrote

>> boxplot(A(:,1)','symbol','','labels',[10]); hold on; boxplot(A(:,2)','symbol','','labels',[20]); boxplot(A(:,3)','symbol','','labels',[30]); boxplot(A(:,4)','symbol','','labels',[40]); boxplot(A(:,5)','symbol','','labels',[50]); boxplot(A(:,6)','symbol','','labels',[60]); boxplot(A(:,7)','symbol','','labels',[70]); hold off;

which makes it run but unfortunately all the 7 boxplots are plotted the one over the other. As you might have guessed already these 7 sets are for the 10,20,30,40,50,60,70 cases.

I would like to thank you for your answers

Best Regards Alex

[Oleg Komarov]

Please see the edit on my answer and stop creating additional answers, use comments.