Why does corr function result depend on the number of columns ?

Hello everyone,

I want to calculate the correlation coefficient between several physical parameters for astrophysical purpose (with Matlab R2019b). Since I have a number of different parameters, instead of using corr(A,B) for each pair of parameters (A,B), I created a matrix whose columns correspond to all the parameters of interest and calculated corr(X,X) to get the correlation matrix and thus look at the non diagonal terms to get the pairwise correlation coefficients. However, I am surprised that the correlation coefficient for the same pair of parameters (A,B) varies depending on the other parameters (columns) I include in the matrix. As far as I understand corr(X,Y) calculates the pairwise correlation coefficient, which should depend only on the pair of columns considered right ?

Thank you in advance if someone can explain if I misunderstood the use of the corr function !

Philippe

Answer 1

[Adam Danz]

Understanding the output of rho=corr(x)

r = corr(x,x);

is the same as

r = corr(x);

If a single value is changed in x in column n, it should affect all of the correlation matrix results in row n and column n.

For example,

% x is an nx3 matrix
% r = corr(x)

% r shows the correlation between 
% the following column pairs:
r =        
      1 & 1     1 & 1     1 & 3
      2 & 1     2 & 2     2 & 3
      3 & 1     3 & 2     3 & 3

If a value changes in column 3, you can see above that it would affect all values in column 3 and row 3 of the correlation matrix.

Here's a demo

x0 = [1 6 5; 9 3 5; 7 5 3; 5 9 5];
x1 = x0;
x1(10) = 9;

NaN infestation

As explained in this answer, a single NaN value in the input matrix of r=corr(x) at x(i,j) will result in all NaN values in row i and column j of the output matrix.

A single NaN value in one of the two matrices x or y of r=corr(x,y) at coordinate (i,j) will result in a column of NaN values in column j of the output matrix but row i will otherwise be OK.

Ignoring missing values (e.g. NaN).

As explained in this answer, to compute column-wise correlation while ignoring missing values, set the 'Rows' property to either 'complete' or 'pairwise'.

[Philippe Garnier]

Hello,

Thank you very much for the answer. But actually, the problem in my case is, with the example given above, that from corr(x0) to corr(x1) the r(1,2) or r(2,1) cell ALSO changes, e.g. from -0.52699 to -0.49. After more investigation, I wonder if the origin is not the presence of NaNs. More precisely, I used instead of corr(x,x) or corr(x): corr(x,x,'Rows','Complete','Tail','Both'), with x being modified with more or less columns (whose number of NaNs varies). I guess the command I used removed part of the columns n-i if I added a column n with more NaNs. If I use corr(x) the problem indeed disappears, I get coherent results from a matrix x to another (but I get NaNs as a result if NaNs exist inside x).

Thank you for your help !

Philippe

[Adam Danz]

Are you saying that when you execute

x0 = [1 6 5; 9 3 5; 7 5 3; 5 9 5]; 
corr(x0)

x1 = [1 6 5; 9 3 9; 7 5 3; 5 9 5]; 
corr(x1)

you get different results from those shown in my answer? If so, you may be shadowing the corr function. To confirm,

which corr -all

Perhaps you could demonstrate the problem by showing me and inputs, outputs, and full line of code.

If there are questions about NaNs in the output, this answer may help.

[Philippe Garnier]

Hi, thanks a lot for the answers. There you can find your example modified with my situation :

x0 = [1 6 5; 9 3 5; 7 5 3; 5 9 5];
x1 = [1 6 5; 9 3 NaN; 7 5 3; 5 9 5];
corr(x0)
corr(x1)
corr(x1,x1,'Rows','Complete','Tail','Both')

I would at first glance assume that the two last lines of code (I used until now the last line) would give the same result for the correlation values of the two first columns, but this is not the case. But overall, as soon as I understand where it comes from, I can cope with that easily. The problem came from the fact that I changed arbitrarily the columns of the matrix to investigate various parameters, including sometimes variables with more or less NaNs, that would impact the other correlations since only the rows with no NaN for all columns were selected. The lines used in the correlation calculations were thus not always the same for a specific column, making the results a bit puzzling :-) But now I know !

Thanks a lot

[Adam Danz]

"the problem in my case is, with the example given above, that from corr(x0) to corr(x1) the r(1,2) or r(2,1) cell ALSO changes, e.g. from -0.52699 to -0.49."

Then you must either be using a different function named corr(), the inputs are incorrectly copied, or you're not using the same syntax.

"After more investigation, I wonder if the origin is not the presence of NaNs.

Yes, NaN values will affect the outputs in the same way as my answer shows.

I just updated this answer to explain what you're seeing this.

For example, the outputs below are completely expected.

x0 = [1 6 5; 9 3 5; 7 5 3; 5 9 5];
x1 = x0; 
x1(10) = NaN; 

corr(x0)
ans =
            1     -0.52699     -0.29277
     -0.52699            1          0.2
     -0.29277          0.2            1
     
corr(x1)
ans =
            1     -0.52699          NaN
     -0.52699            1          NaN
          NaN          NaN          NaN

" I used instead of corr(x,x) or corr(x): corr(x,x,'Rows','Complete','Tail','Both')"

With that syntax, The 'Tail','Both' property is the default property so that doesn't change anything. However, using 'Rows','Complete', instead of the default 'all'results in the removal of rows with missing values (ie NaN) which will change all of the correlation values!

See the link to my other answer above. If you have any other questions, please let me know.

[Philippe Garnier]

Thanks a lot, everything is clear now !

[Adam Danz]

Glad it was useful!

[Ilaria Sani]

Thanks for the above explanations.

I have a follow up question.

What if I add an entire new column?

What I see is that the correlation of untouched columns changes. Is that possible? Is there a correction for number of comarisons?

Thank you so much for your kind reply.

Ilaria

[Adam Danz]

>... the correlation of untouched columns changes...

That shouldn't be the case. Consider this demo below where x1 is the same as x0 except for the addition of a 4th column. The corr results only differ in row 4 and column 4.

x0 = [1 6 5; 9 3 5; 7 5 3; 5 9 5]

x0 = 4×3

1 6 5 9 3 5 7 5 3 5 9 5

corr(x0)

ans = 3×3

1.0000 -0.5270 -0.2928 -0.5270 1.0000 0.2000 -0.2928 0.2000 1.0000

x1 = [x0,[2;3;4;1]]

x1 = 4×4

1 6 5 2 9 3 5 3 7 5 3 4 5 9 5 1

corr(x1)

ans = 4×4

1.0000 -0.5270 -0.2928 0.5292 -0.5270 1.0000 0.2000 -0.7746 -0.2928 0.2000 1.0000 -0.7746 0.5292 -0.7746 -0.7746 1.0000