Discrepancy between anovan and fitlme for interactions

I have run into an issue where anonvan and fitlme give different answers but only for models with interactions.

Here is a minimal example:

% some fake data
X = [1 1 1 1 0 0 0 0]'; % label 1
Y = [1 0 1 0 1 0 1 0]'; % label 2
Z = [
    -1.1719
    1.9764
    0.0093
    1.1649
    0.7713
   -0.0083
    0.3148
   -0.6221]; % data

%% model with interactions - discrepancy
% statistics for main effects are different
% statistics for interaction are identical
[P,T,STATS,TERMS] = anovan(Z, [X Y], 'model', 'interaction', 'varnames', {'X', 'Y'});

tbl = table(Z, X, Y, 'VariableNames',{'Z', 'X', 'Y'} );
mdl = fitlme(tbl,'Z ~ X*Y', 'FitMethod','reml');
anova(mdl,'dfmethod','satterthwaite')

ans = 
    ANOVA marginal tests: DFMethod = 'Satterthwaite'

    Term                   FStat      DF1    DF2    pValue  
    {'(Intercept)'}        0.60238    1      4       0.48101
    {'X'          }         10.782    1      4        0.0304
    {'Y'          }          2.233    1      4        0.2094
    {'X:Y'        }         13.735    1      4      0.020725

Note how p value and fStat are the same for the interaction term (up to rounding error) but different for the main effects terms.

This difference goes away when I remove the interaction term:

%% model without interactions - no discrepancy
% all statistics identical
[P,T,STATS,TERMS] = anovan(Z, [X Y],'varnames', {'X', 'Y'});

tbl = table(Z, X, Y, 'VariableNames',{'Z', 'X', 'Y'} );
mdl = fitlme(tbl,'Z ~ X + Y', 'FitMethod','reml');
anova(mdl, 'dfmethod','satterthwaite')

ans = 
    ANOVA marginal tests: DFMethod = 'Satterthwaite'

    Term                   FStat      DF1    DF2    pValue 
    {'(Intercept)'}        0.43595    1      5      0.53828
    {'X'          }        0.24781    1      5      0.63974
    {'Y'          }        0.71523    1      5      0.43631

Note: I used the Satterthwaite dfmethod and reml FitMethod as described in a previous post on a similar issue (https://www.mathworks.com/matlabcentral/answers/247286-discrepancy-between-anova-and-fitlme).

Answer 1

[Sanjana]

Hi Robert,

I understand you are facing an issue with “annovan” and “fitlme” functions which result in different answers when used for models with interactions.

One reason for this is that they estimate different types of model parameters. “ANOVAN” treats all factors (both continuous and categorical) as fixed-effects parameters and estimates the marginal means of each factor level. If the model includes interactions, “ANOVAN” will also estimate the marginal means for each combination of factor levels.

On the other hand,” fitlme“ treats categorical factors as fixed-effects parameters and continuous factors as random-effects parameters. If the model includes interactions, “fitlme” will estimate and test the interaction between the fixed and random effects and will estimate the variance of the random effects.

Therefore, the two methods are not measuring the same thing, and it's possible that they will give different results. One way to reconcile the results is to make sure that the model structure and the methods used are identical across “ANOVAN” and “fitlme”.

Please refer to the following links, for further information,

https://www.mathworks.com/help/stats/anovan.html

https://www.mathworks.com/help/stats/fitlme.html

Hope this helps!