How can I test for significance between expected and observed outcomes?

5 views (last 30 days)
Josie B
Josie B on 29 Jul 2021
Answered: Jeff Miller on 30 Jul 2021
Hi, I'm working on a project where we are validating a psychometric self-reported tool on personality traits and comparing to other personality measures (self-report and other). Myself and colleagues have identifies which items we expect to correlate on each tool by indicating that a correlation is expected and its direction.
I have run the correlations and now have data to compare what we expected to correlate vs. the actual outcomes.
We would like to run some significance testing to see whether the hits and misses are better than chance. We thought of using a chi-squared, but all the literature I have seen using this method have given actual values for their expected correlations (for example, we expect a correlation of .33 and the observed correlation was .56). We don't have the values for expected outcomes - is there a way we can still carry out a significance test?
I'd be grateful for any advice.

Answers (2)

Scott MacKenzie
Scott MacKenzie on 29 Jul 2021
Edited: Scott MacKenzie on 29 Jul 2021
Since there were only two conditions, you can use a t-test (i.e., an anova isn't needed). Here's an example using test random data. Not surprisingly, the difference between the two conditions was not statistically significant, t(18) = 0.395, p > .05.
data = rand(10,2);
[h, p, ci, stats] = ttest2(data(:,1), data(:,2))
h = 0
p = 0.6975
ci = 2×1
-0.1782 0.2608
stats = struct with fields:
tstat: 0.3950 df: 18 sd: 0.2336

Jeff Miller
Jeff Miller on 30 Jul 2021
Another alternative: You might consider that you have three classes of items: those for which the predicted correlation was positive, zero, or negative. You have observed correlations for each of those items. So, you could use ANOVA to compare the mean observed correlations for the three classes of items, expected that (e.g.) the mean correlation would be larger for those predicted to have a positive correlation than for those predicted to have a negative correlation.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by