Use of Statistical Data in Social Science
Jost dedicates a small section of his article to discuss psychological differences between liberals and conservatives. His hypothesis is that there are significant differences of cognitive and motivational style between liberals and conservatives. The nine variables he uses include fear of death/mortality salience, system instability/threat, dogmatism/ambiguity intolerance, openness to experience, uncertainty avoidance, needs for order/structure/closure, integrative complexity, fear of threat and loss, and self-esteem. Jost claims that there is a “clear tendency for conservatives to score higher on measures of dogmatism, intolerance of ambiguity, need for order, structure, and closure and to be lower in openness to experience and integrative complexity than moderates and liberals.” While Jost seems very adamant about his claims, I am not quite convinced.
A main issue is that statistical significance does not necessarily reflect practical significance. According to Jost, “all effect sizes are statistically reliable, p < .001.” This means that there are statistically significant differences of the variables between the samples. In the referred study, however, the highest effect size listed in Table 2.2 is .50. According to Cohen’s guideline for t-Test, this size represents a medium effect at best. In case of self-esteem, the effect size is -.09, and this number represents small effect. These numbers represent that the significance of the differences among variables is not big in a practical sense. Although it is quite common that effect sizes of many educational studies are in the small to medium range, the writer is obliged to carefully state the meanings of the numbers. Jost, however, seems to be busy solidifying his claims.
Assuming that his study was carefully conducted, I agree that the results may convey some meaning. I actually would have completely believed his claim if I had not seen the appendix. In fact, Jost’s idea is complex enough that many readers may be too exhausted to scrutinize his claims before finishing a few pages. But, at this point, I am afraid to say that I am skeptical about the validity and reliability of his study as well as his claim about psychological differences between two different types of people. I really need to see the details of the studies mentioned in the article.