October, 29 — Regular Seminar
Topic: “Measurement invariance of liberal and authoritarian notions of democracy: Evidence from the World Values Survey and additional methodological considerations”
Speaker: Boris Sokolov (Senior Research Fellow, LCSR)
The Laboratory for Comparative Social Research announces the next regular seminar, which will be held as a zoom session on October, 29 at 16-30 p.m. (GMT+3). Boris Sokolov (Senior Research Fellow, LCSR) will deliver a report “Measurement invariance of liberal and authoritarian notions of democracy: Evidence from the World Values Survey and additional methodological considerations”.
A link to zoom session is available upon request (firstname.lastname@example.org)
Mass support for democracy is often considered as a key prerequisite for democratic persistence (if it exists in an established democracy) or successful democratic transit (if it exists in an autocratic regime). It is however puzzling that even in some countries severely lacking democratic institutions as high as up to 90% of population express support for democracy. To explain this contradiction, Kirsch and Welzel (2018) recently proposed that people in authoritarian countries often confuse a conventional notion of democracy (which they called a Liberal Notion of Democracy, or LND), composed of such elements as free elections, protection of civil rights, and gender equality, with local meanings of the term, contaminated with such arguably non-democratic features as obedience to the rulers, military rule, or superiority of religious authorities (the latter understanding of democracy can be denoted as an Authoritarian Notion of Democracy, or AND).
Using data from the 6th wave of the World Values Survey, Kirsch and Welzel managed to show that it is possible to identify two factors corresponding to these contradicting conceptions of democracy, and that these factors can be related to other relevant political variables, such as emancipative values and the level of democracy. Importantly, Kirsch and Welzel used pooled-data factor analysis to validate their measures but didn’t do much to explore whether these two constructs have similar meanings in different countries, i.e., that they are comparable across WVS countries. The comparability/invariance issue is nevertheless acknowledged by methodologists as being of primary importance in cross-national research.
In this paper I conduct (using the same data) a series of comparability checks for these two scales. I first perform conventional invariance tests of the LND and AND indices using MGCFA. My results suggest that the former index can be considered as roughly invariant across WVS countries. It fails to pass strict tests for metric and especially scalar invariance across all 60 countries of the 6th wave, but after removing a few countries or applying a more liberal Bayesian approximate approach it is possible to achieve partial scalar invariance of the LND index. However, the latter index, AND, do not satisfy even the most general requirement of configural invariance, since factor loadings of its three indicators massively vary cross-nationally in their size, significance and even direction.
Does this suggest that only the first construct out of the two may be used for substantive comparative analysis? It may seem so from the dominating methodological perspective in the literature on cross—national survey research. Still, I argue that comparative validity of the AND index can be justified, yet on grounds different to the LDN index.
Following the recent discussion about measurement validity of the Index of Emancipative Values (Aleman and Woods 2016, Inglehart and Welzel 2016, Sokolov 2018), I conduct additional checks to test whether the apparent problems with invariance of the AND index can be resolved when adjusting for three “misconceptions” of measurement invariance formulated by Inglehart and Welzel: (a) ignoring that the primary level of operation for WVS constructs is culture, not individuals, (b) ignoring formative nature of such constructs, and (c) ignoring strong external linkages of such constructs.
First, following recommendations of Stapleton et al (2016), I show that, while in contrast to the LND, the AND can be, to some extent, considered as a nation-level construct (up to 15-20% of the variation of its indicators can be attributed to the nation-level). Then, using multilevel CFA, I test cross-level invariance of these constructs, finding that for the LND, its construct meaning does not differ between the individual and the national level, which is not true for the AND index. This finding further suggests the possibility that the latter index ihas a separate meaning as a culture-level measure.
Second, I apply four criteria proposed in Jervis et al. 2003 to decide whether the LND and the AND are reflective or formative constructs. My results show that both constructs can be defined as formative ones, though the former one, LND, fits empirically the definition of a reflective construct as well. Still, the AND does not fit measurement invariance requirements for formative constructs, proposed by Diamantopoulos and Papadopoulos. (2010). Nevertheless, I argue that these requirements are themselves controversial since they contradict some existing guidelines for indicator selection for formative constructs, which suggest that insignificant indicators can be used as far as they are believed to be theoretically important. The key point here is that there are no strong theoretical reasons to expect that different people in different societies should misinterpret the term “democracy” in strictly the same way under any circumstances. Paraphrasing Leo Tolstoy, it can be claimed that all democratic people (and countries) have the same understanding of what democracy is, but all non-democratic peoples (and countries) may have their own (mis)concept of democracy. In other words, when measuring misunderstanding we should take into account any deviation from the absolute zero benchmark (correct rejection of some feature as a non-democratic one).
Finally, I investigate nomological validity of the LND and AND indices, or how well are they related to their expected theoretical antecedents and consequences. These relationships are indeed strong, while for the AND they exist mostly on the national level (which resonates well with my previous observations for it). I nevertheless suggest that the AND index may be redefined in a more efficient manner, without the obedience item, which does not actually add predictive power but in the same time is too much ambiguous from the theoretical point of view (e.g., it is hard to imagine that democracies can be successful when their citizens do not obey to their rulers).
The overall conclusion of the study is that WVS-based measurements of LND and AND are each methodologically valid and generally comparable cross-nationally, though the respective grounds of their comparability are different. Besides, this paper not only performs standard invariance-testing approaches, such as full and approximate invariance using MGCFA, with respect to novel constructs, but also highlights additional issues in comparability analysis and provides several guidelines on how to deal with those issues.