Subjective data in political science research: from expert evaluation to artificial intelligence

Empirical research in Comparative Politics and International Relations is often built not only on statistical data, but also on expert evaluation data. However, the methods of data analysis employed in this case often fail to account for the differences between statistical and expert evaluation data, and disregard the extra uncertainty in the latter. This article focuses the state-of-the-art methods for collecting and processing expert evaluation data in political science research, as well as open questions in this area. The article presents Bayesian data analysis as the most natural approach to analyzing subjective data and focuses on the differences between Bayesian and classical approaches. Then the article focuses on the methods for obtaining expert evaluations through prior elicitation for further use in Bayesian analysis. These approaches are illustrated using examples from the research project “Political Atlas of the Modern World 2.0”. The next section discusses the possibility of replacing expert evaluation data with crowdcoding, i.e. the procedures for annotating or coding qualitative features by non-experts based on formalized instructions. The article cites both successful examples of crowdcoding usage in empirical research and potential challenges for its integration into research in Comparative Politics and International Relations. Finally, the author addresses the issues of integrating expert evaluation data, on the one hand, and artificial intelligence and machine learning technologies, on the other. We highlight their compatibility in the framework of Bayesian data analysis.

1. Athey S., Imbens G.W. Machine learning methods that economists should know about. Annual review of economics. 2019, Vol. 11, P. 685–725. DOI: https://doi.org/10.1146/annurev-economics-080217-053433

2. Bansak K. Can nonexperts really emulate statistical learning methods? A comment on “The accuracy, fairness, and limits of predicting recidivism.” Political analysis. 2019, Vol. 27, N 3, P. 370–380. DOI: https://doi.org/10.1017/pan.2018.55

3. Benoit K., Conway D., Lauderdale B.E., Laver M., Mikhaylov S. Crowd-sourced text analysis: reproducible and agile production of political data. American political science review. 2016, Vol. 110, N 2, P. 278–295. DOI: https://doi.org/10.1017/s0003055416000058

4. Brand J.E., Zhou X., Xie Y. Recent developments in causal inference and machine learning. Annual Review of Sociology. 2023, Vol. 49, P. 81–110. DOI: https://doi.org/10.1146/annurev-soc-030420-015345

5. Breiman L. Statistical modeling: the two cultures. Statistical Science. 2001, Vol. 16, N 3, P. 199–215. DOI: https://doi.org/10.1214/ss/1009213726

6. Dressel J., Farid H. The accuracy, fairness, and limits of predicting recidivism. Science advances. 2018, Vol. 4, N 1, P. eaao5580. DOI: https://doi.org/10.1126/sciadv.aao5580

7. Edelmann A., Wolff T., Montagne D., Bail C.A. Computational social science and sociology. Annual review of sociology. 2020, Vol. 46, P. 61–81. DOI: https://doi.org/10.1146/annurev-soc-121919-054621

8. Fleiss J.L., Cohen J., Everitt B.S. Large sample standard errors of kappa and weighted kappa. Psychological bulletin. 1969, Vol. 72, N 5, P. 323–327. DOI: 10.1037/h0028106

9. Franzmann S., Kaiser A. Locating political parties in policy space: a reanalysis of Party Manifesto data. Party politics. 2006, Vol. 12, N 2, P. 163–188. DOI: https://doi.org/10.1177/1354068806061336

10. Gelman A., Carlin J.B., Stern H.S., Dunson D.B., Vehtari A., Rubin D.B. Bayesian data analysis. New York, Chapman and Hall/CRC, 2020, 675 p.

11. Gelman A., Meng X.-L., Stern H. Posterior predictive assessment of model fitness via realized discrepancies. Statistica sinica. 1996, Vol. 6, P. 733–807.

12. Gosling J.P. SHELF: The Sheffield Elicitation Framework. In: Dias L.C., Morton A., Quigley J. (eds). Elicitation: the science and art of structuring judgement. New York: Springer, 2018, P. 61–93.

13. Gosling J.P., Hart A., Mouat D., Sabirovic M., Scanlon S., Simmons A. Quantifying experts’ uncertainty about the future cost of exotic diseases. Risk analysis. 2012, Vol. 32, N 5, P. 881–893. DOI: https://doi.org/10.1111/j.1539-6924.2011.01704.x

14. Grimmer J., Roberts M.E., Stewart B.M. Machine learning for social science: an agnostic approach. Annual review of political science. 2021, Vol. 24, P. 395–419. DOI: https://doi.org/10.1146/annurev-polisci-053119-015921

15. Hastie T., Tibshirani R., Friedman J. The elements of statistical learning: data mining, inference, and prediction. New York: Springer, 2016, 745 p.

16. Imai K., Yamamoto T. Identification and sensitivity analysis for multiple causal mechanisms: revisiting evidence from framing experiments. Political analysis. 2013, Vol. 21, N 2, P. 141–171. DOI: https://doi.org/10.1093/pan/mps040

17. Krippendorff K. Content analysis: an introduction to its methodology. Beverly Hills, CA: Sage, 1980, 441 p.

18. Larichev O.I., Petrovskiy A.B. Decision support systems: state of the art and perspetives for development. Results of science and technology. Moscow: VINITI, 1987, Vol. 21, P. 131–164. (In Russ.)

19. Laver M., Budge I. Party policy and government coalitions. New York: St. Martins Press, 1992, 448 p.

20. Leamer E.E. Let’s take the con out of econometrics. The American economic review. 1983, Vol. 73, N 1, P. 31–43.

21. Lehmann P., Zobel M. Positions and saliency of immigration in party manifestos: a novel dataset using crowd coding. European journal of political research. 2018, Vol. 7, N 4, P. 1056–1083. DOI: https://doi.org/10.1111/1475-6765.12266

22. Lenart-Gansiniec R., Czakon W., Sulkowski L., Jasna Pocek J. Understanding crowdsourcing in science. Review of managerial science. 2023, Vol. 17, P. 2797–2830. DOI: https://doi.org/10.1007/s11846-022-00602-z

23. Melville A.Yu., Malgin A.V., Mironyuk M.G., Stukal D.K. “Political atlas of the modern world 2.0”: formulation of the research problem. Polis. Political studies. 2023, N 2, P. 72–87. DOI: https://doi.org/10.17976/jpps/2023.02.06 (In Russ.)

24. Melville A.Yu., Malgin A.V., Mironyuk M.G., Stukal D.K. Empirical challenges and methodological approaches in comparative politics (through the lens of the Political Atlas of the Modern World 2.0). Polis. Political studies. 2023, N 5, P. 153–171. DOI:10.17976/jpps/2023.05.10 (In Russ.)

25. Molina M., Garip F. Machine learning for sociology. Annual review of sociology. 2019, Vol. 45, P. 27–45. DOI: https://doi.org/10.1146/annurev-soc-073117-041106

26. O’Hagan A. Eliciting expert beliefs in substantial practical applications. Journal of the royal statistical society. Series D (The Statistician). 1998, Vol. 47, N 1, P. 21–35. DOI: https://doi.org/10.1111/1467-9884.00114

27. Orlov A.I. Expert evaluation theory in our country. Polythematic online scientific journal of Kuban State Agrarian University. 2013, N 93, P. 1–11. (In Russ.)

28. Oster E. Unobservable selection and coefficient stability: theory and evidence. Journal of business & Economic statistics. 2019, Vol. 37, N 2, P. 187–204. DOI: https://doi.org/10.1080/07350015.2016.1227711

29. Quigley J., Colson A., Aspinall W., Cooke R.M. Elicitation in the classical model. In: Dias L.C., Morton A., Quigley J. (eds). Elicitation: the science and art of structuring judgement. Springer, 2018, P. 15–36.

30. Perälä T., Vanhatalo J., Chrysafi A. Calibrating expert assessments using hierarchical gaussian process models. Bayesian analysis. 2020, Vol. 15, N 4, P. 1251–1280. DOI: https://doi.org/10.1214/19-ba1180

31. Rufo M.J., Pérez C.J., Martín J. A Bayesian approach to aggregate experts’ initial information. Electronic journal of statistics. 2012, Vol. 6, P. 2362–2382. DOI: https://doi.org/10.1214/12-ejs752

32. Schmerling D.S., Dubrovskiy S.A., Arzhanova T.D., Frenkel A.A. Expert evaluation. Methods and application. In: Statistical methods of expert evaluation analysis. Moscow: Nauka, 1977, P. 290–382. (In Russ.)

33. Starke C., Lünich M. Artificial intelligence for political decision-making in the European Union: effects on citizens’ perceptions of input, throughput, and output legitimacy. Data & Policy. 2020, Vol. 2, E16. DOI: https://doi.org/10.1017/dap.2020.19

34. Stone M. The opinion pool. Annals of mathematical statistics. 1961, Vol. 32, N 4, P. 1339–1342. DOI: https://doi.org/10.1214/aoms/1177704873

35. Tetlock Ph. E. Expert political judgment: how good is it? how can we know? Princeton: Princeton university press, 2006, 321 p.