The basics yet ahead: an overview of the approaches to investigating political bias in large language models

Political bias of Large Language Models has frequently become a topic for scientific investigation. Most of the researchers tend to compete in inventing more original ways of identifying bias rather than posing new research questions related to it besides “Is this model politically biased?” and “What is the character of its bias?”. To properly evaluate possible influence of the models on the political reality and finding answers to some questions regarding regulation of Artificial Intelligence it is essential to be able to study the linkage between the bias and its cause. With regard to how the question of dependence between the identified bias and its possible source is ad- dressed I have grouped the approaches to studying political bias of LLMs into three clusters: approaches that use political orientation surveys and questionnaires, studies devoted to investigating different ways of creating prompts and models’ responses and their interdependence, and interdisciplinary research in which manipulations with possible sources of LLMs’ political bias is conducted. The latter research trajectory seems to be the most promising one, despite its current unpopularity. Yet, it is impossible to advance in this trajectory without it being complemented by further developments in the approaches in the first two clusters. In studying the issue of LLM bias, not only computer science specialists but also philosophers and political scientists and other experts in the social sciences should be involved. Political biases at the intersection of LLM and other generative AI technologies – in particular, technologies for generating images based on prompts composed in natural language, recommendation algorithms, etc. – also require separate research. From a regulatory standpoint, further progress in mitigating, eradicating, and controlling political biases in algorithmic tools will require providing researchers with greater access to existing and actively used technologies. Furthermore, it appears necessary to establish specialized institutions dedicated to research on AI at the intersection of computer science, ethics, the philosophy of mind, neurocognitive and social sciences.

1. Agiza A., Mostagir M., Reda S. Analyzing the impact of data selection and fine-tuning on economic and political biases in LLMs. arXiv preprint arXiv:2404.08699. 2024. DOI: https://doi.org/10.48550/arXiv.2404.08699

2. Bang Y., Chen D., Lee N., Fung P. 2024. Measuring political bias in large language models: what is said and how it is said. arXiv preprint arXiv:2403.18932. 2024. DOI: https://doi.org/10.48550/arXiv.2403.18932

3. Davis R. Typing politics: The role of blogs in American politics. Oxford: Oxford university press, 2009, 241 p.

4. Durmus E., Nyugen K., Liao T.I., Schiefer N., Askell A., Bakhtin A., Chen C., Hatfield-Dodds Z., Hernandez D., Joseph N., Lovitt L. Towards measuring the representation of subjective global opinions in language models. arXiv preprint arXiv:2306.16388. 2023. DOI: https://doi.org/10.48550/arXiv.2306.16388

5. Elmer G., Langlois G., McKelvey F. The permanent campaign: new media, new politics. Lausanne: Peter Lang, 2012, 144 p.

6. Farrell H., Drezner D.W. The power and politics of blogs. Public choice. 2008, N 134, P. 15–30.

7. Feng S., Park C.Y., Liu Y., Tsvetkov Y. From pretraining data to language models to downstream tasks: Tracking the trails of political biases leading to unfair NLP models. arXiv preprint arXiv:2305.08283. 2023. DOI: https://doi.org/10.48550/arXiv.2305.08283

8. Freeden M. Ideologies and political theory: A conceptual approach. Oxford: Clarendon Press, 1996, 604 p.

9. Gover L. Political bias in large language models. The commons: Puget sound journal of politics. 2023, Vol. 4 (1), N 2, P. 11–22.

10. Habermas J. Concluding comments on empirical approaches to deliberative politics. Acta politica. 2005, N 40, P. 384–392.

11. Habermas J. Between facts and norms: Contributions to a discourse theory of law and democracy. Hoboken: John Wiley & Sons, 2015, 631 p.

12. Hargittai E., Gallo J., Kane M. Cross-ideological discussions among conservative and liberal bloggers. Public Choice. 2008, N 134, P. 67–86.

13. Jiang H., Beeferman D., Roy B., Roy D. CommunityLM: Probing partisan worldviews from language models. arXiv preprint arXiv:2209.07065. 2022. DOI: https://doi.org/10.48550/arXiv.2209.07065

14. Kotek H., Dockum R., Sun D. Gender bias and stereotypes in large language models. In: Bernstein M., Savage S., Bozzon A. (eds). Proceedings of The ACM collective intelligence conference. 2023, P. 12–24.

15. Liu R., Jia C., Wei J., Xu G., Wang L., Vosoughi S. Mitigating political bias in language models through reinforced calibration. Proceedings of the AAAI Conference on Artificial Intelligence. 2021, Vol. 35, N 17, P. 14857–14866.

16. Malinova O.Yu. Symbolic policy: contours of the problem field. In: Malinova O.Yu. (ed.). Symbolic politics: Collection of articles. Moscow: INION RAS, 2012, P. 5–16. (In Russ.)

17. Motoki F., Pinho Neto V., Rodrigues V. More human than human: Measuring ChatGPT political bias. Public Choice. 2023, Vol. 198, N 1, P. 3–23.

18. Nadeem M., Bethke A., Reddy S. StereoSet: Measuring stereotypical bias in pretrained language models. arXiv preprint arXiv:2004.09456. 2020. DOI: https://doi.org/10.48550/arXiv.2004.09456

19. Omiye J. A., Lester J. C., Spichak S., Rotemberg V., Daneshjou R. Large language models propagate race-based medicine. NPJ Digital medicine. 2023, Vol. 6, N 1, Article 195. Papacharissi Z. The virtual sphere: The internet as a public sphere. New media & society. 2002, Vol. 4, N 1, P. 9–27.

20. Pit P., Ma X., Conway M., Chen Q., Bailey J., Pit H., Keo P., Diep W., Jiang Y.G. Whose side are you on? Investigating the political stance of large language models. arXiv preprint arXiv:2403.13840. 2024. DOI: https://doi.org/10.48550/arXiv.2403.13840

21. Rasmussen T. Internet and the political public sphere. Sociology Compass. 2014, Vol. 8, N 12, P. 1315–1329.

22. Rozado D. The political biases of ChatGPT. Social Sciences. 2023, Vol. 12, N 3, Article 148.

23. Rozado D. The political preferences of LLMs. PLoS ONE. 2024, Vol. 19, N 7, Article e0306621. DOI: https://doi.org/10.1371/journal.pone.0306621

24. Röttger P., Hofmann V., Pyatkin V., Hinck M., Kirk H.R., Schütze H., Hovy D. Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models. arXiv preprint arXiv:2402.16786. 2024. DOI: https://doi.org/10.48550/arXiv.2402.16786

25. Tulchinskii G.L. Power and making of meaning, or political pragmasemantics. Political science (RU). 2023, N 3, P. 151–169. DOI: http://www.doi.org//10.31249/poln/2023.03.07

26. Urman A., Makhortykh M. The silence of the LLMs: Cross-lingual analysis of guardrailrelated political bias and false information prevalence in ChatGPT, Google Bard (Gemini), and Bing Chat. Telematics and Informatics. 2025, Vol 96, Article 102211.

27. Zhou M., Abhishek V., Derdenger T., Kim J., Srinivasan, K. Bias in Generative AI. arXiv preprint arXiv:2403.02726. 2024. DOI: https://doi.org/10.48550/arXiv.2403.02726