Aktuell
01.04.2025
Institutskolloquium IfP (16.04.25) - “Goodbye human annotators? Content analysis of social policy debates using ChatGPT”
Speaker: Dr. Jakub Sowula, Post-Doctoral Researcher & Lecturer (University of Tübingen)
Wednesday, April 16th 2025, 16:00 c.t. / Room 124, Institut für Politikwissenschaft or online via Zoom
About the lecture: In this talk, Dr. Sowula will present findings from the paper “Goodbye human annotators? Content analysis of social policy debates using ChatGPT”, published in the Journal of Social Policy. The paper evaluates the potential of the most prominent large language model, ChatGPT, to automate content analysis in policy debates. Using two contrasting case studies—the Universal Basic Income debate on Dutch Twitter and pension reform debates in German newspapers—we compare ChatGPT’s classification performance to that of human annotators. While ChatGPT demonstrates high reliability across datasets, its precision and recall vary significantly across argument types, indicating that full automation is not yet feasible. At least for now, human annotators are here to stay.
Nevertheless, the results also underscore the considerable potential of LLMs: when performance is tested on a subset of categories and benchmarked against human-coded data, tools like ChatGPT could substantially increase the volume and speed of qualitative content analysis. This opens up promising avenues for future research.
Moreover, beyond this specific study, Dr. Sowula invites a broader discussion on the potential and limitations of LLMs throughout the stages of the research process. What role should these tools play in the future of social science research?
Dr. Jakub Sowula completed his studies in mathematics, political science & economics (1st state examination degree) at the University of Tübingen. After finishing his 2nd state examination, he pursued his doctorate in political science titled “Young Minds and Political Competencies in the Context of Welfare Support”, jointly supervised by Professors Martin Seeleib-Kaiser and Sebastian Tempelmann. During his doctoral research, he was based at the Bern University of Teacher Education in Switzerland. In 2025, he returned to the University of Tübingen as a postdoctoral researcher and lecturer. His research focuses on various aspects of comparative social policy, including welfare state change and public welfare attitudes. He also works on political education and the application of AI in political science research. His work has been published in leading social science journals such as the Journal of European Social Policy, Social Policy & Administration, Journal of Social Policy, International Journal of Science Education, and Swiss Political Science Review. More info!
The lecture will be held in English