Large Language Models as Lab Agents: Transfer Learning for Building Generalizable Automatic In-House Annotators
Abstract
Automatic data annotation is a common task in social science research, such as labeling social media posts for the presence of hate speech, or new articles for their ideological leaning. When fine-tuned with some manually annotated examples, Large Language Models (LLMs) consistently outperform traditional machine learning models. However, researchers often fine-tune a pre-trained LLM from scratch for each new task at hand. Building on the notion of transfer learning, in this paper we leverage several datasets annotated in the same research lab, for different tasks and projects, to develop a Lab Agent that gradually performs better at zero-shot and few-shot classification on new unseen tasks. We explore the performance of several options in the development of such Lab Agents, with the goal of facilitating and speeding up accurate data annotation for social science research.
About this workshop
The Political Economy + AI seminar is an online seminar focused on political economy research using AI and computational methods. We welcome work in progress and methods talks on these topics.
Formerly the Public Governance working group, the seminar is organized jointly by King's College London and Université Paris-Dauphine – PSL.
More information can be found on the website of the Political Economy + AI seminar.
Periodicity: Mondays from 17h30 to 18h30 (Paris time).