Validating Open-Weight LLM Measures of Language Functions in TV Advertising: A Multitrait-Multimethod Analysis"
Abstract
Large language models increasingly support content analysis, yet plausible annotations do not set valid measurement. We evaluate nine preferred open-weight model configurations as computational raters of four Jakobsonian language functions. The corpus contains 8,938 French automotive advertisements broadcast between 2014 and 2024. Models score informativeness, expressiveness, phaticness and creativeness/poeticness. Median same-trait Pearson correlations range from .706 for phaticness to .793 for informativeness. An MTMM analysis finds stronger monotrait than heterotrait associations, but Gemma exerts substantial method influence. Substantively, electric advertisements are more informative after adjustment for period and advertiser composition. Creativeness/poeticness declines after 2020, while the aggregate increase in informativeness disappears under the same controls. These findings establish a reproducible validation protocol and demonstrate why LLM annotations require measurement diagnostics before substantive interpretation.
About this workshop
The aim of this workshop is to promote technical and practical exchanges between researchers who use NLP methods. There is no hesitation in detailing the code (r/python), sharing tips, and discovering new methods and models.
Periodicity: Thursdays from 12h15 to 13h30, by videoconference.