The Model as One Rater Among Several: Measuring Political Positions in Data-Sparse Regions with a Language-Model Panel
作者: Tarek Gara
分类: cs.CY, cs.AI, cs.CL, stat.AP
发布日期: 2026-06-22
备注: 21 pages, 1 figure, 7 tables. Dataset, rubric, and interactive tools: https://tarekgara.com/tayyar
💡 一句话要点
提出语言模型面板以解决数据稀缺地区的政治立场测量问题
🎯 匹配领域: 支柱九：具身大模型 (Embodied Foundation Models)
关键词: 政治立场测量 语言模型 数据稀缺 评估面板 可靠性分析 中东北非 专家调查
📋 核心要点
- 现有的政治立场测量工具在非西方国家的适用性差，导致数据稀缺地区的测量效果不佳。
- 论文提出将大型语言模型视为评估面板中的一个评估者，通过多个评估者的汇聚来提高测量的可靠性。
- 实验结果显示，书面定义的添加提高了评分一致性，且面板的可靠性在评估者数量增加时保持稳定。
📝 摘要（中文）
大多数用于测量政治立场的工具在西方政党体系中构建和验证，而在其他环境中效果不佳。本文提出了一种新方法，将大型语言模型视为一个单一的、可能出错的评估者，通过多个评估者的汇聚来提高测量的可靠性。研究报告了三项主要结果：首先，书面轴定义的添加显著提高了评分一致性；其次，九个模型的Krippendorff's alpha值为0.86，显示出良好的可靠性；最后，面板的分歧提供了有价值的信息，揭示了对某些问题的解释差异。该方法特别适用于中东和北非地区，未来可推广至其他数据稀缺的区域。
🔬 方法详解
问题定义：本文旨在解决现有政治立场测量工具在数据稀缺地区的适用性不足，尤其是在非西方国家的表现不佳。现有方法往往依赖于单一评估者，缺乏多样性和可靠性。
核心思路：论文的核心思路是将大型语言模型视为一个评估者，类似于专家调查中的单一专家，通过多个评估者的汇聚来提高测量的准确性和可靠性。
技术框架：整体架构包括一个评估面板，设定适用性规则以区分零分和空白分数，并通过镜头系统将行为与言论分开。主要模块包括评估者选择、评分标准设定和结果分析。
关键创新：最重要的创新在于将语言模型作为评估者的一部分，而非单一测量工具，从而利用多个评估者的观点来提高结果的可靠性和一致性。
关键设计：在设计中，设置了书面轴定义以提高评分一致性，并采用Krippendorff's alpha作为可靠性指标，确保在评估者数量增加时，评分的一致性得以保持。
📊 实验亮点
实验结果显示，书面轴定义的添加使评分均值提高了1.8分，评估者之间的一致性显著增强，Krippendorff's alpha值达到了0.86，表明该方法在可靠性方面表现优异。此外，面板的分歧提供了对政治立场的深入理解，揭示了约三分之二的分歧源于解释差异而非错误。
🎯 应用场景
该研究的潜在应用领域包括政治科学、社会研究及数据分析，尤其是在数据稀缺的地区。通过提供一种新的测量方法，研究能够帮助学者和政策制定者更好地理解和分析这些地区的政治立场，从而推动相关领域的研究和实践。
📄 摘要（原文）
Most tools for measuring political positions, manifesto coding, expert surveys, text-scaling models, were built and validated on Western party systems, and outside that setting they work poorly, and often not at all. This paper is an attempt at a method for those settings. It treats a large language model not as a measurement device but as a single, fallible rater in a panel, roughly the way an expert survey treats one expert: the value comes from pooling many judges rather than trusting any one of them. I describe the panel, an applicability rule that keeps a score of zero distinct from a blank, and a lens system that separates what an actor says from what it does. I report three results. First, holding a definition-free round fixed, adding written axis definitions moves scores by a mean of 1.8 points on a 21-point scale and tightens agreement between raters (mean absolute gap 2.81 to 2.50; r 0.81 to 0.89); they make two independent raters agree more closely, which an arbitrary steer would not. Second, across nine models from eight laboratories in two countries, Krippendorff's alpha is 0.86 on both an interval and an ordinal metric, and it stayed put as the panel grew from five raters to nine. That is reliability, the reproducibility of a reading, and not validity, its correctness. Third, where the panel does disagree, the disagreement is informative: the sharpest split, a full-scale divergence on an actor's stance toward its state's foundational order, points to a referent problem, and a blind triple-coding puts about two-thirds of it down to interpretation rather than error. I try to be plain about what the method can't do, including the human validation it still lacks, and I release the instrument and data in full. The worked example is the Middle East and North Africa, but I'd expect the method to carry to any region these standard tools leave out.