基本信息

要点解读

这是什么

该基准把真实社交事件的新闻和社交媒体内容整合成时间线，并将其转化为去标识化的反事实情境，用以评估语言模型在概率校准和时间准确性两个维度上的预测表现。

用在哪里

适用于研究团队系统测评大模型对社会事件走向的理解与预测能力，也可帮助平台构建跨平台事件流的评测框架。

可以推断的

  • 推测：模型在概率校准和时间准确性上可能出现权衡，单一模型难以同时在两方面取得高分。
  • 推测：不同事件上得分波动较大，说明跨事件的泛化仍是模型评估的主要挑战。

来源摘要/节选

Large language models (LLMs), and the agents built on top of them, are now benchmarked heavily on whether they can finish a task – fix a bug, drive a browser, operate a GUI. A complementary social ability, namely how well a model understands and forecasts the way real social events unfold, has barely been measured. We introduce SocietyBench, an end-to-end benchmark that takes a one-line event topic, collects Web news and social-media posts across five platforms, distills them into a date-indexed timeline that keeps factual events and a public-opinion layer separate, and then turns every cutoff date on that timeline into an audited bank of forecasting questions. Questions are scored on two orthogonal 100-point axes: probability calibration and temporal accuracy. Before any model sees a timeline, a three-phase procedure replaces every named entity and shifts every date by a per-event constant, turning a real arc into a counterfactual social world – structurally identical to what happened, but stripped of the surface labels a model could match against pre-training memory. On five heterogeneous events and 125 prediction points in Chinese and English editions, the strongest of six frontier LLMs reaches only 75.0 out of 100, against a trivial anchor of 50. The two axes come apart: a model can be calibration-strong but time-weak, or the reverse. Three agent frameworks built on a shared base model fail to improve on that base, and two model-free heuristics trail every LLM. Per-event gaps reach 21.4 points on a single axis, which is our main argument for evaluating on several events rather than one. All anonymized timelines, question banks, ground truth, and scoring code are released.

来源说明

当前保存的是来源摘要，不代表论文全文。请以原始来源为准。

「要点解读」由 AI Stack 依据上方已保存内容整理，不代表来源的完整表述；标注「推测：」的判断来自编辑，不是来源陈述。