草稿 内容完整但尚未审校,可能还会修改
第三部分主要方法
本部分导读
这一部分逐一讲解主要的监督方法。每章先用“证据与前提”一项说明方法想处理什么、实验测过什么、依赖哪些前提(第 10 章)、不能推出什么,再讲方法本身。读实验论文时,用表 11.1 的四个核心问题检查它。
最后一章把这些方法放在一张表里比较,并用第 1 章的记账程序(已加入转账功能)练习设计一个组合方案。
本部分各章
- 14 拆分任务:分解、迭代放大与递归奖励建模
人判断不了整体时,能不能一块一块地判断?
- 15 辩论
让两个模型互相挑错,较弱的裁判能否判断谁对?
- 16 让模型帮人挑错
模型写的批评能否帮人发现自己发现不了的错误?
- 17 弱监督与强模型:三种不同的设定
监督质量有限时,强模型能否学到比监督者更好的东西?
- 18 模型内部表示
除了看模型说了什么,还能从它的内部计算中得到什么证据?
- 19 检查过程
检查推理步骤,是否比只检查最终答案更可靠?
- 20 机器检查与外部证据
哪些任务可以不靠人的判断,而靠可执行的检查或外部证据?
- 21 按原则给出的 AI 反馈
能否给模型一组原则,让它代替人来评估?
- 22 方法比较
面对一个具体任务,怎样选择和组合监督方法?
第三部分参考文献
本部分各章共引用 44 篇文献。
- (2024). Mapping the mind of a large language model. Anthropic. 非同行评审 第 18 章
- (2024). Training Language Models to Win Debates with Self-Play Improves Judge Accuracy. arXiv. 预印本 第 15 章
- (2022). Constitutional AI: Harmlessness from AI Feedback. arXiv. 预印本 第 21 章 第 22 章
- (2025). Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation. arXiv. 预印本 第 19 章 第 20 章 第 22 章
- (2020). Debate update: Obfuscated arguments problem. AI Alignment Forum. 非同行评审 第 14 章 第 15 章
- (2021). Probing Classifiers: Promises, Shortcomings, and Advances. Computational Linguistics, 2022. 第 18 章
- (2023). Scalable AI Safety via Doubly-Efficient Debate. International Conference on Machine Learning, 2024. 第 15 章
- (2025). Avoiding Obfuscation with Prover-Estimator Debate. arXiv. 预印本 第 15 章
- (2025). An alignment safety case sketch based on debate. arXiv. 预印本 第 15 章
- (2022). Discovering Latent Knowledge in Language Models Without Supervision. International Conference on Learning Representations, 2023. 第 18 章 第 22 章
- (2023). Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision. International Conference on Machine Learning, 2024. 第 17 章 第 22 章
- (2025). Reasoning Models Don't Always Say What They Think. arXiv. 预印本 第 19 章
- (2018). Supervising strong learners by amplifying weak experts. arXiv. 预印本 第 14 章
- (2021). Eliciting latent knowledge: How to tell if your eyes deceive you. Alignment Research Center. 非同行评审 第 18 章
- (2018). Iterated Distillation and Amplification. AI Alignment Forum. 非同行评审 第 14 章
- (2025). DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature. 第 19 章 第 20 章 第 22 章
- (2023). Challenges with unsupervised LLM knowledge discovery. arXiv. 预印本 第 18 章
- (2025). Great Models Think Alike and this Undermines AI Oversight. International Conference on Machine Learning. 第 17 章
- (2025). Detecting Strategic Deception with Linear Probes. International Conference on Machine Learning. 第 18 章
- (2025). Dodging systematic human errors in scalable oversight. AI Alignment Forum. 非同行评审 第 15 章
- (2018). AI safety via debate. arXiv. 预印本 第 15 章
- (2024). On scalable oversight with weak LLMs judging strong LLMs. Advances in Neural Information Processing Systems. 第 15 章 第 22 章
- (2024). Debating with More Persuasive LLMs Leads to More Truthful Answers. International Conference on Machine Learning. 第 15 章 第 22 章
- (2024). Prover-Verifier Games improve legibility of LLM outputs. arXiv. 预印本 第 19 章
- (2025). Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety. arXiv. 预印本 第 19 章
- (2026a). Axioms. Lean Language Reference. 非同行评审 第 20 章
- (2026b). Validating a Lean Proof. Lean Language Reference. 非同行评审 第 20 章
- (2023). RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback. International Conference on Machine Learning, 2024. 第 21 章 第 22 章
- (2018). Scalable agent alignment via reward modeling: a research direction. arXiv. 预印本 第 14 章 第 15 章
- (2023). Let's Verify Step by Step. International Conference on Learning Representations, 2024. 第 19 章 第 22 章
- (2023). Evaluating Verifiability in Generative Search Engines. Findings of the Association for Computational Linguistics: EMNLP 2023. doi:10.18653/v1/2023.findings-emnlp.467. 第 20 章 第 22 章
- (2023). The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets. Conference on Language Modeling, 2024. 第 18 章 第 22 章
- (2024). LLM Critics Help Catch LLM Bugs. arXiv. 预印本 第 16 章 第 22 章
- (2023). Debate Helps Supervise Unreliable Experts. arXiv. 预印本 第 15 章 第 22 章
- (2023). weak-to-strong. GitHub. 软件 第 17 章
- (2021). QuALITY: Question Answering with Long Input Texts, Yes! Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 第 15 章
- (2007). Game Theory. Open Yale Courses. 非同行评审 第 15 章
- (2025b). FindTheFlaws: Annotated Errors for Detecting Flawed Reasoning and Scalable Oversight Research. Proceedings of the AAAI Conference on Artificial Intelligence, 2026. doi:10.1609/aaai.v40i44.41123. 第 16 章
- (2022). Self-critiquing models for assisting human evaluators. arXiv. 预印本 第 16 章 第 22 章
- (2013). Introduction to the Theory of Computation. Cengage. 第 15 章
- (2024). Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervision. Advances in Neural Information Processing Systems. 第 17 章 第 22 章
- (2023). Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting. Advances in Neural Information Processing Systems. 第 19 章
- (2025). Unsupervised Elicitation of Language Models. arXiv. 预印本 第 17 章 第 22 章
- (2021). Recursively Summarizing Books with Human Feedback. arXiv. 预印本 第 14 章 第 22 章