草稿 内容完整但尚未审校,可能还会修改
第四部分当模型针对监督调整策略
本部分导读
前三部分已经涉及对抗性论证和误导评估者。这一部分系统讨论四件事:监督信号被优化之后的变化(第 23 章),多个模型的共同盲点(第 24 章),持续的隐藏目标(第 25 章、第 27 章),以及部署时的防御(第 26 章)。
这一部分的许多证据来自“模型有机体”:为研究而人为构造的对象。它们说明某种行为可以发生,不说明这种行为在普通部署的模型中有多常见。
本部分各章
- 23 监督信号变成优化目标之后
今天能抓住错误的裁判,成为明天的训练目标后,是否仍然可靠?
- 24 多个模型的共同盲点
多个模型意见一致,能否算作独立验证?
- 25 欺骗与伪装
模型会不会在被监督时表现得与平时不同?
- 26 AI control 与长过程智能体
如果无法确认模型可信,能否在部署时限制它可能造成的伤害?
- 27 对齐审计
怎样判断一个表面正常的模型是否藏着别的目标?
第四部分参考文献
本部分各章共引用 26 篇文献。
- (2026). Automated Alignment Researchers: Using large language models to scale scalable oversight. Anthropic. 非同行评审 第 23 章
- (2024). Training Language Models to Win Debates with Self-Play Improves Judge Accuracy. arXiv. 预印本 第 23 章
- (2024). A Critical Analysis of the Largest Source for Generative AI Training Data: Common Crawl. The 2024 ACM Conference on Fairness, Accountability, and Transparency. 第 24 章
- (2025). Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation. arXiv. 预印本 第 23 章
- (2025). Ctrl-Z: Controlling AI Agents via Resampling. arXiv. 预印本 第 26 章
- (2025). Building and evaluating alignment auditing agents. Anthropic Alignment Science Blog. 非同行评审 第 27 章
- (2026). Automated Researchers Can Mitigate Well-Characterized Alignment Failures. Anthropic Alignment Science Blog. 非同行评审 第 23 章
- (2022). Scaling Laws for Reward Model Overoptimization. International Conference on Machine Learning, 2023. 第 23 章
- (2025). Great Models Think Alike and this Undermines AI Oversight. International Conference on Machine Learning. 第 24 章
- (2024). The case for ensuring that powerful AIs are controlled. LessWrong / AI Alignment Forum. 非同行评审 第 26 章
- (2023). AI Control: Improving Safety Despite Intentional Subversion. International Conference on Machine Learning, 2024. 第 26 章
- (2024). Alignment faking in large language models. arXiv. 预印本 第 25 章
- (2019). Risks from Learned Optimization in Advanced Machine Learning Systems. arXiv. 预印本 第 25 章
- (2023). Model Organisms of Misalignment: The Case for a New Pillar of Alignment Research. AI Alignment Forum. 非同行评审 第 25 章
- (2024). Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training. arXiv. 预印本 第 25 章
- (2025). Evaluating Control Protocols for Untrusted AI Agents. arXiv. 预印本 第 26 章
- (2025). Natural Emergent Misalignment from Reward Hacking in Production RL. arXiv. 预印本 第 25 章
- (2025). Auditing language models for hidden objectives. arXiv. 预印本 第 27 章
- (2024a). Llama 3.1 Model Card. GitHub, meta-llama/llama-models. 非同行评审 第 24 章
- (2024b). Llama 3.2-Vision Model Card. GitHub, meta-llama/llama-models. 非同行评审 第 24 章
- (2025). Large Language Models Often Know When They Are Being Evaluated. arXiv. 预印本 第 25 章
- (2019). Do ImageNet Classifiers Generalize to ImageNet? International Conference on Machine Learning. 第 23 章
- (2025). Stress Testing Deliberative Alignment for Anti-Scheming Training. arXiv. 预印本 第 25 章
- (2026). AuditBench: Evaluating Alignment Auditing Techniques on Models with Hidden Behaviors. arXiv. 预印本 第 27 章
- (2024). AI Sandbagging: Language Models can Strategically Underperform on Evaluations. International Conference on Learning Representations, 2025. 第 25 章