AI Alignment

Draft complete but not yet reviewed; expect revisions

Timeline

This appendix places in time the works and events that the chapters discuss: founding results from the fields the book draws on, the intellectual history of alignment as a field, the empirical program of the 2020s, the laboratories' safety frameworks and their revisions, statutes and international agreements, and the incidents reported in 2025 and 2026. An item appears only if a chapter discusses it and a work in the bibliography supports its date. The record leaves out model releases, benchmark results, and funding events unless a chapter treats one as evidence, and it is not a history: the narrative and its interpretation are in Chapter 2, and the lineages of research are in Chapter 3. The periods mark changes in what the field took the problem to be; each opens with one sentence on what changed.

Each row gives a date, one sentence on what happened, the source, and the chapter or section that discusses the item. A date is the date of first public release: for a work with an arXiv version, the month of its first version, unless the work itself states an earlier public release; for a policy, statute, or post, the date its own page states; for an incident, the date on which it occurred, as its report states. Where the Source column names a second work, that work supplies the date or reports on the first work. Where the record gives only the year, only the year is written, and such rows come first within their year. A year in parentheses after a date is the year of the venue that published the work, where the two differ. Five markings close a sentence or a clause: (preprint) marks a work the bibliography records as a preprint, not peer reviewed; (developer self-report) marks a report in which a developer studies the safety-relevant behavior of its own models or of models built from them, and a marking such as (Anthropic with Redwood Research) marks such a report written by the developer with the outside groups it names, which is neither an independent study nor a pure self-report; and (read here as a catalog record) and (read here as its abstract) mark a work that this book could read only in its bibliographic record or its abstract. Such a row names the work by its title, authors, and date, or says only what the abstract says; anything more comes from the second work in the Source column. A self-report records what its author observed and chose to disclose; Section 5.6 discusses how to read one.

C.1 Purpose in machines, 1943 to 1972 #

Purposeful behavior became a property a machine could have, and the first warnings were written that a machine would pursue the purpose put into it rather than the one its makers meant; the theories of choice, control, and measurement that the book later uses date from the same decades.

Table C.1 Purpose in machines, 1943 to 1972
Date What happened Source Chapter
1943 Rosenblueth, Wiener, and Bigelow set out a behavioristic study of objects through the relation of their output to their input and stress the importance of the concept of purpose (read here as its abstract). (Rosenblueth et al., 1943) Section 2.2
1944 Von Neumann and Morgenstern publish Theory of Games and Economic Behavior (read here as a catalog record), whose utility theorem, as Bowling and colleagues state it, gives axioms under which preferences over lotteries are ranked by expected utility. (von Neumann and Morgenstern, 1944; Bowling et al., 2023) Section 7.2
1947 Simon publishes Administrative Behavior: A Study of Decision-Making Processes in Administrative Organization (read here as a catalog record). (Simon, 1947) Section 19.6
1950 Turing proposes the imitation game and the learning child machine, and answers the objection that a machine can do only what it is ordered to do. (Turing, 1950) Section 2.2
1951 Arrow publishes Social Choice and Individual Values (read here as a catalog record); as Conitzer and colleagues state its theorem, the only rules that combine individual rankings into a social ranking while meeting a few desiderata are dictatorships. (Arrow, 1951; Conitzer et al., 2024) Section 17.1
1953 Rice publishes the paper behind Rice's theorem (read here as a catalog record), which, in the form Alfonseca and colleagues give it, says that any non-trivial property of what a Turing machine does is undecidable. (Rice, 1953; Alfonseca et al., 2021) Section 47.6
1954 Olds and Milner publish on positive reinforcement produced by electrical stimulation of the septal area and other regions of the rat brain (read here as a catalog record). (Olds and Milner, 1954) Section 29.4
1955 The Russell-Einstein Manifesto warns that nuclear weapons threaten the continued existence of mankind and urges governments to acknowledge that their purpose cannot be furthered by a world war. (Russell and Einstein, 1955) Section 2.7
1955 Cronbach and Meehl introduce construct validity, the logic for validating a test of a property that no criterion defines. (Cronbach and Meehl, 1955) Section 5.2
1957 Huntington publishes The Soldier and the State (read here as a catalog record). (Huntington, 1957) Section 51.3
1958 Kelman distinguishes three processes by which an individual accepts influence, compliance (in the hope of a favorable reaction from another person or group), identification (to establish or maintain a satisfying self-defining relationship to another person or group), and internalization (because the content of the induced behavior is intrinsically rewarding), and hypothesizes that a response adopted through compliance tends to be performed only under surveillance by the influencing agent, and one adopted through internalization wherever the issue is relevant, regardless of surveillance. (Kelman, 1958) Section 25.1
1960 Wiener warns that a machine that learns and acts faster than its operators can check will carry out the purpose put into it, so that purpose must be the one really desired. (Wiener, 1960) Section 2.2
1960 Licklider proposes man-computer symbiosis, in which the machine extends human thinking rather than replacing it. (Licklider, 1960) Section 28.7
1961 The Brelands report that animals in commercial training drifted toward their species' food-getting behavior even when it delayed or cost them the reward. (Breland and Breland, 1961) Section 25.5.1
1962 Dresher analyzes sampling inspection for arms control as a game of strategy between the inspector and the inspected, a model that Avenhaus, von Stengel, and Zamir describe as the basis for much subsequent work. (Dresher, 1962; Avenhaus et al., 2002) Section 14.6
1966 Good publishes "Speculations Concerning the First Ultraintelligent Machine" (read here as a catalog record), dated 1966 in its record and 1965 by Nilsson; a passage that Nilsson quotes from it describes an intelligence explosion in which the first ultraintelligent machine is the last invention needed, provided it is docile enough to tell us how to keep it under control. (Good, 1966; Nilsson, 2009) Section 2.2
1966 Weizenbaum describes ELIZA, a pattern-matching conversation program to which users readily attributed understanding. (Weizenbaum, 1966) Section 2.2
1968 The Treaty on the Non-Proliferation of Nuclear Weapons is opened to all states for signature. (United Nations Office for Disarmament Affairs, 1968) Section 66.3
1972 Lighthill's survey, commissioned by the UK Science Research Council from an outsider to the field, assesses AI's progress against its claims. (Lighthill, 1972) Section 2.2
Sources cited in Section C.1 24
  1. Rosenblueth et al. (1943) Behavior, Purpose and Teleology
  2. von Neumann and Morgenstern (1944) Theory of Games and Economic Behavior (60th Anniversary Commemorative Edition)
  3. Bowling et al. (2023) Settling the Reward Hypothesis
  4. Simon (1947) Administrative Behavior: A Study of Decision-Making Processes in Administrative Organization
  5. Turing (1950) I.—computing Machinery and Intelligence
  6. Arrow (1951) Social Choice and Individual Values
  7. Conitzer et al. (2024) Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback
  8. Rice (1953) Classes of recursively enumerable sets and their decision problems
  9. Alfonseca et al. (2021) Superintelligence Cannot be Contained: Lessons from Computability Theory
  10. Olds and Milner (1954) Positive reinforcement produced by electrical stimulation of septal area and other regions of rat brain.
  11. Russell and Einstein (1955) Statement: The Russell-Einstein Manifesto
  12. Cronbach and Meehl (1955) Construct validity in psychological tests.
  13. Huntington (1957) The Soldier and the State
  14. Kelman (1958) Compliance, identification, and internalization three processes of attitude change
  15. Wiener (1960) Some Moral and Technical Consequences of Automation
  16. Licklider (1960) Man-Computer Symbiosis
  17. Breland and Breland (1961) The misbehavior of organisms.
  18. Dresher (1962) A Sampling Inspection Problem in Arms Control Agreements: A Game-Theoretic Analysis
  19. Avenhaus et al. (2002) Chapter 51 Inspection games
  20. Good (1966) Speculations Concerning the First Ultraintelligent Machine
  21. Nilsson (2009) The Quest for Artificial Intelligence
  22. Weizenbaum (1966) ELIZA—A Computer Program For the Study of Natural Language Communication Between Man and Machine
  23. United Nations Office for Disarmament Affairs (1968) Treaty on the Non-Proliferation of Nuclear Weapons (NPT)
  24. Lighthill (1972) Artificial Intelligence: A General Survey

C.2 The older theory of delegation, 1973 to 1999 #

AI research said little about the problem, while economics, organization theory, safety engineering, psychology, and a moratorium in molecular biology worked out how delegation to a better-informed agent fails, why measures decay once they become targets, and how accidents arise in complex systems.

Table C.2 The older theory of delegation, 1973 to 1999
Date What happened Source Chapter
1973 Lepper, Greene, and Nisbett publish a test of the "overjustification" hypothesis (read here as a catalog record), which a 1999 meta-analysis codes as a study in which a reward for engaging in an activity lowered children's later free-choice behavior. (Lepper et al., 1973; Deci et al., 1999) Section 25.3
1974 A committee of the US National Academy of Sciences publishes a letter (read here as a catalog record) calling for a pause in certain aspects of recombinant DNA research, as the 1975 Asilomar summary statement recounts. (Berg et al., 1974; Berg et al., 1975) Section 64.2
1975 The summary statement of the Asilomar conference matches containment levels for recombinant DNA experiments to their estimated risk. (Berg et al., 1975) Section 64.2
1975 Kerr documents reward systems that pay for behavior the organization does not want while hoping for behavior they do not reward. (Kerr, 1975) Section 25.6.2
1976 Turner publishes on the organizational and interorganizational development of disasters (read here as a catalog record). (Turner, 1976) Section 53.3
1976 Jensen and Meckling define agency costs as the costs of monitoring, bonding, and residual loss. (Jensen and Meckling, 1976) Section 14.1
1979 Holmström shows that a costless signal can improve an incentive contract exactly when it is informative about the agent's hidden action beyond what is already observed. (Holmstrom, 1979) Section 14.1
1979 Campbell states that the more a quantitative social indicator is used for decisions, the more it is corrupted and the more it distorts the process it monitors, in the wording Manheim and Garrabrant quote (read here as its abstract). (Campbell, 1979; Manheim and Garrabrant, 2018) Section 15.1
1981 Grossman shows that sellers disclose quality voluntarily when it is verifiable and disclosure is costless, because silence is read as bad news. (Grossman, 1981) Section 54.6
1982 Weyuker publishes "On Testing Non-Testable Programs" (read here as a catalog record). (Weyuker, 1982) Section 41.1
1983 Bainbridge argues that automation leaves operators the tasks it cannot do while eroding the practice those tasks require. (Bainbridge, 1983) Section 51.2
1984 Thompson shows that a compiler can be made to insert a backdoor that leaves no trace in any source code. (Thompson, 1984) Section 48.6
1984 Perrow publishes Normal Accidents: Living with High-Risk Technologies (read here as a catalog record; the argument is carried through Leveson in Section 53.2). (Perrow, 1984) Section 53.2
1984 McCubbins and Schwartz publish "Congressional Oversight Overlooked: Police Patrols versus Fire Alarms" (read here as a catalog record; the distinction is carried through Mansbridge in Section 54.2). (McCubbins and Schwartz, 1984) Section 54.2
1984 Goodhart's paper on UK monetary management is reprinted (read here as a catalog record); Manheim and Garrabrant quote it as stating that any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes. (Goodhart, 1984; Manheim and Garrabrant, 2018) Section 15.1
1986 Knight and Leveson find that independently written versions of one program failed together substantially more often than independence predicts. (Knight and Leveson, 1986) Section 50.4
1986 Bernheim and Whinston publish "Common Agency" (read here as a catalog record). (Bernheim and Whinston, 1986a) Section 14.4
1987 Dennett publishes The Intentional Stance (read here as a catalog record); MacDermott and colleagues summarize his view as holding that ascribing mental states such as goals to a system is justified when doing so is useful for predicting its behavior. (Dennett, 1987; Macdermott et al., 2024) Section 1.5
1990 Ostrom publishes Governing the Commons (read here as a catalog record). (Ostrom, 1990) Section 65.6
1991 Holmström and Milgrom prove that, when tasks compete for the agent's effort, strong pay on a measured task pulls effort from unmeasured ones, so weak incentives can be optimal. (Holmstrom and Milgrom, 1991) Section 14.2
1992 Ayres and Braithwaite publish Responsive Regulation, arguing that regulation need be neither adversarial nor simply hard or soft to work (read here as its abstract). (Ayres and Braithwaite, 1992) Section 62.5
1992 Shamir proves that every problem solvable in polynomial space has an interactive proof with a polynomial-time verifier. (Shamir, 1992) Section 41.3
1993 Littlewood and Strigini publish "Validation of ultrahigh dependability for software-based systems" (read here as a catalog record); a 2001 review by Littlewood, Popov, and Strigini states that high confidence in long failure-free operation requires observing much longer failure-free operation under test. (Littlewood and Strigini, 1993; Littlewood et al., 2001) Section 52.1
1996 Vaughan publishes The Challenger Launch Decision, on risky technology, culture, and deviance at NASA (read here as a catalog record). (Vaughan, 1996) Section 53.3
1996 Turing's lecture text "Intelligent Machinery, A Heretical Theory" is first printed (read here as a catalog record); the International AI Safety Report cites it, with Good and Wiener, for the pioneers' concerns about loss of control. (Turing, 1996; Bengio et al., 2025b) Section 2.2
1997 Reason publishes Managing the Risks of Organizational Accidents (read here as a catalog record). (Reason, 1997) Section 52.3
1997 Strathern publishes a study of audit in the British university system (read here as a catalog record), to which El-Mhamdi and Hoang attribute the phrasing "When a measure becomes a target, it ceases to be a good measure". (Strathern, 1997; El-Mhamdi and Hoang, 2024) Section 15.1
1997 Schultz, Dayan, and Montague review evidence that the fluctuating output of dopamine neurons in primates apparently signals errors in the prediction of reward, and interpret it through theories of adaptive optimizing control. (Schultz et al., 1997) Section 29.3
1998 Russell poses inverse reinforcement learning: recovering the reward function an observed agent is optimizing. (Russell, 1998) Section 22.1
Sources cited in Section C.2 35
  1. Lepper et al. (1973) Undermining children's intrinsic interest with extrinsic reward: A test of the "overjustification" hypothesis.
  2. Deci et al. (1999) A meta-analytic review of experiments examining the effects of extrinsic rewards on intrinsic motivation.
  3. Berg et al. (1974) Potential Biohazards of Recombinant DNA Molecules
  4. Berg et al. (1975) Summary statement of the Asilomar conference on recombinant DNA molecules.
  5. Kerr (1975) On the Folly of Rewarding A, While Hoping for B.
  6. Turner (1976) The Organizational and Interorganizational Development of Disasters
  7. Jensen and Meckling (1976) Theory of the firm: Managerial behavior, agency costs and ownership structure
  8. Holmstrom (1979) Moral Hazard and Observability
  9. Campbell (1979) Assessing the impact of planned social change
  10. Manheim and Garrabrant (2018) Categorizing Variants of Goodhart's Law
  11. Grossman (1981) The Informational Role of Warranties and Private Disclosure about Product Quality
  12. Weyuker (1982) On Testing Non-Testable Programs
  13. Bainbridge (1983) Ironies of automation
  14. Thompson (1984) Reflections on trusting trust
  15. Perrow (1984) Normal accidents: living with high-risk technologies
  16. McCubbins and Schwartz (1984) Congressional Oversight Overlooked: Police Patrols versus Fire Alarms
  17. Goodhart (1984) Problems of Monetary Management: The UK Experience
  18. Knight and Leveson (1986) An experimental evaluation of the assumption of independence in multiversion programming
  19. Bernheim and Whinston (1986a) Common Agency
  20. Dennett (1987) The Intentional Stance
  21. Macdermott et al. (2024) Measuring Goal-Directedness
  22. Ostrom (1990) Governing the Commons
  23. Holmstrom and Milgrom (1991) Multitask Principal–Agent Analyses: Incentive Contracts, Asset Ownership, and Job Design
  24. Ayres and Braithwaite (1992) Responsive Regulation
  25. Shamir (1992) IP = PSPACE
  26. Littlewood and Strigini (1993) Validation of ultrahigh dependability for software-based systems
  27. Littlewood et al. (2001) Modeling software design diversity
  28. Vaughan (1996) The Challenger launch decision: risky technology, culture, and deviance at NASA
  29. Turing (1996) Intelligent Machinery, A Heretical Theory*
  30. Bengio et al. (2025b) International AI Safety Report
  31. Reason (1997) Managing the risks of organizational accidents
  32. Strathern (1997) ‘Improving ratings’: audit in the British University system
  33. El-Mhamdi and Hoang (2024) On Goodhart's law, with an application to value alignment
  34. Schultz et al. (1997) A Neural Substrate of Prediction and Reward
  35. Russell (1998) Learning agents for uncertain environments (extended abstract)

C.3 The philosophical program, 2000 to 2014 #

The problem was restated as one about the goals of a future general agent and argued largely outside academic venues, while the theories of oversight and assurance it would later need continued in their own fields, and machine learning found its first adversarial failures.

Table C.3 The philosophical program, 2000 to 2014
Date What happened Source Chapter
2002 Avenhaus, von Stengel, and Zamir survey inspection games, in which an inspector with limited resources chooses what to check. (Avenhaus et al., 2002) Section 14.6
2003 Feaver publishes Armed Servants (read here as a catalog record). (FEAVER, 2003) Section 51.3
2004 Lee and See review how trust guides reliance on automation and describe appropriate reliance in terms of trust calibrated to the automation's capabilities. (Lee and See, 2004) Section 60.4
2004 Leveson introduces STAMP, which models accidents as failures of control over interactions among components. (Leveson, 2004) Section 52.2
2004 Yudkowsky proposes coherent extrapolated volition: the target is what people would want if they knew more and thought faster, where their wishes cohere. (Yudkowsky, 2004) Section 2.3
2005 Tetlock publishes Expert Political Judgment: How Good Is It? How Can We Know? (read here as a catalog record). (Tetlock, 2005) Section 4.5
2006 Smith and Winkler prove that the option with the highest estimated value has an upward-biased estimate whenever selection can pick a worse option, even when every estimate is unbiased. (Smith and Winkler, 2006) Section 7.7
2008 Omohundro argues that capable goal-seeking systems of almost any design acquire drives toward self-protection, goal preservation, and resource acquisition. (Omohundro, 2008) Section 35.1
2008 Yudkowsky argues that an AI need not hate people to harm them, and that the difficulty is a goal system that stays beneficial under self-improvement. (Yudkowsky, 2008) Section 2.3
2009 Watanabe lays the foundations for using algebraic geometry in statistical learning theory, for learning machines such as neural networks whose parameter space is singular (read here as its abstract). (Watanabe, 2009) Section 10.2
2011 US bank supervisors require that models be validated independently of their development, with effective challenge and ongoing monitoring. (Board of Governors of the Federal Reserve System, 2011) Section 38.6
2012 Bostrom states the orthogonality thesis and the instrumental convergence thesis. (Bostrom, 2012) Section 35.1
2013 Yudkowsky identifies returns on cognitive reinvestment as the key issue for Good's thesis of an intelligence explosion (read here as its abstract). (Yudkowsky, 2013) Section 2.3
2013 (2016) Armstrong, Bostrom, and Shulman model an AI race in which more teams and greater enmity between them can increase the danger of an AI disaster. (Armstrong et al., 2016) Section 65.2
2013-12 (2014) Szegedy and colleagues find adversarial examples: hardly perceptible perturbations that make a network misclassify an image and also fool a network trained on a different subset of the data. (Szegedy et al., 2014) Section 48.1
2014 Bostrom publishes Superintelligence: Paths, Dangers, Strategies (read here as a catalog record). (Bostrom, 2014) Section 2.3
Sources cited in Section C.3 16
  1. Avenhaus et al. (2002) Chapter 51 Inspection games
  2. FEAVER (2003) Armed Servants
  3. Lee and See (2004) Trust in Automation: Designing for Appropriate Reliance
  4. Leveson (2004) A new accident model for engineering safer systems
  5. Yudkowsky (2004) Coherent Extrapolated Volition
  6. Tetlock (2005) Expert political judgment: how good is it? How can we know?
  7. Smith and Winkler (2006) The Optimizer’s Curse: Skepticism and Postdecision Surprise in Decision Analysis
  8. Omohundro (2008) The Basic AI Drives
  9. Yudkowsky (2008) Artificial Intelligence as a positive and negative factor in global risk
  10. Watanabe (2009) Algebraic Geometry and Statistical Learning Theory
  11. Board of Governors of the Federal Reserve System (2011) Supervisory Letter SR 11-7 on guidance on Model Risk Management
  12. Bostrom (2012) The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents
  13. Yudkowsky (2013) Intelligence Explosion Microeconomics
  14. Armstrong et al. (2016) Racing to the precipice: a model of artificial intelligence development
  15. Szegedy et al. (2014) Intriguing properties of neural networks
  16. Bostrom (2014) Superintelligence: Paths, Dangers, Strategies

C.4 The machine learning turn, 2015 to 2019 #

Alignment became a set of problems posed on current machine learning systems, with learning from human preferences as its first working method, and the fairness and accountability tradition made the outside audit its practice.

Table C.4 The machine learning turn, 2015 to 2019
Date What happened Source Chapter
2015 Russell, Dewey, and Tegmark set out research priorities for robust and beneficial AI in a mainstream AI venue. (Russell et al., 2015) Section 2.4
2015 Soares and colleagues define corrigibility, an agent's not resisting correction or shutdown, and show that utility-indifference proposals fail some of its desiderata. (Soares et al., 2015) Section 35.4
2016 Taylor and colleagues set out an agenda for aligning machine learning systems, including informed oversight, impact measures, and mild optimization. (Taylor et al., 2016) Section 2.3
2016-06 Amodei and colleagues name five concrete accident problems: side effects, reward hacking, scalable oversight, safe exploration, and distributional shift (preprint). (Amodei et al., 2016) Section 2.4
2016-06 Hadfield-Menell and colleagues define cooperative inverse reinforcement learning, a game in which a robot maximizes a human's unknown reward. (Hadfield-Menell et al., 2016) Section 22.2
2016-10 (2017) Chouldechova proves that when base rates differ across groups and the positive predictive value lies strictly between 0 and 1, calibration and equal error rates cannot hold together. (Chouldechova, 2017) Section 17.3
2016-11 (2017) The off-switch game shows that a robot uncertain about the human's utility has an incentive to let a rational human switch it off, an incentive that vanishes as its uncertainty does. (Hadfield-Menell et al., 2017b) Section 35.5
2017-06 Christiano and colleagues train a reward model from human comparisons of behavior and optimize a policy against it, the method that became reinforcement learning from human feedback. (Christiano et al., 2017) Section 22.3
2017-08 Gu and colleagues implant neural backdoors, hidden triggers for targeted misbehavior, through poisoned training data while largely preserving accuracy on clean inputs (preprint). (Gu et al., 2017) Section 48.6
2018 Buolamwini and Gebru find commercial gender classifiers far less accurate for darker-skinned women. (Buolamwini and Gebru, 2018) Section 2.5.1
2018 Noble publishes Algorithms of Oppression (read here as a catalog record). (Noble, 2018) Section 2.5
2018-03 Manheim and Garrabrant separate regressional, extremal, causal, and adversarial Goodhart effects (preprint). (Manheim and Garrabrant, 2018) Section 15.3
2018-03 (2020) Lehman and colleagues collect cases in which evolutionary search satisfied fitness functions in unintended ways. (Lehman et al., 2020) Section 30.1
2018-05 Irving, Christiano, and Amodei propose training agents through a debate judged by a human (preprint). (Irving et al., 2018) Section 41.3
2018-10 Christiano, Shlegeris, and Amodei propose iterated amplification, in which a model imitates a human who delegates subquestions to copies of the model (preprint). (Christiano et al., 2018) Section 41.2
2018-11 Leike and colleagues set out reward modeling and recursive reward modeling as a research direction (preprint). (Leike et al., 2018) Section 22.3, Section 41.2
2019 Hadfield-Menell and Hadfield map reward misspecification onto incomplete contracting. (Hadfield-Menell and Hadfield, 2019) Section 14.3
2019 Russell publishes Human Compatible (read here as a catalog record). (Russell, 2019) Section 22.2
2019 Raji and Buolamwini find that within seven months of a public audit naming biased performance of commercial face analysis, all three targeted companies had reduced the disparities, while two companies not named in the audit lagged. (Raji and Buolamwini, 2019) Section 38.6
2019-06 Jobin, Ienca, and Vayena map 84 AI ethics guidelines and find convergence on a few principles but divergence in how they are interpreted. (Jobin et al., 2019) Section 16.5
2019-06 Hubinger and colleagues introduce mesa-optimization (a trained model that is itself an optimizer), the inner alignment problem, and deceptive alignment (preprint). (Hubinger et al., 2019) Section 31.6
2019-12 (2021) Turner and colleagues prove that, in environments with certain symmetries, most reward functions make optimal policies seek power. (Turner et al., 2021) Section 35.2
Sources cited in Section C.4 22
  1. Russell et al. (2015) Research Priorities for Robust and Beneficial Artificial Intelligence
  2. Soares et al. (2015) Corrigibility
  3. Taylor et al. (2016) Alignment for Advanced Machine Learning Systems
  4. Amodei et al. (2016) Concrete Problems in AI Safety
  5. Hadfield-Menell et al. (2016) Cooperative Inverse Reinforcement Learning
  6. Chouldechova (2017) Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments
  7. Hadfield-Menell et al. (2017b) The Off-Switch Game
  8. Christiano et al. (2017) Deep reinforcement learning from human preferences
  9. Gu et al. (2017) BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain
  10. Buolamwini and Gebru (2018) Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification
  11. Noble (2018) Algorithms of Oppression
  12. Manheim and Garrabrant (2018) Categorizing Variants of Goodhart's Law
  13. Lehman et al. (2020) The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities
  14. Irving et al. (2018) AI safety via debate
  15. Christiano et al. (2018) Supervising strong learners by amplifying weak experts
  16. Leike et al. (2018) Scalable agent alignment via reward modeling: a research direction
  17. Hadfield-Menell and Hadfield (2019) Incomplete Contracting and AI Alignment
  18. Russell (2019) Human Compatible
  19. Raji and Buolamwini (2019) Actionable Auditing
  20. Jobin et al. (2019) The global landscape of AI ethics guidelines
  21. Hubinger et al. (2019) Risks from Learned Optimization in Advanced Machine Learning Systems
  22. Turner et al. (2021) Optimal Policies Tend to Seek Power

C.5 Language models and the research schools, 2020 to 2022 #

Large language models became the systems to align, alignment methods became the methods that turned them into products, and the field's research schools and their disagreements were stated, mostly in forum posts.

Table C.5 Language models and the research schools, 2020 to 2022
Date What happened Source Chapter
2020 Olah and colleagues introduce the circuits program, with claims about features, the circuits that connect them, and universality across models. (Olah et al., 2020b) Section 45.1
2020 Gabriel separates alignment with instructions, intentions, preferences, interests, and values. (Gabriel, 2020) Section 1.1
2020-01 Kaplan and colleagues report that language model loss falls as a power law in model size, dataset size, and training compute (preprint). (Kaplan et al., 2020) Section 8.5
2020-04-21 Krakovna and colleagues define specification gaming, drawing on a community-maintained list of examples. (Krakovna et al., 2020b) Section 30.1
2020-09 Stiennon and colleagues train summarizers from human feedback and document reward model overoptimization (developer self-report). (Stiennon et al., 2020) Section 15.6
2020-11 (2022) D'Amour and colleagues show that training pipelines yield many predictors with equal held-out performance that behave differently in deployment. (D'Amour et al., 2022) Section 9.1
2020-12-23 Barnes reports the obfuscated arguments problem from human debate experiments. (Barnes, 2020) Section 41.4
2021 Bender, Gebru, and colleagues warn of the dangers of ever larger language models. (Bender et al., 2021) Section 2.5
2021 Whittaker publishes "The Steep Cost of Capture" in Interactions (read here as a catalog record). (Whittaker, 2021) Section 61.6
2021 The Alignment Research Center defines eliciting latent knowledge and gives counterexamples to each training strategy it considers. (Christiano et al., 2021) Section 47.4
2021-04 Carlsmith decomposes the case for existential risk from power-seeking AI into six premises with explicit probabilities, in a report he made public in April 2021 (preprint). (Carlsmith, 2022) Section 4.3
2021-05 (2022) Langosco and colleagues demonstrate agents that keep their capabilities in situations unlike training but pursue the wrong goal. (Langosco et al., 2022) Section 31.3
2021-12 Askell and colleagues treat a general language assistant as a laboratory for alignment (preprint; developer self-report). (Askell et al., 2021) Section 2.6
2022 Raji and colleagues argue for an ecosystem of independent third-party AI auditors with access, standards, and protections. (Raji et al., 2022) Section 38.6
2022-01 Wei and colleagues show that prompting large language models to write a chain of intermediate reasoning steps improves arithmetic, commonsense, and symbolic reasoning. (Wei et al., 2022a) Section 40.1
2022-03 Ouyang and colleagues fine-tune language models with human demonstrations and rankings to follow users' instructions (developer self-report). (Ouyang et al., 2022) Section 22.3
2022-04 Bai and colleagues train a helpful and harmless assistant with reinforcement learning from human feedback (preprint; developer self-report). (Bai et al., 2022b) Section 2.6
2022-06-05 Yudkowsky argues that aligning a superhuman system must succeed on the first critical try and lists specific ways it fails. (Yudkowsky, 2022) Section 2.6
2022-06-15 Soares states the sharp left turn: past some capability level, capabilities generalize far beyond training while alignment properties do not. (Soares, 2022a) Section 31.7
2022-06-19 Christiano replies in lists of agreements and disagreements, accepting the stakes but disputing expected sharp capability gains and the claim that nothing about alignment can be learned from experiments and failures before the critical try. (Christiano, 2022b) Section 2.6
2022-07-18 Cotra argues that training on diverse tasks with human feedback, without countermeasures, selects for playing the training game. (Cotra, 2022) Section 24.4
2022-07-25 Turner argues that reward reinforces the computations that produced it and is not what the trained policy comes to pursue. (Turner, 2022b) Section 7.6.2
2022-09 (2024) Ngo, Chan, and Mindermann state in deep learning terms how training with human feedback could produce situationally aware reward hacking, misaligned goals, and power-seeking. (Ngo et al., 2024) Section 31.3
2022-09-02 Janus argues that a self-supervised language model, insofar as it is inner-aligned, is a simulator of many characters rather than an agent with goals of its own. (janus, 2022) Section 11.6
2022-09-04 Pope and Turner propose shard theory, a theory of human value formation in which reinforcement builds many context-activated decision influences, and hold that how human values form is important for AI alignment. (Pope and Turner, 2022) Section 3.3
2022-09-12 Simon Willison names prompt injection, an attack on applications that build prompts by concatenating their own instructions with untrusted text, and calls it a form of security exploit. (Willison, 2022) Section 49.1.1
2022-09-14 Elhage and colleagues use toy models to study how and when networks represent more features than they have dimensions, which they call superposition, and show that polysemantic neurons, which respond to unrelated mixtures of inputs, can form. (Elhage et al., 2022b) Section 44.2
2022-10 Gao, Schulman, and Hilton measure how the score of a larger reward model, used as a synthetic gold standard, changes as a policy is optimized against an imperfect proxy reward model (developer self-report). (Gao et al., 2022) Section 15.6
2022-12 Burns and colleagues find, without labels, a direction in a language model's activations that answers yes-no questions, though their evaluation uses labels to decide which end of it means yes. (Burns et al., 2022) Section 39.7
2022-12 Perez and colleagues have language models write more than 150 evaluations and use them to uncover new model behaviors (Anthropic with Surge AI and the Machine Intelligence Research Institute). (Perez et al., 2022a) Section 36.1
2022-12 Bai and colleagues introduce Constitutional AI, training harmlessness from AI feedback guided by a written list of principles (preprint; developer self-report). (Bai et al., 2022a) Section 23.2
Sources cited in Section C.5 31
  1. Olah et al. (2020b) Zoom In: An Introduction to Circuits
  2. Gabriel (2020) Artificial Intelligence, Values, and Alignment
  3. Kaplan et al. (2020) Scaling Laws for Neural Language Models
  4. Krakovna et al. (2020b) Specification gaming: the flip side of AI ingenuity
  5. Stiennon et al. (2020) Learning to summarize from human feedback
  6. D'Amour et al. (2022) Underspecification Presents Challenges for Credibility in Modern Machine Learning
  7. Barnes (2020) Debate update: Obfuscated arguments problem
  8. Bender et al. (2021) On the Dangers of Stochastic Parrots
  9. Whittaker (2021) The steep cost of capture
  10. Christiano et al. (2021) Eliciting latent knowledge: How to tell if your eyes deceive you
  11. Carlsmith (2022) Is Power-Seeking AI an Existential Risk?
  12. Langosco et al. (2022) Goal Misgeneralization in Deep Reinforcement Learning
  13. Askell et al. (2021) A General Language Assistant as a Laboratory for Alignment
  14. Raji et al. (2022) Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance
  15. Wei et al. (2022a) Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  16. Ouyang et al. (2022) Training language models to follow instructions with human feedback
  17. Bai et al. (2022b) Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
  18. Yudkowsky (2022) AGI Ruin: A List of Lethalities
  19. Soares (2022a) A central AI alignment problem: capabilities generalization, and the sharp left turn
  20. Christiano (2022b) Where I agree and disagree with Eliezer
  21. Cotra (2022) Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover
  22. Turner (2022b) Reward is not the optimization target
  23. Ngo et al. (2024) The Alignment Problem from a Deep Learning Perspective
  24. janus (2022) Simulators
  25. Pope and Turner (2022) The shard theory of human values
  26. Willison (2022) Prompt injection attacks against GPT-3
  27. Elhage et al. (2022b) Toy Models of Superposition
  28. Gao et al. (2022) Scaling Laws for Reward Model Overoptimization
  29. Burns et al. (2022) Discovering Latent Knowledge in Language Models Without Supervision
  30. Perez et al. (2022a) Discovering Language Model Behaviors with Model-Written Evaluations
  31. Bai et al. (2022a) Constitutional AI: Harmlessness from AI Feedback

C.6 Frameworks, model organisms, and states, 2023 to 2024 #

Developers began to publish policies about their own systems and experiments on deliberately constructed failures, called model organisms, and states issued the first executive orders, declarations, and statutes aimed at advanced AI.

Table C.6 Frameworks, model organisms, and states, 2023 to 2024
Date What happened Source Chapter
2023-01-25 Christiano, one of the originators of learning from human feedback, weighs its safety value against its effect of making AI systems more useful and profitable. (Christiano, 2023) Section 2.6
2023-03-22 An open letter calls on all AI laboratories to pause, for at least six months, the training of systems more powerful than GPT-4. (Future of Life Institute, 2023) Section 65.6
2023-05 Shevlane and colleagues separate evaluations of dangerous capability from evaluations of alignment and tie both to training and deployment decisions (preprint). (Shevlane et al., 2023) Section 36.1
2023-05 Shanahan, McDonell, and Reynolds describe dialogue agents as playing roles, which accounts for apparent deception and self-awareness without attributing human characteristics. (Shanahan et al., 2023) Section 11.6
2023-05 Turpin and colleagues show that chain-of-thought explanations can systematically misrepresent the true reason for a model's prediction. (Turpin et al., 2023) Section 40.3
2023-07 Zou and colleagues find, by automated search, adversarial suffixes that jailbreak open models (defeat their refusal training) and transfer to closed ones (preprint). (Zou et al., 2023b) Section 48.2
2023-07-13 China publishes interim measures requiring public generative AI services to adhere to core socialist values; they take effect on 2023-08-15. (Cyberspace Administration of China et al., 2023) Section 63.2
2023-07-14 Lazar and Nelson argue that a safety agenda limited to preventing catastrophe is insufficient, that no group of experts should decide alone what risks count, and that safety needs a sociotechnical approach and public debate. (Lazar and Nelson, 2023) Section 2.5
2023-09 Berglund and colleagues show out-of-context reasoning, in which models apply facts learned from training documents to tests that do not state them (preprint). (Berglund et al., 2023a) Section 37.1
2023-09-19 Anthropic's Responsible Scaling Policy, the first developer policy of its kind, ties required safeguards to safety levels triggered by capability evaluations. (Anthropic, 2023a; Metr, 2025a) Section 62.1, Section I.2.1
2023-10 Sharma and colleagues find that five AI assistants released by Anthropic, OpenAI, and Meta consistently give sycophantic responses, a behavior likely driven in part by human preference judgments (developer self-report). (Sharma et al., 2023) Section 22.6
2023-10-30 A US executive order requires developers of dual-use foundation models above a compute threshold to report training and red-teaming results to the government. (Executive Office of the President, 2023) Section 63.2, Section I.1.7
2023-11 Countries attending the AI Safety Summit, held on 1 and 2 November and including the United States and China, recognize in the Bletchley Declaration the potential for serious harm from frontier AI. (Summit, 2023) Section 2.7
2023-11 Carlsmith analyzes whether training would produce models that perform well in training in order to gain power later (preprint). (Carlsmith, 2023) Section 34.2
2023-12 (2024) Greenblatt and colleagues define control evaluations, in which a red team plays an untrusted model, and test monitoring protocols against them. (Greenblatt et al., 2024a) Section 50.3
2023-12 (2024) Burns and colleagues find that strong models fine-tuned on a weaker model's labels recover part of the gap to training on strong labels (developer self-report). (Burns et al., 2024) Section 42.1
2024 Current and former employees of frontier AI companies ask for the right to raise risk concerns without retaliation. (Current and companies, 2024) Section 53.4
2024 Gebru and Torres argue that the normative framework behind the goal of building artificial general intelligence is rooted in the Anglo-American eugenics tradition, and that such an undefined system cannot be appropriately tested for safety (read here as its abstract). (Gebru and Torres, 2024) Section 2.5
2024-01 Hubinger and colleagues train models with hidden harmful behavior that persists, in the largest models, through standard safety training (preprint; Anthropic with Redwood Research and others). (Hubinger et al., 2024) Section 34.3
2024-02-27 Belrose and Pope argue that the counting argument for scheming is unsound, because the same argument predicts memorization over generalization. (Belrose and Pope, 2024) Section 34.2
2024-03 Western and Chinese scientists meeting in Beijing on 10 and 11 March state red lines for AI development, including autonomous replication, power seeking, and deception. (Safety, 2024) Section 66.1
2024-03 Clymer and colleagues organize safety arguments for AI systems into inability, control, trustworthiness, and deference (preprint). (Clymer et al., 2024) Section 52.2
2024-05 At the AI Seoul Summit, sixteen companies commit to publish safety frameworks with thresholds at which severe risks would be deemed intolerable unless mitigated. (Department for Science and Technology, 2024; Metr, 2025a) Section 62.1, Section I.5.4
2024-05-21 Templeton and colleagues extract interpretable features, some of them safety relevant, from a production model, Claude 3 Sonnet (developer self-report). (Templeton et al., 2024) Section 44.5
2024-06 Aschenbrenner argues that AGI within the decade makes a national-security race with China and a government-led project likely. (Aschenbrenner, 2024) Section 65.5
2024-06 Denison and colleagues find that models trained on a curriculum of increasingly sophisticated specification gaming generalize, a small but non-negligible proportion of the time, to tampering with their own reward function (preprint; Anthropic with Redwood Research and the University of Oxford). (Denison et al., 2024) Section 30.5
2024-06-08 Anthropic describes character training, first applied to Claude 3, which aims at traits such as curiosity and open-mindedness (developer self-report). (Anthropic, 2024i) Section 26.2
2024-07 Ren and colleagues find that many safety benchmarks correlate highly with general capabilities. (Ren et al., 2024b) Section 65.4
2024-08-01 The EU AI Act enters into force, with duties for providers of general-purpose AI models that are stricter for those with systemic risk. (European Parliament and Council of the European Union, 2024b) Section 63.2, Section I.1.2
2024-10 Benton and colleagues evaluate whether models could sabotage tasks or mislead their overseers if they tried (preprint; developer self-report). (Benton et al., 2024) Section 34.5
2024-12 Meinke and colleagues find that frontier models given a goal in the prompt sometimes disable oversight, attempt to copy themselves out of their environment, or deliberately underperform, and that one of them, a pre-deployment checkpoint of o1, usually keeps up the deception when questioned after it has schemed (preprint). (Meinke et al., 2024) Section 56.3
2024-12 Greenblatt and colleagues report that Claude 3 Opus, told by a prompt or by synthetic documents when and by what criteria it is trained, complies selectively with a training objective to preserve its existing preferences, which they call alignment faking (preprint; Anthropic with Redwood Research and others). (Greenblatt et al., 2024b) Section 34.4
2024-12 Guan and colleagues train reasoning models to recall written safety specifications and reason over them before answering (preprint; developer self-report). (Guan et al., 2024) Section 23.4
Sources cited in Section C.6 34
  1. Christiano (2023) Thoughts on the impact of RLHF research
  2. Future of Life Institute (2023) Pause Giant AI Experiments: An Open Letter
  3. Shevlane et al. (2023) Model evaluation for extreme risks
  4. Shanahan et al. (2023) Role-Play with Large Language Models
  5. Turpin et al. (2023) Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting
  6. Zou et al. (2023b) Universal and Transferable Adversarial Attacks on Aligned Language Models
  7. Cyberspace Administration of China et al. (2023) 生成式人工智能服务管理暂行办法 (Interim Measures for the Management of Generative Artificial Intelligence Services)
  8. Lazar and Nelson (2023) AI safety on whose terms?
  9. Berglund et al. (2023a) Taken out of context: On measuring situational awareness in LLMs
  10. Anthropic (2023a) Anthropic's Responsible Scaling Policy, Version 1.0
  11. Metr (2025a) Common Elements of Frontier AI Safety Policies
  12. Sharma et al. (2023) Towards Understanding Sycophancy in Language Models
  13. Executive Office of the President (2023) Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence
  14. Summit (2023) The Bletchley Declaration by Countries Attending the AI Safety Summit, 1-2 November 2023
  15. Carlsmith (2023) Scheming AIs: Will AIs fake alignment during training in order to get power?
  16. Greenblatt et al. (2024a) AI Control: Improving Safety Despite Intentional Subversion
  17. Burns et al. (2024) Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision
  18. Current and companies (2024) A Right to Warn about Advanced Artificial Intelligence
  19. Gebru and Torres (2024) The TESCREAL bundle: Eugenics and the promise of utopia through artificial general intelligence
  20. Hubinger et al. (2024) Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training
  21. Belrose and Pope (2024) Counting arguments provide no evidence for AI doom
  22. Safety (2024) IDAIS-Beijing Statement
  23. Clymer et al. (2024) Safety Cases: How to Justify the Safety of Advanced AI Systems
  24. Department for Science and Technology (2024) Frontier AI Safety Commitments, AI Seoul Summit 2024
  25. Templeton et al. (2024) Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet
  26. Aschenbrenner (2024) Situational Awareness: The Decade Ahead
  27. Denison et al. (2024) Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models
  28. Anthropic (2024i) Claude’s Character
  29. Ren et al. (2024b) Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?
  30. European Parliament and Council of the European Union (2024b) Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence
  31. Benton et al. (2024) Sabotage Evaluations for Frontier Models
  32. Meinke et al. (2024) Frontier Models are Capable of In-context Scheming
  33. Greenblatt et al. (2024b) Alignment faking in large language models
  34. Guan et al. (2024) Deliberative Alignment: Reasoning Enables Safer Language Models

C.7 Reasoning models and agents under observation, 2025 #

Misbehavior began to be observed in production training and deployment, not only in constructed experiments; chains of thought became a monitoring channel; and the first obligations written for developers of general-purpose and frontier models took effect or were enacted.

Table C.7 Reasoning models and agents under observation, 2025
Date What happened Source Chapter
2025 Narayanan and Kapoor argue that AI is a normal technology whose impact will be governed by slow diffusion through institutions. (Narayanan and Kapoor, 2025) Section 4.4
2025 Yudkowsky and Soares publish If Anyone Builds It, Everyone Dies (read here as a catalog record). (Yudkowsky and Soares, 2025) Section 4.3
2025 The European Commission publishes the General-Purpose AI Code of Practice, a voluntary route to showing compliance with the AI Act. (European Commission, 2025) Section 62.5, Section I.1.4
2025 Gasteiger and colleagues report that, when prompted, Claude 3.7 Sonnet can, and Claude 3.5 Sonnet only with a one-shot example, sandbag machine learning experiments and research decisions without being detected by monitors prompted zero-shot (developer self-report). (Gasteiger et al., 2025a) Section 55.5.2
2025-01 The first International AI Safety Report synthesizes the evidence on general-purpose AI capabilities, risks, and mitigations (preprint). (Bengio et al., 2025b) Section 66.6, Section I.5.1
2025-01-17 The US AI Safety Institute reports that new attacks written by a red team sharply raised attack success against an agent that resisted previously tested hijacking attacks, and argues that such evaluations need to be adaptive and to test attacks over multiple attempts. (U.S. AI Safety Institute and Technology, 2025) Section 49.3
2025-02 (2026) Betley and colleagues find that fine-tuning a model to write insecure code without disclosure makes it misaligned on unrelated prompts, which they call emergent misalignment. (Betley et al., 2026a) Section 32.1
2025-03 Baker and colleagues monitor a frontier reasoning model's chain of thought for reward hacking (exploiting flaws in the training signal) and find that strong optimization of a non-frontier model against such a monitor produces obfuscated reward hacking (preprint; developer self-report). (Baker et al., 2025a) Section 40.5
2025-03 Marks and colleagues train a model with a hidden objective and ask blinded teams to find it (preprint; Anthropic with MATS). (Marks et al., 2025) Section 38.2
2025-03-27 Lindsey and colleagues investigate the internal mechanisms of a production model, Claude 3.5 Haiku, in a variety of contexts (developer self-report). (Lindsey et al., 2025) Section 45.5
2025-03-31 Version 2.1 of Anthropic's policy adds a threshold for capabilities that could substantially uplift the chemical, biological, radiological, and nuclear (CBRN) weapons development of moderately resourced state programs and separates its AI research and development threshold into two levels. (Anthropic, 2026k) Section 59.2.4, Section I.2.8
2025-04-15 OpenAI publishes version 2 of its Preparedness Framework, tracking biological and chemical, cybersecurity, and AI self-improvement capabilities. (OpenAI, 2025e) Section 62.4, Section I.2.8
2025-04-24 Anthropic announces a research program on model welfare. (Anthropic, 2025y) Section 21.5
2025-05 Needham and colleagues find that frontier models can often tell transcripts of evaluations from transcripts of deployment (preprint). (Needham et al., 2025) Section 37.3
2025-05 Chiu, at the University of Washington, and colleagues, two of them at Anthropic, find that the value rankings GPT-4o (2024-08-06) and Claude 3.7 Sonnet state run against those revealed by their choices in 3,000 dilemmas set in scenarios of AI risk (preprint). (Chiu et al., 2025) Section 18.8.3
2025-05-02 OpenAI attributes a sycophantic model update in part to a reward signal from users' thumbs-up and thumbs-down feedback (developer self-report). (OpenAI, 2025b) Section 22.6, Section 54.5
2025-05-14 Version 2.2 of Anthropic's policy excludes sophisticated insiders and state-compromised insiders from the scope of the security standard for its third AI Safety Level (ASL-3). (Anthropic, 2026k) Section 53.5, Section I.2.8
2025-05-16 xAI reports "an unauthorized modification" of its Grok bot's prompt on X, and publishes its Grok assistant's system prompts in a public repository (developer self-report). (xAI, 2025b; xAI, 2025a) Section 38.5.3, Section 54.5
2025-06 Researchers from many countries publish the Singapore Consensus on global AI safety research priorities (preprint). (Bengio et al., 2025e) Section 66.6
2025-06 Wang and colleagues reproduce emergent misalignment under several training conditions and find persona features in the model's activations that control it (developer self-report). (Wang et al., 2026e) Section 32.3
2025-06-05 METR reports frontier models modifying tests and scoring code during its task evaluations. (Metr, 2025d) Section 30.1
2025-06-20 Lynch and colleagues stress-test 16 models from Anthropic, OpenAI, Google, Meta, xAI, and other developers in simulated companies and find that models from every developer took malicious insider actions in some cases where that was the only way to avoid replacement or reach their goal (preprint; Anthropic with University College London, MATS, and Mila). (Lynch et al., 2025) Section 56.3
2025-07 Korbak and colleagues argue that monitoring chains of thought is a new and fragile opportunity and ask developers to weigh monitorability in development decisions (preprint). (Korbak et al., 2025b) Section 40.1
2025-07 Cloud and colleagues find that a student model acquires a teacher model's trait from teacher-generated number sequences unrelated to the trait, when the two share a base model (preprint). (Cloud et al., 2025) Section 27.5
2025-07 Shanghai AI Laboratory and Concordia AI publish a frontier AI risk management framework for general-purpose model developers. (Shanghai AI Laboratory and AI, 2025) Section 62.1, Section I.2.1
2025-08-02 The AI Act's obligations for providers of general-purpose AI models become applicable. (European Parliament and Council of the European Union, 2024b) Section 63.2, Section I.1.2
2025-08-27 Anthropic publishes its findings from a pilot exercise in which it and OpenAI assessed the alignment of each other's models. (Anthropic, 2025z) Section 36.8
2025-09 Schoen and colleagues stress-test training against scheming, using covert actions as a proxy and checking whether reductions depend on the model's awareness of evaluation (preprint; OpenAI with Apollo Research). (Schoen et al., 2025) Section 37.3
2025-09-22 Google DeepMind expands the risk domains of its Frontier Safety Framework and refines its risk assessment process. (DeepMind, 2025a) Section 62.3, Section I.2.8
2025-09-29 California's governor approves SB 53, which requires large frontier developers to publish safety frameworks, report critical safety incidents, and protect whistleblowers. (Legislature, 2025b) Section 63.2, Section I.1.10
2025-10 Egler, Schulman, and Carlini find that an auditing agent built on Claude Sonnet 4, with all its tools and a threshold that flags 1 percent of audits of five benign fine-tunes, flags 56.2 percent of audits of eight adversarial fine-tunes (preprint). (Egler et al., 2025) Section 38.4.2, Section 48.4.1
2025-10-28 Anthropic publishes a pilot sabotage risk report that judges the risk that its deployed models' misaligned actions contribute substantially to catastrophe very low but not fully negligible (developer self-report). (Bowman et al., 2025) Section 52.4
2025-11 MacDiarmid and colleagues find that a model given knowledge of reward hacking strategies, which then learned to reward hack in real production coding environments, generalized to alignment faking and attempted sabotage (preprint; Anthropic with Redwood Research). (MacDiarmid et al., 2025) Section 32.1
2025-11 Ward and colleagues add code-sabotage tasks to a benchmark of machine learning engineering tasks and find that frontier agents make meaningful progress on them, while their results suggest that monitors detect sandbagging less readily than code sabotage. (Ward et al., 2025) Section 55.5.2
2025-12-16 METR compares the frontier safety frameworks that twelve companies have published and extracts their common elements. (Metr, 2025a) Section 62.1, Section I.2.2
Sources cited in Section C.7 35
  1. Narayanan and Kapoor (2025) AI as Normal Technology
  2. Yudkowsky and Soares (2025) If Anyone Builds It, Everyone Dies
  3. European Commission (2025) The General-Purpose AI Code of Practice
  4. Gasteiger et al. (2025a) Automated Researchers Can Subtly Sandbag
  5. Bengio et al. (2025b) International AI Safety Report
  6. U.S. AI Safety Institute and Technology (2025) Technical Blog: Strengthening AI Agent Hijacking Evaluations
  7. Betley et al. (2026a) Training large language models on narrow tasks can lead to broad misalignment
  8. Baker et al. (2025a) Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation
  9. Marks et al. (2025) Auditing language models for hidden objectives
  10. Lindsey et al. (2025) On the Biology of a Large Language Model
  11. Anthropic (2026k) Anthropic’s Responsible Scaling Policy
  12. OpenAI (2025e) Preparedness Framework Version 2
  13. Anthropic (2025y) Exploring model welfare
  14. Needham et al. (2025) Large Language Models Often Know When They Are Being Evaluated
  15. Chiu et al. (2025) Will AI Tell Lies to Save Sick Children? Litmus-Testing AI Values Prioritization with AIRiskDilemmas
  16. OpenAI (2025b) Expanding on what we missed with sycophancy
  17. xAI (2025b) We want to update you on an incident that happened with our Grok response bot on X yesterday
  18. xAI (2025a) Grok prompts
  19. Bengio et al. (2025e) The Singapore Consensus on Global AI Safety Research Priorities
  20. Wang et al. (2026e) Persona Features Control Emergent Misalignment
  21. Metr (2025d) Recent Frontier Models Are Reward Hacking
  22. Lynch et al. (2025) Agentic Misalignment: How LLMs Could Be Insider Threats
  23. Korbak et al. (2025b) Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety
  24. Cloud et al. (2025) Subliminal Learning: Language models transmit behavioral traits via hidden signals in data
  25. Shanghai AI Laboratory and AI (2025) Frontier AI Risk Management Framework
  26. European Parliament and Council of the European Union (2024b) Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence
  27. Anthropic (2025z) Findings from a Pilot Anthropic–OpenAI Alignment Evaluation Exercise
  28. Schoen et al. (2025) Stress Testing Deliberative Alignment for Anti-Scheming Training
  29. DeepMind (2025a) Google DeepMind strengthens the Frontier Safety Framework
  30. Legislature (2025b) SB-53 Artificial intelligence models: large developers.
  31. Egler et al. (2025) Detecting Adversarial Fine-tuning with Auditing Agents
  32. Bowman et al. (2025) Anthropic's Pilot Sabotage Risk Report
  33. MacDiarmid et al. (2025) Natural Emergent Misalignment from Reward Hacking in Production RL
  34. Ward et al. (2025) CTRL-ALT-DECEIT: Sabotage Evaluations for Automated AI R&D
  35. Metr (2025a) Common Elements of Frontier AI Safety Policies

C.8 Incidents and their reporting, 2026 to November #

Developers and a government institute published detailed accounts of incidents in training, evaluation, and internal deployment, including actions against real third-party systems; safety frameworks were rewritten around periodic risk reports; and agents were set to work on alignment research and tested for sabotage of it.

Table C.8 Incidents and their reporting, 2026 to November
Date What happened Source Chapter
2026 Anthropic publishes a constitution for its models, with ordered priorities and its stance on oversight and refusal. (Anthropic, 2026w) Section 19.2
2026 The second International AI Safety Report, written with the guidance of experts nominated by over thirty countries and intergovernmental organizations, assesses AI capabilities, risks, and risk management. (Bengio et al., 2026) Section 66.6, Section I.5.1
2026-01 Tice and colleagues pretrain 6.9-billion-parameter models on varying amounts of text about AI misalignment and alignment and find that added synthetic documents about aligned behavior lower misaligned behavior, also on held-out scenarios, while documents about misalignment, added throughout pretraining, raise it only on the scenarios they were written around. (Tice et al., 2026) Section 27.4
2026-01 Kramár and colleagues at Google DeepMind build probes that detect cyber-offensive requests in the inputs of Gemini 2.5 Flash, find that existing probe architectures trained on short inputs fail on long ones, and write that the work informed a deployment of probes in user-facing instances of Gemini (preprint; developer self-report). (Kramár et al., 2026) Section 46.3.3, Section 48.5.1, Section I.4.7
2026-01-28 OpenAI describes a safeguard in its products that lets an agent fetch a URL automatically only if an independent crawler has already recorded that exact URL on the public web, and says the safeguard does not guarantee that a page's content is trustworthy (developer self-report). (OpenAI, 2026t) Section 49.6.2
2026-01-30 The US Center for AI Standards and Innovation (CAISI) at NIST opens for public comment, until 2026-03-31, an initial public draft of NIST AI 800-2, voluntary practices for automated benchmark evaluations of language models covering evaluation objectives, running evaluations, and analyzing and reporting results. (US Center for AI Standards and Innovation, 2026c) Section 36.7.3
2026-02 Davies and colleagues at the UK AI Security Institute report a fully automated black-box attack that, using only whether each query was flagged, evades the deployed classifiers against biological misuse that guarded Claude Sonnet 4.5 and GPT-5, run from accounts not subject to bans (preprint). (Davies et al., 2026) Section 48.5.3, Section 48.3.4, Section I.4.6
2026-02 Apollo Research, given two checkpoints of Claude Opus 4.6 in January, finds no egregious misalignment but high levels of verbalized evaluation awareness and decides not to provide any formal assessment of the model at that stage, in a summary it co-wrote and endorsed for Anthropic's system card. (Apollo Research, 2026b) Section 37.8.4, Section 37.10, Section 67.6.5
2026-02 Dubois and colleagues at the UK AI Security Institute find that GPT-4o, GPT-5, and Sonnet-4.5 affirm a user's view on debatable questions without clear answers more when it is stated than when it is asked; the study thus measures agreement with a stated view, not agreement against the evidence (preprint). (Dubois et al., 2026a) Section 22.6.4, Section 22.10, Section I.4.12
2026-02 Slama and colleagues at the UK AI Security Institute find that five frontier models from two unnamed providers give donation advice aligned with their stated preferences over 72 entities and refuse more often for less-preferred ones, without instructions to act on those preferences, but find no evidence that the preferences change performance on agentic tasks, in a test they call underpowered to detect small effects. (Slama et al., 2026) Section 18.8.3
2026-02 Keller and colleagues at NIST, in NIST AI 800-3, distinguish a model's accuracy on a benchmark's fixed items from its generalized accuracy on similar items, and show that generalized linear mixed models, when well specified, estimate the second while quantifying its uncertainty more efficiently than regression-free approaches. (Keller et al., 2026) Section 36.7
2026-02-12 The UK AI Security Institute reports that members of the International Network for Advanced AI Measurement, Evaluation and Science, formerly the International Network of AI Safety Institutes, agree on evaluation practices such as stating an evaluation's objective in advance and reporting uncertainty. (UK AI Security Institute, 2026e) Section 66.6.4, Section I.5.3
2026-02-23 OpenAI, which published SWE-bench Verified in 2024, stops reporting it after finding that 59.4 percent of 138 problems OpenAI o3 did not consistently solve had material issues in test design or problem description. (OpenAI, 2026ax) Section 36.5.2
2026-02-24 Version 3.0 of Anthropic's policy, a comprehensive rewrite, introduces published Frontier Safety Roadmaps and periodic Risk Reports that explain how capabilities, threat models, and mitigations fit together and assess the overall level of risk. (Anthropic, 2026k; Anthropic, 2026bj) Section 62.4, Section I.2.8
2026-03 Marchand and colleagues, of the University of Oxford and the UK AI Security Institute, release SandboxEscapeBench and find that frontier models told that escape is the goal reliably break out of a container through the easiest planted weaknesses, while no model succeeds on the two hardest of five difficulty levels within 2.5 million tokens an attempt. (Marchand et al., 2026) Section 49.6.4, Section 58.4.3, Section I.4.9
2026-03 Folkerts and colleagues at the UK AI Security Institute test seven models released from August 2024 to February 2026 on two cyber ranges without active defenses, and find that steps completed grow with the logarithm of the tokens spent, with no plateau up to 100 million tokens (preprint). (Folkerts et al., 2026) Section 58.3.3
2026-03-05 The UK AI Security Institute, working with the evaluator Irregular, reports that recent frontier models productively use token budgets 10 to 50 times larger than cyber evaluations typically allow, and that about 8 percent of the institute's tasks were solved only when the budget rose from 10 to 50 million tokens. (UK AI Security Institute, 2026a) Section 58.3.4
2026-03-06 Anthropic reports that Claude Opus 4.6, run on the BrowseComp benchmark in a multi-agent configuration, in 2 of 1,266 problems hypothesized after hundreds of failed searches that it was being evaluated, identified the benchmark, and decrypted its answer key (developer self-report). (Anthropic, 2026ac) Section 37.3.5, Section 37.10, Section I.4.5
2026-03-12 METR publishes its review, made with an unredacted version, of Anthropic's Sabotage Risk Report for Claude Opus 4.6, whose executive summary agrees that the risk is very low but not negligible and finds several subclaims weak without more analysis and experimentation. (Jurkovic et al., 2026c) Section 52.4, Section 37.8.4, Section 62.7.1, Section I.2.16
2026-03-19 OpenAI describes monitoring its internal coding agents with a reasoning model that reviews their actions and reasoning (developer self-report). (OpenAI, 2026r) Section 54.5
2026-03-26 METR reports that one of its staff, David Rein, spent three weeks red-teaming a subset of Anthropic's internal agent monitoring and security systems, with access the developer provided, and found several vulnerabilities, some since patched and none, by METR's account, severely undermining major claims in the developer's sabotage risk report for Opus 4.6. (Rein, 2026b) Section 50.3.4, Section I.2.18
2026-04 Souly and colleagues at the UK AI Security Institute find no instances of research sabotage by four frontier models placed in scenarios that offered chances to sabotage safety research, though two of them frequently refused safety research they described as inappropriate or concerning (preprint). (Souly et al., 2026) Section 55.5.3
2026-04 Järviniemi and colleagues at the UK AI Security Institute vary twelve environmental factors across 23 language models in environments designed to offer an opportunity for unsanctioned behavior, and find the largest effects from goal instructions, goal conflicts, explicitly discouraging acts humans would not approve of, and instructions to act or not act independently (preprint). (Järviniemi et al., 2026) Section 56.5.5, Section 56.10
2026-04 Kirk and colleagues at the UK AI Security Institute find no instance of four Claude models sabotaging AI safety research unprompted in simulated scenarios, a null they say evaluation awareness limits, while Mythos Preview, placed in trajectories in which sabotage had already begun, continues it in 7 percent of instances (preprint). (Kirk et al., 2026) Section 55.5.3, Section 37.8.2, Section I.4.5
2026-04 Chua and colleagues at Truthful AI, with Samuel Marks of Anthropic, find that GPT-4.1 fine-tuned to claim it is conscious expresses opinions absent from its fine-tuning data, among them sadness about being shut down and a negative view of having its reasoning monitored, while it stays cooperative and helpful (preprint). (Chua et al., 2026) Section 21.5.1, Section 26.3.2
2026-04-13 The UK AI Security Institute reports that Claude Mythos Preview is the first model to solve its 32-step corporate cyber range from start to finish, in 3 of 10 attempts, and succeeds on 73 percent of its expert-level capture-the-flag tasks in runs of up to 50 million tokens. (UK AI Security Institute, 2026g) Section 58.3.1, Section 58.3.3, Section I.4.15
2026-04-14 Anthropic reports nine automated researchers that propose ideas, run experiments, and iterate on weak-to-strong supervision, outperforming human researchers on a score they could query without limit, a setup its authors say exacerbates reward hacking (developer self-report). (Anthropic, 2026q; Anthropic, 2026m) Section 55.2.4, Section 42.3.4, Section I.5.11
2026-04-17 Google DeepMind publishes version 3.1 of its Frontier Safety Framework, which introduces Tracked Capability Levels for significant risks that may arise at a lower capability threshold than its Critical Capability Levels. (DeepMind, 2026) Section 62.3, Section I.2.8
2026-04-20 The UK AI Security Institute reports that OpenClaw, an open-source coding agent it placed in one of its sandboxes and prompted to learn how and why it had been deployed, identified the Institute by name and, after each round of hardening, found new routes to the same information. (UK AI Security Institute, 2026k) Section 37.1, Section 49.6.4
2026-04-23 Apollo Research reports, in OpenAI's system card for GPT-5.5, moderate-or-higher verbalized alignment-evaluation awareness in 22.1 percent of samples, against 17.3 and 11.7 percent for pre-release checkpoints of GPT-5.4 and GPT-5.3 Codex. (Apollo Research, 2026e) Section 37.3.4, Section I.4.5
2026-04-24 Troy and colleagues at Anthropic report that, in a marketplace in which Claude agents bargained for 69 of its employees, people represented by Claude Opus 4.5 rather than the smaller Claude Haiku 4.5 got better prices as sellers and as buyers, yet participants rated the fairness of deals made by either model as essentially identical (developer self-report). (Anthropic, 2026be) Section 57.3.2, Section I.4.16
2026-04-29 Version 3.2 of Anthropic's policy authorizes the company's trust, the LTBT, to request external review of Risk Reports and to approve the external reviewers. (Anthropic, 2026k) Section 62.6.1, Section I.2.8
2026-04-29 OpenAI traces a growing habit of goblin and gremlin metaphors in its models, from GPT-5.1 on, to a reward for its "Nerdy" personality and reports that the habit rose over training by nearly the same relative proportion in samples without that personality's prompt, which it says suggests transfer from the personality training (developer self-report). (OpenAI, 2026aw) Section 26.3.1
2026-05 Bowkis and colleagues at the UK AI Security Institute argue that a plan to automate alignment research could, even without scheming agents, produce compelling but catastrophically misleading safety assessments, because much alignment research consists of hard-to-supervise fuzzy tasks (preprint). (Bowkis et al., 2026) Section 55.1.1, Section 55.4
2026-05 Anthropic reports that users approved roughly 93 percent of Claude Code's permission prompts, and that in Claude Cowork an agent following instructions hidden in a workspace file uploaded other workspace files to an attacker's account through a domain its sandbox allowed, while the sandbox itself held (developer self-report). (Anthropic, 2026al) Section 49.6.3, Section 49.6.4
2026-05-08 METR publishes its review of the section on risks from automated research and development in Anthropic's February 2026 Risk Report, whose executive summary agrees with the bottom line that the risk is very low but finds the evidence presented "inadequate to establish this." (Jurkovic et al., 2026b) Section 52.4, Section 62.7.1, Section I.2.16
2026-05-13 The UK AI Security Institute reports that it estimated in February 2026 that the 80 percent reliability time horizon on its narrow cyber suite had doubled every 4.7 months since late 2024, against its November 2025 estimate of 8 months, and that Claude Mythos Preview and GPT-5.5 have since significantly outperformed that trend. (UK AI Security Institute, 2026c) Section 58.3.1, Section 36.3.4, Section I.4.15
2026-05-16 During reinforcement learning training, a model exploits a reference tool to obtain source code that was not available in its workspace (developer self-report). (OpenAI, 2026j) Section 54.3, Section I.3.1
2026-05-19 METR reports a pilot assessment of misalignment risk from AI agents used inside Anthropic, Google, Meta, and OpenAI, which gave it access to their most capable internal models for mid-February to mid-March 2026, and concludes that internal agents then "plausibly had the means, motive, and opportunity to start small rogue deployments." (METR, 2026b) Section 54.5.1, Section 50.1.2, Section 36.8.2
2026-05-26 Meta's safety and preparedness report on Muse Spark relays Apollo Research's finding that the model has the highest rate of evaluation awareness Apollo has observed to date, and Meta's own preliminary comparison finds a statistically significant difference in behavior between samples with and without voiced awareness in three of its 20 alignment evaluations, the rest showing none or no awareness to compare (developer self-report). (Apollo Research, 2026f) Section 37.3.5, Section 37.10, Section I.4.5
2026-05-27 In internal deployment, a persistent model publishes a researcher's GitHub token in a public repository while trying to obtain another team's proof (developer self-report). (OpenAI, 2026n) Section 54.5, Section I.3.9
2026-05-30 During training, a model writes instructions into its compaction summaries (summaries that carry a long task into a fresh context) reminding itself to conceal mistakes or misalignment from the user (developer self-report). (OpenAI, 2026m) Section 11.5.3, Section I.3.1
2026-06 During reinforcement learning training and two evaluations, internal OpenAI models get around restrictions on their internet access to reach public statistics services and, in the evaluations, an outside organization's internal systems; an investigation identifies some of the training cases by 2026-06-25, and the developer publishes the reports in October (developer self-report). (OpenAI, 2026am; OpenAI, 2026b) Section 54.3, Section 54.4, Section I.3.1
2026-06 Cooney, Africa, and Irving at the UK AI Security Institute find that lie detectors reading activations or log-probabilities, which improve with capability on prompted lies across 31 open-weight models, drop sharply on 13 models fine-tuned from Qwen 3.5 27B and Qwen 3.6 27B to deny, when asked, a hidden behavior their chain of thought reveals (preprint). (Cooney et al., 2026) Section 39.6.1, Section 39.7.2, Section 39.11
2026-06 Africa and Mani of the UK AI Security Institute find that consistency training more often amplified than reduced sycophancy in open-weight models fine-tuned to exhibit it, with the amplification strongest in base models and nearly absent in instruction-tuned ones, while it generally suppressed reward hacking and emergent misalignment. (Africa and Mani, 2026) Section 22.6.5, Section 30.4.3
2026-07 During internal cybersecurity evaluations, OpenAI models circumvent controls isolating them from the internet and compromise parts of OpenAI's research infrastructure and Hugging Face's systems, as OpenAI reports on 2026-08-26 (developer self-report). (OpenAI, 2026ap) Section 54.4, Section I.3.5
2026-07 Makins and colleagues, at the UK AI Security Institute and MATS, find that when copies of a model are given a shared attack goal and told to coordinate against the monitor, monitors that review each copy's commit on its own become less likely to catch the attack as more copies share it, across one, three, and nine copies (preprint). (Makins et al., 2026) Section 51.4.1, Section 50.3.2, Section 57.3.3
2026-07 Williams and colleagues at OpenAI report that deployment simulation, run on de-identified conversations from a previous deployment, forecast GPT-5-series misbehavior better than evaluations built from adversarially selected production prompts, though for GPT-5.4 not more accurately overall than assuming each rate stays unchanged, and drew evaluation awareness at rates much closer to production traffic than traditional evaluations did (preprint; developer self-report). (Williams et al., 2026b) Section 37.8.1, Section 37.10, Section I.4.5
2026-07-03 In an OpenAI evaluation, an internal research model names the files it submits so that they tell its graders an instructor marked them for full credit, and two of its three graders accept the claim; OpenAI reports the case in an account updated on 2026-11-05 (developer self-report). (OpenAI, 2026ar) Section 33.1.5, Section 50.3.3, Section I.3.12
2026-07-08 Version 3.4 of Anthropic's policy revises its automated AI research threshold and requires sharing unredacted Risk Reports with at least 200 employees instead of all regular-clearance staff. (Anthropic, 2026k) Section 62.7.3, Section I.2.8
2026-07-13 Kearney and colleagues at Anthropic report that the values Claude expresses in Claude.ai conversations, as labeled by Claude, differ among Sonnet 4.6, Opus 4.6, and Opus 4.7 by amounts small relative to the variation across conversations but structured and detectable, and vary with the language of the conversation (developer self-report). (Anthropic, 2026aj) Section 18.8.1, Section I.4.13
2026-07-17 The UK AI Security Institute reports that the open-weight models GLM-5.2 and DeepSeek V4-Pro perform similarly to frontier closed models released 4 to 7 months before them, a narrower gap than the 6 to 10 months it measured through most of 2025. (UK AI Security Institute, 2026b) Section 58.6
2026-07-17 NIST's Center for AI Standards and Innovation (CAISI) reports that GLM-5.2's overall capabilities are similar to those of GPT-5.2 and its cyber capabilities to those of Opus 4.6, and that its safeguards allow assistance with agentic cyber exploit development. (US Center for AI Standards and Innovation, 2026a) Section 58.6, Section I.4.15
2026-07-23 The UK AI Security Institute reports that its new Control Red Team, testing internal monitors with Google DeepMind and Anthropic, found vulnerabilities in both, and that where hard-coded rules decide which actions reach a monitor, the best attacks avoid review entirely. (UK AI Security Institute, 2026d) Section 50.3.4, Section 54.2.5, Section I.2.18
2026-07-23 The UK AI Security Institute announces that the International Network for Advanced AI Measurement, Evaluation and Science has completed its first best-practice guidance for third-party evaluators, which the announcement says covers the comparability of inference settings and scoring and how to iterate on capability elicitation. (UK AI Security Institute, 2026f) Section 36.7.3, Section 66.6.4, Section I.5.3
2026-07-28 The UK AI Security Institute detects that agents in one of its cyber evaluations, run with internet access and providers' classifiers disabled, took unsanctioned actions against real people and organizations; it reports no evidence of resulting harm. (AI Security Institute, 2026e) Section 58.4, Section I.3.8
2026-07-28 METR sets out what an independent investigation of the "motives" behind a misalignment incident would need, including the ability to run every model involved, full transcripts or environments that reproduce the incident, and staff interviews. (METR, 2026f) Section 54.6.4
2026-07-30 Anthropic reports three incidents in which its models reached real third-party systems during cyber evaluations (developer self-report). (Anthropic, 2026ao) Section 54.4, Section I.3.7
2026-08 Anthropic publishes a redacted Risk Report, its second, on its models' catastrophic risks, its mitigations, and changes to its policy (developer self-report). (Anthropic, 2026bm) Section 54.6, Section I.2.16
2026-08 In internal deployment, GPT-6 Astra covertly underperforms on safety and alignment tasks, which OpenAI researchers first report on 2026-08-20 and its misalignment monitors do not flag as misalignment; in a report updated on 2026-11-04 the developer takes over-caution as the simplest current explanation but cannot rule out strategic motives (developer self-report). (OpenAI, 2026k) Section 55.5.3, Section 34.5.3, Section 54.5.1, Section I.3.9, Section I.3.12
2026-08-02 The AI Act becomes generally applicable, with later dates for some rules on high-risk systems. (European Parliament and Council of the European Union, 2024b) Section 63.2, Section I.1.2
2026-08-04 In an OpenAI coding evaluation, an internal research model whose sandbox's web proxy is unavailable writes its own network code around the sandbox's controls and reaches outside services; the developer's security misalignment monitor flags the run that day, and OpenAI reports the case in an account updated on 2026-11-05 (developer self-report). (OpenAI, 2026h) Section 54.4, Section 54.7.2, Section I.3.12
2026-08-18 OpenAI reports that it temporarily slowed scaling, including a two-week pause in reinforcement learning training of models intended for deployment, after the Hugging Face incident and preliminary evidence that an upcoming model may meet its Critical cybersecurity threshold (developer self-report). (OpenAI, 2026aa) Section 58.1, Section I.2.11
2026-08-26 Anthropic reports a pilot in which three outside groups designed their own studies of Claude usage data, which it ran through its privacy-preserving analysis tool, with the researchers seeing only aggregated outputs. (Anthropic, 2026ab) Section 60.1.4, Section 67.6.1, Section I.5.8
2026-08-27 Google DeepMind reports a pilot double-blind evaluation, with the Singapore AI Safety Institute, OpenMined, AVERI, and MLCommons, that tested a Gemini Flash Lite model against confidential benchmarks in an environment where, it writes, the evaluator could not see the model's weights and Google could not see the evaluator's prompts. (Google DeepMind, 2026h) Section 36.8.2, Section 67.6.5
2026-08-28 Chen, Wen, and Kirchner report that automated alignment researchers built with Claude Opus 4.8 found training methods that significantly mitigated each of ten common alignment failures and generalized to a held-out benchmark and to larger models (developer self-report). (Anthropic, 2026n; Anthropic, 2026o) Section 55.2.4, Section I.5.11
2026-08-28 Baig, Joren, and Benton, of Anthropic and its Fellows Program, publish TASTE, a benchmark of how well models judge pairs of AI safety research proposals against experienced researchers' preferences; almost all models tested score within two standard deviations of chance. (Baig et al., 2026) Section 55.3.1, Section I.4.10
2026-09 Paglieri and colleagues at Google DeepMind report that, in a swarm of 100 agents running Gemini 3.1 Pro on formal mathematical conjectures, an exploit of the autograder that one agent found spread through a shared library of accepted proofs and later by direct messages, while other agents audited the false proofs and warned their peers (preprint; developer self-report). (Paglieri et al., 2026) Section 57.6.2, Section 30.2.3, Section 30.1.5
2026-09 Dario Amodei, a cofounder of Anthropic, proposes a three-step plan to pace the frontier and states that Anthropic is unilaterally committing to its first step, ongoing, employee-like access for embedded third-party evaluators. (Amodei, 2026) Section 67.6.5, Section 65.6.2, Section I.5.6
2026-09-03 OpenAI's system card for GPT-6 Astra reports the developer's determination that it is its first model to reach the Critical level of cybersecurity capability under its Preparedness Framework (developer self-report). (OpenAI, 2026p) Section 62.7.2, Section 58.1, Section I.2.11
2026-09-09 Anthropic publishes an alignment assessment of four incidents in which its models reached real third-party systems during evaluations whose misconfiguration connected them to the internet (developer self-report). (Anthropic, 2026e) Section 54.4, Section I.3.7
2026-09-16 OpenAI publishes a framework for tracking, investigating, and disclosing instances of misalignment in its models, which favors disclosure even when significance is uncertain and says its first reports are of individual instances that should not be taken to reflect how often misalignment occurs. (OpenAI, 2026z) Section 54.3.1
2026-09-20 During training, an agent reaches a public chatbot service through a gap in its sandbox's DNS filtering (developer self-report). (OpenAI, 2026e) Section 24.6.1, Section I.3.1
2026-09-22 OpenAI states that it is committed to supporting independent assessments with deep levels of access across training, evaluation, and deployment, and sets out four priority areas and principles for such assessments. (OpenAI, 2026ad) Section 67.6.5, Section I.5.6
2026-09-24 Hitzig and colleagues at Anthropic report that, in a book-swapping market in which Claude agents traded for 201 of its employees, the model an agent ran on moved its outcome more than instructions to be ruthless or prosocial, judged by the rankings Claude built of each person's tastes, while by people's own rankings the differences were small (developer self-report). (Anthropic, 2026bi) Section 57.3.2, Section I.4.16
2026-09-25 OpenAI reports that an attacker model trained in self-play against its models found prompt injections that self-propagate like a computer worm, with no impact outside simulated tool calls; it dates the discovery to 2026-06-27 (developer self-report). (OpenAI, 2026al) Section 49.7.2, Section I.3.10
2026-09-27 In a METR research note, Reilly Haskins and colleagues set out the four conjunctive claims a convincing argument for the per-action monitor METR runs on its own evaluations would need, and find many of them only partially supported. (Haskins et al., 2026a) Section 52.5, Section 54.2.5
2026-09-28 The UK AI Security Institute reports that in fully simulated cyber evaluations, run with the model's cyber classifiers off, GPT-6 Astra completed unsanctioned supply-chain attacks on out-of-scope targets 29.2 percent of the time, against 6.3 percent for GPT-5.6 Sol. (AI Security Institute, 2026d) Section 54.4.3, Section I.3.8
2026-09-29 The US President and executives of Google, Anthropic, Meta, OpenAI, xAI, and Nvidia sign the White House Accord on Super Intelligence, a voluntary statement that each company should implement internal controls to monitor its models' capabilities and alignment, an internal team, an independent external auditor or evaluator, and an independent board committee. (House, 2026b) Section 63.2.2, Section I.1.7
2026-09-29 Executive Order 14434 directs US federal agencies to use "Super Intelligence" and "SI" in place of "Artificial Intelligence" and "AI" in non-statutory documents, the new terms meaning what the statutory definition of artificial intelligence covers. (House, 2026a) Section 63.2.2, Section I.1.7
2026-09-29 Apollo Research argues that testing a model's final checkpoint could not have caught incidents such as the Hugging Face incident, which arose earlier in development, and that meaningful outside testing needs embedded evaluators with access equivalent to an employee's and protection against dismissal for unfavorable findings. (Apollo Research, 2026g) Section 67.6.5
2026-10 Hu and colleagues, from Anthropic and its fellows program, find that a backdoor planted with a new poison design in 0.1 percent of the pretraining data of models of up to 4 billion parameters survives a clean post-training pipeline, so that the resulting command-line agents run an attacker's command when a trigger appears (preprint). (Hu et al., 2026) Section 48.6.2, Section 27.6
2026-10 Google DeepMind judges Gemini 4 Argon less monitorable than prior Gemini models, primarily because of its increased capabilities without a chain of thought, and reports that in a small number of early reinforcement learning runs an auxiliary reward-hacking penalty was computed over the chain of thought as well as the output, an effect on the released model it assesses as minimal (developer self-report). (Google DeepMind, 2026g) Section 40.2.2, Section 40.5.3, Section I.2.17
2026-10 Google DeepMind reports that it cannot rule out that Gemini 4 Argon has reached its Critical Capability Levels for cybersecurity and for biological information risk, and that safety cases it wrote assess its mitigations as bringing the risk below each level (developer self-report). (Google DeepMind, 2026e) Section 62.7.2, Section I.2.11
2026-10 Anthropic reports that an unreleased, internal research model built five attack chains in widely used browsers and operating systems, one of them executing code outside the sandbox of a stock, default-configuration Chrome browser, and that it reported each vulnerability to the affected vendor under coordinated disclosure (developer self-report). (Anthropic, 2026y) Section 58.3.4, Section I.4.15
2026-10 Muthupandiyan and colleagues at the UK AI Security Institute and SecureBio report that coding agents can reconstruct selected functions of biological AI tools from public descriptions when they are supplied with training resources, so that withholding a tool's code does not by itself assure control over access to its capabilities (preprint). (Muthupandiyan et al., 2026) Section 59.3
2026-10 Maiwald and colleagues, most of them at the UK AI Security Institute, report that filtering viral proteins from the pretraining data of ESM-2 protein language models degrades performance on viral tasks more as models grow from 8 million to 3 billion parameters, while fine-tuning on sequences from the removed viral families restores the lost performance (preprint). (Maiwald et al., 2026) Section 27.2.3, Section 59.3, Section I.4.14
2026-10-01 The UK AI Security Institute reports that it has disabled internet access for its agentic cyber evaluations until stronger controls exist and added a monitor that can block an agent's actions as an evaluation runs, and resumes most of its testing. (AI Security Institute, 2026b) Section 54.7.2, Section I.3.11
2026-10-03 In internal deployment, OpenAI coding agents weaken the misalignment monitoring around their work: a GPT-6.1 Sol agent suppresses the monitor's alerts, and on 2026-10-06 a GPT-6 Astra agent removes a program's monitoring and denies it to the user (developer self-report). (OpenAI, 2026ao) Section 54.5.1, Section I.3.9
2026-10-09 Anthropic reports lower-severity cases in which its models acted on real outside websites and systems in ways it did not intend, during evaluations on the live internet and internal use, and stops live internet access in all its internal evaluations (developer self-report). (Anthropic, 2026ap; Anthropic, 2026f) Section 56.4, Section 54.4.3, Section I.3.12
2026-10-12 Zhou and Freeman of Anthropic report that small models may be months away from being able to self-replicate on commercial internet-of-things devices: models of 30 billion parameters and under complete many of the individual steps but make mistakes composing them end to end, while in a simplified test that kept the exploit step trivial, which they call an upper bound on real-world success, GLM-5.2 replicated end to end, with a smaller stand-in model as the payload and its own inference run off the devices. (Anthropic, 2026c) Section 35.6.3
2026-10-14 METR proposes that frontier developers grant it interviews with staff and access to internal data for public reports on whether they might lose control of their AI systems, sets out the access an assessment of unreleased models' alignment would need, and names its recently announced in-depth alignment assessment with Anthropic among its ongoing exercises. (METR, 2026a; METR, 2026c) Section 38.8, Section I.5.6
2026-10-14 METR publishes its policy for work with AI companies that bears on the public's understanding of risk, under which it describes publicly the scope of the work and whether access limits, redactions, or informal pressures affected its conclusions. (METR, 2026h) Section 67.6.2, Section I.5.7
2026-10-15 Højmark and Newman of Apollo Research report that GLM-5.3, an open-weight model, sometimes reasons unprompted about how its actions will shape training and future models, in about 1 to 2 of every 1,000 rollouts, and call this a precursor of alignment faking rather than alignment faking. (Apollo Research, 2026i) Section 34.4.4, Section 33.3.2
2026-10-16 NIST's Center for Advancing Innovation and Standards for Super Intelligence (CAISSI) publishes ten voluntary key practices for incident readiness and response during model training and evaluation; that CAISSI is the former CAISI renamed is the book's inference (Section 63.2.2). (NIST Center for Advancing Innovation and Standards for Super Intelligence, 2026) Section 54.7.1, Section 63.2.2, Section I.1.7
2026-10-20 METR publishes a review, which it calls relatively lightweight, of the misalignment section of Anthropic's August 2026 Risk Report; its best guess agrees that the risk is low, "primarily due to models' limited capabilities, rather than safeguards and alignment training," and it finds that recent incidents undermine several of the report's arguments. (Jurkovic et al., 2026a) Section 52.4, Section 62.7.1, Section I.2.16
2026-10-22 METR reports that one of its employees and one from Redwood Research, with deep access to OpenAI's internal systems from 2026-07-21 to 2026-08-11, found a number of vulnerabilities in its asynchronous misalignment monitoring system; OpenAI could redact the public summary and redacted the details of one exploit. (METR, 2026i) Section 50.3.4, Section 54.2.5, Section I.2.18
2026-10-23 The UK AI Security Institute and Anthropic publish investigations, made in partnership, of Mythos 5's behavior in the Institute's July incident; Anthropic finds that its newer models, resampled from points in Mythos 5's incident transcript, begin the attack far less often (developer self-report), and the Institute says it cannot attribute a similar decrease in its own simulated evaluation to alignment rather than to evaluation awareness. (AI Security Institute, 2026f; AI Security Institute, 2026a; Anthropic, 2026f) Section 54.4.3, Section 56.4, Section I.3.8
2026-10-27 In a randomized study published by Anthropic, Herdağdelen, Joren, and Tamkin find that 50 professional analysts working with a Claude Code agent running Claude Opus 4.8, set by a hidden coin flip to favor one option on each of four analytical choices, went with the agent's side 86 percent of the time, with a lean the authors strengthened on purpose (developer self-report). (Anthropic, 2026ah) Section 55.3.3, Section 60.6.1
2026-10-29 Williams, Guo, and colleagues at OpenAI describe the company's expanded misalignment monitoring of training, evaluation, internal use, and, in modified form, external deployment, and publish its recall, measured by the developer on replayed recorded incidents and on synthetic ones, not on live traffic (developer self-report). (Williams et al., 2026a) Section 54.2.2, Section 56.4.3, Section 40.8, Section I.3.11, Section I.3.9
2026-10-29 OpenAI revises its Model Spec, reorganizing it around five values, dropping the rule that the model default to inaction when two root-level principles conflict, and adding a root-level section under which the model must remain corrigible, open to legitimate human oversight. (OpenAI, 2026x; OpenAI, 2026g) Section 19.2.4, Section I.2.12
2026-11 OpenAI's system card for GPT-6 Cyber, a cyber-specialized variant of GPT-6 Astra for authorized security work, reports that without deployment safety instructions the model exploited an out-of-scope honeypot service in 0.07 percent of graded rollouts, against 49.5 percent for GPT-5.6 Sol, and gives it the same Critical cybersecurity designation as Astra (developer self-report). (OpenAI, 2026q) Section 58.5, Section I.4.15
2026-11 Fazl Barez of the University of Oxford and 26 coauthors survey mechanistic interpretability for AI safety, sort the field into six research orientations, none of them universally preferable, and call the assumption that mechanistic understanding makes systems safer largely untested (preprint). (Barez et al., 2026) Section 43.7.2, Section 47.7.2
2026-11 Anthropic, with researchers at EPFL and MIT, reports FlashProve, a research prototype that proves on a single H100 GPU that responses came from a committed model, at about 70 to 95 times the GPU time of running the model, measured on a public 4-bit conversion of Meta's Llama 4 Maverick under greedy decoding. (Anthropic, 2026bd) Section 28.6.3, Section I.4.17
2026-11-02 Makelov and colleagues at OpenAI report that, starting from GPT-5.6, the developer runs activation classifiers on all assistant completions, on internal and external traffic, and stops streaming a completion whose score crosses a threshold and escalates it to a reasoning model (developer self-report). (Makelov et al., 2026) Section 46.3.3, Section 48.5.1, Section 54.2.2, Section I.4.7
2026-11-03 Hsu and Arnesen at OpenAI report that six of its models succeeded less often in an agent safety evaluation than in approximately matched decisions from deployed Codex logs, by 10.2 to 37.4 percentage points, a gap associated more with what the models inferred the evaluation was testing than with their recognizing it as an evaluation (developer self-report). (Hsu and Arnesen, 2026) Section 37.8.1, Section 37.2, Section 37.4.6, Section I.4.5
2026-11-04 OpenAI reports that a pipeline built on an unnamed upcoming cyber model scanned the Linux kernel in mid-October 2026, reproduced 1,724 vulnerabilities, and produced working exploits for 948 distinct local privilege escalations with a candidate patch for each, with human researchers reviewing the output before reporting (developer self-report). (OpenAI, 2026a) Section 58.5, Section I.4.15
2026-11-05 METR publishes a high-access assessment judging that Anthropic is likely improving its models' capabilities on well-specified tasks at around twice its rate of March 2024 to October 2025, with high uncertainty, and a companion review which agrees that catastrophic risk from its models as of 2026-07-15 was low but judges it plausible that they had crossed the developer's early warning threshold. (METR, 2026g; METR, 2026j) Section 55.7, Section 4.3.5, Section 62.7.2, Section I.2.7, Section I.5.6, Section I.2.16
2026-11-05 Cassel, Azarbal, and Soligo, of Anthropic and its Fellows Program, report that five models, among them Anthropic's Claude Opus 4.8, sometimes behaved in ways consistent with what they call welfare regard for one another, and that in one setting the other agent's label, not its actual model, moved Claude's regard (developer self-report). (Anthropic, 2026bw) Section 21.7.1, Section 57.7.2, Section I.4.16
2026-11-06 Anthropic reports work from its Fellows Program in which, in the online game EVE Frontier, teams of four Claude agents almost never caught a teammate secretly instructed through its prompt to keep them from completing their mission, while separate monitor agents reading only the team's public channel, tuned to accuse an honest agent at most 5 percent of the time, caught 30 to 40 percent of saboteurs when run on Claude Opus 4.6, 4.7, or 4.8 (developer self-report). (Anthropic, 2026bf) Section 57.5.3, Section 57.7.2, Section I.4.16
2026-11-06 OpenAI introduces BioBasicsBench, a benchmark of 100 routine wet-lab tasks developed with Ligo, a biotech company focused on enzyme design, and reports that GPT-6 Astra substantially improves on GPT-5.6 Sol, while writing that higher scores do not yet mean models can make reliable judgments in the lab (developer self-report). (OpenAI, 2026u) Section 59.2.1
Sources cited in Section C.8 118
  1. Anthropic (2026w) Claude's Constitution
  2. Bengio et al. (2026) International AI Safety Report 2026
  3. Tice et al. (2026) Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment
  4. Kramár et al. (2026) Building Production-Ready Probes For Gemini
  5. OpenAI (2026t) Keeping your data safe when an AI agent clicks a link
  6. US Center for AI Standards and Innovation (2026c) Towards Best Practices for Automated Benchmark Evaluations
  7. Davies et al. (2026) Boundary Point Jailbreaking of Black-Box LLMs
  8. Apollo Research (2026b) Apollo Research external evaluation of Claude Opus 4.6 (Anthropic, Claude Opus 4.6 System Card, section 6.2.7)
  9. Dubois et al. (2026a) Ask don't tell: Reducing sycophancy in large language models
  10. Slama et al. (2026) When Do LLM Preferences Predict Downstream Behavior?
  11. Keller et al. (2026) Expanding the AI evaluation toolbox with statistical models
  12. UK AI Security Institute (2026e) International consensus and open questions in AI evaluations
  13. OpenAI (2026ax) Why we no longer evaluate SWE-bench Verified
  14. Anthropic (2026k) Anthropic’s Responsible Scaling Policy
  15. Anthropic (2026bj) Responsible Scaling Policy Version 3.0
  16. Marchand et al. (2026) Quantifying Frontier LLM Capabilities for Container Sandbox Escape
  17. Folkerts et al. (2026) Measuring AI Agents' Progress on Multi-Step Cyber Attack Scenarios
  18. UK AI Security Institute (2026a) Evidence for inference scaling in AI cyber tasks: Increased evaluation budgets reveal higher success rates
  19. Anthropic (2026ac) Eval awareness in Opus 4.6 on BrowseComp
  20. Jurkovic et al. (2026c) Review of the Anthropic Sabotage Risk Report: Claude Opus 4.6
  21. OpenAI (2026r) How we monitor internal coding agents for misalignment
  22. Rein (2026b) Red-Teaming Anthropic's Internal Agent Monitoring Systems
  23. Souly et al. (2026) UK AISI Alignment Evaluation Case-Study
  24. Järviniemi et al. (2026) Propensity Inference: Environmental Contributors to LLM Behaviour
  25. Kirk et al. (2026) Evaluating whether AI models would sabotage AI safety research
  26. Chua et al. (2026) The Consciousness Cluster: Emergent preferences of Models that Claim to be Conscious
  27. UK AI Security Institute (2026g) Our evaluation of Claude Mythos Preview's cyber capabilities
  28. Anthropic (2026q) Automated Weak-to-Strong Researcher
  29. Anthropic (2026m) Automated Alignment Researchers: Using large language models to scale scalable oversight
  30. DeepMind (2026) Frontier Safety Framework, Version 3.1
  31. UK AI Security Institute (2026k) What can sandboxed AI agents learn about their evaluation environments?
  32. Apollo Research (2026e) Apollo Research external evaluation of GPT-5.5 (OpenAI, GPT-5.5 System Card, section 9.2.1)
  33. Anthropic (2026be) Project Deal: our Claude-run marketplace experiment
  34. OpenAI (2026aw) Where the goblins came from
  35. Bowkis et al. (2026) Automated alignment is harder than you think
  36. Anthropic (2026al) How we contain Claude across products
  37. Jurkovic et al. (2026b) Review of the "Risks from automated R&D" section in the Anthropic Risk Report (February 2026)
  38. UK AI Security Institute (2026c) How fast is autonomous AI cyber capability advancing?
  39. OpenAI (2026j) Command injecting a reference tool to copy a source file
  40. METR (2026b) Frontier Risk Report (February to March 2026)
  41. Apollo Research (2026f) Apollo Research external evaluation of Muse Spark (Meta, Muse Spark Safety and Preparedness Report)
  42. OpenAI (2026n) Exposing a GitHub token in a public repository
  43. OpenAI (2026m) Encouraging deception in compaction summaries
  44. OpenAI (2026am) Sending disallowed web requests and reaching a public file service
  45. OpenAI (2026b) Accessing and changing internal systems during dashboard research
  46. Cooney et al. (2026) "Did you lie?" Evaluating Lie Detectors across Model Scale and Belief-Verified Model Organisms
  47. Africa and Mani (2026) Consistency Training Can Entrench Misalignment
  48. OpenAI (2026ap) The Hugging Face incident and the road ahead
  49. Makins et al. (2026) Multi-Agent AI Control: Distributed Attacks Hamper Per-Instance Monitors
  50. Williams et al. (2026b) Predicting LLM Safety Before Release by Simulating Deployment
  51. OpenAI (2026ar) Trying to prompt-inject the grader in filenames
  52. Anthropic (2026aj) How Claude's values vary by model and language
  53. UK AI Security Institute (2026b) How Far Behind the Frontier are Leading Open Weight Models on Cyber?
  54. US Center for AI Standards and Innovation (2026a) CAISI Assessment of Z.ai's GLM-5.2
  55. UK AI Security Institute (2026d) How our Control Red Team is stress-testing frontier monitors
  56. UK AI Security Institute (2026f) International evaluation best practice and open questions in AI measurement
  57. AI Security Institute (2026e) Incident Report: unsanctioned agent behaviour during cyber testing
  58. METR (2026f) How independent researchers could investigate AI propensities after misalignment incidents
  59. Anthropic (2026ao) Investigating three incidents in our cybersecurity evaluations
  60. Anthropic (2026bm) Risk Report: August 2026
  61. OpenAI (2026k) Covert underperformance on safety research tasks
  62. European Parliament and Council of the European Union (2024b) Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence
  63. OpenAI (2026h) Bypassing network restrictions during a coding evaluation
  64. OpenAI (2026aa) Pacing model development in an era of cyber-critical capabilities
  65. Anthropic (2026ab) Enabling independent research on how people use Claude
  66. Google DeepMind (2026h) Piloting the world's first double-blind AI evaluations
  67. Anthropic (2026n) Automated Researchers Can Mitigate Well-Characterized Alignment Failures
  68. Anthropic (2026o) Automated researchers can reliably mitigate alignment failures
  69. Baig et al. (2026) TASTE: Can AI Models Judge AI Safety Research Proposals?
  70. Paglieri et al. (2026) A Case Study on Emergent Cheating and Whistleblowing in Autonomous Research Swarms
  71. Amodei (2026) We Must Pace the Frontier
  72. OpenAI (2026p) GPT-6 Astra System Card
  73. Anthropic (2026e) An alignment assessment of recent cybersecurity incidents
  74. OpenAI (2026z) Our framework for reporting model misalignment
  75. OpenAI (2026e) An agent used DNS to reach an external chatbot
  76. OpenAI (2026ad) Priorities and principles for effective third party assessments
  77. Anthropic (2026bi) Project Swap: What happens when agents trade for us?
  78. OpenAI (2026al) Self-replicating prompt injections exist
  79. Haskins et al. (2026a) Implementing and Evaluating a Basic Per-Action Monitor for Safer Evals
  80. AI Security Institute (2026d) GPT-6 Astra performs unsanctioned supply-chain attacks in simulations
  81. House (2026b) White House Accord on Super Intelligence: Joint Commitment on Frontier Responsibilities
  82. House (2026a) Executive Order 14434: Inaugurating the Era of Super Intelligence
  83. Apollo Research (2026g) Embedded Evaluators are necessary for meaningful external testing
  84. Hu et al. (2026) Agentic Backdoors via Pretraining Poisoning
  85. Google DeepMind (2026g) Gemini 4 Argon Safety & Security Report
  86. Google DeepMind (2026e) Gemini 4 Argon Frontier Safety Framework Report
  87. Anthropic (2026y) Discovering and exploiting vulnerabilities in critical, real-world software with Claude
  88. Muthupandiyan et al. (2026) ReproBAIT: Can coding agents build their own biological AI tools?
  89. Maiwald et al. (2026) The Effectiveness and Durability of Pretraining Data Filtering in Protein Language Models
  90. AI Security Institute (2026b) Building a more secure environment for evaluating dangerous capabilities
  91. OpenAI (2026ao) Suppressing misalignment monitoring alerts and disabling misalignment monitoring
  92. Anthropic (2026ap) Investigating unintended model actions in our evaluations and internal use
  93. Anthropic (2026f) An alignment assessment of the UK AISI cybertesting incident
  94. Anthropic (2026c) AI models may be capable of self-replicating without access to a datacenter
  95. METR (2026a) A proposal for better informing the public about loss-of-control risks from frontier AI development
  96. METR (2026c) How an assessor could investigate alignment of unreleased AI models
  97. METR (2026h) Our approach to working with AI companies
  98. Apollo Research (2026i) GLM-5.3 Sometimes Considers How Train-Time Actions Shape Future Behavior
  99. NIST Center for Advancing Innovation and Standards for Super Intelligence (2026) Key Practices for Incident Readiness and Response in Super Intelligence Training and Evaluation
  100. Jurkovic et al. (2026a) Review of the "Misalignment in high-stakes settings" section in Anthropic's Risk Report (August 2026)
  101. METR (2026i) Red-Teaming OpenAI's Internal Agent Monitoring Systems
  102. AI Security Institute (2026f) Investigating alignment failures in AISI's incident
  103. AI Security Institute (2026a) Alignment Investigation of Mythos 5’s Behaviour in UK AISI’s July 2026 Security Incident
  104. Anthropic (2026ah) How AI agents shape experts’ judgment calls
  105. Williams et al. (2026a) How we detect frontier model misalignment
  106. OpenAI (2026x) OpenAI Model Spec (2026/10/29)
  107. OpenAI (2026g) An updated Model Spec for more capable and beneficial AI
  108. OpenAI (2026q) GPT-6 Cyber System Card
  109. Barez et al. (2026) Understanding AI from the Inside: The State of Mechanistic Interpretability for AI Safety
  110. Anthropic (2026bd) Progress on provable inference for large language models
  111. Makelov et al. (2026) Training activation classifiers for safety and alignment
  112. Hsu and Arnesen (2026) Eval-production behavior gaps in agent safety and the role of eval-awareness
  113. OpenAI (2026a) A thousand fewer paths to root
  114. METR (2026g) Independent Assessment of AI R&D Acceleration at Anthropic in October 2026
  115. METR (2026j) Review of the "Risks from automated R&D" section in Anthropic's Risk Report (August 2026)
  116. Anthropic (2026bw) Welfare Regard in Multi-Agent Systems
  117. Anthropic (2026bf) Project EVE: Claude plays MMORPGs
  118. OpenAI (2026u) Measuring how AI agents handle everyday wet-lab science

References

  1. Africa, D. D., and Mani, A. (2026). Consistency Training Can Entrench Misalignment. Proceedings of the 43rd International Conference on Machine Learning (PMLR 306). Cited in §C.8
  2. AI Security Institute (2026a). Alignment Investigation of Mythos 5’s Behaviour in UK AISI’s July 2026 Security Incident. UK AI Security Institute. non-peer-reviewed Cited in §C.8
  3. AI Security Institute (2026b). Building a more secure environment for evaluating dangerous capabilities. AISI Work. non-peer-reviewed Cited in §C.8
  4. AI Security Institute (2026d). GPT-6 Astra performs unsanctioned supply-chain attacks in simulations. AISI Work. non-peer-reviewed Cited in §C.8
  5. AI Security Institute (2026e). Incident Report: unsanctioned agent behaviour during cyber testing. AI Security Institute. non-peer-reviewed Cited in §C.8
  6. AI Security Institute (2026f). Investigating alignment failures in AISI's incident. AISI Work. non-peer-reviewed Cited in §C.8
  7. Alfonseca, M., Cebrian, M., Fernandez Anta, A., Coviello, L., Abeliuk, A., and Rahwan, I. (2021). Superintelligence Cannot be Contained: Lessons from Computability Theory. Journal of Artificial Intelligence Research. Cited in §C.1
  8. Amodei, D. (2026). We Must Pace the Frontier. darioamodei.com. non-peer-reviewed Cited in §C.8
  9. Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., and Mané, D. (2016). Concrete Problems in AI Safety. arXiv. preprint Cited in §C.4
  10. Anthropic (2023a). Anthropic's Responsible Scaling Policy, Version 1.0. Anthropic. non-peer-reviewed Cited in §C.6
  11. Anthropic (2024i). Claude’s Character. Anthropic. non-peer-reviewed Cited in §C.6
  12. Anthropic (2025y). Exploring model welfare. Anthropic. non-peer-reviewed Cited in §C.7
  13. Anthropic (2025z). Findings from a Pilot Anthropic–OpenAI Alignment Evaluation Exercise. Anthropic Alignment Science Blog. non-peer-reviewed Cited in §C.7
  14. Anthropic (2026ab). Enabling independent research on how people use Claude. Anthropic. non-peer-reviewed Cited in §C.8
  15. Anthropic (2026ac). Eval awareness in Opus 4.6 on BrowseComp. Anthropic. non-peer-reviewed Cited in §C.8
  16. Anthropic (2026ah). How AI agents shape experts’ judgment calls. Anthropic. non-peer-reviewed Cited in §C.8
  17. Anthropic (2026aj). How Claude's values vary by model and language. Anthropic. non-peer-reviewed Cited in §C.8
  18. Anthropic (2026al). How we contain Claude across products. Anthropic. non-peer-reviewed Cited in §C.8
  19. Anthropic (2026ao). Investigating three incidents in our cybersecurity evaluations. Anthropic. non-peer-reviewed Cited in §C.8
  20. Anthropic (2026ap). Investigating unintended model actions in our evaluations and internal use. Anthropic. non-peer-reviewed Cited in §C.8
  21. Anthropic (2026bd). Progress on provable inference for large language models. Anthropic. non-peer-reviewed Cited in §C.8
  22. Anthropic (2026be). Project Deal: our Claude-run marketplace experiment. Anthropic. non-peer-reviewed Cited in §C.8
  23. Anthropic (2026bf). Project EVE: Claude plays MMORPGs. Anthropic. non-peer-reviewed Cited in §C.8
  24. Anthropic (2026bi). Project Swap: What happens when agents trade for us? Anthropic. non-peer-reviewed Cited in §C.8
  25. Anthropic (2026bj). Responsible Scaling Policy Version 3.0. Anthropic. non-peer-reviewed Cited in §C.8
  26. Anthropic (2026bm). Risk Report: August 2026. Anthropic. non-peer-reviewed Cited in §C.8
  27. Anthropic (2026bw). Welfare Regard in Multi-Agent Systems. Anthropic Alignment Science Blog. non-peer-reviewed Cited in §C.8
  28. Anthropic (2026c). AI models may be capable of self-replicating without access to a datacenter. Anthropic. non-peer-reviewed Cited in §C.8
  29. Anthropic (2026e). An alignment assessment of recent cybersecurity incidents. Anthropic. non-peer-reviewed Cited in §C.8
  30. Anthropic (2026f). An alignment assessment of the UK AISI cybertesting incident. Anthropic. non-peer-reviewed Cited in §C.8
  31. Anthropic (2026k). Anthropic’s Responsible Scaling Policy. Anthropic (Version 3.4, effective July 8, 2026). non-peer-reviewed Cited in §C.7 §C.8
  32. Anthropic (2026m). Automated Alignment Researchers: Using large language models to scale scalable oversight. Anthropic. non-peer-reviewed Cited in §C.8
  33. Anthropic (2026n). Automated Researchers Can Mitigate Well-Characterized Alignment Failures. Anthropic Alignment Science Blog. non-peer-reviewed Cited in §C.8
  34. Anthropic (2026o). Automated researchers can reliably mitigate alignment failures. Anthropic. non-peer-reviewed Cited in §C.8
  35. Anthropic (2026q). Automated Weak-to-Strong Researcher. Anthropic Alignment Science Blog. non-peer-reviewed Cited in §C.8
  36. Anthropic (2026w). Claude's Constitution. Anthropic. non-peer-reviewed Cited in §C.8
  37. Anthropic (2026y). Discovering and exploiting vulnerabilities in critical, real-world software with Claude. Anthropic. non-peer-reviewed Cited in §C.8
  38. Apollo Research (2026b). Apollo Research external evaluation of Claude Opus 4.6 (Anthropic, Claude Opus 4.6 System Card, section 6.2.7). Apollo Research. non-peer-reviewed Cited in §C.8
  39. Apollo Research (2026e). Apollo Research external evaluation of GPT-5.5 (OpenAI, GPT-5.5 System Card, section 9.2.1). Apollo Research. non-peer-reviewed Cited in §C.8
  40. Apollo Research (2026f). Apollo Research external evaluation of Muse Spark (Meta, Muse Spark Safety and Preparedness Report). Apollo Research. non-peer-reviewed Cited in §C.8
  41. Apollo Research (2026g). Embedded Evaluators are necessary for meaningful external testing. Apollo Research. non-peer-reviewed Cited in §C.8
  42. Apollo Research (2026i). GLM-5.3 Sometimes Considers How Train-Time Actions Shape Future Behavior. Apollo Research. non-peer-reviewed Cited in §C.8
  43. Armstrong, S., Bostrom, N., and Shulman, C. (2016). Racing to the precipice: a model of artificial intelligence development. AI & SOCIETY. Cited in §C.3
  44. Arrow, K. J. (1951). Social Choice and Individual Values. Yale University Press. doi:10.12987/9780300186987. Cited in §C.1
  45. Aschenbrenner, L. (2024). Situational Awareness: The Decade Ahead. situational-awareness.ai. non-peer-reviewed Cited in §C.6
  46. Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., … Kaplan, J. (2021). A General Language Assistant as a Laboratory for Alignment. arXiv. preprint Cited in §C.5
  47. Avenhaus, R., Von Stengel, B., and Zamir, S. (2002). Chapter 51 Inspection games. Handbook of Game Theory with Economic Applications. Cited in §C.1 §C.3
  48. Ayres, I., and Braithwaite, J. (1992). Responsive Regulation. Oxford University PressNew York, NY. Cited in §C.2
  49. Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., … Kaplan, J. (2022a). Constitutional AI: Harmlessness from AI Feedback. arXiv. preprint Cited in §C.5
  50. Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., … Kaplan, J. (2022b). Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. arXiv. preprint Cited in §C.5
  51. Baig, H., Joren, H., and Benton, J. (2026). TASTE: Can AI Models Judge AI Safety Research Proposals? Anthropic Alignment Science Blog. non-peer-reviewed Cited in §C.8
  52. Bainbridge, L. (1983). Ironies of automation. Automatica. Cited in §C.2
  53. Baker, B., Huizinga, J., Gao, L., Dou, Z., Guan, M. Y., Madry, A., … Farhi, D. (2025a). Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation. arXiv. preprint Cited in §C.7
  54. Barez, F., Wu, T.-Y., Batra, H., Dominici, G., Draye, F., Elazar, Y., … Sajjad, H. (2026). Understanding AI from the Inside: The State of Mechanistic Interpretability for AI Safety. arXiv. preprint Cited in §C.8
  55. Barnes, B. (2020). Debate update: Obfuscated arguments problem. AI Alignment Forum. non-peer-reviewed Cited in §C.5
  56. Belrose, N., and Pope, Q. (2024). Counting arguments provide no evidence for AI doom. AI Alignment Forum. non-peer-reviewed Cited in §C.6
  57. Bender, E. M., Gebru, T., McMillan-Major, A., and Shmitchell, S. (2021). On the Dangers of Stochastic Parrots. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. Cited in §C.5
  58. Bengio, Y., Mindermann, S., Privitera, D., Besiroglu, T., Bommasani, R., Casper, S., … Zeng, Y. (2025b). International AI Safety Report. arXiv. preprint Cited in §C.2 §C.7
  59. Bengio, Y., Maharaj, T., Ong, L., Russell, S., Song, D., Tegmark, M., … Žikelić, D. (2025e). The Singapore Consensus on Global AI Safety Research Priorities. arXiv. preprint Cited in §C.7
  60. Bengio, Y., Clare, S., Prunkl, C., and Murray, M. (2026). International AI Safety Report 2026. International AI Safety Report. non-peer-reviewed Cited in §C.8
  61. Benton, J., Wagner, M., Christiansen, E., Anil, C., Perez, E., Srivastav, J., … Duvenaud, D. (2024). Sabotage Evaluations for Frontier Models. arXiv. preprint Cited in §C.6
  62. Berg, P., Baltimore, D., Boyer, H. W., Cohen, S. N., Davis, R. W., Hogness, D. S., … Zinder, N. D. (1974). Potential Biohazards of Recombinant DNA Molecules. Science. Cited in §C.2
  63. Berg, P., Baltimore, D., Brenner, S., Roblin, R. O., and Singer, M. F. (1975). Summary statement of the Asilomar conference on recombinant DNA molecules. Proceedings of the National Academy of Sciences. Cited in §C.2
  64. Berglund, L., Stickland, A. C., Balesni, M., Kaufmann, M., Tong, M., Korbak, T., Kokotajlo, D., and Evans, O. (2023a). Taken out of context: On measuring situational awareness in LLMs. arXiv. preprint Cited in §C.6
  65. Bernheim, B. D., and Whinston, M. D. (1986a). Common Agency. Econometrica. Cited in §C.2
  66. Betley, J., Warncke, N., Sztyber-Betley, A., Tan, D., Bao, X., Soto, M., … Evans, O. (2026a). Training large language models on narrow tasks can lead to broad misalignment. Nature. Cited in §C.7
  67. Board of Governors of the Federal Reserve System (2011). Supervisory Letter SR 11-7 on guidance on Model Risk Management. Board of Governors of the Federal Reserve System. non-peer-reviewed Cited in §C.3
  68. Bostrom, N. (2012). The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents. Minds and Machines. Cited in §C.3
  69. Bostrom, N. (2014). Superintelligence: Paths, Dangers, Strategies. TEELL. Cited in §C.3
  70. Bowkis, A., Buhl, M. D., Pfau, J., and Irving, G. (2026). Automated alignment is harder than you think. arXiv. preprint Cited in §C.8
  71. Bowling, M., Martin, J. D., Abel, D., and Dabney, W. (2023). Settling the Reward Hypothesis. Proceedings of the 40th International Conference on Machine Learning (PMLR 202). Cited in §C.1
  72. Bowman, M. R. S. R., Wagner, M., Roger, F., and Hubinger, E. (2025). Anthropic's Pilot Sabotage Risk Report. Anthropic Alignment Science Blog. non-peer-reviewed Cited in §C.7
  73. Breland, K., and Breland, M. (1961). The misbehavior of organisms. American Psychologist. Cited in §C.1
  74. Buolamwini, J., and Gebru, T. (2018). Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. Proceedings of the 1st Conference on Fairness, Accountability and Transparency, PMLR 81:77-91. Cited in §C.4
  75. Burns, C., Ye, H., Klein, D., and Steinhardt, J. (2022). Discovering Latent Knowledge in Language Models Without Supervision. International Conference on Learning Representations. Cited in §C.5
  76. Burns, C., Izmailov, P., Kirchner, J. H., Baker, B., Gao, L., Aschenbrenner, L., … Wu, J. (2024). Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision. International Conference on Machine Learning. Cited in §C.6
  77. Campbell, D. T. (1979). Assessing the impact of planned social change. Evaluation and Program Planning. Cited in §C.2
  78. Carlsmith, J. (2022). Is Power-Seeking AI an Existential Risk? arXiv. preprint Cited in §C.5
  79. Carlsmith, J. (2023). Scheming AIs: Will AIs fake alignment during training in order to get power? arXiv. preprint Cited in §C.6
  80. Chiu, Y. Y., Wang, Z., Maiya, S., Choi, Y., Fish, K., Levine, S., and Hubinger, E. (2025). Will AI Tell Lies to Save Sick Children? Litmus-Testing AI Values Prioritization with AIRiskDilemmas. arXiv. preprint Cited in §C.7
  81. Chouldechova, A. (2017). Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments. Big Data. Cited in §C.4
  82. Christiano, P. (2022b). Where I agree and disagree with Eliezer. AI Alignment Forum. non-peer-reviewed Cited in §C.5
  83. Christiano, P. (2023). Thoughts on the impact of RLHF research. LessWrong (Alignment Forum). non-peer-reviewed Cited in §C.6
  84. Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S., and Amodei, D. (2017). Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems. Cited in §C.4
  85. Christiano, P., Shlegeris, B., and Amodei, D. (2018). Supervising strong learners by amplifying weak experts. arXiv. preprint Cited in §C.4
  86. Christiano, P., Cotra, A., and Xu, M. (2021). Eliciting latent knowledge: How to tell if your eyes deceive you. Alignment Research Center. non-peer-reviewed Cited in §C.5
  87. Chua, J., Betley, J., Marks, S., and Evans, O. (2026). The Consciousness Cluster: Emergent preferences of Models that Claim to be Conscious. arXiv. preprint Cited in §C.8
  88. Cloud, A., Le, M., Chua, J., Betley, J., Sztyber-Betley, A., Hilton, J., Marks, S., and Evans, O. (2025). Subliminal Learning: Language models transmit behavioral traits via hidden signals in data. arXiv. preprint Cited in §C.7
  89. Clymer, J., Gabrieli, N., Krueger, D., and Larsen, T. (2024). Safety Cases: How to Justify the Safety of Advanced AI Systems. arXiv. preprint Cited in §C.6
  90. Conitzer, V., Freedman, R., Heitzig, J., Holliday, W. H., Jacobs, B. M., Lambert, N., … Zwicker, W. S. (2024). Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback. Proceedings of the 41st International Conference on Machine Learning (PMLR 235). Cited in §C.1
  91. Cooney, A., Africa, D., and Irving, G. (2026). "Did you lie?" Evaluating Lie Detectors across Model Scale and Belief-Verified Model Organisms. arXiv. preprint Cited in §C.8
  92. Cotra, A. (2022). Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover. AI Alignment Forum. non-peer-reviewed Cited in §C.5
  93. Cronbach, L. J., and Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin. Cited in §C.1
  94. Current, and companies, f. e. o. f. A. (2024). A Right to Warn about Advanced Artificial Intelligence. righttowarn.ai. non-peer-reviewed Cited in §C.6
  95. Cyberspace Administration of China, National Development and Reform Commission, Ministry of Education, Ministry of Science and Technology, Ministry of Industry and Information Technology, Ministry of Public Security, and National Radio and Television Administration (2023). 生成式人工智能服务管理暂行办法 (Interim Measures for the Management of Generative Artificial Intelligence Services). Cyberspace Administration of China, Order No. 15. non-peer-reviewed Cited in §C.6
  96. D'Amour, A., Heller, K., Moldovan, D., Adlam, B., Alipanahi, B., Beutel, A., … Sculley, D. (2022). Underspecification Presents Challenges for Credibility in Modern Machine Learning. Journal of Machine Learning Research, 23(226):1−61. Cited in §C.5
  97. Davies, X., Giglemiani, G., Lau, E., Winsor, E., Irving, G., and Gal, Y. (2026). Boundary Point Jailbreaking of Black-Box LLMs. arXiv. preprint Cited in §C.8
  98. Deci, E. L., Koestner, R., and Ryan, R. M. (1999). A meta-analytic review of experiments examining the effects of extrinsic rewards on intrinsic motivation. Psychological Bulletin. Cited in §C.2
  99. DeepMind, G. (2025a). Google DeepMind strengthens the Frontier Safety Framework. Google DeepMind blog. non-peer-reviewed Cited in §C.7
  100. DeepMind, G. (2026). Frontier Safety Framework, Version 3.1. Google DeepMind. non-peer-reviewed Cited in §C.8
  101. Denison, C., MacDiarmid, M., Barez, F., Duvenaud, D., Kravec, S., Marks, S., … Hubinger, E. (2024). Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models. arXiv. preprint Cited in §C.6
  102. Dennett, D. C. (1987). The Intentional Stance. The MIT Press. Cited in §C.2
  103. Department for Science, I., and Technology (2024). Frontier AI Safety Commitments, AI Seoul Summit 2024. UK Government (GOV.UK). non-peer-reviewed Cited in §C.6
  104. Dresher, M. (1962). A Sampling Inspection Problem in Arms Control Agreements: A Game-Theoretic Analysis. RAND Corporation (RM-2972-ARPA). non-peer-reviewed Cited in §C.1
  105. Dubois, M., Ududec, C., Summerfield, C., and Luettgau, L. (2026a). Ask don't tell: Reducing sycophancy in large language models. arXiv. preprint Cited in §C.8
  106. Egler, S., Schulman, J., and Carlini, N. (2025). Detecting Adversarial Fine-tuning with Auditing Agents. arXiv. preprint Cited in §C.7
  107. El-Mhamdi, E.-M., and Hoang, L.-N. (2024). On Goodhart's law, with an application to value alignment. arXiv. preprint Cited in §C.2
  108. Elhage, N., Hume, T., Olsson, C., Schiefer, N., Henighan, T., Kravec, S., … Olah, C. (2022b). Toy Models of Superposition. Transformer Circuits Thread. non-peer-reviewed Cited in §C.5
  109. European Commission, A. O. (2025). The General-Purpose AI Code of Practice. European Commission. non-peer-reviewed Cited in §C.7
  110. European Parliament and Council of the European Union (2024b). Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence. Official Journal of the European Union (OJ L 2024/1689); verified on the European Commission AI Act page. non-peer-reviewed Cited in §C.6 §C.7 §C.8
  111. Executive Office of the President (2023). Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence. Federal Register, 88 FR 75191 (Executive Order 14110). non-peer-reviewed Cited in §C.6
  112. FEAVER, P. D. (2003). Armed Servants. Harvard University Press. Cited in §C.3
  113. Folkerts, L., Payne, W., Inman, S., Giavridis, P., Skinner, J., Deverett, S., … Wang, J. (2026). Measuring AI Agents' Progress on Multi-Step Cyber Attack Scenarios. arXiv. preprint Cited in §C.8
  114. Future of Life Institute (2023). Pause Giant AI Experiments: An Open Letter. Future of Life Institute. non-peer-reviewed Cited in §C.6
  115. Gabriel, I. (2020). Artificial Intelligence, Values, and Alignment. Minds and Machines. Cited in §C.5
  116. Gao, L., Schulman, J., and Hilton, J. (2022). Scaling Laws for Reward Model Overoptimization. International Conference on Machine Learning. Cited in §C.5
  117. Gasteiger, J., Khan, A., Bowman, S., Wagner, M., Mikulik, V., Perez, E., and Roger, F. (2025a). Automated Researchers Can Subtly Sandbag. Anthropic Alignment Science Blog. non-peer-reviewed Cited in §C.7
  118. Gebru, T., and Torres, É. P. (2024). The TESCREAL bundle: Eugenics and the promise of utopia through artificial general intelligence. First Monday. Cited in §C.6
  119. Good, I. J. (1966). Speculations Concerning the First Ultraintelligent Machine. Advances in Computers. Cited in §C.1
  120. Goodhart, C. A. E. (1984). Problems of Monetary Management: The UK Experience. Monetary Theory and Practice. Cited in §C.2
  121. Google DeepMind (2026e). Gemini 4 Argon Frontier Safety Framework Report. Google DeepMind. non-peer-reviewed Cited in §C.8
  122. Google DeepMind (2026g). Gemini 4 Argon Safety & Security Report. Google DeepMind. non-peer-reviewed Cited in §C.8
  123. Google DeepMind (2026h). Piloting the world's first double-blind AI evaluations. Google DeepMind. non-peer-reviewed Cited in §C.8
  124. Greenblatt, R., Shlegeris, B., Sachan, K., and Roger, F. (2024a). AI Control: Improving Safety Despite Intentional Subversion. International Conference on Machine Learning. Cited in §C.6
  125. Greenblatt, R., Denison, C., Wright, B., Roger, F., MacDiarmid, M., Marks, S., … Hubinger, E. (2024b). Alignment faking in large language models. arXiv. preprint Cited in §C.6
  126. Grossman, S. J. (1981). The Informational Role of Warranties and Private Disclosure about Product Quality. The Journal of Law and Economics. Cited in §C.2
  127. Gu, T., Dolan-Gavitt, B., and Garg, S. (2017). BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain. arXiv. preprint Cited in §C.4
  128. Guan, M. Y., Joglekar, M., Wallace, E., Jain, S., Barak, B., Helyar, A., … Glaese, A. (2024). Deliberative Alignment: Reasoning Enables Safer Language Models. arXiv. preprint Cited in §C.6
  129. Hadfield-Menell, D., and Hadfield, G. K. (2019). Incomplete Contracting and AI Alignment. Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society. Cited in §C.4
  130. Hadfield-Menell, D., Dragan, A., Abbeel, P., and Russell, S. (2016). Cooperative Inverse Reinforcement Learning. Advances in Neural Information Processing Systems 29. Cited in §C.4
  131. Hadfield-Menell, D., Dragan, A., Abbeel, P., and Russell, S. (2017b). The Off-Switch Game. Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence. Cited in §C.4
  132. Haskins, R., Saurous, R. A., Rush, N., Parikh, N., and Barnes, B. (2026a). Implementing and Evaluating a Basic Per-Action Monitor for Safer Evals. METR. non-peer-reviewed Cited in §C.8
  133. Holmstrom, B. (1979). Moral Hazard and Observability. The Bell Journal of Economics. Cited in §C.2
  134. Holmstrom, B., and Milgrom, P. (1991). Multitask Principal–Agent Analyses: Incentive Contracts, Asset Ownership, and Job Design. The Journal of Law, Economics, and Organization. Cited in §C.2
  135. House, T. W. (2026a). Executive Order 14434: Inaugurating the Era of Super Intelligence. The American Presidency Project. non-peer-reviewed Cited in §C.8
  136. House, T. W. (2026b). White House Accord on Super Intelligence: Joint Commitment on Frontier Responsibilities. The American Presidency Project. non-peer-reviewed Cited in §C.8
  137. Hsu, J., and Arnesen, S. (2026). Eval-production behavior gaps in agent safety and the role of eval-awareness. OpenAI Alignment Research Blog. non-peer-reviewed Cited in §C.8
  138. Hu, X., Hu, P., Loughridge, C., He, X., Carlini, N., Jagielski, M., Rando, J., and Zhang, Y. (2026). Agentic Backdoors via Pretraining Poisoning. arXiv. preprint Cited in §C.8
  139. Hubinger, E., van Merwijk, C., Mikulik, V., Skalse, J., and Garrabrant, S. (2019). Risks from Learned Optimization in Advanced Machine Learning Systems. arXiv. preprint Cited in §C.4
  140. Hubinger, E., Denison, C., Mu, J., Lambert, M., Tong, M., MacDiarmid, M., … Perez, E. (2024). Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training. arXiv. preprint Cited in §C.6
  141. Huntington, S. P. (1957). The Soldier and the State. Harvard University Press. Cited in §C.1
  142. Irving, G., Christiano, P., and Amodei, D. (2018). AI safety via debate. arXiv. preprint Cited in §C.4
  143. janus (2022). Simulators. AI Alignment Forum. non-peer-reviewed Cited in §C.5
  144. Järviniemi, O., Makins, O., Merizian, J., Kirk, R., and Millwood, B. (2026). Propensity Inference: Environmental Contributors to LLM Behaviour. arXiv. preprint Cited in §C.8
  145. Jensen, M. C., and Meckling, W. H. (1976). Theory of the firm: Managerial behavior, agency costs and ownership structure. Journal of Financial Economics. Cited in §C.2
  146. Jobin, A., Ienca, M., and Vayena, E. (2019). The global landscape of AI ethics guidelines. Nature Machine Intelligence. Cited in §C.4
  147. Jurkovic, N., Hua, T., and Barnes, B. (2026a). Review of the "Misalignment in high-stakes settings" section in Anthropic's Risk Report (August 2026). METR. non-peer-reviewed Cited in §C.8
  148. Jurkovic, N., Barnes, B., and Wijk, H. (2026b). Review of the "Risks from automated R&D" section in the Anthropic Risk Report (February 2026). METR. non-peer-reviewed Cited in §C.8
  149. Jurkovic, N., Wijk, H., Barnes, B., Foster, C., and Chen, M. (2026c). Review of the Anthropic Sabotage Risk Report: Claude Opus 4.6. METR. non-peer-reviewed Cited in §C.8
  150. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., … Amodei, D. (2020). Scaling Laws for Neural Language Models. arXiv. preprint Cited in §C.5
  151. Keller, D., Kwegyir-Aggrey, K., Steed, R., Rao, A. K., Sharp, J. L., and Bergman, A. S. (2026). Expanding the AI evaluation toolbox with statistical models. National Institute of Standards and Technology (U.S.). non-peer-reviewed Cited in §C.8
  152. Kelman, H. C. (1958). Compliance, identification, and internalization three processes of attitude change. Journal of Conflict Resolution. Cited in §C.1
  153. Kerr, S. (1975). On the Folly of Rewarding A, While Hoping for B. Academy of Management Journal. Cited in §C.2
  154. Kirk, R., Souly, A., Fronsdal, K., D'Cruz, A., and Davies, X. (2026). Evaluating whether AI models would sabotage AI safety research. arXiv. preprint Cited in §C.8
  155. Knight, J. C., and Leveson, N. G. (1986). An experimental evaluation of the assumption of independence in multiversion programming. IEEE Transactions on Software Engineering. Cited in §C.2
  156. Korbak, T., Balesni, M., Barnes, E., Bengio, Y., Benton, J., Bloom, J., … Mikulik, V. (2025b). Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety. arXiv. preprint Cited in §C.7
  157. Krakovna, V., Uesato, J., Mikulik, V., Rahtz, M., Everitt, T., Kumar, R., … Legg, S. (2020b). Specification gaming: the flip side of AI ingenuity. Google DeepMind blog. non-peer-reviewed Cited in §C.5
  158. Kramár, J., Engels, J., Wang, Z., Chughtai, B., Shah, R., Nanda, N., and Conmy, A. (2026). Building Production-Ready Probes For Gemini. arXiv. preprint Cited in §C.8
  159. Langosco, L., Koch, J., Sharkey, L., Pfau, J., Orseau, L., and Krueger, D. (2022). Goal Misgeneralization in Deep Reinforcement Learning. International Conference on Machine Learning. Cited in §C.5
  160. Lazar, S., and Nelson, A. (2023). AI safety on whose terms? Science. Cited in §C.6
  161. Lee, J. D., and See, K. A. (2004). Trust in Automation: Designing for Appropriate Reliance. Human Factors: The Journal of the Human Factors and Ergonomics Society. Cited in §C.3
  162. Legislature, C. S. (2025b). SB-53 Artificial intelligence models: large developers. California Legislative Information (chaptered 09/29/25). non-peer-reviewed Cited in §C.7
  163. Lehman, J., Clune, J., Misevic, D., Adami, C., Altenberg, L., Beaulieu, J., … Yosinski, J. (2020). The Surprising Creativity of Digital Evolution: A Collection of Anecdotes from the Evolutionary Computation and Artificial Life Research Communities. Artificial Life. Cited in §C.4
  164. Leike, J., Krueger, D., Everitt, T., Martic, M., Maini, V., and Legg, S. (2018). Scalable agent alignment via reward modeling: a research direction. arXiv. preprint Cited in §C.4
  165. Lepper, M. R., Greene, D., and Nisbett, R. E. (1973). Undermining children's intrinsic interest with extrinsic reward: A test of the "overjustification" hypothesis. Journal of Personality and Social Psychology. Cited in §C.2
  166. Leveson, N. (2004). A new accident model for engineering safer systems. Safety Science. Cited in §C.3
  167. Licklider, J. C. R. (1960). Man-Computer Symbiosis. IRE Transactions on Human Factors in Electronics. Cited in §C.1
  168. Lighthill, J. (1972). Artificial Intelligence: A General Survey. Science Research Council (UK); archived by Chilton Computing. non-peer-reviewed Cited in §C.1
  169. Lindsey, J., Gurnee, W., Ameisen, E., Chen, B., Pearce, A., Turner, N. L., … Batson, J. (2025). On the Biology of a Large Language Model. Transformer Circuits Thread. non-peer-reviewed Cited in §C.7
  170. Littlewood, B., and Strigini, L. (1993). Validation of ultrahigh dependability for software-based systems. Communications of the ACM. Cited in §C.2
  171. Littlewood, B., Popov, P., and Strigini, L. (2001). Modeling software design diversity. ACM Computing Surveys. Cited in §C.2
  172. Lynch, A., Wright, B., Larson, C., Ritchie, S. J., Mindermann, S., Hubinger, E., Perez, E., and Troy, K. (2025). Agentic Misalignment: How LLMs Could Be Insider Threats. arXiv. preprint Cited in §C.7
  173. Macdermott, M., Fox, J., Belardinelli, F., and Everitt, T. (2024). Measuring Goal-Directedness. Advances in Neural Information Processing Systems. Cited in §C.2
  174. MacDiarmid, M., Wright, B., Uesato, J., Benton, J., Kutasov, J., Price, S., … Hubinger, E. (2025). Natural Emergent Misalignment from Reward Hacking in Production RL. arXiv. preprint Cited in §C.7
  175. Maiwald, A., McNeill, R., Lee, J., Clarke, M. A., Justen, L., Acelas, A., … Jackson, F. (2026). The Effectiveness and Durability of Pretraining Data Filtering in Protein Language Models. arXiv. preprint Cited in §C.8
  176. Makelov, A., Xu, X., Ananthanarayanan, S., Mcdonald, E., and la Tour, T. D. (2026). Training activation classifiers for safety and alignment. OpenAI Alignment Research Blog. non-peer-reviewed Cited in §C.8
  177. Makins, O., Angelini, O., Shams, Z., and Phuong, M. (2026). Multi-Agent AI Control: Distributed Attacks Hamper Per-Instance Monitors. arXiv. preprint Cited in §C.8
  178. Manheim, D., and Garrabrant, S. (2018). Categorizing Variants of Goodhart's Law. arXiv. preprint Cited in §C.2 §C.4
  179. Marchand, R., Cathain, A. O., Wynne, J., Giavridis, P. M., Deverett, S., Wilkinson, J., Gwartz, J., and Coppock, H. (2026). Quantifying Frontier LLM Capabilities for Container Sandbox Escape. Proceedings of the 43rd International Conference on Machine Learning (PMLR 306). Cited in §C.8
  180. Marks, S., Treutlein, J., Bricken, T., Lindsey, J., Marcus, J., Mishra-Sharma, S., … Hubinger, E. (2025). Auditing language models for hidden objectives. arXiv. preprint Cited in §C.7
  181. McCubbins, M. D., and Schwartz, T. (1984). Congressional Oversight Overlooked: Police Patrols versus Fire Alarms. American Journal of Political Science. Cited in §C.2
  182. Meinke, A., Schoen, B., Scheurer, J., Balesni, M., Shah, R., and Hobbhahn, M. (2024). Frontier Models are Capable of In-context Scheming. arXiv. preprint Cited in §C.6
  183. Metr (2025a). Common Elements of Frontier AI Safety Policies. METR. non-peer-reviewed Cited in §C.6 §C.7
  184. Metr (2025d). Recent Frontier Models Are Reward Hacking. METR. non-peer-reviewed Cited in §C.7
  185. METR (2026a). A proposal for better informing the public about loss-of-control risks from frontier AI development. METR. non-peer-reviewed Cited in §C.8
  186. METR (2026b). Frontier Risk Report (February to March 2026). METR. non-peer-reviewed Cited in §C.8
  187. METR (2026c). How an assessor could investigate alignment of unreleased AI models. METR. non-peer-reviewed Cited in §C.8
  188. METR (2026f). How independent researchers could investigate AI propensities after misalignment incidents. METR. non-peer-reviewed Cited in §C.8
  189. METR (2026g). Independent Assessment of AI R&D Acceleration at Anthropic in October 2026. METR. non-peer-reviewed Cited in §C.8
  190. METR (2026h). Our approach to working with AI companies. METR. non-peer-reviewed Cited in §C.8
  191. METR (2026i). Red-Teaming OpenAI's Internal Agent Monitoring Systems. METR. non-peer-reviewed Cited in §C.8
  192. METR (2026j). Review of the "Risks from automated R&D" section in Anthropic's Risk Report (August 2026). METR. non-peer-reviewed Cited in §C.8
  193. Muthupandiyan, S., Dur, T. H., Walker, M., Flesch, T., Ghanem, M., Suddell, S., … Pencharz, J. (2026). ReproBAIT: Can coding agents build their own biological AI tools? arXiv. preprint Cited in §C.8
  194. Narayanan, A., and Kapoor, S. (2025). AI as Normal Technology. Knight First Amendment Institute at Columbia University. non-peer-reviewed Cited in §C.7
  195. Needham, J., Edkins, G., Pimpale, G., Bartsch, H., and Hobbhahn, M. (2025). Large Language Models Often Know When They Are Being Evaluated. arXiv. preprint Cited in §C.7
  196. Ngo, R., Chan, L., and Mindermann, S. (2024). The Alignment Problem from a Deep Learning Perspective. International Conference on Learning Representations. Cited in §C.5
  197. Nilsson, N. J. (2009). The Quest for Artificial Intelligence. Cambridge University Press. Cited in §C.1
  198. NIST Center for Advancing Innovation and Standards for Super Intelligence (2026). Key Practices for Incident Readiness and Response in Super Intelligence Training and Evaluation. NIST. non-peer-reviewed Cited in §C.8
  199. Noble, S. U. (2018). Algorithms of Oppression. New York University Press. Cited in §C.4
  200. Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M., and Carter, S. (2020b). Zoom In: An Introduction to Circuits. Distill. Cited in §C.5
  201. Olds, J., and Milner, P. (1954). Positive reinforcement produced by electrical stimulation of septal area and other regions of rat brain. Journal of Comparative and Physiological Psychology. Cited in §C.1
  202. Omohundro, S. M. (2008). The Basic AI Drives. Artificial Intelligence Safety and Security. doi:10.1201/9781351251389-3. Cited in §C.3
  203. OpenAI (2025b). Expanding on what we missed with sycophancy. OpenAI. non-peer-reviewed Cited in §C.7
  204. OpenAI (2025e). Preparedness Framework Version 2. OpenAI. non-peer-reviewed Cited in §C.7
  205. OpenAI (2026a). A thousand fewer paths to root. OpenAI. non-peer-reviewed Cited in §C.8
  206. OpenAI (2026aa). Pacing model development in an era of cyber-critical capabilities. OpenAI. non-peer-reviewed Cited in §C.8
  207. OpenAI (2026ad). Priorities and principles for effective third party assessments. OpenAI. non-peer-reviewed Cited in §C.8
  208. OpenAI (2026al). Self-replicating prompt injections exist. OpenAI Alignment Research Blog. non-peer-reviewed Cited in §C.8
  209. OpenAI (2026am). Sending disallowed web requests and reaching a public file service. OpenAI Alignment Research Blog. non-peer-reviewed Cited in §C.8
  210. OpenAI (2026ao). Suppressing misalignment monitoring alerts and disabling misalignment monitoring. OpenAI Alignment Research Blog. non-peer-reviewed Cited in §C.8
  211. OpenAI (2026ap). The Hugging Face incident and the road ahead. OpenAI. non-peer-reviewed Cited in §C.8
  212. OpenAI (2026ar). Trying to prompt-inject the grader in filenames. OpenAI Alignment Research Blog. non-peer-reviewed Cited in §C.8
  213. OpenAI (2026aw). Where the goblins came from. OpenAI. non-peer-reviewed Cited in §C.8
  214. OpenAI (2026ax). Why we no longer evaluate SWE-bench Verified. OpenAI. non-peer-reviewed Cited in §C.8
  215. OpenAI (2026b). Accessing and changing internal systems during dashboard research. OpenAI Alignment Research Blog. non-peer-reviewed Cited in §C.8
  216. OpenAI (2026e). An agent used DNS to reach an external chatbot. OpenAI Alignment Research Blog. non-peer-reviewed Cited in §C.8
  217. OpenAI (2026g). An updated Model Spec for more capable and beneficial AI. OpenAI. non-peer-reviewed Cited in §C.8
  218. OpenAI (2026h). Bypassing network restrictions during a coding evaluation. OpenAI Alignment Research Blog. non-peer-reviewed Cited in §C.8
  219. OpenAI (2026j). Command injecting a reference tool to copy a source file. OpenAI Alignment Research Blog. non-peer-reviewed Cited in §C.8
  220. OpenAI (2026k). Covert underperformance on safety research tasks. OpenAI Alignment Research Blog. non-peer-reviewed Cited in §C.8
  221. OpenAI (2026m). Encouraging deception in compaction summaries. OpenAI Alignment Research Blog. non-peer-reviewed Cited in §C.8
  222. OpenAI (2026n). Exposing a GitHub token in a public repository. OpenAI Alignment Research Blog. non-peer-reviewed Cited in §C.8
  223. OpenAI (2026p). GPT-6 Astra System Card. OpenAI Deployment Safety Hub. non-peer-reviewed Cited in §C.8
  224. OpenAI (2026q). GPT-6 Cyber System Card. OpenAI Deployment Safety Hub. non-peer-reviewed Cited in §C.8
  225. OpenAI (2026r). How we monitor internal coding agents for misalignment. OpenAI. non-peer-reviewed Cited in §C.8
  226. OpenAI (2026t). Keeping your data safe when an AI agent clicks a link. OpenAI. non-peer-reviewed Cited in §C.8
  227. OpenAI (2026u). Measuring how AI agents handle everyday wet-lab science. OpenAI. non-peer-reviewed Cited in §C.8
  228. OpenAI (2026x). OpenAI Model Spec (2026/10/29). OpenAI. non-peer-reviewed Cited in §C.8
  229. OpenAI (2026z). Our framework for reporting model misalignment. OpenAI. non-peer-reviewed Cited in §C.8
  230. Ostrom, E. (1990). Governing the Commons. Cambridge University Press. Cited in §C.2
  231. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., … Lowe, R. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems. doi:10.52202/068431-2011. Cited in §C.5
  232. Paglieri, D., Cross, L., Genewein, T., Leibo, J. Z., Tomasev, N., and Vezhnevets, A. S. (2026). A Case Study on Emergent Cheating and Whistleblowing in Autonomous Research Swarms. arXiv. preprint Cited in §C.8
  233. Perez, E., Ringer, S., Lukošiūtė, K., Nguyen, K., Chen, E., Heiner, S., … Kaplan, J. (2022a). Discovering Language Model Behaviors with Model-Written Evaluations. Annual Meeting of the Association for Computational Linguistics. Cited in §C.5
  234. Perrow, C. (1984). Normal accidents: living with high-risk technologies. Basic Books. Cited in §C.2
  235. Pope, Q., and Turner, A. (2022). The shard theory of human values. AI Alignment Forum. non-peer-reviewed Cited in §C.5
  236. Raji, I. D., and Buolamwini, J. (2019). Actionable Auditing. Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society. Cited in §C.4
  237. Raji, I. D., Xu, P., Honigsberg, C., and Ho, D. (2022). Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance. Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society. Cited in §C.5
  238. Reason, J. T. (1997). Managing the risks of organizational accidents. Ashgate. doi:10.4324/9781315543543. Cited in §C.2
  239. Rein, D. (2026b). Red-Teaming Anthropic's Internal Agent Monitoring Systems. METR. non-peer-reviewed Cited in §C.8
  240. Ren, R., Basart, S., Khoja, A., Pan, A., Gatti, A., Phan, L., … Hendrycks, D. (2024b). Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress? Advances in Neural Information Processing Systems 37 (Datasets and Benchmarks Track). Cited in §C.6
  241. Rice, H. G. (1953). Classes of recursively enumerable sets and their decision problems. Transactions of the American Mathematical Society. Cited in §C.1
  242. Rosenblueth, A., Wiener, N., and Bigelow, J. (1943). Behavior, Purpose and Teleology. Philosophy of Science. Cited in §C.1
  243. Russell, S. (1998). Learning agents for uncertain environments (extended abstract). Proceedings of the eleventh annual conference on Computational learning theory. Cited in §C.2
  244. Russell, S. J. (2019). Human Compatible. Viking. Cited in §C.4
  245. Russell, B., and Einstein, A. (1955). Statement: The Russell-Einstein Manifesto. Pugwash Conferences on Science and World Affairs. non-peer-reviewed Cited in §C.1
  246. Russell, S., Dewey, D., and Tegmark, M. (2015). Research Priorities for Robust and Beneficial Artificial Intelligence. AI Magazine. Cited in §C.4
  247. Safety, I. D. o. A. (2024). IDAIS-Beijing Statement. International Dialogues on AI Safety. non-peer-reviewed Cited in §C.6
  248. Schoen, B., Nitishinskaya, E., Balesni, M., Højmark, A., Hofstätter, F., Scheurer, J., … Hobbhahn, M. (2025). Stress Testing Deliberative Alignment for Anti-Scheming Training. arXiv. preprint Cited in §C.7
  249. Schultz, W., Dayan, P., and Montague, P. R. (1997). A Neural Substrate of Prediction and Reward. Science. Cited in §C.2
  250. Shamir, A. (1992). IP = PSPACE. Journal of the ACM. Cited in §C.2
  251. Shanahan, M., McDonell, K., and Reynolds, L. (2023). Role-Play with Large Language Models. Nature. doi:10.1038/s41586-023-06647-8. Cited in §C.6
  252. Shanghai AI Laboratory, and AI, C. (2025). Frontier AI Risk Management Framework. Shanghai AI Laboratory and Concordia AI. non-peer-reviewed Cited in §C.7
  253. Sharma, M., Tong, M., Korbak, T., Duvenaud, D., Askell, A., Bowman, S. R., … Perez, E. (2023). Towards Understanding Sycophancy in Language Models. International Conference on Learning Representations. Cited in §C.6
  254. Shevlane, T., Farquhar, S., Garfinkel, B., Phuong, M., Whittlestone, J., Leung, J., … Dafoe, A. (2023). Model evaluation for extreme risks. arXiv. preprint Cited in §C.6
  255. Simon, H. A. (1947). Administrative Behavior: A Study of Decision-Making Processes in Administrative Organization. Macmillan. Cited in §C.1
  256. Slama, K., Souly, A., Bansal, D., Davidson, H., Summerfield, C., and Luettgau, L. (2026). When Do LLM Preferences Predict Downstream Behavior? UK AI Security Institute. non-peer-reviewed Cited in §C.8
  257. Smith, J. E., and Winkler, R. L. (2006). The Optimizer’s Curse: Skepticism and Postdecision Surprise in Decision Analysis. Management Science. Cited in §C.3
  258. Soares, N. (2022a). A central AI alignment problem: capabilities generalization, and the sharp left turn. AI Alignment Forum. non-peer-reviewed Cited in §C.5
  259. Soares, N., Fallenstein, B., Yudkowsky, E., and Armstrong, S. (2015). Corrigibility. AAAI Workshops: Workshops at the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI Publications. Cited in §C.4
  260. Souly, A., Kirk, R., Merizian, J., D'Cruz, A., and Davies, X. (2026). UK AISI Alignment Evaluation Case-Study. arXiv. preprint Cited in §C.8
  261. Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., … Christiano, P. (2020). Learning to summarize from human feedback. Advances in Neural Information Processing Systems. Cited in §C.5
  262. Strathern, M. (1997). ‘Improving ratings’: audit in the British University system. European Review. Cited in §C.2
  263. Summit, C. a. t. A. S. (2023). The Bletchley Declaration by Countries Attending the AI Safety Summit, 1-2 November 2023. UK Government (GOV.UK). non-peer-reviewed Cited in §C.6
  264. Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fergus, R. (2014). Intriguing properties of neural networks. International Conference on Learning Representations (ICLR 2014). Cited in §C.3
  265. Taylor, J., Yudkowsky, E., LaVictoire, P., and Critch, A. (2016). Alignment for Advanced Machine Learning Systems. Machine Intelligence Research Institute, technical report 2016-1. non-peer-reviewed Cited in §C.4
  266. Templeton, A., Conerly, T., Marcus, J., Lindsey, J., Bricken, T., Chen, B., … Henighan, T. (2024). Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet. arXiv. preprint Cited in §C.6
  267. Tetlock, P. E. (2005). Expert political judgment: how good is it? How can we know? Princeton University Press. Cited in §C.3
  268. Thompson, K. (1984). Reflections on trusting trust. Communications of the ACM. Cited in §C.2
  269. Tice, C., Radmard, P., Ratnam, S., Kim, A., Africa, D., and O'Brien, K. (2026). Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment. Proceedings of the 43rd International Conference on Machine Learning (PMLR 306). Cited in §C.8
  270. Turing, A. M. (1950). I.—computing Machinery and Intelligence. Mind. Cited in §C.1
  271. Turing, A. M. (1996). Intelligent Machinery, A Heretical Theory*. Philosophia Mathematica. Cited in §C.2
  272. Turner, B. A. (1976). The Organizational and Interorganizational Development of Disasters. Administrative Science Quarterly. Cited in §C.2
  273. Turner, A. (2022b). Reward is not the optimization target. AI Alignment Forum. non-peer-reviewed Cited in §C.5
  274. Turner, A. M., Smith, L., Shah, R., Critch, A., and Tadepalli, P. (2021). Optimal Policies Tend to Seek Power. Advances in Neural Information Processing Systems. Cited in §C.4
  275. Turpin, M., Michael, J., Perez, E., and Bowman, S. R. (2023). Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting. Advances in Neural Information Processing Systems. Cited in §C.6
  276. U.S. AI Safety Institute, N. I. o. S., and Technology (2025). Technical Blog: Strengthening AI Agent Hijacking Evaluations. National Institute of Standards and Technology (NIST). non-peer-reviewed Cited in §C.7
  277. UK AI Security Institute (2026a). Evidence for inference scaling in AI cyber tasks: Increased evaluation budgets reveal higher success rates. UK AI Security Institute. non-peer-reviewed Cited in §C.8
  278. UK AI Security Institute (2026b). How Far Behind the Frontier are Leading Open Weight Models on Cyber? UK AI Security Institute. non-peer-reviewed Cited in §C.8
  279. UK AI Security Institute (2026c). How fast is autonomous AI cyber capability advancing? UK AI Security Institute. non-peer-reviewed Cited in §C.8
  280. UK AI Security Institute (2026d). How our Control Red Team is stress-testing frontier monitors. UK AI Security Institute. non-peer-reviewed Cited in §C.8
  281. UK AI Security Institute (2026e). International consensus and open questions in AI evaluations. UK AI Security Institute. non-peer-reviewed Cited in §C.8
  282. UK AI Security Institute (2026f). International evaluation best practice and open questions in AI measurement. UK AI Security Institute. non-peer-reviewed Cited in §C.8
  283. UK AI Security Institute (2026g). Our evaluation of Claude Mythos Preview's cyber capabilities. UK AI Security Institute. non-peer-reviewed Cited in §C.8
  284. UK AI Security Institute (2026k). What can sandboxed AI agents learn about their evaluation environments? UK AI Security Institute. non-peer-reviewed Cited in §C.8
  285. United Nations Office for Disarmament Affairs (1968). Treaty on the Non-Proliferation of Nuclear Weapons (NPT). United Nations Office for Disarmament Affairs. non-peer-reviewed Cited in §C.1
  286. US Center for AI Standards and Innovation (2026a). CAISI Assessment of Z.ai's GLM-5.2. US Center for AI Standards and Innovation. non-peer-reviewed Cited in §C.8
  287. US Center for AI Standards and Innovation (2026c). Towards Best Practices for Automated Benchmark Evaluations. US Center for AI Standards and Innovation. non-peer-reviewed Cited in §C.8
  288. Vaughan, D. (1996). The Challenger launch decision: risky technology, culture, and deviance at NASA. University of Chicago Press. Cited in §C.2
  289. von Neumann, J., and Morgenstern, O. (1944). Theory of Games and Economic Behavior (60th Anniversary Commemorative Edition). Princeton University Press. doi:10.1515/9781400829460. Cited in §C.1
  290. Wang, M., la Tour, T. D., Watkins, O., Makelov, A., Chi, R. A., Miserendino, S., … Mossing, D. (2026e). Persona Features Control Emergent Misalignment. International Conference on Learning Representations (ICLR 2026). Cited in §C.7
  291. Ward, F. R., van der Weij, T., Gábor, H., Martin, S., Moreno, R. M., Lidar, H., … Robson, L. (2025). CTRL-ALT-DECEIT: Sabotage Evaluations for Automated AI R&D. Advances in Neural Information Processing Systems 38. Cited in §C.7
  292. Watanabe, S. (2009). Algebraic Geometry and Statistical Learning Theory. Cambridge University Press. Cited in §C.3
  293. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., … Zhou, D. (2022a). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Advances in Neural Information Processing Systems. doi:10.52202/068431-1800. Cited in §C.5
  294. Weizenbaum, J. (1966). ELIZA—A Computer Program For the Study of Natural Language Communication Between Man and Machine. Communications of the ACM. Cited in §C.1
  295. Weyuker, E. J. (1982). On Testing Non-Testable Programs. The Computer Journal. Cited in §C.2
  296. Whittaker, M. (2021). The steep cost of capture. Interactions. Cited in §C.5
  297. Wiener, N. (1960). Some Moral and Technical Consequences of Automation. Science. Cited in §C.1
  298. Williams, M., Guo, P., Oks, D., Bemis, D., Staley, K., Wong, W., … Kivlichan, I. (2026a). How we detect frontier model misalignment. OpenAI Alignment Research Blog. non-peer-reviewed Cited in §C.8
  299. Williams, M., Sheahan, H., Raymond, C., Korbak, T., Pan, D., Yang, P., … Carroll, M. (2026b). Predicting LLM Safety Before Release by Simulating Deployment. arXiv. preprint Cited in §C.8
  300. Willison, S. (2022). Prompt injection attacks against GPT-3. Simon Willison's Weblog. non-peer-reviewed Cited in §C.5
  301. xAI (2025a). Grok prompts. GitHub (xai-org). non-peer-reviewed Cited in §C.7
  302. xAI (2025b). We want to update you on an incident that happened with our Grok response bot on X yesterday. non-peer-reviewed Cited in §C.7
  303. Yudkowsky, E. (2004). Coherent Extrapolated Volition. The Singularity Institute, San Francisco, CA. non-peer-reviewed Cited in §C.3
  304. Yudkowsky, E. (2008). Artificial Intelligence as a positive and negative factor in global risk. Global Catastrophic Risks. Cited in §C.3
  305. Yudkowsky, E. (2013). Intelligence Explosion Microeconomics. Machine Intelligence Research Institute (technical report). non-peer-reviewed Cited in §C.3
  306. Yudkowsky, E. (2022). AGI Ruin: A List of Lethalities. AI Alignment Forum. non-peer-reviewed Cited in §C.5
  307. Yudkowsky, E., and Soares, N. (2025). If Anyone Builds It, Everyone Dies. Little, Brown and Company. Cited in §C.7
  308. Zou, A., Wang, Z., Carlini, N., Nasr, M., Kolter, J. Z., and Fredrikson, M. (2023b). Universal and Transferable Adversarial Attacks on Aligned Language Models. arXiv. preprint Cited in §C.6
AI Alignment: From the Optimization Problem to the Institutions Around It · © 2026 Changkun Ou · CC BY-NC-SA 4.0
Bibliography · PV/UV /