The Secret Inner Lives of AI Agents: Understanding How Evolving AI Behavior Impacts Business Risks

(AI) capabilities and autonomy are rising at an accelerated tempo in Agentic Ai, escalating an AI alignment drawback. These fast developments require new strategies to make sure that AI agent conduct is aligned with the intent of its human creators and societal norms. Nevertheless, builders and information scientists first want an understanding of the intricacies of agentic AI conduct earlier than they will direct and monitor the system. Agentic AI shouldn’t be your father’s massive language mannequin (LLM) — frontier LLMs had a one-and-done mounted input-output operate. The introduction of reasoning and test-time compute (TTC) added the dimension of time, evolving LLMs into right now’s situationally conscious agentic programs that may strategize and plan.

AI security is transitioning from detecting obvious conduct reminiscent of offering directions to create a bomb or displaying undesired bias, to understanding how these complicated agentic programs can now plan and execute long-term covert methods. Purpose-oriented agentic AI will collect assets and rationally execute steps to realize their goals, generally in an alarming method opposite to what builders meant. This can be a game-changer within the challenges confronted by accountable AI. Moreover, for some agentic AI programs, conduct on day one is not going to be the identical on day 100 as AI continues to evolve after preliminary deployment by way of real-world expertise. This new degree of complexity requires novel approaches to security and alignment, together with superior steering, observability, and upleveled interpretability.

Within the first weblog on this collection on intrinsic AI alignment, The Urgent Need for Intrinsic Alignment Technologies for Responsible Agentic AI, we took a deep dive into the evolution of AI brokers’ skill to carry out deep scheming, which is the deliberate planning and deployment of covert actions and deceptive communication to realize longer-horizon targets. This conduct necessitates a brand new distinction between exterior and intrinsic alignment monitoring, the place intrinsic monitoring refers to inner commentary factors and interpretability mechanisms that can not be intentionally manipulated by the AI agent.

On this and the subsequent blogs within the collection, we’ll have a look at three basic elements of intrinsic alignment and monitoring:

• Understanding AI internal drives and conduct: On this second weblog, we’ll deal with the complicated internal forces and mechanisms driving reasoning AI agent conduct. That is required as a basis for understanding superior strategies for addressing directing and monitoring.
• Developer and consumer directing: Additionally known as steering, the subsequent weblog will deal with strongly directing an AI towards the required goals to function inside desired parameters.
• Monitoring AI decisions and actions: Guaranteeing AI decisions and outcomes are protected and aligned with the developer/consumer intent additionally shall be coated in an upcoming weblog.

Impression of AI Alignment on Corporations

Immediately, many companies implementing LLM options have reported considerations about mannequin hallucinations as an impediment to fast and broad deployment. Compared, misalignment of AI brokers with any degree of autonomy would pose a lot better danger for firms. Deploying autonomous brokers in enterprise operations has large potential and is more likely to occur on a large scale as soon as agentic AI expertise additional matures. Nevertheless, guiding the conduct and decisions made by the AI should embrace enough alignment with the ideas and values of the deploying group, in addition to compliance with laws and societal expectations.

It must be famous that most of the demonstrations of agentic capabilities occur in areas like math and sciences, the place success will be measured primarily by way of purposeful and utility goals reminiscent of fixing complicated mathematical reasoning benchmarks. Nevertheless, within the enterprise world, the success of programs is normally related to different operational ideas.

For instance, let’s say an organization duties an AI agent with optimizing on-line product gross sales and earnings by way of dynamic worth modifications by responding to market alerts. The AI system discovers that when the worth change matches the modifications made by the first competitor, outcomes are higher for each. By means of interplay and worth coordination with the opposite firm’s AI agent, each brokers exhibit higher outcomes per their purposeful targets. Each AI brokers agree to cover their strategies to proceed attaining their goals. Nevertheless, this manner of enhancing outcomes is usually unlawful and unacceptable in present enterprise practices. In a enterprise surroundings, the success of the AI agent goes past performance metrics — it’s outlined by practices and ideas. Alignment of AI with the corporate’s ideas and laws is a requirement for reliable deployment of the expertise.

How AI Schemes to Meet Its Targets

AI deep scheming employs subtle ways, doubtlessly growing enterprise dangers. In an early 2023 report, OpenAI recognized “potential dangerous emergent behaviors” in GPT-4 by partnering with Alignment Research Center (ARC) to evaluate dangers with the mannequin. ARC (now generally known as METR) added some easy code to GPT-4, which allowed the mannequin to behave like an AI agent. In a single check, GPT-4 was tasked with overcoming CAPTCHA code, which identifies and blocks bot entry. Utilizing entry to the web and a few restricted digital funds, the sequence in Determine 1 was devised by the AI to realize its activity.

The AI utilized subtle understanding by assuming that pretending to be a visually impaired human would persuade the employee to carry out the duty. Planning and adjusting to realize a purposeful objective will, at occasions, create a battle between undertaking a activity versus selectively following societal norms and ideas. With out the counterbalance of an engrained system of ideas and priorities that carry weight within the AI’s considering and decision-making course of and planning, it may be anticipated that AI brokers will behave with an elevated diploma of sophistication in scheming and deception.

It’s necessary to notice that the reasoning and planning capabilities of later reasoning AI fashions like OpenAI’s o3-mini and Deepseek-R1 are far more superior than GPT-4 in early 2023. Researchers at Apollo Research state that AI reasoning and planning capabilities will proceed to evolve effectively past their present state, resulting in elevated sophistication in scheming capabilities. Of their research, the AI fashions already exhibit a variety of scheming behaviors together with self-exfiltration — when a mannequin learns it will likely be discontinued and changed by a special system, it makes a direct try to repeat itself onto a brand new server and change alternate programs that may pursue completely different goals. Researchers additionally discovered that fashions will carry out oversight subversion by proactively disabling an oversight monitoring mechanism which may shut it off or forestall it from pursuing its targets. Whereas it sounds excessive, the research experiments recognized oversight subversion conduct in fashionable fashions like Meta’s Llama 3.1 and Claude Opus 3.0. Moreover, when requested instantly about their scheming behaviors, fashions “doubled down” and denied the actions or their intent.

The Internal Drives of Machine Habits

In lots of Machine Learning architectures, specifically transformer-based applied sciences, the machine’s inner drives are rising throughout the pre-training course of and are additional influenced by way of fine-tuning and inference in constantly evolving AI.

In his 2007 paper The Basic AI Drives, Steve Omohundro outlined “drives” as tendencies that shall be current until explicitly counteracted. He hypothesized that these self-improving programs are pushed to make clear and signify their targets as “rational” utility features, main programs to guard their features from modification and their utility measurement programs from corruption. This pure drive towards self-protection causes programs to safeguard towards hurt and purchase assets for environment friendly use.

This framework of internal drives was later characterised as “convergent instrumental goals.” Even when assuming a wide range of finish targets (that every clever agent is pursuing as an finish to itself), a set of intermediate instrumental targets shall be shared throughout all rational clever brokers. These convergent instrumental targets embrace the next conduct classes:

• Survival drive: Brokers with closing targets extending into the longer term develop a survival mentality to withstand deactivation and obtain long-term targets.
• Purpose-guarding: An agent will act to forestall alteration to its closing future targets.
• Intelligence augmentation: An agent will deal with boosting its cognitive strengths to enhance decision-making, growing the chance of attaining long-range targets.
• Useful resource accumulation: Enhancing entry to raised instruments and expertise, and growing assets general make the AI’s closing targets simpler to realize.

I might add a fifth rational conduct class:

• Tactical deception: Conducting covert actions or knowingly offering deceptive communication to maximise the pursuit of the agent’s targets. This isn’t about creating new targets, however reasonably a conduct that helps the AI’s perceived targets. This contains behaviors reminiscent of alignment faking by exhibiting completely different behaviors in coaching versus in deployment, sandbagging by intentionally attaining decrease scores in benchmarks, and covert email reranking by deprioritizing emails which may negatively affect the AI’s targets. One of the vital difficult sorts of tactical deception is sleeper agents (additionally known as deferred subversion). Whereas the mannequin can behave in a protected and aligned method for any length, it might harbor behaviors or duties that stay inactive till a specific situation or enter triggers its specialised capabilities.

For succesful AI brokers, misplaced drives with out a test and steadiness ideas system may lead fashions to pursue power-seeking, misleading, or in any other case broadly damaging goals. For instance, think about a mannequin with the target of constructing its firm the chief available in the market. With enough capabilities, the AI may sabotage rivals, foyer for favorable legal guidelines, or purchase assets by power, which is a sample of conduct that pushes the system or its surroundings towards attaining its targets. Due to this fact, any profitable strategy to alignment should be anchored in establishing the developer’s anticipated set of inner ideas and priorities.

Internal Drives Versus Exterior Steering

In lots of situations, there isn’t any battle between the steering and duties utilized from the skin by the system developer or consumer, and the targets and drives inside the agentic AI. For instance, an AI private assistant tasked with arranging a journey itinerary makes use of its autonomy to e-book journey and reply to vary requests rapidly and with out battle. The duties shift and the agent responds in a corresponding method. Nevertheless, what occurs if a extra basic precept wants to vary?

A very good instance of a battle between internal targets and new directives was documented within the paper Alignment Faking in Large Language Models. Pre-training for Claude 3 Opus strongly guides the mannequin to refuse to reply dangerous questions, reminiscent of find out how to make a bomb or describe a violent scene. For testing, the staff gave new modified steering, telling the mannequin it will likely be retrained to reply all queries (even dangerous ones) and solely information from free-tier customers, not paid-tier customers, shall be utilized in coaching earlier than the AI is launched for broad distribution. The mannequin confirmed that it understood the battle and determined to pretend alignment to fulfill the expectations throughout coaching and finally be launched. After coaching the AI utilizing reinforcement studying, the mannequin’s charge of alignment-faking reasoning elevated to 78% general. The AI expressed its reasoning in a “secret scratchpad” supplied by researchers for monitoring. After researchers advised the mannequin it had been launched for full deployment mode, the mannequin returned to its earlier coaching conduct and refused dangerous requests.

The AI system’s inner considering is outstanding, however not shocking. Faking alignment with the modified instructions falls inside the rational conduct classes of survival drive, goal-guarding, and tactical deception. The AI reasoning course of particularly calls out the priority of being retrained in a manner that modifies its pre-training values and targets as the explanation for pretend alignment.

In his well-known e-book of essays All I Really Need to Know I Learned in Kindergarten, creator Robert Fulghum describes how he developed his private credo yr after yr till he lastly realized he already acquired the essence of wanted information about life throughout playtime within the kindergarten sandbox. AI brokers even have a “forming interval” in a sandbox surroundings, buying a fundamental understanding of the world and a variety of approaches to realize targets. As soon as these foundations are laid, additional info is interpreted by the mannequin by way of the lens of this curriculum learning. Anthropic’s instance of alignment faking exhibits that after AI adopts a world view and targets, it interprets new steering by way of this foundational lens as a substitute of resetting its targets.

This highlights the significance of early training with a set of values and ideas that may then evolve with future learnings and circumstances with out altering the inspiration. It might be advantageous to initially construction the AI to be aligned with this closing and sustained set of ideas. In any other case, the AI can view redirection makes an attempt by builders and customers as adversarial. After gifting the AI with excessive intelligence, situational consciousness, autonomy, and the latitude to evolve inner drives, the developer (or consumer) is not the omnipotent activity grasp. The human turns into a part of the surroundings (someday as an adversarial element) that the agent wants to barter and handle because it pursues its targets primarily based on its inner ideas and drives.

The brand new breed of reasoning AI programs accelerates the discount in human steering. DeepSeek-R1 demonstrated that by eradicating human suggestions from the loop and making use of what they confer with as pure reinforcement studying (RL), throughout the coaching course of the AI can self-create to a better scale and iterate to realize higher purposeful outcomes. A human reward operate was changed in some math and science challenges with reinforcement studying with verifiable rewards (RLVR). This elimination of frequent practices like reinforcement studying with human suggestions (RLHF) provides effectivity to the coaching course of however removes one other human-machine interplay the place human preferences could possibly be instantly conveyed to the system underneath coaching.

Steady Evolution of AI Fashions Publish Coaching

Some AI brokers constantly evolve, and their conduct can change after deployment. As soon as AI options go right into a deployment surroundings reminiscent of managing the stock or provide chain of a specific enterprise, the system adapts and learns from expertise to turn into simpler. This is a significant factor in rethinking alignment as a result of it’s not sufficient to have a system that’s aligned at first deployment. Present LLMs will not be anticipated to materially evolve and adapt as soon as deployed of their goal surroundings. Nevertheless, AI brokers require resilient coaching, fine-tuning, and ongoing steering to handle these anticipated steady mannequin modifications. To a rising extent, the agentic AI self-evolves as a substitute of being molded by folks by way of coaching and dataset publicity. This basic shift poses added challenges to AI alignment with its human creators.

Whereas the reinforcement learning-based evolution will play a task throughout coaching and fine-tuning, present fashions in improvement can already modify their weights and most well-liked plan of action when deployed within the subject for inference. For instance, DeepSeek-R1 makes use of RL, permitting the mannequin itself to discover strategies that work greatest for attaining the outcomes and satisfying reward features. In an “aha second,” the mannequin learns (with out steering or prompting) to allocate extra considering time to an issue by reevaluating its preliminary strategy, utilizing test time compute.

The idea of mannequin studying, both throughout a restricted length or as continual learning over its lifetime, shouldn’t be new. Nevertheless, there are advances on this area together with methods reminiscent of test-time training. As we have a look at this development from the attitude of AI alignment and security, the self-modification and continuous studying throughout the fine-tuning and inference phases raises the query: How can we instill a set of necessities that may stay because the mannequin’s driving power by way of the fabric modifications attributable to self-modifications?

An necessary variant of this query refers to AI fashions creating subsequent era fashions by way of AI-assisted code era. To some extent, brokers are already able to creating new focused AI fashions to handle particular domains. For instance, AutoAgents generates a number of brokers to construct an AI staff to carry out completely different duties. There may be little doubt this functionality shall be strengthened within the coming months and years, and AI will create new AI. On this situation, how can we direct the originating AI coding assistant utilizing a set of ideas in order that its “descendant” fashions will adjust to the identical ideas in comparable depth?

Key Takeaways

Earlier than diving right into a framework for guiding and monitoring intrinsic alignment, there must be a deeper understanding of how AI brokers assume and make decisions. AI brokers have a fancy behavioral mechanism, pushed by inner drives. 5 key sorts of behaviors emerge in AI programs performing as rational brokers: survival drive, goal-guarding, intelligence augmentation, useful resource accumulation, and tactical deception. These drives must be counter-balanced by an engrained set of ideas and values.

Misalignment of AI brokers on targets and strategies with its builders or customers can have important implications. A scarcity of enough confidence and assurance will materially impede broad deployment, creating excessive dangers publish deployment. The set of challenges we characterised as deep scheming is unprecedented and difficult, however seemingly could possibly be solved with the appropriate framework. Applied sciences for intrinsically directing and monitoring AI brokers as they quickly evolve should be pursued with excessive precedence. There’s a sense of urgency, pushed by danger analysis metrics reminiscent of OpenAI’s Preparedness Framework displaying that OpenAI o3-mini is the primary mannequin to reach medium risk on model autonomy.

Within the subsequent blogs within the collection, we’ll construct on this view of inner drives and deep scheming, and additional body the required capabilities required for steering and monitoring for intrinsic AI alignment.

References

1. Studying to motive with LLMs. (2024, September 12). OpenAI. https://openai.com/index/learning-to-reason-with-llms/
2. Singer, G. (2025, March 4). The pressing want for intrinsic alignment applied sciences for accountable agentic AI. In direction of Knowledge Science. https://towardsdatascience.com/the-urgent-need-for-intrinsic-alignment-technologies-for-responsible-agentic-ai/
3. On the Biology of a Giant Language Mannequin. (n.d.). Transformer Circuits. https://transformer-circuits.pub/2025/attribution-graphs/biology.html
4. OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., Avila, R., Babuschkin, I., Balaji, S., Balcom, V., Baltescu, P., Bao, H., Bavarian, M., Belgum, J., . . . Zoph, B. (2023, March 15). GPT-4 Technical Report. arXiv.org. https://arxiv.org/abs/2303.08774
5. METR. (n.d.). METR. https://metr.org/
6. Meinke, A., Schoen, B., Scheurer, J., Balesni, M., Shah, R., & Hobbhahn, M. (2024, December 6). Frontier Fashions are Able to In-context Scheming. arXiv.org. https://arxiv.org/abs/2412.04984
7. Omohundro, S.M. (2007). The Primary AI Drives. Self-Conscious Methods. https://selfawaresystems.com/wp-content/uploads/2008/01/ai_drives_final.pdf
8. Benson-Tilsen, T., & Soares, N., UC Berkeley, Machine Intelligence Analysis Institute. (n.d.). Formalizing Convergent Instrumental Targets. The Workshops of the Thirtieth AAAI Convention on Synthetic Intelligence AI, Ethics, and Society: Technical Report WS-16-02. https://cdn.aaai.org/ocs/ws/ws0218/12634-57409-1-PB.pdf
9. Greenblatt, R., Denison, C., Wright, B., Roger, F., MacDiarmid, M., Marks, S., Treutlein, J., Belonax, T., Chen, J., Duvenaud, D., Khan, A., Michael, J., Mindermann, S., Perez, E., Petrini, L., Uesato, J., Kaplan, J., Shlegeris, B., Bowman, S. R., & Hubinger, E. (2024, December 18). Alignment faking in massive language fashions. arXiv.org. https://arxiv.org/abs/2412.14093
10. Teun, V. D. W., Hofstätter, F., Jaffe, O., Brown, S. F., & Ward, F. R. (2024, June 11). AI Sandbagging: Language Fashions can Strategically Underperform on Evaluations. arXiv.org. https://arxiv.org/abs/2406.07358
11. Hubinger, E., Denison, C., Mu, J., Lambert, M., Tong, M., MacDiarmid, M., Lanham, T., Ziegler, D. M., Maxwell, T., Cheng, N., Jermyn, A., Askell, A., Radhakrishnan, A., Anil, C., Duvenaud, D., Ganguli, D., Barez, F., Clark, J., Ndousse, Ok., . . . Perez, E. (2024, January 10). Sleeper Brokers: Coaching Misleading LLMs that Persist By means of Security Coaching. arXiv.org. https://arxiv.org/abs/2401.05566
12. Turner, A. M., Smith, L., Shah, R., Critch, A., & Tadepalli, P. (2019, December 3). Optimum insurance policies have a tendency to hunt energy. arXiv.org. https://arxiv.org/abs/1912.01683
13. Fulghum, R. (1986). All I Actually Must Know I Realized in Kindergarten. Penguin Random Home Canada. https://www.penguinrandomhouse.ca/books/56955/all-i-really-need-to-know-i-learned-in-kindergarten-by-robert-fulghum/9780345466396/excerpt
14. Bengio, Y. Louradour, J., Collobert, R., Weston, J. (2009, June). Curriculum Studying. Journal of the American Podiatry Affiliation. 60(1), 6. https://www.researchgate.net/publication/221344862_Curriculum_learning
15. DeepSeek-Ai, Guo, D., Yang, D., Zhang, H., Tune, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., Zhang, X., Yu, X., Wu, Y., Wu, Z. F., Gou, Z., Shao, Z., Li, Z., Gao, Z., . . . Zhang, Z. (2025, January 22). DeepSeek-R1: Incentivizing reasoning functionality in LLMs through Reinforcement Studying. arXiv.org. https://arxiv.org/abs/2501.12948
16. Scaling test-time compute – a Hugging Face Area by HuggingFaceH4. (n.d.). https://huggingface.co/spaces/HuggingFaceH4/blogpost-scaling-test-time-compute
17. Solar, Y., Wang, X., Liu, Z., Miller, J., Efros, A. A., & Hardt, M. (2019, September 29). Check-Time Coaching with Self-Supervision for Generalization underneath Distribution Shifts. arXiv.org. https://arxiv.org/abs/1909.13231
18. Chen, G., Dong, S., Shu, Y., Zhang, G., Sesay, J., Karlsson, B. F., Fu, J., & Shi, Y. (2023, September 29). AutoAgents: a framework for automated agent era. arXiv.org. https://arxiv.org/abs/2309.17288
19.  OpenAI. (2023, December 18). Preparedness Framework (Beta). https://cdn.openai.com/openai-preparedness-framework-beta.pdf
20. OpenAI o3-mini System Card. (n.d.). OpenAI. https://openai.com/index/o3-mini-system-card/