Demystifying Policy Optimization in RL: An Introduction to PPO and GRPO

Introduction

studying (RL) has achieved outstanding success in educating brokers to resolve advanced duties, from mastering Atari video games and Go to coaching useful language fashions. Two necessary methods behind many of those advances are coverage optimization algorithms known as Proximal Coverage Optimization (PPO) and the newer Generalized Reinforcement Coverage Optimization (GRPO). On this article, we’ll clarify what these algorithms are, why they matter, and the way they work – in beginner-friendly phrases. We’ll begin with a fast overview of reinforcement studying and Policy Gradient strategies, then introduce GRPO (together with its motivation and core concepts), and dive deeper into PPO’s design, math, and benefits. Alongside the best way, we’ll examine PPO (and GRPO) with different in style RL algorithms like DQN, A3C, TRPO, and DDPG. Lastly, we’ll have a look at some code to see how PPO is utilized in apply. Let’s get began!

Background: Reinforcement Studying and Coverage Gradients

Reinforcement studying is a framework the place an agent learns by interacting with an atmosphere via trial and error. The agent observes the state of the atmosphere, takes an motion, after which receives a reward sign and probably a brand new state in return. Over time, by attempting actions and observing rewards, the agent adapts its behaviour to maximise the cumulative reward it receives​. This loop of state → motion → reward → subsequent state is the essence of RL, and the agent’s purpose is to find a great coverage (a technique of selecting actions based mostly on states) that yields excessive rewards.

In policy-based RL strategies (also called coverage gradient strategies), we instantly optimize the agent’s coverage. As a substitute of studying “worth” estimates for every state or state-action (as in value-based strategies like Q-learning), coverage gradient algorithms modify the parameters of a coverage (usually a neural community) within the course that improves efficiency. A basic instance is the REINFORCE algorithm, which updates the coverage parameters in proportion to the reward-weighted gradient of the log-policy​. In apply, to scale back variance, we use an benefit operate (the additional reward of taking motion a in state s in comparison with common) or a baseline (like a worth operate) when computing the gradient​. This results in actor-critic strategies, the place the “actor” is the coverage being realized, and the “critic” is a worth operate that estimates how good states (or state-action pairs) are to supply a baseline for the actor’s updates​. Many superior algorithms, together with Ppo, fall into this actor-critic household: they keep a coverage (actor) and use a realized worth operate (critic) to help the coverage replace.

Generalized Reinforcement Coverage Optimization (GRPO)

One of many newer developments in coverage optimization is Generalized Reinforcement Coverage Optimization (GRPO) – typically referred to in literature as Group Relative Coverage Optimization. GRPO was launched in latest analysis (notably by the DeepSeek workforce) to handle some limitations of PPO when coaching massive fashions (akin to language fashions for reasoning)​. At its core, GRPO is a variant of coverage gradient RL that eliminates the necessity for a separate critic/worth community and as an alternative optimizes the coverage by evaluating a group of motion outcomes towards one another.

Motivation: Why take away the critic? In advanced environments (e.g. lengthy textual content era duties), coaching a worth operate may be arduous and resource-intensive. By “foregoing the critic,” GRPO avoids the challenges of studying an correct worth mannequin and saves roughly half the reminiscence/computation since we don’t keep further mannequin parameters for the critic​. This makes RL coaching easier and extra possible in memory-constrained settings. Actually, GRPO was proven to chop the compute necessities for Reinforcement Learning from human suggestions almost in half in comparison with PPO​.

Core concept: As a substitute of counting on a critic to inform us how good every motion was, GRPO evaluates the coverage by evaluating a number of actions’ outcomes relative to one another. Think about the agent (coverage) generates a set of attainable outcomes for a similar state (or immediate) a group of responses. These are all evaluated by the atmosphere or a reward operate, yielding rewards. GRPO then computes a bonus for every motion based mostly on how its reward compares to the others. One easy manner is to take every motion’s reward minus the typical reward of the group (optionally dividing by the group’s reward commonplace deviation for normalization)​. This tells us which actions did higher than common and which did worse. The coverage is then up to date to assign larger chance to the better-than-average actions and decrease chance to the more serious ones​. In essence, “the mannequin learns to develop into extra just like the solutions marked as appropriate and fewer just like the others”​.

How does this look in apply? It seems the loss/goal in GRPO seems similar to PPO’s. GRPO nonetheless makes use of the thought of a “surrogate” goal with chance ratios (we’ll clarify this below PPO) and even makes use of the identical clipping mechanism to restrict how far the coverage strikes in a single replace​. The important thing distinction is that the benefit is computed from these group-based relative rewards quite than a separate worth estimator​. Additionally, implementations of GRPO usually embody a KL-divergence time period within the loss to maintain the brand new coverage near a reference (or previous) coverage, just like PPO’s non-obligatory KL penalty​.

PPO vs. GRPO — Prime: In PPO, the agent’s Coverage Mannequin is educated with the assistance of a separate Worth Mannequin (critic) to estimate benefit, together with a Reward Mannequin and a set Reference Mannequin (for KL penalty). Backside: GRPO removes the worth community and as an alternative computes benefits by evaluating a gaggle of sampled outcomes reward scores for a similar enter by way of a easy “group computation.” The coverage replace then makes use of these relative scores because the benefit indicators. By dropping the worth mannequin, GRPO considerably simplifies the coaching pipeline and reduces reminiscence utilization, at the price of utilizing extra samples per replace (to type the teams)

In abstract, GRPO may be seen as a PPO-like strategy with out a realized critic. It trades off some pattern effectivity (because it wants a number of samples from the identical state to match rewards) in change for larger simplicity and stability when worth operate studying is tough. Initially designed for big language mannequin coaching with human suggestions (the place getting dependable worth estimates is difficult), GRPO’s concepts are extra typically relevant to different RL situations the place relative comparisons throughout a batch of actions may be made. By understanding GRPO at a excessive stage, we additionally set the stage for understanding PPO, since GRPO is basically constructed on PPO’s basis.

Proximal Coverage Optimization (PPO)

Now let’s flip to Proximal Coverage Optimization (PPO) – probably the most in style and profitable coverage gradient algorithms in fashionable RL. PPO was launched by OpenAI in 2017 as a solution to a sensible query: how can we replace an RL agent as a lot as attainable with the information now we have, whereas guaranteeing we don’t destabilize coaching by making too massive a change? In different phrases, we would like huge enchancment steps with out “falling off a cliff” in efficiency. Its predecessors, like Belief Area Coverage Optimization (TRPO), tackled this by imposing a tough constraint on the scale of the coverage replace (utilizing advanced second-order optimization). PPO achieves an identical impact in a a lot easier manner – utilizing first-order gradient updates with a intelligent clipped goal – which is simpler to implement and empirically simply pretty much as good.

In apply, PPO is applied as an on-policy actor-critic algorithm. A typical PPO coaching iteration seems like this:

1. Run the present coverage within the atmosphere to gather a batch of trajectories (state, motion, reward sequences). For instance, play 2048 steps of the sport or have the agent simulate a couple of episodes.
2. Use the collected information to compute the benefit for every state-action (usually utilizing Generalized Benefit Estimation (GAE) or an identical methodology to mix the critic’s worth predictions with precise rewards).
3. Replace the coverage by maximizing the PPO goal above (often by gradient ascent, which in apply means doing a couple of epochs of stochastic gradient descent on the collected batch).
4. Optionally, replace the worth operate (critic) by minimizing a worth loss, since PPO sometimes trains the critic concurrently to enhance benefit estimates.

As a result of PPO is on-policy (it makes use of recent information from the present coverage for every replace), it forgoes the pattern effectivity of off-policy algorithms like DQN. Nevertheless, PPO usually makes up for this by being steady and scalable it’s simple to parallelize (gather information from a number of atmosphere cases) and doesn’t require advanced expertise replay or goal networks. It has been proven to work robustly throughout many domains (robotics, video games, and many others.) with comparatively minimal hyperparameter tuning. Actually, PPO grew to become one thing of a default selection for a lot of RL issues attributable to its reliability.

PPO variants: There are two main variants of PPO that have been mentioned within the authentic papers:

• PPO-penalty: which provides a penalty to the target proportional to the KL-divergence between new and previous coverage (and adapts this penalty coefficient throughout coaching). That is nearer in spirit to TRPO’s strategy (hold KL small by specific penalty).
• PPO-clip: which is the variant we described above utilizing clipped goal and no specific KL time period. That is by far the extra in style model and what folks often imply by “PPO”.

Each variants goal to limit coverage change; PPO-clip grew to become commonplace due to its simplicity and powerful efficiency. PPO additionally sometimes contains entropy bonus regularization (to encourage exploration by not making the coverage too deterministic too rapidly) and different sensible tweaks, however these are particulars past our scope right here.

Why PPO is in style – benefits: To sum up, PPO presents a compelling mixture of stability and simplicity. It doesn’t collapse or diverge simply throughout coaching due to the clipped updates, and but it’s a lot simpler to implement than older trust-region strategies. Researchers and practitioners have used PPO for the whole lot from controlling robots to coaching game-playing brokers. Notably, PPO (with slight modifications) was utilized in OpenAI’s InstructGPT and different large-scale RL from human suggestions initiatives to fine-tune language fashions, attributable to its stability in dealing with high-dimensional motion areas like textual content. It might not at all times be absolutely the most sample-efficient or fastest-learning algorithm on each process, however when unsure, PPO is usually a dependable selection.

PPO and GRPO vs Different RL Algorithms

To place issues in perspective, let’s briefly examine PPO (and by extension GRPO) with another in style RL algorithms, highlighting key variations:

• DQN (Deep Q-Community, 2015): DQN is a value-based methodology, not a coverage gradient. It learns a Q-value operate (by way of deep neural community) for discrete actions, and the coverage is implicitly “take the motion with highest Q”. DQN makes use of tips like an expertise replay buffer (to reuse previous experiences and break correlations) and a goal community (to stabilize Q-value updates). In contrast to PPO which is on-policy and updates a parametric coverage instantly, DQN is off-policy and doesn’t parameterize a coverage in any respect (the coverage is grasping w.r.t. Q). PPO sometimes handles massive or steady motion areas higher than DQN, whereas DQN excels in discrete issues (like Atari) and may be extra sample-efficient because of replay.
• A3C (Asynchronous Benefit Actor-Critic, 2016): A3C is an earlier coverage gradient/actor-critic algorithm that makes use of a number of employee brokers in parallel to gather expertise and replace a worldwide mannequin asynchronously. Every employee runs by itself atmosphere occasion, and their updates are aggregated to a central set of parameters. This parallelism decorrelates information and hurries up studying, serving to to stabilize coaching in comparison with a single agent working sequentially. A3C makes use of a bonus actor-critic replace (usually with n-step returns) however doesn’t have the express “clipping” mechanism of PPO. Actually, PPO may be seen as an evolution of concepts from A3C/A2C – it retains the on-policy benefit actor-critic strategy however provides the surrogate clipping to enhance stability. Empirically, PPO tends to outperform A3C, because it did on many Atari video games with far much less wall-clock coaching time, attributable to extra environment friendly use of batch updates (A2C, a synchronous model of A3C, plus PPO’s clipping equals sturdy efficiency). A3C’s asynchronous strategy is much less widespread now, since you possibly can obtain comparable advantages with batched environments and steady algorithms like PPO.
• TRPO (Belief Area Coverage Optimization, 2015): TRPO is the direct predecessor of PPO. It launched the thought of a “belief area” constraint on coverage updates basically guaranteeing the brand new coverage isn’t too removed from the previous coverage by imposing a constraint on the KL divergence between them. TRPO makes use of a fancy optimization (fixing a constrained optimization drawback with a KL constraint) and requires computing approximate second order gradients (by way of conjugate gradient). It was a breakthrough in enabling bigger coverage updates with out chaos, and it improved stability and reliability over vanilla coverage gradient. Nevertheless, TRPO is sophisticated to implement and may be slower as a result of second-order math. PPO was born as a less complicated, extra environment friendly different that achieves comparable outcomes with first-order strategies. As a substitute of a tough KL constraint, PPO both softens it right into a penalty or replaces it with the clip methodology. Consequently, PPO is simpler to make use of and has largely supplanted TRPO in apply. When it comes to efficiency, PPO and TRPO usually obtain comparable returns, however PPO’s simplicity provides it an edge for growth pace. (Within the context of GRPO: GRPO’s replace rule is basically a PPO-like replace, so it additionally advantages from these insights while not having TRPO’s equipment).
• DDPG (Deep Deterministic Coverage Gradient, 2015): DDPG is an off-policy actor-critic algorithm for steady motion areas. It combines concepts from DQN and coverage gradients. DDPG maintains two networks: a critic (like DQN’s Q-function) and an actor that deterministically outputs an motion. Throughout coaching, DDPG makes use of a replay buffer and a goal community (like DQN) for stability, and it updates the actor utilizing the gradient of the Q-function (therefore “deterministic coverage gradient”). In easy phrases, DDPG extends Q-learning to steady actions by utilizing a differentiable coverage (actor) to pick out actions, and it learns that coverage by gradients via the Q critic. The draw back is that off-policy actor-critic strategies like DDPG may be considerably finicky – they might get caught in native optima or diverge with out cautious tuning (enhancements like TD3 and SAC have been later developed to handle a few of DDPG’s weaknesses). In comparison with PPO, DDPG may be extra sample-efficient (replaying experiences) and might converge to deterministic insurance policies which is likely to be optimum in noise-free settings, however PPO’s on-policy nature and stochastic coverage could make it extra sturdy in environments requiring exploration. In apply, for steady management duties, one may select PPO for ease and robustness or DDPG/TD3/SAC for effectivity and efficiency if tuned effectively.

In abstract, PPO (and GRPO) vs others: PPO is an on-policy, coverage gradient methodology targeted on steady updates, whereas DQN and DDPG are off-policy value-based or actor-critic strategies targeted on pattern effectivity. A3C/A2C are earlier on-policy actor-critic strategies that launched helpful tips like multi-environment coaching, however PPO improved on their stability. TRPO laid the theoretical groundwork for protected coverage updates, and PPO made it sensible. GRPO, being a spinoff of PPO, shares PPO’s benefits however simplifies the pipeline additional by eradicating the worth operate making it an intriguing choice for situations like large-scale language mannequin coaching the place utilizing a worth community is problematic. Every algorithm has its personal area of interest, however PPO’s normal reliability is why it’s usually a baseline selection in lots of comparisons.

PPO in Apply: Code Instance

To solidify our understanding, let’s see a fast instance of how one would use PPO in apply. We’ll use a preferred RL library (Secure Baselines3) and practice a easy agent on a basic management process (CartPole). This instance can be in Python utilizing PyTorch below the hood, however you received’t have to implement the PPO replace equations your self – the library handles it.

Within the code above, we first create the CartPole atmosphere (a basic balancing pole toy drawback). We then create a PPO mannequin with an MLP (multi-layer perceptron) coverage community. Below the hood, this units up each the coverage (actor) and worth operate (critic) networks. Calling mannequin.be taught(...) launches the coaching loop: the agent will work together with the atmosphere, gather observations, calculate benefits, and replace its coverage utilizing the PPO algorithm. The verbose=1 simply prints out coaching progress. After coaching, we run a fast check: the agent makes use of its realized coverage (mannequin.predict(obs)) to pick out actions and we step via the atmosphere to see the way it performs. If all went effectively, the CartPole ought to stability for an honest variety of steps.

import gymnasium as fitness center
from stable_baselines3 import PPO

env = fitness center.make("CartPole-v1")

mannequin = PPO(coverage="MlpPolicy", env=env, verbose=1)

mannequin.be taught(total_timesteps=50000)

# Check the educated agent
obs, _ = env.reset()
for step in vary(1000):
    motion, _state = mannequin.predict(obs, deterministic=True)
    obs, reward, terminated, truncated, data = env.step(motion)
    if terminated or truncated:
        obs, _ = env.reset()

This instance is deliberately easy and domain-generic. In additional advanced environments, you may want to regulate hyperparameters (just like the clipping, studying fee, or use reward normalization) for PPO to work effectively. However the high-level utilization stays the identical outline your atmosphere, choose the PPO algorithm, and practice. PPO’s relative simplicity means you don’t should fiddle with replay buffers or different equipment, making it a handy start line for a lot of issues.

Conclusion

On this article, we explored the panorama of coverage optimization in reinforcement studying via the lens of PPO and GRPO. We started with a refresher on how RL works and why coverage gradient strategies are helpful for instantly optimizing resolution insurance policies. We then launched GRPO, studying the way it forgoes a critic and as an alternative learns from relative comparisons in a gaggle of actions – a technique that brings effectivity and ease in sure settings. We took a deep dive into PPO, understanding its clipped surrogate goal and why that helps keep coaching stability. We additionally in contrast these algorithms to different well-known approaches (DQN, A3C, TRPO, DDPG), to focus on when and why one may select coverage gradient strategies like PPO/GRPO over others.

Each PPO and GRPO exemplify a core theme in fashionable RL: discover methods to get huge studying enhancements whereas avoiding instability. PPO does this with light nudges (clipped updates), and GRPO does it by simplifying what we be taught (no worth community, simply relative rewards). As you proceed your RL journey, hold these rules in thoughts. Whether or not you’re coaching a sport agent or a conversational AI, strategies like PPO have develop into go-to workhorses, and newer variants like GRPO present that there’s nonetheless room to innovate on stability and effectivity.

Sources:

1. Sutton, R. & Barto, A. Reinforcement Learning: An Introduction. (Background on RL basics).
2. Schulman et al. Proximal Policy Optimization Algorithms. arXiv:1707.06347 (PPO authentic paper).
3. OpenAI Spinning Up – PPO (PPO rationalization and equations).
4. RLHF Handbook – Policy Gradient Algorithms (Particulars on GRPO formulation and instinct).
5. Stable Baselines3 Documentation(DQN description) (PPO vs others).