Policy Gradient methods are a class of reinforcement learning algorithms that directly optimize policies by computing gradients of expected returns with respect to policy parameters. Unlike value-based methods that learn action-value functions and derive policies from them, policy gradient approaches parameterize and optimize policies directly, making them particularly effective for continuous action spaces and stochastic policies.
Mathematical Foundation
Policy gradient methods are grounded in the policy gradient theorem, which provides a computable expression for the gradient of the expected return. For a parameterized policy πθ(a|s), the gradient of the expected return J(θ) = Eπ_θ[R(τ)] is given by:
∇θ J(θ) = Eπθ[∇θ log πθ(a|s) Qπ(s,a)]
This fundamental result shows that the policy gradient can be estimated by sampling trajectories and computing the product of the log-derivative of the policy and the action-value function. The log-derivative term ∇θ log πθ(a|s) represents the sensitivity of the policy to parameter changes, while Q_π(s,a) measures the quality of state-action pairs.
REINFORCE Algorithm
The REINFORCE algorithm, introduced by Williams in 1992, is the canonical policy gradient method. It estimates the policy gradient using Monte Carlo returns:
∇_θ J(θ) ≈ (1/T) Σt=1^T ∇θ log π_θ(a_t|s_t) G_t
where G_t is the return from time step t onwards. REINFORCE is unbiased but suffers from high variance due to the use of raw returns. The algorithm requires complete episodes before updates can be computed, making it suitable for episodic tasks but inefficient for continuing problems.
Actor-Critic Methods
Actor-critic architectures combine policy gradient methods with value function approximation to reduce variance. The "actor" represents the policy πθ(a|s), while the "critic" estimates the value function Vφ(s) or action-value function Q_φ(s,a). The critic provides low-variance estimates of policy performance, replacing the high-variance returns in basic policy gradient methods.
The advantage function A_π(s,a) = Q_π(s,a) - V_π(s) is commonly used in actor-critic methods, leading to the update:
∇θ J(θ) ≈ Eπθ[∇θ log πθ(a|s) Aπ(s,a)]
Advantage estimation reduces variance by subtracting a baseline that depends only on state, as the baseline does not affect the expected gradient but can significantly reduce variance.
Variance Reduction Techniques
Several techniques address the high variance inherent in policy gradient methods. Baselines, such as state-value functions, subtract terms that do not bias the gradient but reduce variance. Generalized Advantage Estimation (GAE) combines multiple n-step returns with exponentially decaying weights to balance bias and variance.
Importance sampling techniques allow policy updates using data from different policies, enabling off-policy learning. Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO) constrain policy updates to prevent destructive large changes, improving training stability.
Applications and Variants
Policy gradient methods excel in continuous control domains where action spaces are high-dimensional or continuous. They naturally handle stochastic policies, making them suitable for partially observable environments where deterministic policies are suboptimal. Deep deterministic policy gradient (DDPG) extends policy gradients to continuous deterministic policies using deep neural networks.
Asynchronous Advantage Actor-Critic (A3C) parallelizes training across multiple environments, improving sample efficiency. Soft Actor-Critic (SAC) incorporates maximum entropy principles, encouraging exploration while optimizing policies. These variants demonstrate the flexibility of the policy gradient framework in addressing specific challenges in reinforcement learning.
Theoretical Properties
Policy gradient methods offer several theoretical advantages. They can converge to local optima of the policy objective, with convergence guarantees under appropriate conditions. The direct policy optimization approach avoids issues with policy degradation that can occur in value-based methods when approximation errors compound.
However, policy gradients face challenges including high variance, sensitivity to hyperparameters, and potential for getting trapped in poor local optima. The choice of baseline, learning rate, and policy parameterization significantly impacts performance. Despite these limitations, policy gradient methods remain fundamental tools in modern reinforcement learning, particularly for complex control tasks and continuous domains where their advantages outweigh their drawbacks.