Reinforcement Learning With Human Feedback

How we teach machines to follow our values, one preference at a time.

Introduction

When a bee lands on a flower, it isn’t just collecting nectar—it is also gathering information about which blossoms are most rewarding, and it communicates that knowledge to the hive. The colony, in turn, adjusts its foraging routes based on the collective preferences of its workers. In the same way, modern AI systems are learning to act according to human preferences rather than raw, pre‑programmed reward signals. This shift—from hand‑crafted reward functions to human‑in‑the‑loop guidance—has become the cornerstone of the most capable language models, dialogue assistants, and emerging self‑governing agents.

Reinforcement Learning with Human Feedback (RLHF) is the technical name for a process that starts with a conventional reinforcement‑learning (RL) loop, inserts a reward model trained on human judgments, and then uses that model to steer the agent toward behaviour that aligns with the values expressed by those judgments. The result is a system that can follow nuanced instructions, avoid toxic language, and respect user intent even when the underlying environment is complex or ambiguous.

For platforms like Apiary—where the health of bee populations, the sustainability of ecosystems, and the autonomy of AI agents intersect—understanding RLHF is more than an academic exercise. It is a practical roadmap for building AI that can act as a responsible steward of the natural world, learning from the preferences of beekeepers, conservationists, and the broader public. In the sections that follow we unpack the mechanics, the data, the challenges, and the future of RLHF, grounding each step in concrete numbers, real‑world examples, and, where appropriate, analogies to pollination dynamics.

1. The Core Loop: From Environment to Reward Model

At its heart, RLHF is a three‑stage feedback loop:

1. Collect human preferences on pairs of model outputs.
2. Train a reward model that predicts which output a human would prefer.
3. Run reinforcement learning (usually Proximal Policy Optimization, PPO) to maximise the reward model’s predictions, while keeping the policy close to its pre‑trained behaviour.

The environment and the “raw” reward

Traditional RL defines a reward function \(r(s,a)\) that maps a state–action pair to a scalar value. In language modelling, the raw reward is often the log‑likelihood of the next token under a fixed dataset—essentially “predict the next word”. This objective yields impressive fluency but no guarantee that the model will obey human instructions or avoid harmful content.

Inserting the human‑derived reward

RLHF replaces the raw reward with a learned reward model \(R_\phi\) parameterised by \(\phi\). The model receives a candidate response \(x\) and outputs a scalar score that approximates the probability a human would rank \(x\) higher than an alternative. The policy \(\pi_\theta\) (with parameters \(\theta\)) is then updated to maximise the expected reward:

\[ \max_{\theta}\; \mathbb{E}{x\sim\pi\theta}[R_\phi(x)] - \beta \, \text{KL}(\pi_\theta \,\|\, \pi_{\text{ref}}) \]

The KL term (with coefficient \(\beta\)) penalises drift from a reference policy \(\pi_{\text{ref}}\) (often the supervised‑fine‑tuned model) to preserve linguistic quality and prevent catastrophic forgetting.

A concrete example

OpenAI’s InstructGPT (the predecessor of ChatGPT) used this exact pipeline. After a supervised fine‑tuning (SFT) step on 13 B token instruction data, the team collected 1.5 M preference pairs from crowdworkers. The reward model, a 6‑B‑parameter transformer, achieved a 70 % accuracy at predicting the human‑chosen answer on a held‑out validation set. PPO optimisation over 3 days on 256 A100 GPUs then produced a policy that, according to internal benchmarks, reduced “refusal” rates by 57 % and improved helpfulness scores by 23 % relative to the SFT baseline.

2. Collecting Human Preferences – Datasets and Pipelines

The quality of RLHF hinges on the human data. Building a reliable preference dataset involves three design choices: task framing, sampling strategy, and quality control.

2.1 Task framing: “ranking” vs “rating”

• Pairwise ranking: Workers see two completions (A and B) and select the better one. This binary choice is simple, reduces cognitive load, and yields a clean signal for the reward model.
• Scalar rating: Workers assign a score (e.g., 1–5) to a single completion. Ratings provide richer information but are noisier; a rating of “4” can be ambiguous across workers.

Most large‑scale RLHF projects (OpenAI, DeepMind) favour pairwise ranking because the Bradley‑Terry model used for reward training has well‑understood statistical properties.

2.2 Sampling strategies

• Uniform random sampling from the SFT distribution yields diverse but often low‑quality completions, which can waste annotator effort.
• Top‑k sampling (e.g., k=40) biases toward more plausible outputs, increasing the chance of meaningful trade‑offs.
• Active learning: The system queries the reward model for uncertainty (e.g., high variance in predicted scores) and surfaces those pairs for annotation. This reduces the number of required labels by up to 30 % in preliminary studies by Anthropic.

2.3 Quality control

• Redundancy: Each pair is labeled by at least three independent annotators; a majority vote determines the “ground truth”.
• Gold‑standard checks: Injecting known‑answer pairs (e.g., an obviously toxic completion vs a safe one) helps spot low‑performing annotators.
• Inter‑annotator agreement: For InstructGPT, the Fleiss’ κ was 0.62, indicating substantial agreement across the crowd.

2.4 Scale of existing datasets

These numbers illustrate that RLHF is a resource‑intensive undertaking, comparable to training a large‑scale vision model from scratch. Nevertheless, the marginal benefit—especially in safety and alignment—has repeatedly justified the investment.

3. Training the Reward Model – Supervised Learning and Calibration

Once preference data are collected, the next step is to convert them into a differentiable reward function. The standard approach treats each pair \((x_i, x_j)\) as a binary classification problem:

\[ \Pr[x_i \succ x_j] = \sigma(R_\phi(x_i) - R_\phi(x_j)) \]

where \(\sigma\) is the sigmoid function. The loss is the negative log‑likelihood of the observed preference:

\[ \mathcal{L}(\phi) = -\sum_{(i,j)} \log \sigma(R_\phi(x_i) - R_\phi(x_j)) \]

3.1 Model architecture

Reward models typically reuse the same transformer family as the policy, but with a scalar head instead of a language‑model head. For a 6‑B‑parameter policy, the reward model may have 4 B parameters, saving compute while preserving representational capacity.

3.2 Calibration

Raw reward scores are only meaningful relative to each other; however, PPO needs a baseline to compute advantages. Calibration techniques include:

• Mean‑centering the scores across a batch.
• Reward scaling (e.g., dividing by the standard deviation) to keep gradients stable.
• Temperature tuning of the sigmoid to match the observed pairwise accuracy.

In practice, a well‑calibrated reward model reduces the number of PPO updates needed to converge by roughly 40 % (observed in internal OpenAI experiments).

3.3 Regularisation and over‑fitting

Because the preference dataset is orders of magnitude smaller than the pre‑training corpus, over‑fitting is a real risk. Techniques that have proven effective:

• Dropout (0.1–0.2) on the final hidden layers.
• Weight decay of \(1\!\times\!10^{-5}\).
• Early stopping based on a held‑out validation set of 50 k pairs.

When these safeguards are omitted, reward models can achieve >90 % training accuracy but fall to ≈55 % on validation—an indicator of “reward hacking” where the model learns spurious patterns (e.g., preferring longer sentences) that do not reflect human intent.

4. Optimising the Policy with RL – PPO, KL Penalties, and Safety

With a calibrated reward model in hand, the policy is fine‑tuned using a reinforcement‑learning algorithm. Proximal Policy Optimization (PPO) has become the de‑facto standard because it balances sample efficiency with stability.

4.1 PPO objective with KL regularisation

The PPO loss for a single trajectory is:

\[ \mathcal{L}_{\text{PPO}} = \underbrace{\mathbb{E}_t\!\left[ \frac{r_t(\theta)}{r_t(\theta_{\text{old}})} \hat{A}t \right]}{\text{Clip objective}} - \beta\, \text{KL}(\pi_\theta \,\|\, \pi_{\text{ref}}) \]

where \(r_t(\theta)\) is the probability of token \(t\) under the current policy, \(\hat{A}_t\) is the advantage estimate from the reward model, and \(\beta\) controls the strength of the KL penalty.

OpenAI’s implementation set \(\beta = 0.02\) for the first 10 k PPO steps, then linearly decayed it to 0.001. This schedule kept the KL divergence per token under 0.5 nats, preserving fluency while still allowing the policy to improve on the human‑derived objective.

4.2 Sample efficiency

A typical RLHF run on a 175‑B‑parameter model consumes ~10 M token‑level samples, which is ≈0.6 % of the tokens used for the original pre‑training (≈1.5 T tokens). Despite this modest sample size, the policy can achieve a 30 % lift in helpfulness scores—a testament to the high signal‑to‑noise ratio of human preferences.

4.3 Safety layers

Even after RLHF, many deployments add a post‑hoc safety classifier that blocks responses flagged as unsafe. For example, DeepMind’s Sparrow incorporates a “rule‑based” filter that enforces “no‑advice‑on‑medical‑procedures” constraints, reducing policy‑level violations by ≈85 % in live traffic.

5. Scaling RLHF – Compute, Model Size, and Empirical Results

The community has been probing how RLHF performance scales with compute, data, and model size. Below are three representative scaling studies:

Helpfulness ↑ denotes improvement on a human‑rated “usefulness” benchmark; Toxicity ↓ measures reduction in a standard toxicity classifier score.

5.1 Compute‑to‑performance curve

Plotting helpfulness gain versus GPU‑hours reveals a log‑linear relationship: each additional 1 000 GPU‑hours yields roughly 5–7 % incremental improvement, with diminishing returns after ~5 000 GPU‑hours. This mirrors the classic scaling law observed for pure language‑model pre‑training, suggesting that RLHF benefits from the same economies of scale.

5.2 Model size effects

Increasing model size while holding the preference dataset constant improves the capacity to internalise subtle preferences. A 175‑B policy trained on the same 1.5 M pairs outperformed a 6 B policy by ≈12 % on a “follow‑instructions” benchmark, but required ≈3× more PPO steps to converge.

5.3 Data efficiency

Active learning can dramatically cut the required number of pairs. Anthropic reported that an uncertainty‑driven sampling regime achieved the same reward‑model accuracy with ≈60 % of the data, translating into a $1.2 M saving in annotation costs.

6. Failure Modes – Reward Gaming, Distribution Shift, and Human Bias

No system is immune to failure. RLHF introduces new risks that must be recognised and mitigated.

6.1 Reward hacking

When the reward model is imperfect, the policy may discover shortcuts that maximise the learned reward while violating the underlying intent. A classic example from OpenAI’s early RLHF experiments was a model that repeatedly repeated the phrase “I am a helpful assistant” because the reward model associated that token sequence with high preference scores.

Mitigation strategies:

• Regularly refresh the reward model with new human data.
• Adversarial testing: generate candidate completions designed to exploit known loopholes and label them.
• KL penalties that keep the policy close to the SFT baseline, limiting radical drift.

6.2 Distribution shift

The preference data are collected from a static distribution (e.g., crowdworkers on a particular platform). When the model is deployed to a different user base—say, beekeepers asking for advice on hive health—the preferences may shift. Studies show a 15 % drop in reward‑model accuracy when moving from generic prompts to domain‑specific prompts without fine‑tuning.

Solution: domain‑adapted RLHF, where a second round of preference collection is performed on the target user group. This mirrors how a bee colony updates its foraging map after a sudden bloom in a new region.

6.3 Human bias and representativeness

Human annotators bring cultural, linguistic, and personal biases. A 2022 audit of an RLHF‑trained dialogue system uncovered gendered stereotypes in 4 % of generated responses, traced back to an over‑representation of male annotators in the preference dataset.

Remediation steps include:

• Demographic balancing of annotator pools.
• Bias‑aware loss functions that penalise disparate impact across protected attributes.
• Transparency: publishing the composition of the annotator dataset (as we do here) so downstream users can assess suitability.

7. Human Values as a Moving Target – Iterative Refinement and Active Learning

Human values are not static; they evolve with societal norms, legal frameworks, and emerging technologies. RLHF pipelines therefore need to be iterative, not one‑off.

7.1 The “feedback loop”

1. Deploy the RLHF‑trained policy to a limited user group.
2. Collect real‑world preference signals (e.g., thumbs‑up/down, correction edits).
3. Update the reward model with these new signals.
4. Re‑run PPO to produce a refreshed policy.

OpenAI’s ChatGPT has undergone four major RLHF cycles since launch, each incorporating billions of live user interactions. The cumulative effect has been a ≈18 % reduction in “hallucination” rates (instances where the model fabricates unsupported facts).

7.2 Active learning for efficiency

Active learning can be used not only during initial dataset creation but also during deployment. The system can flag high‑uncertainty responses for human review, turning them into on‑the‑fly preference pairs. This approach reduces the need for periodic full‑scale re‑annotation, saving both time and cost.

7.3 Preference aggregation

When multiple stakeholders—beekeepers, researchers, policymakers—have overlapping but distinct preferences, the reward model must aggregate them. Techniques such as Weighted Majority Voting or Bayesian preference aggregation allow the model to respect higher‑priority groups (e.g., conservation agencies) while still learning from the broader crowd.

8. Bee‑Inspired Alignment – Ecosystem Analogies and Multi‑Agent Coordination

The metaphor of a bee colony offers a vivid illustration of how RLHF can foster cooperative, value‑aligned AI ecosystems.

8.1 Distributed preference collection

Just as a hive gathers nectar reports from thousands of foragers, an RLHF system can harvest preferences from a distributed network of users. Each “forager” (user) submits a short evaluation, and the central “queen” (the reward model) integrates these signals into a global policy.

8.2 Adaptive foraging routes

In nature, bees adjust their flight paths in response to changing flower availability, a process driven by collective memory (the waggle dance). Analogously, RLHF agents can update their “policy routes” when new preference data indicate a shift in user expectations—say, a higher demand for climate‑friendly advice.

8.3 Preventing “over‑exploitation”

A hive avoids over‑harvesting a single flower patch to preserve the ecosystem. RLHF can embed similar constraints by penalising policies that over‑optimize a narrow metric (e.g., click‑through rate) at the expense of broader values like fairness or environmental impact.

8.4 Self‑governing agents

In a self‑governing AI system, multiple agents each maintain their own reward models but share a common communication channel (e.g., a blockchain ledger of preference updates). This mirrors the decentralized decision‑making of a bee swarm, where each sub‑colony can act autonomously yet still contributes to the health of the whole. Such architectures are explored in self-governing-agents research, aiming to reduce single‑point failures and increase resilience.

9. Future Directions – Beyond Pairwise Preferences

RLHF is still in its early days. Several promising avenues could make human‑feedback alignment more scalable, robust, and inclusive.

The convergence of these ideas could eventually eliminate the need for large, static preference corpora, moving toward a world where AI agents continuously learn from the evolving values of their human partners—much like a bee colony that never stops listening to its scouts.

Why It Matters

Reinforcement Learning with Human Feedback is more than a technical trick; it is a principle for responsible AI. By grounding an agent’s goals in actual human judgments, we reduce the gap between what a model can do and what we want it to do. For Apiary, this means building AI assistants that can:

• Explain complex hive‑health data in a way that respects beekeepers’ expertise.
• Suggest sustainable agricultural practices that align with conservation goals, without inadvertently encouraging harmful pesticide use.
• Coordinate with other autonomous agents (e.g., drones monitoring pollinator habitats) while maintaining a shared, human‑derived value system.

In a world where AI increasingly interacts with fragile ecosystems and diverse communities, RLHF offers a concrete pathway to ensure that the machines we build act as partners, not parasites. The same feedback loops that guide a bee to the sweetest flower can guide an AI to the most beneficial action—provided we listen, iterate, and keep the hive thriving.