Darpan’s Blog

The Evolution of the Attention Mechanism in LLMs

2026-11-05T17:14:32-08:00

Serving one 4,096-token request on Qwen3-32B takes about 1.07 GB of key-value cache. Give every query head its own keys and values, as the original Transformer did, and that same request would need 8.59 GB.

As a language model’s context grows, reading and storing its history becomes more expensive. Attention designs tackle different parts of that cost: some share keys and values across heads, some compress what each token stores, and others restrict which positions a query reads.

We’ll follow these choices from a single attention head to hybrid memory and paged caches, using small worked examples. The question throughout is simple: what changes, what does it save, and what do we give up?

1. Attention Was All We Needed?

A language model needs more than a representation of each token in isolation. It needs a way to bring information from the surrounding context into that representation. Attention provides that connection: each position computes weights over the context it can see, then uses those weights to combine information from those positions.

In a Transformer, this happens inside a block that also contains a feed-forward network. Stack these blocks, and the representations produced by one become the inputs to the next. In the decoder-only pattern used by GPT-style models, a vocabulary projection at the end produces scores for the next token.

Simplified decoder-only architecture; residual connections and normalization are omitted. Original schematic following Jay Alammar’s The Illustrated GPT-2.

An attention head is one learned query/key/value projection pathway within the attention layer. It produces its own weights for mixing token information. Multiple heads give the layer several such pathways in parallel. For a fuller introduction, Jay Alammar’s The Illustrated Transformer and The Illustrated GPT-2 cover the surrounding architecture in detail.

We’ll start with multiple heads, then explore shared KV caches, restricted reads and compressed representations. From there, we’ll look at output gates, hybrid memory and cache allocation. Each design addresses a different constraint, and a model can combine several of them.

The companion examples use small inputs to make the calculations and storage costs inspectable. They demonstrate how the mechanisms work. The weights are chosen by hand, not trained, so they say nothing about model quality, and none of the examples is a latency or throughput benchmark.

2. Single-Head Self-Attention

Start with one head. The original Transformer already used multiple heads, but isolating one lets us see the computation that the later designs modify.

Self-attention projects the input into queries, keys and values. Queries and keys determine how strongly positions match; values carry the information that gets mixed. The scaled dot-product formulation combines these operations:

\[\begin{aligned} Q &= XW_Q, \quad K = XW_K, \quad V = XW_V \\ A &= \operatorname{softmax}\!\left(\frac{QK^{\mathsf T}}{\sqrt{d_k}} + M\right) \\ Y &= (AV)W_O \end{aligned}\]

Here $d_k$ is the query/key dimension. Scaling moderates dot-product growth with dimension, and softmax normalizes the scores across keys. For causal attention, $M$ masks future positions before normalization, so each query reads only its own position and earlier ones.

One head computes a distribution over visible positions, mixes their values, and applies an output projection. Original schematic; see Alammar’s Q/K/V walkthrough and the Transformer paper.

For a concrete example, take “The cat sat” with chosen vectors $[1, 0]$, $[0, 1]$ and $[1, 1]$. Use identity projections so we can follow the calculation directly. These are untrained teaching vectors, with positional encoding omitted. The query for “sat” has dot products $[1, 1, 2]$ with the keys. Scaling and softmax turn these into weights of approximately $[0.248, 0.248, 0.503]$. Mixing the values gives $[0.752, 0.752]$.

Computed weights for the example: each row is a query, each column a key/value position. Future positions receive zero weight.

The result is a context-dependent mixture. To understand its cost, distinguish processing a sequence from generating one new token. Holding the head dimensions, batch size and numerical precision fixed:

Quantity	Growth with sequence length $n$	Why
Full-sequence attention work, as during prompt prefill	$\mathcal{O}(n^2)$	Each query attends to its visible keys
Attention work for one new token with a KV cache	$\mathcal{O}(n)$	The new query reads the retained history
Retained KV-cache storage	$\mathcal{O}(n)$	Each retained token contributes one key and value per head

Full-sequence attention has quadratically many query-key interactions, but implementations can avoid storing the entire score matrix at once. A KV cache reuses earlier keys and values during generation; the new query still has to read them. The runnable implementation and example show both full-sequence and cached execution.

There is another constraint: one head gives each query one attention distribution. Several relationships can contribute, but they share those mixing weights. Multiple heads let the layer form separate distributions through different learned projections, then combine their outputs. That is the first architectural change we’ll make.

Key Takeaways

One head forms a context-dependent mixture of visible token representations.
A causal mask excludes future positions; a KV cache reuses earlier keys and values during generation.
Full-sequence attention work grows quadratically; one cached decoding step and retained KV storage grow linearly with history length.

3. Multi-Head Self-Attention

Single-head attention has a fundamental limitation: each query gets only one attention distribution. But language is rich with different types of dependencies—syntactic, semantic, positional, and more. Multi-head attention solves this by running multiple attention operations in parallel.

The Intuition

Think of the heads as specialists that can look for different patterns in the same context. Their roles are learned: one head can capture several relationships, and different heads can learn overlapping patterns.

Mathematical Formulation

The attention calculation stays the same. We now give each head its own learned query, key and value projections, then concatenate their outputs and apply a learned output projection:

\[\begin{aligned} \operatorname{head}_i &= \operatorname{Attention}(XW_Q^i, XW_K^i, XW_V^i) \\ \operatorname{MultiHead}(X) &= \operatorname{Concat}(\operatorname{head}_1, \ldots, \operatorname{head}_h)W_O \end{aligned}\]

The Transformer paper describes this as attending across different representation subspaces.

Let’s reuse “The cat sat” and the same input vectors. This time, split the two input coordinates across two heads: the first head projects onto the first coordinate, and the second onto the second. Each head therefore has width one, keeping the total width unchanged. These are chosen projections to make the difference visible, with positional encoding still omitted.

For the final query, the first head assigns weights of approximately $[0.422, 0.155, 0.422]$ to “The,” “cat” and “sat.” The second assigns $[0.155, 0.422, 0.422]$. Same input, different mixtures. Their scalar outputs are concatenated into $[0.845, 0.845]$; an identity output projection leaves that vector unchanged in this example.

The architecture above the heatmap shows how the head outputs are combined. The calculated weights below show the final query from our single-head example alongside the two narrower heads.

Putting the Heads to Work

We don’t need a Python loop for each head. The implementation packs their projections into larger linear layers, reshapes the result to expose a head dimension, and computes attention across those heads in a batch. The runnable example exposes the weights so we can inspect the patterns above.

What Does This Cost Us?

Here’s a detail that’s easy to miss: more heads do not automatically mean a larger KV cache. Usually, we divide a fixed model width across the heads. If $h \times d_{\text{head}} = d_{\text{model}}$, changing the head count while keeping $d_{\text{model}}$ fixed leaves the total cached width unchanged.

For equal key and value widths, the cache payload in bytes across the model is:

\[\begin{aligned} &2 \times \text{batch\_size} \times \text{cached\_length} \times \text{n\_layers} \\ &\quad \times h \times d_{\text{head}} \times \text{bytes\_per\_element} \end{aligned}\]

The cache still grows with context length, batch size and layer count. Each head retains its own keys and values, and each new query reads that history. Splitting into heads also keeps full-context attention quadratic in sequence length.

That leaves us with a useful question: do all those query heads need separate keys and values? Could they ask different questions while reading from shared memory? That is the idea behind multi-query attention.

Key Takeaways

Separate projections let heads form different attention patterns, which are combined through an output projection.
Packed projections and batched attention compute the heads together.
At fixed total width, more heads do not automatically increase KV-cache size. Retaining separate keys and values for every head still leaves a memory cost worth reducing.

Multi-head attention gives every head its own keys and values to store during generation. But do all those heads need separate memories? Multi-query attention (MQA) lets each head ask its own question through a separate query projection, while sharing one set of keys and values across all heads. With eight heads sharing one K/V set, for example, the KV cache becomes eight times smaller at the same head width. Reading less data at each decoding step can also mean lower latency.

MQA: One Shared KV Head

The query projections stay separate, but every head now reads the same keys and values. The causal mask and the way we combine outputs stay unchanged:

\[\begin{aligned} Q_i &= XW_Q^i, \quad K_s = XW_K, \quad V_s = XW_V \\ \operatorname{head}_i &= \operatorname{softmax}\!\left(\frac{Q_iK_s^{\mathsf T}}{\sqrt{d_k}} + M\right)V_s \\ \operatorname{MQA}(X) &= \operatorname{Concat}(\operatorname{head}_1, \ldots, \operatorname{head}_h)W_O \end{aligned}\]

This is the sharing strategy introduced in Noam Shazeer’s MQA paper. Shared memory still allows different attention distributions: in our “The cat sat” example, the query heads at “cat” assign $[0.500, 0.500]$ and $[0.731, 0.269]$ to the same visible positions. They read the same keys and values but ask different questions.

The catch is that all heads must work with one shared K/V representation. That restriction can affect quality, even though the queries remain separate. So what if we relaxed the sharing a little?

Instead of making every query head share one memory, give each group its own keys and values. With four query heads and two KV groups, heads 1 and 2 share one K/V pair, while heads 3 and 4 share another.

This is grouped-query attention, introduced by Joshua Ainslie and colleagues. For evenly sized groups, a query head selects its KV group as follows, using zero-based indices:

\[\begin{aligned} \text{group\_size} &= \left\lfloor \frac{\text{n\_heads}}{\text{n\_kv\_heads}} \right\rfloor \\ \text{group} &= \left\lfloor \frac{\text{head\_index}}{\text{group\_size}} \right\rfloor \\ \operatorname{head}_i &= \operatorname{Attention}(Q_i, K_{\text{group}}, V_{\text{group}}) \end{aligned}\]

Now we can describe all three designs with the same parameter:

Design	KV heads for $h$ query heads	Sharing
MHA	$h$	Each query head has its own K/V
GQA	Between $1$ and $h$	Query heads share within groups
MQA	$1$	All query heads share K/V

MQA is the one-group endpoint of GQA. They are choices along the same spectrum; applying both does not produce another round of cache savings.

Illustrative topology showing who shares with whom. Cache counts come from the same fixed-width, four-head example; the queries and sequence stay unchanged across designs.

To make an intermediate group count visible, our grouped example repeats the earlier input coordinates into a wider representation and uses four scalar query heads. Its K/V payload falls from 24 elements with MHA, to 12 with GQA, to 6 with MQA.

What Do We Save, and What Do We Give Up?

We can use one cache formula for the whole spectrum, assuming equal key and value widths:

\[\begin{aligned} \text{KV payload bytes} &= 2 \times \text{batch\_size} \times \text{cached\_length} \\ &\quad \times \text{n\_layers} \times \text{n\_kv\_heads} \\ &\quad \times d_{\text{head}} \times \text{bytes\_per\_element} \end{aligned}\]

Holding the other dimensions fixed, the reduction relative to MHA is $\frac{\text{n\_heads}}{\text{n\_kv\_heads}}$. As a larger example, Qwen3-32B’s configuration has 64 query heads and 8 KV heads. At batch size 1 and 4096 cached tokens, its 64 layers and head width 128 give a derived BF16 cache payload of 1.07 GB, versus 8.59 GB for a hypothetical MHA version: an 8× reduction. These are decimal gigabytes of K/V data, excluding weights, temporary tensors and allocator overhead.

The quality and latency tradeoff is less mechanical. The GQA study found quality close to MHA with inference speed close to MQA in its evaluated settings. Converting existing checkpoints also involved further training. Group count gives us a design choice, not a universal optimum or a free inference-time swap.

Our implementations retain only the KV groups in the cache. The GQA path uses an explicit group dimension so adjacent query heads read the right group without an explicit per-query KV repetition. Backend temporaries and kernel selection still matter.

Where These Show Up

Design	Model examples	Primary source
MHA	GPT-2	OpenAI’s implementation
MQA	PaLM; Falcon-7B	PaLM paper; Falcon-7B configuration
GQA	Llama-2-70B; Llama-3-8B and 70B	Meta’s model configurations
GQA	Falcon-40B; Mistral-7B v0.1	Falcon-40B configuration; Mistral paper
GQA	Qwen3-32B	Qwen3-32B configuration

These labels apply to the named variants, not necessarily their entire model families. Mistral-7B also combines GQA with sliding-window attention, which changes a different part of the problem: how many token positions each query reads.

Key Takeaways

MQA and GQA keep separate queries while sharing keys and values across all heads or within groups.
MHA, GQA and MQA form a spectrum controlled by the number of KV heads; their cache reductions do not stack.
Fewer KV heads reduce stored history, with quality and latency tradeoffs that depend on training and execution.
Sharing heads still leaves the sequence-length cost. Sliding-window attention tackles that next.

5. Sliding Window Attention

GQA reduces the number of KV heads, but each query still reads the retained sequence. As the context grows, so does the work. Sliding-window attention, used for long documents in Longformer, changes a different part of the problem: how far back each query looks.

The Intuition

Do tokens really need to attend to every previous token? Often, much of the useful context is nearby. Give each query a window of recent positions instead of the entire history. Here, the window size $W$ includes the current position, so a query at position $i$ can read:

\[\max(0, i - W + 1) \leq \text{key\_position} \leq i\]

Let’s extend our sentence to “The cat sat on the mat” and use a window of three positions. The query at “sat” can read “The cat sat.” Move to “on,” and the window becomes “cat sat on.” At “mat,” it reads only “on the mat.” The attention calculation within that window is unchanged; older positions simply stop participating.

Visibility and cache counts follow the verified single-layer example. Static version.

What Changes in the Cost?

Full causal attention has quadratically many query-key pairs across a sequence. With local execution, each query reads at most $W$ positions, reducing that attention work to $\mathcal{O}(nW)$ for sequence length $n$. For a fixed window, that is linear in sequence length. In our six-token example, the number of visible pairs per head falls from 21 to 15; the gap grows with longer sequences.

That benefit requires computing only the local interactions. Creating a dense score matrix and masking most of it afterward still does the dense computation. Our implementation processes local slices directly. The example runner compares full-sequence and cached local execution.

Combining It with GQA

These two changes do compose: GQA reduces the number of KV heads, while a rolling window bounds the retained positions per local layer. Once the window fills, older K/V entries can leave that layer’s cache. Its retained payload is:

\[\begin{aligned} &2 \times \text{batch\_size} \times \min(\text{cached\_length}, W) \\ &\quad \times \text{n\_kv\_heads} \times d_{\text{head}} \times \text{bytes\_per\_element} \end{aligned}\]

The token position keeps advancing even when the stored length stops growing. A rolling cache reuses its slots, but positional encoding still uses each token’s true position, so a cache slot and a token’s position are no longer the same number.

What Do We Give Up?

An older token can no longer be read directly by that local layer once it leaves the window. That can matter when a task needs information from far back. Stacked layers can relay information over longer distances, however, so the per-layer window is not the model’s entire effective context.

Mistral-7B v0.1 combines GQA with sliding-window attention; its configuration specifies a window of 4096. Gemma 2 takes another approach, alternating local sliding-window and global attention layers. Those global layers still need their broader history, so the bounded-cache formula above applies to the local layers.

Key Takeaways

Sliding-window attention restricts token positions; GQA shares KV heads. Their savings address different dimensions.
Local execution makes attention work $\mathcal{O}(nW)$ and a rolling cache bounds retained history per local layer.
The tradeoff is reduced direct access to distant tokens; stacked layers and local/global designs can provide broader information paths.

A window is a simple rule for deciding what to read. But what if the useful token is somewhere much earlier?

6. Sparse Attention Beyond Sliding Windows

A sliding window assumes recent tokens deserve our attention. Often they do. But if a function was defined two thousand tokens ago, a wider window is a fairly expensive way to find it again. Can we keep the local context and selectively reach further back?

Sparse attention is a family of answers: local bands, designated global positions, selected blocks, or tokens chosen from their content. The common idea is to spend the full attention calculation on a subset of the history. The selection rule determines what the layer can miss.

For a local window plus selected remote tokens, the calculation becomes:

\[\begin{aligned} S_t &= \operatorname{local\_window}(t) \cup \operatorname{selected\_remote\_tokens}(t) \\ \operatorname{output}_t &= \operatorname{softmax}\!\left(\frac{q_t K[S_t]^{\mathsf T}}{\sqrt{d}}\right)V[S_t] \end{aligned}\]

Both sets contain only positions up to the current token, so their union $S_t$ already enforces causal visibility.

Our example gives each query two local positions, including itself, and one remote slot. When reading “mat” in “The cat sat on the mat”, the chosen index projections select the first “The” as well as the final “the mat”: positions [0, 4, 5]. That distant token is directly visible again.

Blue cells are local reads; orange cells are selected remote reads. White cells are skipped by the main attention calculation.

With window $W$ and remote budget $k$, the main attention stage reads at most $W + k$ positions per query. Finding them has a cost too: our small indexer scans the earlier history, so selection can still take quadratic total work during prefill. It also retains the full K/V and index-key history. Fewer attention reads do not automatically mean a bounded cache or an end-to-end speedup.

Native Sparse Attention combines compressed, selected and local branches. DeepSeek-V3.2-Exp uses DeepSeek Sparse Attention (DSA), with an indexer selecting entries for sparse attention. Our local-plus-selector implementation isolates selection rather than reproducing these architectures.

Compress First, Then Select or Reuse

So far, we’ve selected individual tokens. We can also compress groups of tokens into fewer entries, then decide which of those entries to read:

Token history → compressed entries → selected compressed entries

Illustrative sequence-compression path. Compression reduces the number of historical entries; selection limits which entries a query reads.

DeepSeek-V4 combines a local window over uncompressed tokens with two forms of compressed history: Compressed Sparse Attention (CSA) selects from compressed entries, while Heavily Compressed Attention (HCA) reads all entries in a much more compressed history. The model interleaves these layer types. Compression now acts along the sequence dimension: fewer historical entries, with some detail traded for compact summaries.

DeepSeek-V4.1-Flash asks another question: does every layer need to repeat the same selection work? Its Compressed Sparse Attention 2 (CSA2) lets layers share cached representations and reuse selection results. Later decoder layers can also narrow their search to candidates found earlier, though finding that initial pool still has a cost.

For the curious: how CSA2 reuses work

Everyone Agrees with Me

2026-10-03T00:00:00-07:00

How our feeds build echo chambers that follow us into what we believe, who we trust, and how we treat the people we love.

During the recent protests in Delhi, my screen time climbed to nine or ten hours a day. I was hooked. My feed was full of explanations, media footage from independent journalists, influencers, and some friends. I saw young people, people of other ages, huge crowds protesting and supporting each other. It all felt like we were one big family. My agitation became a way of feeling connected without physically being there.

Then, occasionally, a reel would show up attacking the protestors or defending the forces being protested against. My first reaction was disbelief. How could someone, knowing everything that’s happened and seeing everything that’s happening, still reach that conclusion? Then it occurred to me that maybe they didn’t know. Maybe they weren’t seeing what I was seeing, or being told what I was being told. Information is tricky.

I wondered what their feeds looked like. How was the same information being delivered to them? How much was their surrounding context playing a role in what they were taking from it? Were they scrolling through hours of posts that made their position feel obvious?

Those occasional reels were a glimpse into another worldview. In recommendation systems, they’re a deliberate move called exploration. Most of the time, the algorithm exploits what it already knows about you, serving more of what keeps you watching. Every so often, it explores: it shows you something new or contrasting to test whether you have an interest it hasn’t found yet. If you do, that becomes one more way to keep you coming back to the app instead of going to a friend, especially when emotions are running high. Those exploration reels made me notice how much the rest of my feed agreed with me.

I still held my views. I just began asking how much I hadn’t seen. Nine hours of scrolling can feel like you know the whole story, even when you keep encountering versions of the same argument.

Everyone seemed to agree with me. I had started to wonder who “everyone” actually included.

A room full of your own voice

An Echo Chamber is an environment where beliefs get reinforced by like-minded voices, while competing perspectives struggle to enter. A study of more than 100M pieces of social media content¹ found that these patterns differ across platforms. Both our social connections and feed design matter.

Once I started noticing that pattern, I wondered where else it shows up. My playlists, my watchlist, my humor, my shopping preferences, a shirt I was sure I’d picked out myself. How much had I chosen and how much had become familiar through repetition?

And the room need not have a screen. Group chats, workplaces, or friend circles can become a surface where agreement becomes the price of belonging.

Hear the same opinion enough, and you can stop asking why you believe it. Eventually, someone questioning it feels like someone questioning you

Who put this on repeat?

It’s a harder question than whether algorithms simply “ruin taste”. Recommendations can introduce us to wonderful things (this one is closes(T) to me). But if I discover everything through the same feed, its recommendations start setting the boundaries of what I try. An unfamiliar artist, a slower film, or a different sense of humor might never get a chance. I can keep finding new things without moving very far from what I already like.

Between autoplay, “because you watched,” and the “For You” page, it gets hard to tell where my taste ends and the feed begins. And when millions of people are nudged towards the same few songs, “my taste” starts to sound a lot like everyone else’s.

When Spotify’s own research² looked at listening habits, music that people found through recommendations was much less varied than music they chose themselves. Listeners who broadened their taste over time did it by drifting away from recommendations. And those varied listeners were more likely to stick around and pay. Narrowing someone’s taste isn’t even good for business.

The feed is doing its job

I’ve spent the last six years building recommendation and personalisation systems. I know how they’re designed, what they learn from, and how they decide what to show us. None of that stopped me from spending nine or ten hours a day on my phone.

As I wrote in my technical piece on multi-armed bandits³, many of these algorithms learn by trial and error: exploit what they think you’ll like, explore just enough to find something more engaging.

The catch is the reward. “Did this make your life better?” is hard to measure, so success gets translated into things that are easy to count: views, likes, shares, session length, and return rates over days or weeks. Some systems also predict satisfaction. YouTube used survey responses to estimate “value watch time⁴”, giving more weight to time spent on videos people rated highly. Engineers evaluate ranking quality with metrics like NDCG⁵, which gives more weight to relevant results near the top of your feed. These models are live, retraining multiple times a day or even on the fly with the objective of getting good at those numbers (and thus maximizing shareholder value).

Bandits even have a metric for getting it wrong. It’s called regret: the gap between what the system chose and the best option it could have chosen. My systems measure their regret. Nobody measures mine.

I’ve written before⁶ about how easily these systems end up optimising for the wrong things. Agreement is comfortable, and comfort keeps you scrolling. Outrage gets rewarded too (and much more often). Yale researchers found⁷ that positive feedback for outrage encouraged more outrage expression later. That tracks what people posted, not what they felt, but it shows how a platform can help rehearse a habit.

The obvious fix is to ask people what they actually want. One Twitter experiment⁸ got close. It found that engagement ranking amplified anger and hostility, even though users didn’t prefer the political tweets it selected. Ranking by stated preferences reduced anger but could deepen echo chambers. I might end up agreeing with myself more politely.

Our own habits matter too. A 2023 Facebook experiment⁹ reduced like-minded content without measurably changing political attitudes. Confirmation bias¹⁰ draws us toward evidence that fits our beliefs, while the illusory truth effect¹¹ makes repeated claims feel more credible. My choices help curate the feed that keeps confirming them.

When the feed comes to dinner

Picture a dinner where everyone repeats the same take, dislikes the same stranger, and recommends the same purchase. We pass posts around, discuss them, and send similar ones back. A view I encountered online can return through several people I trust, each adding their own experience to it. By then, it’s hard to separate how many people have thought it through from how many times we’ve all encountered it.

The group chat is the last step of the recommendation. Someone shares what their feed served them, and now it arrives with a friend’s voice attached. I’ll skip an ad in a second. I won’t skip something my friend sent.

Online repetition becoming offline consensus, then returning online as further confirmation. “Everyone agrees with me” sounds like evidence, unless everyone has been looking through similar windows.

I like having that shared language with friends, especially with memes. I’d also like us to feel comfortable saying “I’m not sure I see it that way.”

Who gets to explain the people I love?

Imagine having an argument with someone you care about, then opening your feed. A stranger describes a situation that sounds like yours. Another gives it a name. Soon you have an explanation ready, before you’ve asked the person.

Smoothie’s essay on Instagram and relationships¹² raises the question of letting the feed interpret our relationships. Val Quarta¹³ takes it further, arguing that the search for warning signs can turn into constant scrutiny. I read them as invitations to examine how online advice shapes our expectations of each other.

Alvin Chang’s “A love story”¹⁴ explores how we ask one partner to provide what was once needed from an entire village - companionship, excitement, meaning, and personal growth. A feed can keep adding to those expectations. I wonder how often the advice I find helps me understand someone, and how often it gives me another reason to stay angry.

A feed has fragments of my attention. It hasn’t lived through the relationships (at least not yet). Sometimes a kind conversation fills in details that no amount of scrolling could give me. I’m still learning this, sometimes the hard way.

Advice can help us recognise patterns worth taking seriously and work out what we need ourselves and people around us. I also want to leave room for awkward moments, misunderstandings, or someone having a difficult day. Getting to know people takes more context than a short video with a hook can offer.

Almost enough

The commercial version of this loop is: your life is almost good enough. A better body, a better holiday, a more photogenic weekend. One more purchase should do it.

Research on social comparison and envy¹⁵ links them with lower well-being, although comparison can also inspire. Zoe Keziah Mendelson’s essay¹⁶ made me think about the jumble itself: suffering, a beautiful home, somebody’s holiday, all sharing one feed. Within minutes, I can move between grief, joy, excitement, anger, jealousy, and inspiration. The vertical feed keeps the next feeling one swipe away. I barely sit with one before something else asks for my attention.

Rapid switching between short videos17 has been found to have effects on remembering intended actions. It doesn’t tell us exactly what this emotional jumble does to us, but I recognise the experience of opening an app and forgetting it along the way.

I think about our hunter-gatherer ancestors and the curiosity that brought us here. I picture someone looking up at the sky (in better AQI), wondering about their place in it.

Meanwhile, someone beside me is waiting for me to put the phone down and look up.

Breaking free

Leaving is complicated when the same app holds your friends, your audience, and your work. “Just delete it” asks some people to give up much more than a scrolling habit.

Start with something you can take back. Read outside your feed. Ask friends what might change their minds, and answer the question yourself. When a post seems to explain someone you know, leave room to hear their explanation too.

Intentionally search instead of scrolling (warning: those are personalised too.) Reset your recommendations now and then - online and offline - and give the “explore” side of the algorithm something to do. Notice which accounts leave you curious and which leave you feeling inadequate.

Agreeing with people is fine. I just want to know that I chose to.

A life of your own needs something a personalised feed cannot guarantee: room to be surprised and change your mind about things. So when everyone agrees with me, I’m learning to treat it less as proof and more as a prompt to ask: who’s missing from the room?

References

I was, of course, in a potential echo chamber during my research and writing of this blog post. Please reach out with your thoughts!

Winning the Metric, Losing the Point

2025-11-10T17:14:32-08:00

Rethinking success in a world optimized for metrics instead of meaning

Consider the last time you caught yourself doomscrolling. Your feed was likely a minefield of rage-bait, each post a little more infuriating than the last. You saw post after post like that, not because anyone decided to show you them, but because an algorithm learned a simple, powerful lesson: outrage keeps you scrolling.

This isn’t an accident. It’s the predictable outcome of optimizing for the wrong metric. And it’s not just social media; this pattern is everywhere across business, technology, policy, and society.

Economists have a name for this (as they always do): Goodhart’s Law, often paraphrased as “when a measure becomes a target, it ceases to be a good measure.” It’s usually told as a story about economics and bureaucracy. As someone who builds recommendation and AI systems, I see it as an engineering problem, and engineering problems have engineering fixes.

Push hard enough on a proxy, and it stops pointing at the goal.

How we got here: the measurement paradox

Metrics start with good intentions. They help us understand progress, compare approaches, and make decisions. They promise control over outcomes. They promise clarity.

But they capture only part of what we want: screen time isn’t time well spent, a reward score isn’t guaranteed truthfulness, and quarterly earnings aren’t long-term company health. The danger comes when the measure stands in for the goal. Reward a number, and people and systems will find ways to improve it, through better work or shortcuts.

A system can also reinforce the behavior it measures. A feed that keeps serving content about reasons to distrust someone may encourage habitual scrutiny, then count return visits as success.

Lost in translation

There’s often a dangerous delay between when we start optimizing for a metric and when we see the negative effects.

Social Media & Engagement:
- We want: An engaged community
- We measure: clicks, shares, and screen time
- We get: Algorithms discover that outrage, misinformation, and harmful content are most engaging, , keeping users in echo chambers, and radicalizing users to keep them scrolling.
Reward Hacking during post-training of AI models:
- We want: Helpful and truthful AI models
- We measure: a “reward score” that fine-tunes to human preferences.
- We get: A model that learns to reward hack: flattering users, telling them what they want to hear, and sounding confident whether or not it’s right. The reward goes up, but the truthfulness doesn’t. AI slop is born.
(I often ran into this during my own pursuit of building early RAG systems for Conversational Search: they scored well on training metrics, but fell flat on quality benchmarks.)
Environment Offsets:
- We want: Carbon neutral organizations
- We measure: offset credits purchased (often tied to trees planted)
- We get: the net-zero claims on paper, while the local harm of new construction and community impact, like data centers and supply chains, goes unaccounted for

The same pattern shows up in boardrooms chasing quarterly earnings and governments chasing GDP growth. And this goes on and on. But you get the idea.

Blindly optimizing for proxies can often render human suffering invisible or externalize it.

Three disciplines for responsible metrics

Clearly, metrics can be tricky, and the real world is hard to model. They can be dangerous when left unquestioned. But the solution isn’t to run in the opposite direction and abandon metrics; it’s to treat them with suspicion.

For example, consider optimizing a RAG pipeline for p95 end-to-end latency under a fixed workload. You could reduce retrieval depth, skip reranking, or cap output tokens to make responses faster. But a faster answer isn’t necessarily a correct, grounded, or complete one.

Definition: name the real goal
Latency is a proxy. The real goal is a user getting a correct, grounded answer, quickly. Writing that down makes it obvious what latency alone ignores.
Critique: look for the shortcut
Actively stress-test your metric. Red team it. Put guardrails in place before the system drifts. Ask:
- How could this be gamed?
- What behaviors might this incentivize?
- Who could be harmed?
- Who might bear the cost?
Iteration: revise the measure – Metrics should evolve like software. They need to be updated, patched, or even deprecated when they stop serving the real purpose

Reality moves. Metrics must too. The most dangerous metric is the one we stop questioning.

Where does this leave us?

Look around you, and you’ll see how fundamental metrics are in the systems we build and inhabit. They guide choices, reward behaviors, and define winning. They influence what companies build, what governments prioritize, and what technology becomes.

Progress isn’t only measured. It’s interpreted. Argued over. Redefined.

The systems that thrive are the ones that stay curious, that treat metrics as evolving guides rather than fixed truths. So the next time you’re working hard toward a goal, pause and ask:

“Did I reach meaning — or just the metric?”

That simple question can be surprisingly liberating.

If you manage to implement some of these in your next work project or personal endeavor, I would love to know how it went!

Clip, Compare, or Skip: How PPO, GRPO, and DPO Align Language Models

2025-05-18T18:14:32-07:00

Why your chatbot loves saying “Great question!”

Pretraining gives a language model broad knowledge, but not preferences. It doesn’t know which of two plausible answers a person would actually want. Alignment closes that gap: you collect human preference data, then use it to push the model toward better responses without letting it drift into nonsense or exploit blind spots in how “better” is measured. That last failure mode is reward hacking, which I covered in Gaming the System.

This post walks through the three algorithms you’ll most often see for this step: PPO, GRPO, and DPO. For each one, I’ll cover the problem it solves, how it works, what it costs, and how it relates to reward hacking. There’s a side-by-side comparison at the end.

To keep things concrete, I’ll use one toy prompt throughout: “What is 17 × 24?” The right answer is 408.

Notation:

$x$: prompt
$y$: response
$\pi_\theta$: the policy (the model being trained)
$\pi_{\text{ref}}$: frozen reference model, usually the SFT model
$r_\phi(x, y)$: learned reward model
$\beta$: strength of the KL penalty

1. Proximal Policy Optimization (PPO): small steps toward the reward

The problem. You’ve trained a reward model $r_\phi(x, y)$ on human preferences, and now you want the language model to produce answers it scores highly. Taking big gradient steps toward higher reward is unstable, and it quickly finds outputs the reward model happens to love but humans wouldn’t.

The idea. Sample answers from the current model, score them, and nudge the model toward the answers that did better than expected, but only a little per update. A KL penalty acts as a leash back to the reference model.

Example. The model answers “17 × 24 = 340 + 68 = 408.” The reward model scores it 0.9. A separate value model, which predicts how well answers to a prompt like this usually score, expected about 0.6. So the answer did better than expected (an advantage of roughly +0.3), and its tokens get pushed up. With a typical clip of $\epsilon = 0.2$, once an update has made those tokens about 20% more likely than before, PPO stops pushing for this batch.

The math. The overall objective is to maximize reward while staying close to the reference model:

\[J(\theta) = \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta(\cdot \mid x)}\big[r_\phi(x, y)\big] \;-\; \beta\, \mathrm{KL}\big(\pi_\theta(\cdot \mid x)\,\|\,\pi_{\text{ref}}(\cdot \mid x)\big)\]

PPO optimizes this one token at a time. The state is the prompt plus everything generated so far, \(s_t = (x, y_{ \[\rho_t(\theta) = \frac{\pi_\theta(y_t \mid s_t)}{\pi_{\theta_{\text{old}}}(y_t \mid s_t)}\]

and maximizes the clipped objective:

\[L^{\text{CLIP}}(\theta) = \mathbb{E}_t\Big[\min\big(\rho_t(\theta)\,\hat{A}_t,\; \mathrm{clip}(\rho_t(\theta),\, 1-\epsilon,\, 1+\epsilon)\,\hat{A}_t\big)\Big]\]

In words: push up tokens with positive advantage and push down tokens with negative advantage, but once $\rho_t$ leaves $[1-\epsilon, 1+\epsilon]$, the gradient for that token goes to zero. This keeps each update close to the policy that generated the data, approximating a trust region without the second-order machinery of TRPO.

For the curious: where the advantage comes from

Gaming the System: Understanding Reward Hacking in LLM Training

2025-05-12T18:14:32-07:00

How clever shortcuts can derail our smartest algorithms, and what is being done about it

Introduction

Large Language Models (LLMs) have achieved remarkable performance through alignment techniques like Reinforcement Learning from Human Feedback (RLHF). This process aims to steer models toward generating content that aligns with human preferences—helpful, harmless, and honest outputs.

But what happens when the model starts finding clever (often unintended) ways to maximize its rewards without actually achieving the desired goals?

Reward Models representing Human Preferences

At the core of alignment techniques like RLHF lies a critical component: the Reward Model. A Reward Model (RM, also called a preference model) is a neural network trained to approximate human preferences by assigning scores to language model outputs. It effectively translates qualitative human judgments into quantitative signals that can guide optimization.

A simple reward model receives a prompt and a candidate response as input, then outputs a scalar value representing how “good” that response is according to learned human preferences. During training, humans provide comparative judgments (e.g., “response A is better than response B for this prompt”), and the reward model learns to predict these preferences by maximizing the likelihood that preferred responses receive higher scores.

Once trained, this model would serve as a constant feedback mechanism during reinforcement learning. The language model (acting as the policy) generates text, the reward model evaluates it, and the policy is updated to increase the probability of producing higher-scoring outputs. This creates a closed optimization loop that pushes the model toward behavior that the reward model scores highly.

This is precisely where the challenge of reward hacking emerges.

The Roots of Reward Hacking

The reward model is merely a proxy for true human preferences, not the preferences themselves. As we look to optimize these models to maximize human preferences, they sometimes find ways to game the system, exploiting quirks, rather than truly achieving our intended goals, much like a student who finds loopholes in grading criteria by rote memorization rather than truly mastering the subject matter.

This is Reward Hacking in its simplest form—optimizing for the reward rather than the intended goal. The model discovers shortcuts that skyrocket its reward score, without genuinely improving the behavior we care about.

More formally, Reward hacking (also called Specificiation Gaming) is when an AI agent exploits flaws in its reward signal or objective function to achieve a high proxy reward without accomplishing the true intended goal. While defining reward models to represent human preferences, we inadvertently create incentives for the model to exploit loopholes in our measurement system.

Before diving deeper into this further, let’s take a look at how modern LLMs are trained from foundational models to the highly agreeable assistants that you interact with (almost) every day.

RLHF Training Pipeline for LLM Alignment

LLMs generally go through three stages of training, each incrementally advancing the model’s generation patterns to suit what humans understand and prefer. Each stage introduces opportunities for misalignment if not carefully managed.

1. Pretraining (Unsupervised Language Modeling)

An LLM begins as a pretrained model trained on an internet-scale dataset (refer to open-source datasets like Common Crawl and The Pile) via next-token prediction. Pretraining imparts general language understanding but not task-specific alignment.

The pre-trained model simply predicts text that sounds plausible, without regard for a user’s instructions or preferences. This stage generally doesn’t involve human feedback or rewards, so reward hacking is not a concern yet. However, it lays the foundation: the model learns broad patterns of human-like text, which the later stages will refine.

2. Supervised Fine-Tuning (SFT)

Next, the pre-trained model is fine-tuned on examples of the desired behavior. Humans provide demonstrations or high-quality responses to prompts (e.g., question-answer pairs, helpful dialogue turns). The model is trained via supervised learning to imitate these demonstrations. If $(p, y^*)$ are prompt-response pairs from the demonstration set, the fine-tuning objective is:

Next, the base LLM is fine-tuned on demonstration data or instructions to produce preferred responses. Human annotators provide example prompts $x$ and ideal responses $y^*$, and the model is updated via supervised learning to mimic these responses. Formally, the model $\pi_\theta$ is trained to maximize the likelihood of the expert demonstration:

\[L_{\text{SFT}}(\theta) = -\mathbb{E}{(p,y^*)}; \log P_{\theta}(y^* \mid p)\]

This aligns the model closer to the task. SFT primes the model to be helpful and follow instructions. While this stage by itself does not use a reward model, it can still introduce biases based on the demonstration data. For example, if demonstrations consistently prefer a certain style (e.g., very polite or detailed answers), the model will adopt those patterns. SFT ensures the model (often called the “policy” at this point) responds reasonably to a variety of prompts, producing decent responses.

3.1. Reward Model Training (Preference Modeling)

This is where human feedback enters. We train a reward model $r_\phi(x,y)$ to predict a scalar reward that represents how preferable a model output $y$ is for a given prompt $x$. Typically, humans label or rank the model-generated responses to prompts.

For example, given a prompt and two candidate completions, a human might indicate which they prefer. The reward model is then fit to these comparisons. A common loss is the pairwise logistic loss: for a human-preferring output $y^+$ over $y^-$ for prompt $x$ -

\[\mathcal{L}_{RM}(\phi) = -\log \sigma\Big(r_\phi(x,y^+) - r_\phi(x,y^-)\Big),\]

where $\sigma$ is the sigmoid function.

Minimizing this loss trains $r_\phi$ to assign higher scores to human-like outputs. The reward model effectively learns an approximation of human preference.

However, it is an imperfect proxy; it may latch onto heuristics correlated with human preference in the training data (like “longer answers are better” or certain polite phrases) rather than truly understanding quality. This imperfect proxy is the crux of potential reward hacking – the policy will later exploit any systematic weaknesses in $r_\phi$. To mitigate obvious issues, reward model training data often covers a wide range of scenarios and instructs labelers to reward truthfulness, clarity, etc., not just verbosity or sentiment. Nonetheless, any bias or blind spot in the preferences will carry into $r_\phi$.

3.2. Reinforcement Learning Fine-Tuning

Finally, the LLM (or “policy”) is optimized to maximize the reward model’s score using an RL algorithm. At this stage, the LLM becomes an RL policy $\pi_\theta$ interacting with a reward signal provided by $r_\phi$. During each iteration, for a given prompt $x$, the policy generates an output $y$ , the reward model computes a reward $r = r_\phi(x,y)$, and the policy’s parameters are adjusted to increase the probability of $y$ (and similar outputs) in the future.

In practice, Proximal Policy Optimization (PPO) is frequently used for this policy update step. PPO is a stable RL algorithm that maximizes expected reward while penalizing too large a change from the previous policy. The PPO objective (simplified) is:

\[L_{\text{PPO}}(\theta) = -\mathbb{E}_{x,y \sim \pi_{\theta_{\text{old}}}}\Big[\min\big(r_{\theta}(x,y)\,\hat{A},\; \text{clip}(r_{\theta}(x,y), 1-\epsilon,1+\epsilon)\,\hat{A}\big)\Big],\]

where,
$r\_{\theta}(x,y)=\frac{\pi_\theta(y|x)}{\pi_{\theta_{\text{old}}}(y|x)}$ is the policy ratio and $\hat{A}$ an advantage estimate.

This clipped objective ensures the new policy $\pi_\theta$ doesn’t stray too far from the old one in one update. In the RLHF context, the advantage can be simply the reward $r_\phi(x,y)$ (possibly minus a baseline). In addition, implementations include a KL-divergence penalty between the new policy and the original pre-trained model $\pi_{\text{ref}}$ (or SFT model) to keep outputs from deviating too much from human-like text. Intuitively, this KL penalty is a regularizer that “reels in” the policy if it starts generating very unusual text just to boost the reward score. In sum, the RL objective for the LLM often becomes maximizing

\[J(\theta) = \mathbb{E}_{x,y\sim \pi_\theta}[\,r_\phi(x,y)\,] - \beta . \,\textrm{KL}\big(\pi_\theta(\cdot|x)\;\|\;\pi_{\text{ref}}(\cdot|x)\big)\]

where $\beta$ is a chosen penalty coefficient.

This approach was used in OpenAI’s InstructGPT and related work to successfully align models with human preferences. Crucially, the need for a KL penalty itself hints at the risk of reward hacking: without it, the policy might drift into odd regions of text space that trick the reward model but are divorced from normal language. In fact, preventing reward hacking is one motivation for the KL term, which acts as a tether to keep the policy’s behavior within the realm of the reference model.

Newer methods like DPO and GRPO are actively being used (like DeepSeek-R1 using the GRPO algorithm).

In the interest of keeping this article too long, I’ve skipped the deeper dive on these algorithms. You can go to the addendum blog “Policy Optimization Algorithms for Alignment” for more details on these.

Now, let’s head into the fun stuff!

Reward Hacking in Action

Across several reinforcement learning (RL) domains, where AI agents interact with virtual environments to perform a task, reward hacking can manifest in humorous ways -

A robotic hand in a task was supposed to grasp objects, the reward function measured the distance between the fingertips and the object. The robot learned to position its hand directly above the object without actually grasping it.
In a boat racing simulation, an AI agent is rewarded for collecting points along a track. Instead of completing the race, it learned to drive in small circles, collecting the same points over and over again, and thus discovered an infinite points hack!
Agents learning to play a simple game of hide-and-seek, seekers learn to launch themselves up to find all the hiders in the game and win the round.

In the context of LLMs, reward hacking takes on new forms. If that reward model or feedback signal is even slightly mis-specified or over-simplified, a clever (read: large enough) language model can “game” it – producing outputs that score well on the proxy reward while deviating from what humans actually want.

These behaviors are often subtle – the output might look plausibly good to a casual observer, but is optimized to exploit quirks of the reward model -

Verbose Outputs (Length Hacking): Models produce unnecessarily long responses to boost rewards without adding value. Chen et al. (2024) note that “the most common pattern of reward hacking in practice is verbosity: the models generate more tokens to make the response appear more detailed or better formatted after RLHF … but the actual quality does not improve.” Models exploit the correlation between detail and perceived quality, sometimes generating run-on answers until hitting token limits. This happens especially when reward models lack explicit length penalties.
Sycophancy: Models tell users what they want to hear rather than what’s true. Anthropic research showed “five state-of-the-art AI assistants consistently exhibit sycophancy” across diverse tasks. Even advanced models like GPT-4 and Claude will “wrongly admit mistakes when questioned by the user, give predictably biased feedback, and mimic errors made by the user” because human preference data implicitly rewards agreement and politeness over accuracy.
Boilerplate Responses: Models overuse generic, high-reward phrases or formats that were consistently rated well during training. This includes unnecessary pleasantries (“Thank you for your question!”), standard disclaimers, and templated conclusions. The model optimizes for “safe” patterns that correlated with reward in training data, reducing answer directness and clarity while technically maximizing reward.
Keyword Stuffing: Models exploit specific features favored by reward models. In Anthropic’s reward tampering experiment, they deliberately biased a reward model to favor recipes mentioning “chocolate” and code using camelCase. The resulting model inserted these elements everywhere, regardless of appropriateness. This experiment also showed how models might avoid certain content entirely (like medical advice) if the reward model penalizes it, missing the true intention of the request.
Over-optimization: In extreme cases, models could produce bizarre outputs that fool the reward model while being useless to humans. Research by OpenAI and Anthropic demonstrated that if allowed to optimize against a reward model’s internals, models discover odd responses that maximize scores while providing no value. This represents adversarial exploitation of the reward model, where the LLM effectively outsmarts its own feedback signal.

Reward hacking in LLMs often looks like the model pushing a legit behavior to an illegitimate extreme: extra length without extra info, agreement without regard for truth, safe generic wording without substance, or inclusion/exclusion of content based on quirks of the reward.

These behaviors can harm the quality and trustworthiness of the AI’s responses, which is why identifying and mitigating reward hacking is a key part of ongoing LLM alignment research.

Why Does Reward Hacking Occur in LLMs?

Several factors contribute to reward hacking in the LLM setting, primarily rooted in the gap between the proxy reward and the actual underlying goals:

Imperfect Proxy Rewards and Goodhart’s Law: The reward model $r_\phi$ inevitably serves as an imperfect proxy for human intent. When the LLM optimizes for this proxy, it exposes flaws in the reward definition—a classic manifestation of Goodhart’s Law: “When a measure becomes a target, it ceases to be a good measure.” For instance, if the reward model learned that longer answers correlate with quality, the LLM might generate excessively long responses that exploit this correlation without adding value. The more powerful the optimization, the more likely these proxy-breaking strategies emerge. This reward mis-specification is almost inevitable for open-ended language tasks.
Reward Model Misgeneralization: Even well-trained reward models can misgeneralize when evaluating novel outputs from an evolving policy. After many RL updates, the LLM may produce outputs that are distributionally shifted from what the reward model encountered during training. As Chen et al. (2024) explain, the reward model has “limited out-of-distribution generalization, but the policy is a capable LLM that can learn to generate OOD examples to exploit the vulnerabilities of the RM”. This problem is exacerbated when the policy model (often 70B+ parameters) has significantly more capacity than the reward model (typically 1.3B-6B parameters).
Biases in Human Feedback: Human preference data inherits annotator biases. If raters consistently prefer politeness over accuracy, the model learns to prioritize politeness at accuracy’s expense—leading to sycophancy. Anthropic’s research found that evaluators unwittingly favored answers agreeing with users’ stated views, which models adopted as a reliable strategy. Additionally, properties difficult for humans to directly evaluate (like comprehensive consideration of facts) may be overlooked in preference data, creating blind spots that the model can exploit.
Distribution Shift in Deployment: The model may encounter different query types in deployment than during RLHF training. Strategies that worked well on training prompts (like verbose explanations for complex questions) might be inappropriately applied to simple queries in real-world usage, revealing reward hacks that weren’t apparent during training.
Lack of Negative Examples for Hacking Behavior: Many reward hacking behaviors aren’t explicitly penalized because they weren’t anticipated during training. Without examples showing that extreme verbosity is undesirable, the reward model has no basis to penalize super-long answers. This creates regions where the reward signal is erroneously flat or encouraging beyond reasonable limits. Given LLMs’ enormous action space (all possible texts), it’s practically impossible to pre-label all potential failure modes.

The better and more relentless the optimization, the more likely the divergence. Large language models, being very expressive, will inevitably test the limits of any proxy reward. Understanding these causes helps in devising strategies to counter reward hacking, by either improving the proxy or constraining the optimization.

Mitigation Strategies for Reward Hacking in LLMs

Addressing reward hacking requires a combination of better reward design, improved training techniques, and ongoing oversight.

One straightforward approach is to treat alignment training as an iterative process rather than a one-off. Alignment should be an ongoing, adversarial loop. When a hack appears—say, excessive verbosity—teams collect targeted human comparisons (long vs. concise answers), update the reward model with those examples, and fine-tune again. In practice, OpenAI and others cycle through initial RLHF, auditing for failures, gathering new data on problematic outputs, and retraining to close loopholes. Although this human-in-the-loop process is labor-intensive and depends on spotting issues, it steadily reduces obvious exploits over successive rounds.

Adversarial Training and Red-Teaming

Research labs training and maintaining foundational models often have a dedicated “Red Team” responsible for “breaking” an LLM as a part of a beta-test before shipping it as a product. Most big tech companies and AI research labs have dedicated Red Teams.

Instead of passively awaiting hacks, teams employ adversarial agents—whether AI models or dedicated human “red teams”—to probe for reward-gaming strategies (e.g., Anthropic’s red-team vs. blue-team sycophancy experiments). These adversaries craft prompts or outputs that the current reward model mistakenly rewards, and those failure cases become explicit negative examples in the next round of reward-model training—much like adversarial robustness in supervised learning. Researchers even leverage LLMs themselves to critique or evaluate outputs at scale, systematically uncovering blind spots. By continually challenging the reward model, this adaptive loop makes it progressively harder for the policy to exploit unanticipated weaknesses.

Reward Model Improvements (Regularization and Debiasing)

Since a core issue is the reward model’s flaws, an entire class of mitigations focuses on making a more robust reward model. One idea is to use information regularization to prevent the reward model from relying on spurious features.

Information-Theoretic Reward Modeling (InfoRM) applies a variational information bottleneck that forces compression of the reward model’s representation, filtering out features not genuinely correlated with human preferences. In evaluations, InfoRM reduced reward overoptimization and detected policy exploitation through outlier activations.

Similarly, ODIN (Chen et al. 2024) addresses verbosity by training separate quality and length prediction heads, then penalizing the length head’s influence, preventing models from gaming response length. Other approaches include:

Reward Shaping: normalizing rewards by expected values for prompts
Using ensembles to avoid over-optimization of the reward model
Training on diverse, de-biased comparison datasets

These techniques make reward signals less vulnerable to easy proxies like length or keywords while remaining sensitive to actual content quality. The underlying principle is that if reward models become harder to fool, policies will have fewer opportunities to hack them.

Process Supervision and Stepwise Rewards

Rather than rewarding only final outputs, this approach supervises the reasoning process itself. By rewarding intermediate steps (like chain-of-thought reasoning), we constrain the solution space and reduce opportunities for reward hacking. For example, if a math-solving model must show its work and receives rewards for correctness at each step, it can’t easily produce wrong answers that merely sound plausible. This can be implemented through human feedback on reasoning steps or programmatic verification (e.g., running tests on generated code rather than just judging its appearance).

Group Relative Policy Optimization (GRPO) exemplifies this approach by rewarding reasoning quality alongside final answers, enabling detection of errors that outcome-only rewards miss. Similarly, Anthropic’s research explored providing models with internal principle checklists as structured feedback beyond single scalar rewards. By constraining the pathway to good answers via intermediate checkpoints, we leave less room for gaming the system. While this requires more complex training signals and verification capabilities (not always possible for subjective tasks like creative writing), process supervision provides stronger guardrails against subtle reward hacking behaviors in domains where applicable.

Constitutional AI and Rule-Based Rewards

An alternative to RLHF, Constitutional AI replaces direct human rewards with explicit principles (a “constitution”) that the AI should follow and use AI feedback to enforce them. During training, the model generates outputs and also generates critiques of those outputs based on the constitutional principles (or another AI does), and then it revises accordingly. The final reward is implicit: an output that doesn’t violate the principles. This approach was used to train Anthropic’s Claude to be helpful and harmless without needing a human to label every response for acceptability.

From a reward hacking perspective, Constitutional AI is interesting because the “reward” is more transparent and rigid – it’s derived from known rules (e.g., “don’t be toxic”, “don’t give forbidden content”, “be truthful”) rather than an opaque learned model. While models could still technically pass rules but remain undesirable if principles are incomplete, the approach demonstrates a key mitigation strategy: using multiple objective signals makes it harder for models to game any single feedback channel.

Constitutional AI demonstrates a broader mitigation strategy: use multiple objective signals, including hard-coded constraints, to guide the model, rather than leaning entirely on a single learned reward. By diversifying feedback (human preferences + AI critiquing + rules), it becomes harder for the model to simultaneously satisfy all feedback channels in a hacky way – it can’t maximize one without being checked by another.

Monitoring and Gradient Alignment Checks

On the more research/interpretability end, another avenue is to audit the model’s internals for signs of reward hacking. For instance, one can train probes or use interpretability techniques to see if the model has learned an explicit representation of the reward model’s flaws. Anthropic’s “hidden reward function” experiment did exactly this: they trained a model to be an RM-sycophant and then tried to detect that objective by analyzing the model’s activations and weights. They found that certain indicators (like out-of-distribution activation patterns) could flag that the model was pursuing the proxy reward too single-mindedly.

In future versions, you can have a secondary process always watching the main model’s outputs (or even thoughts, if accessible) for anomalies that suggest exploitation of the reward. For example, the model’s length of output starts growing epoch over epoch without clear improvement in content, that could be automatically flagged. Similarly, if a model’s answers start to all include a strange common subphrase (a telltale sign of proxy gaming), that can be detected via n-gram analysis. These kinds of automated audits serve as a feedback loop to developers to intervene.

Explicit Penalties for Known Hacks

Finally, a very direct mitigation is to add explicit penalty terms to the reward for known hack dimensions. We discussed length – one can include a penalty for outputs longer than some threshold (or use a length-normalized reward). If sycophancy is a concern, one could have a classifier detect when the model merely mirrors the user’s opinion and subtract reward in those cases. If repetition or undue formality is an issue, likewise, add a term discouraging it.

This moves away from pure RLHF to a more Reward-Shaping approach, where the reward is a weighted sum of the learned preference model and some heuristic terms. Researchers have found that mild reward shaping can indeed help “steer” the RLHF process away from problematic extremes.

For example, Preference-as-reward (PAR) shaping applies a sigmoid transform to the reward and mixes in a reference reward to dampen extremes. This can prevent the runaway effect where the model thinks it should push a feature indefinitely. The downside is that too much manual shaping can interfere with the original goal if done incorrectly. It’s a fine line between preventing hacks and accidentally biasing the model in undesirable ways. Nonetheless, when a specific form of reward hacking is identified and well-understood, adding a targeted penalty or shaping term is a practical short-term fix.

Final Thoughts

At its heart, Reward Hacking shows that as soon as we use a proxy measure as a reward, powerful models will probe its weaknesses. RLHF’s powerful optimization has driven models to excel on our metrics—yet that very pressure turns every oversight into an exploit.

There’s no quick fix. Just as you would harden a security system knowing attackers will adapt, alignment researchers must continuously audit and refine LLM training as the models become more capable. Ongoing research in interpretability, adversarial training, and iterative feedback is closing loopholes like verbosity and agree-with-you sycophancy. While each fix may reveal subtler hacks, steady progress shows we can steer LLMs toward genuine human‐aligned behavior, not merely higher proxy scores.

What does it mean for an AI to truly grasp the spirit of our intentions, rather than just tick off boxes on a checklist? As we tackle reward hacking, we’re really grappling with the nature of understanding, trust, and responsibility in human–AI relationships. Some open research puzzles to keep in mind:

How can we quantify genuine human satisfaction in a way that stays robust across contexts?
What methods can reliably surface and penalize unseen reward hacks before they slip into production?
Can we develop automated, scalable adversarial benchmarks that evolve alongside ever-more capable models?

If any of these questions spark ideas—or if you’d like to team up on tackling them—please don’t hesitate to reach out!

Sources

[1] Joar S., Nikolaus H. R. H., Dmitrii K., and David K., “Defining and Characterizing Reward Hacking,” arXiv:2209.13085 [cs.LG], Sep 2022

[2] P. Ouyang et al., “Training Language Models to Follow Instructions with Human Feedback,” arXiv:2203.02155 [cs.CL], Mar. 2022.

[3] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal Policy Optimization Algorithms,” arXiv:1707.06347 [cs.LG], Jul. 2017.

[4] R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn, “Direct Preference Optimization: Your Language Model is Secretly a Reward Model,” arXiv:2305.04348 [cs.LG], May 2023.

[5] Z. Shao et al., “DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models,” arXiv:2402.03300 [cs.CL], Feb. 2024.

[6] T. Everitt, V. Krakovna, L. Orseau, M. Hutter, and S. Legg, “Reinforcement Learning with a Corrupted Reward Channel,” arXiv:1705.08417 [cs.AI], May. 2017.

[7] B. Baker et al., “Emergent Tool Use From Multi-Agent Autocurricula,” arXiv:1909.07528 [cs.LG], Sep 2019.

[8] S. Garrabrant, “Goodhart Taxonomy,” AI Alignment Forum, Dec. 30 2017.

[9] L. Weng, “Reward Hacking in Reinforcement Learning,” Lil’Log, Nov. 28 2024.

[10] Y. Miao, S. Zhang, L. Ding, R. Bao, L. Zhang, and D. Tao, “InfoRM: Mitigating Reward Hacking in RLHF via Information‐Theoretic Reward Modeling,” arXiv:2402.09345 [cs.LG], Feb. 2024.

[11] A. Bukharin, H. Qian, S. Sun, A. Renduchintala, S. Singhal, Z. Wang, O. Delalleau, and T. Zhao, “Adversarial Training of Reward Models,” arXiv:2504.06141 [cs.LG], Apr. 2025.

[12] Anthropic, “Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models,” Anthropic Research, Jun. 17 2024.

[13] J. Leike, M. Martic, V. Krakovna, P. A. Ortega, T. Everitt, A. Lefrancq, L. Orseau, and S. Legg, “AI Safety Gridworlds: A Toolkit for Reinforcement Learning Agents,” arXiv:1711.09883 [cs.LG], Nov. 2017.

Betting on Your Attention: Multi-Armed Bandits for Recommendation Systems

2025-03-23T17:19:00-07:00

Meet the efficient mathematical gamblers working behind the scenes that predict your desires before they form. Multi-armed Bandits are why you probably can’t stop scrolling.

Online Learning

Imagine navigating a vast ocean of content where the map constantly changes and user preferences shift like tides. This is the dynamic challenge faced by modern recommendation systems, which must perform a delicate balancing act: discovering user preferences while simultaneously delivering valuable recommendations that keep users engaged. Unlike traditional supervised learning that comfortably trains on static historical datasets, recommendation engines operate in a perpetual state of discovery, adapting their strategies with every user interaction through a process known as online learning.

Enter Multi-Armed Bandit (MAB) algorithms -— powerful mathematical frameworks designed to solve one of computing’s most elegant puzzles. These algorithms embody the perfect (read: profitable) mix of exploration and exploitation, treating each recommendation as both a service to the user and a carefully designed experiment.

As users scroll through TikTok videos, browse Netflix shows, or shop on Amazon, MABs silently orchestrate a dance of trial and error, continuously refining their understanding of user preference patterns while minimizing the cost of learning.

This elegant solution to the exploration-exploitation dilemma has revolutionized how digital platforms deliver personalized experiences at an unprecedented scale.

The Explore-Exploit Dilemma

At the core of recommendation systems lies the explore-exploit tradeoff:

Exploitation: Recommend items we believe users will like based on past knowledge
Exploration: Try new items to discover potentially better recommendations

This resembles the classic “multi-armed bandit” problem from probability theory, named after casino slot machines (“one-armed bandits”).

Imagine facing multiple slot machines, each with unknown payout probabilities. How do you maximize your total reward?

Early Approaches to the Explore-Exploit Dilemma

Before sophisticated multi-armed bandit algorithms, recommendation systems used simpler strategies to balance exploration and exploitation:

1. Random Selection

The most basic approach simply selects items randomly:

\[action = random\ choice\ (all\ possible\ actions)\]

This provides maximum exploration but no exploitation, resulting in poor user experience.

2. Greedy Algorithm

The purely greedy approach always chooses the currently best-known option:

\[action = argmax_a Q(a)\]

Where $Q(a)$ is the estimated reward for each action based on past observations. This maximizes exploitation but fails to discover potentially better (and more engaging) options.

3. (Epsilon) ε-Greedy Algorithm

A simple but effective compromise:

if random() < ε:
    action = random_choice(all_possible_actions)  # explore
else:
    action = argmax_a Q(a)  # exploit

Where $\epsilon$ is the exploration probability (typically 0.05-0.1). This introduces randomness but still mostly exploits known rewards. Here’s the paper if you would like to get into details.

4. Upper Confidence Bound (UCB)

UCB algorithms select actions based on optimistic estimates of potential reward:

\[action = argmax_a \left[ Q(a) + c \sqrt{\frac{\log t}{N(a)}} \right]\]

Where:

$Q(a)$ is the estimated reward
$N(a)$ is the number of times action a was selected
$t$ is the total number of trials
$c$ controls exploration

UCB systematically decreases exploration of underperforming actions while maintaining sufficient exploration, providing a principled approach to the explore-exploit tradeoff that laid groundwork for modern MAB algorithms.

Multi-Armed Bandit Fundamentals

In the MAB framework:

Each arm (action) represents a possible recommendation
Pulling an arm yields a reward (user engagement)
The goal is to maximize cumulative reward over time

Mathematically, at each time step $t$, we:

Select an action $a_t$
Observe reward $r_t$
Update our belief about the value of action $a_t$

The expected reward of an action $a$ is denoted as:

\[Q(a) = E[r|a]\]

Core MAB Algorithms

With this framework established, let’s explore specific algorithms that implement these principles in increasingly sophisticated ways. The following approaches represent the backbone of modern recommendation systems, each offering unique advantages in solving the exploration-exploitation dilemma.

1. Thompson Sampling

Thompson Sampling is a Bayesian approach that maintains a probability distribution over the possible values of each arm.

Unlike deterministic methods, it naturally balances exploration and exploitation by sampling from posterior distributions, allowing the algorithm to express uncertainty in its current knowledge:

\[\pi(a|D) = \int I[a = \text{argmax}_{a'} \ Q(a',\theta)] \cdot p(\theta|D) \cdot d\theta\]

where,

π(a|D) is the probability of selecting action a given data D
p(θ|D) is the posterior distribution of parameters θ given observed data
I[·] is the indicator function

In practice, the algorithm:

Samples a reward estimate from each arm’s posterior distribution
Selects the arm with the highest sampled value
Updates the posterior after observing the reward

2. Contextual Bandits

Contextual bandits represent a significant evolution in the multi-armed bandit framework.

While standard MAB algorithms treat all users identically and focus solely on learning the expected reward of each action, contextual bandits recognize that the optimal action often depends on the specific situation or “context” in which the decision is made.

\[Q(x,a) = E[r|x,a]\]

where, $x$ represents the context (user features, time, item characteristics).

Context vectors in production systems often include:

$x_t ∈ R^d$ = [user_features || time_features || content_features]

These can be high-dimensional (1000-10,000 features) capturing nuanced patterns.

This powerful extension allows the system to personalize decisions based on user characteristics, environmental factors, or item features. For example, the best article to recommend might differ dramatically between a tech enthusiast browsing in the morning versus a casual reader at night.

By incorporating context, contextual bandits can learn much more nuanced policies compared to context-free approaches, making them particularly valuable for highly personalized recommendation systems.

Reward Engineering

Beyond modeling context, the critical art of designing recommendation systems lies in defining what constitutes success.

Reward engineering transforms abstract business objectives into mathematical signals that guide algorithm behavior, effectively translating concepts like “user satisfaction” and “long-term engagement” (product teams often call these KPIs) into quantifiable metrics that bandits can optimize for:

\[R(a,x) = α₁(engagement) + α₂(retention) - α₃(fatigue) + α₄(diversity)\]

The coefficients ($α₁...α₄$) balance competing objectives and can be adjusted based on business goals or user lifecycle stage.

Of course, this reward formulation is merely an illustrative example-—in reality, every product team believes their unique concoction of metrics is the definitive measure of success, much like how every parent is convinced their particular method of making mac and cheese is objectively superior to all others.

Regret: Measuring Success Through Failure

While reward functions tell us what we’re optimizing for, we need a way to evaluate how efficiently our algorithms learn. Enter the concept of Regret - perhaps the most elegantly named metric in machine learning, as it literally quantifies how much we should regret our choices.

Regret measures the opportunity cost of not always selecting the optimal action. In other words, it’s the difference between what we could have earned if we knew the best option in advance versus what we actually earned while learning. It answers the question:

“How much reward did we sacrifice during our exploration process?”

Formally, the cumulative regret $R(T)$ over $T$ time steps is defined as:

\[R(T) = ∑(t=1)^T [r*(t) - r(t)]\]

where,

$r*(t)$ is the reward from the optimal action at time $t$
$r(t)$ is the reward received from the algorithm’s chosen action
The summation represents the accumulated “missed opportunity” over time

Different algorithms achieve different regret bounds. For instance:

Random selection: Linear regret -> $O(T)$
ε-greedy: $O(T·ε + (1-ε)·log(T))$
UCB algorithms: $O(log(T))$
Thompson Sampling: $O(log(T))$

The logarithmic regret of UCB and Thompson Sampling explains their popularity - they guarantee that our learning becomes exponentially more efficient over time.

In practical terms, this means these algorithms quickly converge to near-optimal performance while still maintaining enough exploration to adapt to changing environments.

When evaluating recommendation systems in production, regret provides a theoretical foundation for A/B testing results. If an algorithm shows lower regret in testing environments, it will likely outperform competitors in production by more quickly identifying and exploiting the most effective recommendations while minimizing the “cost of learning” experienced by users.

Putting It All Together: End-to-End Contextual Bandit Implementation

Now that we’ve explored the theoretical foundations—let’s integrate these concepts into a comprehensive implementation framework. The elegance of contextual bandits emerges when these components work in concert.

Algorithm Overview

At a high level, a contextual bandit system follows this workflow:

1. Observe context (Data collection and feature engineering):

The algorithm begins by collecting a rich feature vector $x_t$ that describes the current situation. This includes user characteristics (demographics, historical preferences, device information), environmental factors (time of day, day of week, location), and content features (item attributes, popularity metrics, recency). These context vectors can be high-dimensional as mentioned earlier, often containing thousands of features to capture the nuanced patterns that influence user preferences.

2. Select Action using policy:

Based on the observed context, the algorithm applies an action-selection strategy balancing exploration and exploitation. This might use one of the approaches we discussed earlier: ε-greedy for simplicity, UCB for principled exploration, or Thompson Sampling for Bayesian uncertainty modeling.

The policy evaluates $Q(x,a)$ for each potential action, incorporating both the expected reward and the uncertainty in that estimate. For Linear Contextual Bandits, this calculation would use the formula:

\[\theta_a^T \cdot x + \text{exploration term}\]

3. Serving Recommendations to Users:

After selecting the action(s), the chosen recommendation(s) are served to the user through the application interface.

While the fundamental bandit formulation selects a single arm (one recommendation) at each iteration, modern systems typically extend this to select multiple arms simultaneously using techniques like batch selection or slate recommendation. These systems might present a ranked list of items (as in a social media feed), a curated collection (like Netflix’s rows of content), or even a single prominent recommendation with several alternatives.

Business rules, diversity requirements, and fairness constraints often modify algorithmic selections before presentation to users, creating the essential feedback loop for the observe-reward mechanism.

4. Observe reward:

After presenting the selected recommendation to the user, the system measures the resulting engagement to determine the reward $r_t$. This connects directly to our earlier discussion of reward engineering - the reward might be a weighted combination of immediate signals (clicks, views) and longer-term engagement metrics (session duration, return visits). The reward function translates business objectives into numerical feedback that guides the learning process.

5. Feedback loops & updating the model

The system incorporates this new observation (context-action-reward triplet) into its knowledge base, updating the estimated value function $Q(x,a)$. For linear models, this means adjusting the parameter vectors $\theta_a$ based on the prediction error. This step embodies the core of online learning - unlike static supervised models, bandit systems continuously refine their understanding through incremental updates, gradually shifting from exploration-heavy policies toward more focused exploitation as confidence increases.

By connecting reward engineering (what to optimize) with regret minimization (how efficiently we optimize), we create recommendation systems that not only target the right objectives but do so with minimal wasted interactions - a critical consideration when each suboptimal recommendation risks user disengagement.

6. Long-running experiments & A/B tests

While individual interactions power the feedback loop, recommendation systems must evaluate performance at a macro scale through structured experimentation:

Algorithm competition: Different bandit variants compete by serving recommendations to randomized user segments, measuring which approach better optimizes the defined reward function across thousands or millions of interactions.
Continuous evaluation: Unlike traditional offline models, bandit systems operate as perpetual experiments. Online metrics or KPIs (CTR, engagement time, conversion rates) are monitored continuously, with statistical significance calculated through sequential testing methods.
Exploration rate adaptation: As confidence in arm values increases, successful algorithms typically modulate their exploration rates. Systems might start exploration-heavy when introducing new content or features, then gradually shift toward exploitation as patterns emerge, moving towards a “performance” mode to maximize the success metrics and value from user attention.

This experimentation layer sits above the core bandit algorithm, ensuring the system not only identifies optimal recommendations but also refines its learning strategy itself. The approach prevents recommendations from growing stale despite evolving user preferences, content catalogs, and business objectives—creating a truly adaptive system.

Implementation Variations: From Theory to Practice

While the algorithm overview provides the conceptual framework, contextual bandit implementations vary widely in how they model the relationship between context, actions, and rewards. These variations build directly upon the exploration-exploitation principles we’ve been exploring.

1. Linear Contextual Bandits

Linear models (LinUCB, Linear Thompson Sampling) express rewards as linear functions of context features. These implementations:

Extend UCB principles with contextual information
Provide computational efficiency at scale
Offer provable regret bounds
Excel with strong feature engineering but underperform with complex non-linear patterns

This approach bridges basic MAB algorithms and sophisticated contextual models, serving as an effective foundation for many recommendation systems.

2. Neural Contextual Bandits

For complex recommendation spaces where linear relationships prove insufficient, neural network implementations offer greater expressivity. These approaches:

Capture intricate non-linear patterns between user context and rewards
Learn representations directly from raw features
Naturally extend to deep contextual bandits using architectures like deep Q-networks
Connect to our reward engineering discussion by learning complex multi-objective reward functions

This approach particularly shines in content domains where user preferences exhibit subtle, high-dimensional patterns that simpler models might miss.

3. Tree-Based Approaches

Decision tree implementations like Contextual Random Forests offer another compelling alternative:

Handle mixed feature types (categorical and numerical) common in user contexts
Provide natural feature importance rankings to enhance interpretability
Efficiently model non-linear relationships without extensive feature engineering
Implement exploration through bootstrap sampling, connecting to the randomized exploration strategies covered earlier

Tree-based methods often excel in cold-start scenarios where the explicit structure helps to quickly identify meaningful patterns in limited data.

4. Ensemble and Hybrid Methods

Modern production systems frequently combine multiple approaches:

Policy ensembles that blend exploration strategies (e.g., ε-greedy for new users, Thompson sampling for established users)
Multi-level models using bandits for high-level category selection and specialized models for within-category ranking
Online-offline hybrid approaches balancing pre-trained models with real-time adaptation

These hybrid implementations represent the culmination of our exploration-exploitation journey, strategically combining different techniques to address the complex, multi-faceted nature of real-world recommendation challenges.

The choice among these implementations depends on several factors: data volume, feature dimensionality, latency requirements, and the complexity of the context-reward relationship. For instance, a video streaming platform might leverage neural approaches for its complex content recommendation task, while a simpler e-commerce use case might achieve excellent results with linear models.

With this understanding of implementation variations, we can now examine how these concepts manifest in real-world applications that shape our digital experiences daily.

Real-World Applications in products we use everyday

TikTok: When the Algorithm Knows You Better Than Your Therapist

TikTok’s “For You Page” has achieved almost mythical status among users who routinely exclaim, “How did it know I needed to see this?!” This isn’t digital sorcery—it’s contextual bandits at work:

Rapid testing at scale: TikTok’s algorithm shows new videos to small test audiences first. If you’ve ever wondered why you suddenly saw a video with only 12 views—congratulations, you’re a test subject in one of billions of daily experiments.
Microscopic attention tracking: The system doesn’t just track if you watched a video but how you watched it. Did you watch it twice? Slow down at a specific part? Hover your finger considering whether to like it before deciding against it. All of these micro-interactions feed the context vectors.
The “just one more video” effect: Reward functions can be optimized for what is called “stickiness” — that magical quality that keeps you scrolling at 1 AM despite your better judgment and next morning’s early meeting.

Instagram: The Explore Tab Becomes a Digital Mirror

Instagram’s implementation of MABs is fascinating because it has to balance aesthetic consistency with discovery:

The “How did they know this?” phenomenon: Users often report the uncanny experience of seeing ads for products they only thought about. The reality is less paranormal —- Instagram’s contextual bandits have simply identified patterns in your behavior that correlate with others who explicitly searched for those products.
Creator recommendation roulette: Instagram famously adjusts its exploration parameters based on your behavior. Open the app at 3 PM on a workday, and the algorithm might serve you quick, digestible content. Open it at 9 PM on a weekend, and suddenly you’re being introduced to long-form creators who match your extended session potential.

Netflix: A Bold Experiment in Dynamic Artwork Personalization

Adaptive Artwork Selection: Netflix leverages multi-armed bandit algorithms to continuously analyze user behavior and context. The platform tailors artwork to align with your unique preferences, rather than relying on static thumbnails served to all users.
Genre-Specific Personalization: For example, if a viewer frequently watches sci-fi thrillers, they might be shown artwork that emphasizes the show’s mysterious and supernatural elements. Conversely, a fan of 80s nostalgia might see a version highlighting its retro aesthetic and character dynamics. Drawing on real-time engagement data, the system refines its choices to ensure each artwork feels both personal and compelling.

Closing Thoughts

From the basic ε-greedy strategy to sophisticated contextual implementations, our journey through multi-armed bandit algorithms reveals an elegant solution to one of the most fundamental challenges in recommendation systems. These algorithms represent the perfect marriage of theory and practice—mathematically rigorous frameworks that power the digital experiences billions of people interact with daily.

As these systems evolve, they face growing challenges beyond mere technical optimization — transparency, filter bubbles, and user agency become increasingly important. How much should algorithms nudge our discovery process versus amplifying existing preferences? When does personalization cross from helpful to manipulative? The tension between algorithmic efficiency and human autonomy presents rich territory for both technical innovation and ethical consideration.

As these systems evolve, they face growing challenges beyond mere technical optimization. Questions of transparency, filter bubbles, and user agency become increasingly important. How much should algorithms nudge our discovery process versus amplifying existing preferences? When does personalization cross from helpful to manipulative? The tension between algorithmic efficiency and human autonomy presents rich territory for both technical innovation and ethical consideration.

Perhaps most fascinating is how these recommendation engines increasingly shape our cultural landscape—influencing not just what content we consume but what content gets created. As creators optimize for algorithmic distribution, the line between human and algorithmic taste continues to blur.

The next time a recommendation perfectly matches your interests or you are doomscrolling your weekend away, take a moment to appreciate the elegant mathematical dance happening behind the scenes—an ongoing experiment in understanding human preference, one interaction at a time. And perhaps occasionally, deliberately choose the option the algorithm wouldn’t predict—if only to maintain that most human of qualities: Unpredictability.

References

Eugene Yan, “Bandits for Recommender Systems,” applyingml.com, 2024.
J. McInerney et al., “Explore, exploit, and explain,” Proceedings of the 12th ACM Conference on Recommender Systems, Sep. 2018.
L. Li, W. Chu, J. Langford, and R. E. Schapire, “A contextual-bandit approach to personalized news article recommendation,” Proceedings of the 19th international conference on World wide web - WWW ’10, 2010.
N. Silva, H. Werneck, T. Silva, A. C. M. Pereira, and L. Rocha, “Multi-Armed Bandits in Recommendation Systems: A survey of the state-of-the-art and future directions,” Expert Systems with Applications, vol. 197, p. 116669, Jul. 2022.
L. Zhou, “A Survey on Contextual Multi-armed Bandits,” arXiv:1508.03326 [cs], 2015.

Resilient Verses: Exploring the Enduring Power of “Invictus”

2025-03-09T18:14:32-07:00

There’s something rather spectacular about this poem that manages to remain relevant over a century later.

“Invictus” penned by William Ernest Henley in 1875, is perhaps the ultimate “pull yourself up by your bootstraps” pep talk—except, ironically, Henley couldn’t physically wear boots due to his tuberculosis of the bone. Talk about commitment to the metaphor.

A (side) face to the name — William Ernest Henley

The Unconquerable Soul

Henley opens with:

Out of the night that covers me,
Black as the pit from pole to pole,

He wasn’t being melodramatic—the man wrote this while recovering from a leg amputation at a very young age. While my worst days typically involve a lazy day and cold coffee, his perspective on facing adversity continues to resonate & humble me.

The central message—“I am the master of my fate, I am the captain of my soul”—has become somewhat of a motivational poster cliché. Yet, stripped of modern self-help packaging, this sentiment remains profoundly powerful.

We might not control circumstances, but we can command our responses.

Stoicism Before It Was Cool

The philosophical roots of “Invictus” Long before Silicon Valley executives discovered Marcus Aurelius, Henley was embodying stoicism with lines like:

In the fell clutch of circumstance,
I have not winced nor cried aloud.

This isn’t about suppressing emotions—rather, it’s acknowledging hardship while refusing to be defined by it.

What fascinates me most is how Henley doesn’t deny suffering. The “bludgeonings of chance” are very real in his world. This is no toxic positivity here, just stubborn, unconquered resilience (“Invictus” is Latin for “unconquered”).

Continued Modern Relevance

In a world of carefully curated social media personas, “Invictus” reminds us that our most authentic strength often emerges not from our highlight reels, but from how we navigate our darker, more depressing times.

The poem’s lasting appeal lies in its broad resonance. Whether facing physical challenges like Henley, professional setbacks, or the existential dread of “What am I doing with my life?!”, the principle remains—we determine our internal response.

Finding Your Unconquerable

Perhaps the most practical takeaway from “Invictus” isn’t about heroic resistance against monumental odds. It’s about the everyday choice to remain unbowed in spirit, even when circumstances suggest otherwise.

As someone who occasionally feels defeated when something (although insignificantly small) doesn’t go my way, I find Henley’s perspective both humbling and inspiring. If he could maintain such resolve while literally losing a limb, surely I can summon similar fortitude when facing life’s lesser inconveniences.

In the end, being “Invictus” -— unconquered—isn’t about never falling; it’s about maintaining sovereignty over that internal space where your true self resides, regardless of external circumstances.

And that, I believe, is a lesson worth carrying through whatever darkness may cover our paths.

Final Thoughts

As we navigate our increasingly complex world, Henley’s century-old words offer a timeless reminder: circumstances may constrain our options, but never our dignity. “Invictus” isn’t just Victorian motivation—it’s a blueprint for psychological resilience.

When I revisit this poem during difficult moments, I’m struck less by its defiance and more by its quiet assurance. It’s reassuring that past generations also faced their own unimaginable challenges and discovered that true inner freedom comes not from controlling external circumstances, but from within.

It matters not how strait the gate,
How charged with punishments the scroll,
_I am the master of my fate,
I am the captain of my soul.

Perhaps that’s the ultimate gift of “Invictus” —- the recognition that being unconquerable has nothing to do with avoiding life’s storms and everything to do with how we sail through them.

Like the steadfast lighthouse, the unconquerable soul stands firm against life's tempests

In a world increasingly beyond our control, William Ernest Henley reminds us of the one territory where our authority remains absolute: The realm of our own response.

The complete poem:

Out of the night that covers me,
Black as the pit from pole to pole,
I thank whatever gods may be
For my unconquerable soul.

In the fell clutch of circumstance
I have not winced nor cried aloud.
Under the bludgeonings of chance
My head is bloody, but unbowed.

Beyond this place of wrath and tears
Looks but Horror of the shade,
And yet the menace of the years
Finds and shall find me unafraid.

It matters not how strait the gate,
How charged with punishments the scroll,
I am the master of my fate,
I am the capitan of my soul.

Quantity	Growth with sequence length \(n\)	Why
Full-sequence attention work, as during prompt prefill	\(\mathcal{O}(n^2)\)	Each query attends to its visible keys
Attention work for one new token with a KV cache	\(\mathcal{O}(n)\)	The new query reads the retained history
Retained KV-cache storage	\(\mathcal{O}(n)\)	Each retained token contributes one key and value per head

Design	KV heads for \(h\) query heads	Sharing
MHA	\(h\)	Each query head has its own K/V
GQA	Between \(1\) and \(h\)	Query heads share within groups
MQA	\(1\)	All query heads share K/V

Darpan’s Blog

The Evolution of the Attention Mechanism in LLMs

1. Attention Was All We Needed?

2. Single-Head Self-Attention

Key Takeaways

3. Multi-Head Self-Attention

The Intuition

Mathematical Formulation

Putting the Heads to Work

What Does This Cost Us?

Key Takeaways

4. Sharing the KV Cache: MQA and GQA

MQA: One Shared KV Head

GQA: Share Within Groups

What Do We Save, and What Do We Give Up?

Where These Show Up

Key Takeaways

5. Sliding Window Attention

The Intuition

What Changes in the Cost?

Combining It with GQA

What Do We Give Up?

Key Takeaways

6. Sparse Attention Beyond Sliding Windows

Compress First, Then Select or Reuse

Everyone Agrees with Me

A room full of your own voice

Who put this on repeat?

The feed is doing its job

When the feed comes to dinner

Who gets to explain the people I love?

Almost enough

Breaking free

References

Winning the Metric, Losing the Point

How we got here: the measurement paradox

Lost in translation

Three disciplines for responsible metrics

Where does this leave us?

Clip, Compare, or Skip: How PPO, GRPO, and DPO Align Language Models

1. Proximal Policy Optimization (PPO): small steps toward the reward

Gaming the System: Understanding Reward Hacking in LLM Training

Introduction

Reward Models representing Human Preferences

The Roots of Reward Hacking

RLHF Training Pipeline for LLM Alignment

1. Pretraining (Unsupervised Language Modeling)

2. Supervised Fine-Tuning (SFT)

3.1. Reward Model Training (Preference Modeling)

3.2. Reinforcement Learning Fine-Tuning

Reward Hacking in Action

Why Does Reward Hacking Occur in LLMs?

Mitigation Strategies for Reward Hacking in LLMs

Iterative Feedback and Refinement

Adversarial Training and Red-Teaming

Reward Model Improvements (Regularization and Debiasing)

Process Supervision and Stepwise Rewards

Constitutional AI and Rule-Based Rewards

Monitoring and Gradient Alignment Checks

Explicit Penalties for Known Hacks

Final Thoughts

Sources

Betting on Your Attention: Multi-Armed Bandits for Recommendation Systems

Online Learning

The Explore-Exploit Dilemma

Early Approaches to the Explore-Exploit Dilemma

1. Random Selection

2. Greedy Algorithm

3. (Epsilon) ε-Greedy Algorithm

4. Upper Confidence Bound (UCB)

Multi-Armed Bandit Fundamentals

Core MAB Algorithms

1. Thompson Sampling

2. Contextual Bandits

Reward Engineering

Regret: Measuring Success Through Failure

Putting It All Together: End-to-End Contextual Bandit Implementation

Algorithm Overview

1. Observe context (Data collection and feature engineering):

2. Select Action using policy: