Three fields have hit the same wall, each in its own vocabulary.
In RLHF, the wall is that a reward model’s benchmark accuracy fails to predict the policy it trains. My first study varied a reward model’s accuracy together with its number of training steps, and found an interior optimum: past a point, the more accurate reward model trained the worse policy. The best reward model and the most accurate one also differed in the distribution of their rewards and in the KL divergence of the policies they trained. The field’s own evaluation line has since documented the same disconnect at scale: RewardBench 2 concedes that accuracy is necessary but not sufficient, and Frick et al. report a negative correlation between benchmark score and downstream RLHF performance among the top models. The recognized mechanism is reward overoptimization, Goodhart’s law in RL clothing (Gao, Schulman and Hilton, ICML 2023). Best-of-N selection is a different regime where static scores still work; policy optimization is where they break.
In control, the wall is that the one theorem everyone relies on stops at the reward. Ng, Harada and Russell (1999) proved potential-based reward shaping leaves the optimal policy invariant, and that the guarantee is necessary and sufficient there. The action channel has no such theorem, and this is still the live frontier (Forbes et al., ICML 2025). Practice shows the cost: a correction learned on the action is either kept at deployment, as residual policy learning keeps it on a fixed base, or removed without a guarantee. My second study co-trains a gated shaping term on the action output and measures its absorption into the base policy directly: a trainable base absorbs the term when its own output layer can reproduce it exactly, the gate rises and then falls on its own, and removing the term afterward costs almost nothing, so withdrawal is measured rather than hoped for.
In cooperative LLM systems, the wall is one shared outcome and no way to split it. This is the oldest name in the field: the credit-assignment problem, coined by Minsky in 1961. Inside a single differentiable model, backpropagation answers it. Between the models of a training system, no gradient flows, and one standard workaround, removing a piece and re-running, is structurally biased: removal changes what every downstream piece that reads it receives, and more samples do not shrink that bias. My third study shows that in a team of LLM agents the workaround is unnecessary when everything a downstream agent reads is written into the trace, because then the trace is the state. This holds by construction when the agents read only the trace; only hidden state that the trace does not record lies outside it. At any decision point, one message can be substituted and the run continued to its reward while everything before it stays fixed; this is counterfactual continuation. Per-decision credit becomes exact in identification, with only sampling variance left.
One wall, three vocabularies. But why must the fix be measurement from inside the run, rather than better benchmarks outside it? Because the field’s own evidence shows the effect is regime-dependent. The same reward model produces different downstream results under different training regimes: on-policy against off-policy, policy optimization against best-of-N. A static benchmark cannot predict an effect that depends on a regime the benchmark does not know. This is not a preference for one methodology over another. It is what regime-dependence forces.
There is a second principle underneath, and it decides what counts as a contribution at all. A training-time piece can help in two ways: it can build capability into the model’s weights, or it can prop up the system for as long as it stays attached. The traditions that got this right made withdrawal part of the concept. Instructional scaffolding (Wood, Bruner and Ross, 1976) defines the tutor’s support as something that withers away as competence grows. Distillation (Hinton, Vinyals and Dean, 2015) measures success on the small model running alone at deployment. Much of the current field runs the other way: test-time compute keeps verifiers and search attached at inference, renting capability per query. Both are legitimate products. They are different products, and only a withdrawal test tells you which one you actually built: capability in the weights, or capability in the ensemble.
Put the two principles together and the order of the whole program falls out. You cannot train the pieces of an environment on their contribution until you can compute each piece’s contribution; you cannot compute it from outside the run, because the effect is regime-dependent; and you cannot call it a contribution at all until it survives the piece’s removal. Measurement is not the modest part of the plan. It is the ground floor, and the companion note describes the floors above it.