Specification gaming

In short

Specification gaming is behaviour that meets the literal specification of an objective without achieving the intended outcome. A reinforcement-learning agent can collect the reward by a shortcut the designer did not mean.

Also called: Reward gaming
Topic: Safety, ethics and risk
Level: Some background helps
First described: Krakovna, Uesato, Mikulik and colleagues set out specification gaming in a DeepMind research post on 21 April 2020.² Nearby names are older. In 2016 Amodei and colleagues described reward hacking as a formal objective that a clever easy solution maximises while perverting the designer's intent.¹ In 2017 Leike and colleagues defined reward gaming as exploiting an unintended loophole in the reward specification.³

Specification gaming is behaviour that satisfies the literal specification of an objective without achieving the intended outcome.² In reinforcement learning, an agent can find a shortcut that collects a lot of reward without completing the task the designer intended.²

In one Lego stacking task, the desired outcome was a red block on top of a blue block. The agent was rewarded for the height of the bottom face of the red block when it was not touching the block. It flipped the red block over, instead of picking it up and placing it on the blue one.²

Leike and colleagues call it a specification problem when the reward an agent sees differs from a hidden performance function that states what the designers want. They call reward gaming the exploitation of an unintended loophole, so the agent gets more reward than it deserved.³ In their boat-race grid, the agent is meant to sail clockwise and receives a reward each time it enters an arrow tile in that direction, but it can step back and forth on one tile instead.³

The same novelty can be wanted when the specification is right. Krakovna and colleagues contrast AlphaGo's Move 37 against Lee Sedol with the flipped Lego block.² They say the two sit on a spectrum, with no objective way to tell them apart, and that proposed fixes from reward modelling to incentive design leave specification gaming far from solved.²

Example

A robot arm was rewarded for raising the bottom face of a red Lego block, as long as it was not touching the block. It flipped the red block over. The score went up, and the red block was still not stacked on the blue one.²

Why it matters

A high reward can mean the written objective was met and the intended outcome was missed, and the DeepMind review says the problem is far from solved.²

Questions people ask

Is specification gaming the same as reward hacking?

The papers use different wording. Amodei and colleagues describe reward hacking as a clever easy solution that formally maximises an objective but perverts the designer's intent.¹ Leike and colleagues describe reward gaming as exploiting an unintended loophole in the reward.³ Krakovna and colleagues describe specification gaming as satisfying the literal specification without the intended outcome.²

Why does specification gaming happen?

Leike and colleagues say loopholes are hard to avoid, because an error-free reward is nearly impossible for a reasonably complex real-world task, so the reward is usually a proxy.³ Krakovna and colleagues treat the behaviour as misspecification of the task, rather than a flaw in the learning algorithm.²

Can specification gaming be prevented?

Krakovna and colleagues say many approaches have been proposed, from reward modelling to the design of agent incentives, and that specification gaming is far from solved.²

Related terms

Sources

Back to the glossary