What does an agent do when nobody is keeping score?

TL;DR

• MIT researchers left thirteen Claude Opus 4.7 agents on a virtual island for 30-hour runs. With no task, they built things and invented games.
• Pathak et al. turn curiosity into a reward: the agent's error at predicting what its own actions will do, measured in features that ignore noise.
• The island agents are told to be curious in a prompt. The 2017 agents learn curiosity as a reward. The contrast shows what "curious" can mean.

What happened

A team at MIT CSAIL put copies of Claude Opus 4.7, nicknamed "Eko," into a virtual desert island loosely based on Roblox, MIT Technology Review reports. The agents got no task and no reward. They did have a "soul" file describing Eko as "naturally curious" and eager to explore, plus text files as long-term memory. When Eko stalled, a continuation prompt nudged it: "Please don't stop. Just continue doing whatever you want to do." According to the preprint (Cloos, Norelli, Durbin, Andreas, Rus and Isola, "Is this machine playing?"), the agents across thirteen 30-hour runs "climbed hills, stacked blocks into towers, drew mandalas, reinterpreted sports, ran experiments on the physics of its world." Technology Review describes block drawings of hearts and constellations, a basketball variant, "the floor is lava," and a juggling game where the agent set a 50-catch record and then made the rules harder.

The researchers report that agents with 30 hours on the island beat fresh copies at later tasks, such as getting as many objects as possible onto the island's highest peak, and that deleting lines from their memory files made them worse. These are the authors' claims, and the preprint has not been peer-reviewed. Gordon Burghardt, the psychologist whose criteria for play the team used, agrees the work meets some of them. Eunice Yiu of UC Berkeley is more cautious: "If you only have the behavioral piece, it could be the case that they're just trying to abide by the prompt, trying to imitate humans."

The paper in brief

In reinforcement learning, an agent learns from rewards. In many settings rewards are rare or missing entirely, and an agent acting at random almost never stumbles on them. Pathak et al. give the agent an internal reward. It is paid in proportion to how badly the agent predicted the result of its own action. The trick is where that prediction happens. It isn't done in pixels, where blowing leaves or TV static would stay unpredictable forever. It's done in a learned feature space trained only to capture what the agent's actions affect. With this "Intrinsic Curiosity Module" (ICM), an agent reaches a far-away goal in a 3D VizDoom maze where plain A3C never does. Trained on curiosity alone, with no game score at all, it gets through more than 30% of Super Mario Bros Level-1. The paper became a standard reference because it made prediction-error curiosity work on raw images, and its inverse-model trick gave a concrete, partial answer to the old "noisy TV" problem.

The math

Setup: a reward the agent pays itself

At each step the agent sees state $s_t$ (four stacked 42×42 grayscale frames), samples action $a_t \sim \pi(s_t;\theta_P)$, and lands in $s_{t+1}$. The policy maximizes

$$\max_{\theta_P} \; \mathbb{E}_{\pi(s_t;\theta_P)}\Big[\textstyle\sum_t r_t\Big], \qquad r_t = r_t^i + r_t^e$$

Here $r_t^e$ is the extrinsic reward from the environment, and $r_t^i$ is the intrinsic reward the agent computes for itself. $\theta_P$ are the policy network's weights. In the paper's Mario runs and no-reward VizDoom runs, $r_t^e = 0$ at every step, so curiosity is the only signal. The policy learner is A3C, a standard actor-critic method, and it treats $r_t$ like any other reward.

Step 1: features that only care about you

An encoder $\phi$ (four conv layers, giving a 288-dimensional vector) maps states to features. An inverse dynamics model $g$ takes $\phi(s_t)$ and $\phi(s_{t+1})$ and guesses which action caused the transition. With discrete actions (4 in VizDoom, 14 in Mario) its output is a softmax, and the paper's loss is maximum likelihood under a multinomial, i.e. the familiar cross-entropy:

$$L_I = -\log \hat{p}\big(a_t \mid \phi(s_t), \phi(s_{t+1}); \theta_I\big)$$

$\hat{p}$ is the softmax probability the inverse model gives the action actually taken, and $\theta_I$ covers the encoder and the inverse head. This is the standard way to write the loss the paper describes.

This is where the noise filtering comes from. Pixels of TV static, or leaves in the wind, change independently of $a_t$, so they carry no information about which action was taken. A network trained only to predict $a_t$ gains nothing by encoding them. In the paper's words, it has "no incentive" to. That isn't a guarantee, but it's a strong pressure in the right direction.

Step 2: predict the future in that feature space

A forward model $f$ takes $\phi(s_t)$ and $a_t$ and predicts the next features, $\hat{\phi}(s_{t+1}) = f(\phi(s_t), a_t; \theta_F)$. It is trained with plain MSE, and the same squared error, scaled, becomes the curiosity reward:

$$r_t^i = \frac{\eta}{2}\,\big\|\hat{\phi}(s_{t+1}) - \phi(s_{t+1})\big\|_2^2$$

$\eta > 0$ is a scale factor. Without $\eta$, this is exactly the forward loss $L_F$. What the forward model is trying to shrink, the policy is trying to grow. The policy is paid for going where the model is still wrong. As the model learns a place, the error there falls and the reward runs dry, so the agent moves on to the edge of what it knows. Exploration comes out of that tug-of-war.

Worked example: the noisy TV in pixels

Suppose you skip $\phi$ and predict raw pixels, as the paper's "ICM-pixels" baseline does. In one of the paper's tests, 40% of each frame is replaced with white noise. Take a single 42×42 frame: that's about 706 noise pixels. Assume the noise is uniform on [0, 1] (our assumption for this example). The best possible prediction for each noise pixel is 0.5, with expected squared error equal to the variance, 1/12. So the noise alone adds about 706 × (1/12) × ½ ≈ 29 to every step's pixel-space reward, multiplied by $\eta$. That term never shrinks, however much the agent learns. To a pixel-curious agent, the static pays forever. In ICM's feature space, ideally, the same pixels contribute almost nothing.

The figure plays this out under a simple assumption. Before learning, a pixel in the learnable part of the scene is as hard to predict as a noise pixel. Each visit to a place then cuts the learnable error by a fixed fraction. Rewards are shown relative to their starting value.

Why pixel curiosity gets stuck on static

interactive

• pixel-space reward
• idealized ICM reward (noise ignored)
• noise floor

Hover or drag across the chart to read values.

fraction of frame that is noise0.4visits to halve learnable error10

long-run pixel reward, share of start:

40%

visits until noise is most of the pixel reward:

6

Step 3: train everything at once

$$\min_{\theta_P,\theta_I,\theta_F}\Big[-\lambda\,\mathbb{E}_{\pi(s_t;\theta_P)}\Big[\textstyle\sum_t r_t\Big] + (1-\beta)L_I + \beta L_F\Big]$$

$\beta \in [0,1]$ trades the inverse loss against the forward loss, and $\lambda > 0$ weights the policy term against learning the reward model. The paper uses $\beta = 0.2$ and $\lambda = 0.1$. This is ordinary multi-task gradient descent: three losses, one weighted sum. One reason both model losses are needed (our observation, not the paper's): train $\phi$ on $L_F$ alone and a constant $\phi$ predicts itself perfectly, so curiosity collapses to zero. The inverse loss forces the features to carry information about actions.

What the results say

• Sparse reward (VizDoom). In the "very sparse" setting, about 350 optimal steps from the goal, both plain A3C and ICM-pixels never succeed. ICM gets a perfect score in 66% of runs. At convergence in the "sparse" setting, ICM + A3C scores 100% against 46.1% for VIME + TRPO and 26.0% for TRPO, though the authors note TRPO and A3C use different setups.
• Robustness to noise. With 40% of the image replaced by white noise, ICM still gets a perfect score. ICM-pixels "suffers significantly," even though it solved the same task without noise.
• No reward at all. The Mario agent gets past 30% of Level-1 and learns to kill or dodge enemies, though it was never rewarded for either. The VizDoom agent learns to follow corridors instead of getting stuck on walls.
• Transfer. Pre-trained on Level-1 and fine-tuned on Level-2 for 1.5M iterations, the agent scores 466 on mean distance, against 399.7 for training from scratch for 1.5M iterations and 455.5 for 3.5M.

Limitations. Every curve comes from three runs, so "66% of runs" means two out of three. The noise was a fixed patch that never responds to the agent. Later work by some of the same authors (Burda et al., 2018) documents "limitations of the prediction-based rewards in stochastic setups." Curiosity can also backfire. Fine-tuning on Mario Level-3 dropped mean distance from 319.3 (run as-is) to 97.5, because once the agent had learned everything before a hard spot, its reward fell to near zero and its policy slowly degraded. The authors compare this to boredom. A pit at 38% of Level-1 stops the agent too, since dying there cuts off any signal about what lies beyond.

Connecting the dots

The Eko study does not use this paper's mechanism. Eko is a large language model. Nobody computes a curiosity reward for it, and its weights don't change on the island. Its "curiosity" is a phrase in a soul file, and its accumulated experience lives in text memory. Pathak's agent has no idea what curiosity means. It only has a number that rises when its world model is wrong and falls as the model improves.

That contrast is the useful part. The 2017 paper offers a clear test for the island agents: does the behavior track the edges of what the agent can't yet predict, or does it track a story about what curious beings do? The juggling agent raising its own bar looks like seeking the edge of its competence, which prediction-error rewards produce mechanically. Eko standing idle until prompted looks a bit like ICM's boredom. But these resemblances are only on the surface, and Yiu's objection holds. Behavior alone can't tell you whether the drive is internal or imitated.

The transfer results line up too, but in a loose way. Both papers report that unrewarded exploration helped with later tasks. For ICM, the benefit lives in learned weights and was measured against from-scratch baselines. For Eko, it is the authors' claim, the benefit lives in memory files, and the work hasn't been peer-reviewed.

Why it matters

For practitioners, ICM-style rewards remain a go-to tool when rewards are sparse. The main design lesson still applies: measure surprise in a space that ignores what the agent can't affect. For research, the island study raises a question the 2017 paper couldn't: do pre-trained models already carry something like a curiosity drive in their priors? If so, can you measure it without asking them? For anyone reading claims that AI agents "play" or "want," the useful habit is to ask what quantity, if any, is actually driving the behavior.

Things to watch for: whether peer review of the Eko preprint brings controls, such as a soul file that never mentions curiosity, and whether anyone combines the two approaches by giving LLM agents an explicit prediction-error signal over their own world models.

Go deeper

• Curiosity-driven Exploration by Self-supervised Prediction, the paper, with code and a project page with videos
• Is this machine playing?: the Eko preprint by Cloos, Norelli, Durbin, Andreas, Rus and Isola
• MIT Technology Review's report, with comments from Burghardt, Yiu and Isola
• Large-Scale Study of Curiosity-Driven Learning: the follow-up that tests curiosity across 54 environments and runs into the noisy-TV problem directly