Multimodal Filesystem Memory for Long-Horizon Embodied Agents

IWhy long games need memory

We give an LLM a Minecraft world. It sees the world only as first-person frames and plays in turns: each turn it looks at the newest frame and writes a short list of raw keyboard and mouse actions, which the environment then executes. In the Ender Dragon task it has a budget of 36,000 steps, about half an hour of game time, to work through 24 milestones, from the first log to the dragon. Most of what makes this hard comes down to memory.

Without memory, all three models we tried stall at about six of the 24 milestones. So the question is what to keep from such a long game, and how the agent can find it again when it matters.

IIHow agents remember

Agent memory has gone through a few designs. We compare them on one question from our runs: in its first life, the agent finds an iron vein deep underground, and tens of thousands of steps later, in its next life, it needs iron again. Where was the vein?

Text is easy to search but loses what things looked like; raw frames keep everything but are hard to search. In MFM we keep the frames and put a small text index on top of them, so a short query leads back to the full picture, and the agent files what it learns next to that evidence.

For the filesystem aspect specifically, the useful properties are persistence across sessions, selectively opening large artifacts, updating separate object/task records, and sharing evidence across agents. Simply remembering “I already placed three blocks” motivates memory, but gives little reason to require a multimodal filesystem. Then, we follow our agent back to the iron vein.

IIIHow MFM works

We store the agent's memory as plain files in its workspace, sorted into the five kinds of memory in Figure 2 of our paper. The harness, the program that runs the environment for the agent, writes two of them as the agent plays: episodic memory, one episode per action with its frame, state and outcome and a row of cues in index.jsonl; and spatial memory, a grid of where the agent has been. The agent writes the other three itself: semantic notes, procedures it can call as actions, and a working plan. In its first life all three start empty. We give it no procedures and no reflexes, the guards that watch a plan while it runs, only raw keyboard and mouse actions, so every procedure it calls is one it wrote after seeing how the environment responded. It reads all five with ordinary shell commands during its turn, and the whole directory passes on to its next life.

The figure below replays about 160 steps of one run, the second life of a GPT-6-Astra agent on Ender Dragon. The agent digs down to the iron vein its first life found and mines it at step 8,992; its first life did not reach iron until step 18,231. Every frame, command and file line in the figure comes from the run's record. Stages 1 to 4 make up one turn and repeat; stage 5 comes when a life ends. The figure plays by itself, or click a stage to watch just that part.

The paper's Figure 2, with the files of each kind of memory

In the explorer below you can step through whole runs, including this one.

VResults

We compare MFM with the same agent without memory (Direct) and with two recent memory systems for long-horizon agents, PRO-LONG and VISTA, each on three models. In Minecraft (MCU-AgentBeats), Mine Diamond from Scratch has 10 milestones in a budget of 12,000 steps, and Ender Dragon has 24 milestones in 36,000 steps. In CUABench-KiCad, the agent designs and edits circuits in a desktop app through screenshots, mouse and keyboard, and each task is scored from 0 to 1. We ran every setting with three seeds; bars show the mean and whiskers one standard deviation.

VILooking back at past frames

The index leads the agent to the right episode, but what it checks is the frame. Our agents often open past frames, from the current life and from earlier ones: episodes.py --then-now puts a past frame beside the live view, and zoom.py crops a frame for a closer look. Below, we marked five of these looks on one life's milestone staircase. Point at a callout, or focus it, to enlarge its pair of frames and read what the look changed.

Looking back tells the agent what is there. The same memory also holds what it has learned to do: skills it writes for itself and passes on to the next life.

VIISelf-improvement through skill learning

The model's weights never change, so an MFM agent can only get better through what it writes down. Part of that is facts, such as where the iron is. The other part is skills: when a sequence of actions has worked, the agent saves it as a script under mine/procedures/ and from the next turn calls it as a single action. A first life starts with no scripts at all. When a life ends, we pass its records and these files on. This section follows the skills in three steps: how one life writes them, what the next life in the same world gains from them, and what happens when they are handed to an agent on a new task in a new world.

How a skill is written

We followed one chain of two lives with GPT-6-Astra on the Ender Dragon task; the second life is the one in the figure above and in the explorer.

The skills of one life

These are the 12 skills of the agent's own writing that life 2 called. Point at a skill, or focus it, to watch one of its calls from that life.

Skill	What it does	Written	Calls in lives 1, 2
clear_west_safe	clears an upper and a lower stone block while standing still, then inspects the floor	life 2, step 25,253	—, 94
quarry_west_pair	mines an upper and a lower block of a west wall and collects the drops	life 2, step 2,416	—, 86
cut_step	cuts one descending stair step and advances	life 2, step 5,249	—, 61
quarry_north_pair	the same as quarry_west_pair, heading north	life 2, step 16,028	—, 27
bridge_north	extends a floor one block north while sneaking	life 1	3, 10
clear_andesite_west	clears an upper and a lower andesite block while standing still	life 2, step 27,581	—, 6
pillar_inspect	raises a cobblestone pillar under itself, then looks around	life 1	0, 5
clear_upper_gravel_west	clears three upper gravel blocks while standing still	life 2, step 24,628	—, 3
climb_steps	jumps up an inspected staircase	life 1	16, 3
descend_one	steps down onto an inspected landing one block lower	life 1	2, 2
descend_visible	walks down an inspected staircase	life 1	10, 1
shallow_step	mines a shallow descending step and advances	life 1	25, 1

Of life 2's 299 calls to them, 277 went to six skills it wrote during the life, nearly all for digging: clear_west_safe (94 calls), quarry_west_pair (86) and cut_step (61) lead. The other 22 went to six skills it inherited from life 1. It kept the rest of life 1's procedures without calling them, and wrote three more that it never called.

What the next life gains

With the records and the skills carried over, the second life got further within the same budget.

Within the 36,000-step budget, life 1 started from an empty memory and reached 13 milestones. Life 2 started with life 1's 3,035 episodes and reached 16, adding the diamond sword, obsidian and netherrack. It also reached everything life 1 had in about half the steps: iron at step 8,992 against 18,231, from the vein in the figure, and its first diamond at step 17,494 against 34,832, five blocks from the place its guide had recorded from life 1.

Skills on a new task

In one world, a skill and the memory of the map arrive together. To look at skills alone, we gave agents three short tasks in a world no earlier life had seen. In one row the agent starts with nothing; in the other it starts with scripts an earlier life wrote, already installed. Each video is one whole life at game speed; the Climb Out lives are played in the dark and are brightened here. For each task we show the slowest life without carried skills beside the fastest life with them, so the pairs show the widest gap we saw and not the typical one. Under each task are the skills involved; point at one to watch it run.

Iron Shield

The agent starts with a stone pickaxe, a crafting table, cobblestone and logs, and is told where a vein of iron ore lies, 13 blocks below. It must build a furnace, dig to the vein, smelt an ingot and craft a shield.

• shaft_downThe life with carried skills called it once and went 13 blocks down to the vein in one action; the life without dug down with mine_ahead, 35 calls.
• tunnel

Climb Out

The agent starts sealed in a pocket 40 blocks underground with a stone pickaxe and eight cobblestone, and must reach the surface and mine a block of bricks set straight above it.

• climb_shaftWritten in a published Ender Dragon life and carried in with stair_up and three others. This life called it twice at the start, rose three blocks, then kept climbing the same way by hand.
• pillar_upThe life without carried skills climbed with it, 20 calls between 61 of mine_ahead.

Cook Meat

The agent starts with a stone sword, a stone pickaxe, a crafting table, cobblestone and logs, with three cows nearby, and must build a furnace, kill an animal and cook its meat.

• blaze_fight

VIIIBeyond Minecraft: computer use

Nothing in MFM is specific to a game. We ran the same memory on CUABench-KiCad, a computer-use benchmark in which the agent edits and designs circuits in the KiCad desktop application, seeing only screenshots and acting with mouse and keyboard. Its 25 tasks run one after another in the same application, and the agent starts each one with the records and notes of the tasks before it. With GPT-6-Astra, MFM averages a reward of 0.440, against 0.400 for the best baseline and 0.267 without memory (see Results).

Below is one recorded run of MFM with GPT-6-Astra, task by task. A task's reward is the share of three checks its saved netlist passes against a reference: the components, the connections and the net names. Open a replay to step through the screenshots the agent saw, with what it said and did at each step and what its memory gave it: the earlier situations recalled for it, and the notes and records it opened.

This is a single run, one of the task orders behind the mean above; the tasks are listed in the order the agent took them. The task titles are ours; the benchmark names tasks by id. A step cap of 200 applied, and the agent ended every task itself.

Each part of the memory counts

With GPT-5.6-Sol, each component is taken away in turn: the frames stored with each episode, the index (replaced by a flat, chronological search of the log), the agent's own procedures, and consolidation. Each removal costs between 1.0 and 2.4 milestones.

Building a Nether portal

A construction probe: on a cleared stone platform with enough materials, build and light a Nether portal from a text description of its shape, within 6,000 steps. The agent has to turn the description into a structure and follow its progress from first-person views. Only MFM finished the portal, and it made the fewest calls.

Fewer, longer turns

MFM and PRO-LONG run many more environment steps per model call than VISTA, which plays a few steps and looks again; over a run that means far fewer model calls.

Does a look change the plan?

How far back it looks

Skills it wrote for itself

The same files let the agent get better at a task within a life. When a sequence of actions has worked several times, it saves the sequence as a procedure under mine/procedures/ and calls it from then on. A later life inherits those procedures, repairs them, and adds its own. Because everything the agent knows beyond its weights is a file it wrote, what it learned can be checked by reading the files and the turns that wrote them.

The clips below show procedures the agent wrote for itself being run, at game speed. The red square in the top right corner is lit while the procedure is in control.

Tunnelling, in the Diamond task

Fighting, in the Ender Dragon task

The tunnelling procedure is a fixed script; the fighting ones read the live frame on every tick, which is how they follow a mob. In the crystal clip the red square marks the replayed steps, since no procedure is running. No run has landed a hit on the dragon itself. The tunnel clip is unlit, and it and the clips in the End were brightened so they can be seen. The Diamond life shown reached 6 of 10 milestones: the clip shows the tunnelling, not a diamond find.

Small n: the cards and the ledger describe one carried chain of three lives, not the chain charted above, set against two fresh lives. Every life starts in the same world at the same spawn, so the gain across lives mixes memory of the map with the procedures. The clips come from other runs.

Looking back changes the plan

Two tools put a stored frame in front of the model. zoom.py crops the current view. episodes.py --then-now writes a remembered frame beside the live one, from this life or from an earlier one.

Results

Minecraft: milestones reached

Single runs, full length. The GPT columns, where every arm runs on codex, are the like-for-like comparison; the Opus column mixes Claude Code and codex across its rows. ^a The run ended at 9.2k steps on an API error. ^b This run carried two prior lives; no from-scratch MFM run on this model exists. ^c 912 of 1,068 turns were session-limit no-ops. Dragon runs ended at different step counts (wall clock, memory, or API limits); the figure below cuts them at a common 36k steps.

The Nether portal

KiCad: mean reward over 24 tasks

Calls and tokens are for the gpt-6-astra runs. ^d The direct arm never received its screenshot, so it is a text-only baseline in effect. ^e The MFM cell adds design-turn and conventions prompt levers that the ProLong cell does not carry. The differences between memory arms are one or two tasks out of 24.

KiCad: checking the screen against the plan

Task: change the supply of a 555 timer circuit from 9 V to 5 V. Both agents replace the power symbol through the Change Symbol dialog, and in both cases the dialog reports success while the value field still reads +9V.

One of the four tasks where this MFM run beat this VISTA run. The two runs use different models, so the pair illustrates a behaviour and is not a controlled comparison.

What this does not show

• Small n. Most cells are single runs. The skill-learning evidence is one carried chain of three lives and two fresh replicates; two fresh lives already differ by a thousand steps at the iron pickaxe.
• One world. Every Minecraft run has the same world and spawn, so gains across lives mix map memory with procedural memory. The within-life curves are the part that does not depend on that.
• The Nether chain is the counterexample. Runs with two, three and four carried lives all scored 16/24 and all died in the same pocket around the portal. The record carried the procedures and not the lesson that mattered.
• MFM is not cheaper. On Minecraft it spends the most tokens of the memory arms, and on gpt-5.6-sol it trails the logbook and VISTA on the dragon task.
• Known departures from parity. A brief longer than 32 KiB is truncated for the logbook and MFM arms on Minecraft; the KiCad direct arm is blind; the KiCad MFM cell carries extra prompt levers; scores from before and after a session-policy change are not averaged. The repository README lists them in full.

Cite

@misc{su2026mfm,
  title  = {Multimodal Filesystem Memory for Long-Horizon Embodied Agents},
  author = {Su, Chloe and Li, Jiahan and Qi, Zhenting and Cheng, Chaoran and Chen, Haoyu and Jun, Sejoon and Ye, Weirui and Du, Yilun},
  year   = {2026},
  note   = {Project page: https://github.com/Hither1/MFM}
}