Memory Without Curation: Learning Multi-Stream Latent Memory for Robot Policies

Approach

Method

Trace how visual, action, and state histories become compact latent memory. Interact with each component to inspect its exact RMBench ablation.

Episode timestep t = 00:34 / 01:08

Encoded: 0 → t Future: t → T

Input

Dust Table example

Multi-stream history

Bottleneck size B

Policy-agnostic

Memory encoder

›

Latent history encoded from 0 → t (50%) Schematic token view · B = 256

Learn

Memory objectives

Teach memory what to retain

Memory decoder Sample and reconstruct the episode

Adapt

Joint adaptation

Read memory inside the action expert

Current head-camera observation from the Dust Table rollout — Current observation

Task instruction Move the objects away to dust the table and put them back in the original positions.

Native observation + language pathway Vision-language model

Frozen image encoder · Trainable VLM transformer

Memory tokens Visual · Action · State Fixed-size

Flow-matching action expert Lightweight memory adapter at each layer

Action queries · Memory K/V Cross-attention Feature modulation FiLM γ, β Native block Layer L

Action queries · Memory K/V Cross-attention Feature modulation FiLM γ, β Native block Layer 1

a_t:t+h

Native attention → memory cross-attention + FiLM → feedforward

RMBench ablation

Exact appendix row

All objectives (IMPRINT)

72.7% Best reported configuration

Per-task success from the matching appendix row.

View the paper method figure

Overview of multi-stream history tokenization, memory learning, and joint adaptation with a vision-language-action model — Visual history is tokenized alongside executed actions and proprioceptive states. A memory encoder compresses these streams into latent tokens; Stage 1 reconstructs sampled history and estimates task progress, while Stage 2 jointly adapts the memory and VLA for action generation.

Interactive Example

Choose an in-context folding example

Compare all four demonstrated styles across four towel colors. Each cell shows only the external-camera stream; select one color × style combination to compare its IMPRINT execution with a matched OOD execution. Successes and failures are labeled explicitly.

Orange

Dark blue

Blue

Pink

Selected demonstration

Orange towel · Style 1 executions

The selected IMPRINT and matched OOD executions use the same demonstration; badges mark each outcome.

Interactive attention trace

How memory attends over time

Compare one Cover Blocks episode across E2E, Frozen, and Ours.

Selected recipe

Ours 100%

Joint memory-policy adaptation · task success

Attention over head-camera, wrist-camera, action-history, and robot-state streams — Current observation **Head camera**

Episode start Frame 1 / 20 Episode end

Each retrieved card shows the top-attended item for its stream; w is its attention weight, and the outline locates it in the temporal map.

Experiments

Result Highlights

Success versus inference latency highlight showing EventVLA as 7.8 times slower

Success improvements across three VLA backbones

Success improvements on non-memory and memory task groups

Contributions

We propose IMPRINT, a policy-agnostic latent memory framework that efficiently compresses multi-stream histories and trains with history reconstruction and progress estimation before joint fine-tuning with the policy.
We systematically analyze how memory capacity, encoding, training objectives, and training recipes affect downstream performance.
Across RMBench, a 12-task RoboCasa benchmark, and three real-world tasks, IMPRINT improves success over the base VLA by 47.4% on RMBench and 50% in the real world, while achieving the lowest average latency and highest success among the memory baselines in the latency comparison.

Evaluation

Real-World Evaluation

Select a task, compare the reported result, and browse every associated rollout. Green and red labels indicate success and failure under the paper's evaluation criteria.

Task prompt

The robot must place the objects down upright, dust the table exactly once, and return the objects upright to the small table in the correct left/right arrangement.

Reported task result

Dust Table

Result analysis IMPRINT reaches 80%, 30 points above uniform sampling; the unseen-object setting reaches 45%.

Dust Table results
Method	Success (%)	Successes / trials

Method

IMPRINT

16 / 20 successful

Success

Dust Table

Trial 01

IMPRINT · Success

Associated trials

Select a rollout

01 / 20

Success Failure

Evaluation

RMBench Simulation Results

Per-task and average success on the nine memory-dependent tasks in RMBench.

RMBench task success distributions for no-memory, heuristic, hierarchical, latent, and proposed memory methods — Full-attention and recurrent IMPRINT reach 72.7% and 72.8% average success, respectively.

RMBench task success rate (%)
Method	Obs. / Pick	Rearrange	Put Back	Swap Blocks	Swap T	Battery	Ranking	Cover	Press	Average
π_0.5	8	63	20	16	8	30	66	15	3	25.4
Uniform sampling	5	79	100	62	30	28	82	13	3	44.7
EventVLA	21	96	95	96	87	35	81	97	3	67.8
MemER	7	17	0	14	7	27	0	6	0	8.7
Mem-0	4	89	90	67	14	28	18	68	0	42.0
HAMLET	6	76	23	25	13	26	70	14	0	28.1
MemoryVLA	2	53	81	76	9	33	53	69	0	41.7
Ours (full)	9	100	100	100	100	41	83	100	21	72.7
Ours (recurrent)	6	99	100	100	100	35	84	100	31	72.8

Result analysis Recurrent IMPRINT leads the strongest baseline by 5.0 points and reaches 99–100% on five tasks; remaining errors cluster in Observe / Pick, Battery, and Press Button.

Evaluation rollouts

Representative RMBench trials

Choose a task and drag the method bar to compare logged evaluations.

Task

Method Ours (recurrent) 6%

Task prompt

Pick the object matching a target shown before occlusion.

Success

Representative successful rollout

Failure

Representative failed rollout

One valid logged camera mosaic is shown for each available outcome. Aggregate rates are computed from all 100 evaluation trials.

Evaluation

RoboCasa: Memory and Non-Memory Tasks

Average success on memory-dependent and memory-free task groups in RoboCasa.

RoboCasa success distributions for non-memory, memory, and overall task groups — Recurrent IMPRINT reaches 65.3% on memory tasks and 68.0% on non-memory tasks, for 66.7% overall.

RoboCasa per-task success rate (%)
Task	Type	π_0.5	Heuristic action expert	Heuristic VLM context	Ours (full)	Ours (recurrent)
Retrieve fruit	Memory	42	88	60	84	90
Retrieve oil	Memory	48	98	50	76	80
Wash and return left/right	Memory	46	58	46	86	96
Wash and return same location	Memory	46	36	44	62	58
Rinse cutting board	Memory	4	8	0	0	4
Sweeten hot chocolate	Memory	46	36	34	58	64
Pan transfer	Non-memory	74	48	44	76	72
Frying pan adjustment	Non-memory	76	52	22	72	70
Organize condiments	Non-memory	72	32	48	62	74
Place equal ice cubes	Non-memory	16	6	6	6	20
Scrub cutting board	Non-memory	60	50	56	66	76
Move from counter to sink	Non-memory	96	74	88	98	96

Result analysis Heuristic history helps memory tasks but hurts non-memory tasks. Recurrent IMPRINT improves both, suggesting its latent memory is used only when needed.

Full evaluation archive

Browse RoboCasa rollouts

12 tasks · 5 methods · 50 rollouts · 3 synchronized camera views

Task

Method Ours (recurrent) Archive: 45 / 50 successes

Task prompt

Place the {fruit} in the sink; recall which side of the sink it was observed on.

Memory task

Retrieve Fruit · Rollout 01

Mem Fruit In Sink Left Far · episode 000

Success

Loading rollout…

Success

Left agent view

Loading rollout…

Success

Right agent view

Loading rollout…

Success

Wrist view

Associated rollouts

Select a trial

01 / 50

Success Failure

The three views are synchronized and show the same rollout. Success and failure labels follow the supplied archive manifest.

Evaluation

Success vs. Inference Efficiency

Average RMBench success versus inference latency per action chunk.

Inference runtime versus average RMBench success rate — Recurrent IMPRINT reaches 72.8% success at 122.8 ms per action chunk.

RMBench success and inference runtime
Method	Success (%)	Runtime (ms)
π_0.5 (no memory)	25.4	91.0
HAMLET	28.1	144.9
MemoryVLA (QwenOFT)	41.7	321.6
EventVLA	67.8	960.0
Ours (full attention)	72.7	130.8
Ours (recurrent)	72.8	122.8

Result analysis Recurrent IMPRINT achieves the best success–latency tradeoff: 72.8% success at 122.8 ms, only 31.8 ms slower than the no-memory policy and faster than full attention.

Ablations & analysis

Memory Encoding

Memory bottleneck capacity and full-attention versus recurrent encoding.

Success rate as the latent-memory bottleneck scales from 1 to 1024 tokens — Performance and Stage-1 supervision loss as the bottleneck scales from 1 to 1024 latent tokens.

Per-task success rate by memory bottleneck size (%)
Task	Benchmark	Type	1 token	4 tokens	16 tokens	64 tokens	256 tokens	1024 tokens
Obs. / Pick	RMBench	Memory	7	8	8	8	9	10
Rearrange	RMBench	Memory	100	100	100	100	100	99
Put Back	RMBench	Memory	100	100	100	100	100	100
Swap Blocks	RMBench	Memory	97	97	99	99	100	93
Swap T	RMBench	Memory	99	99	99	99	100	99
Battery	RMBench	Memory	33	40	44	39	41	46
Ranking	RMBench	Memory	64	73	79	81	83	68
Cover	RMBench	Memory	74	68	83	83	100	100
Press	RMBench	Memory	29	18	15	26	21	15
RMBench average	RMBench	Memory	67.0	67.0	69.7	70.6	72.7	70.0
Retrieve fruit	RoboCasa	Memory	—	80	74	80	84	—
Retrieve oil	RoboCasa	Memory	—	66	78	70	76	—
Wash and return	RoboCasa	Memory	—	92	84	82	86	—
Wash and return same location	RoboCasa	Memory	—	46	62	60	62	—
Rinse cutting board	RoboCasa	Memory	—	0	0	0	0	—
Sweeten hot chocolate	RoboCasa	Memory	—	40	44	54	58	—
RoboCasa memory average	RoboCasa	Memory	—	54.0	57.0	57.7	61.0	—
Pan transfer	RoboCasa	Non-memory	—	78	68	70	76	—
Frying pan adjustment	RoboCasa	Non-memory	—	58	58	58	72	—
Organize condiments	RoboCasa	Non-memory	—	48	52	60	62	—
Place equal ice cubes	RoboCasa	Non-memory	—	4	10	6	6	—
Scrub cutting board	RoboCasa	Non-memory	—	60	60	56	66	—
Counter to sink	RoboCasa	Non-memory	—	98	94	96	98	—
RoboCasa non-memory average	RoboCasa	Non-memory	—	57.7	57.0	57.7	63.3	—

Stage-1 validation losses by memory capacity
Tokens	Total ×10⁻²	Vision ×10⁻²	Action ×10⁻⁴	State ×10⁻⁴	Progress ×10⁻⁴
1	3.649 ± 0.295	3.599	2.37	2.43	5.13
4	3.308 ± 0.273	3.278	1.31	1.35	3.31
16	3.406 ± 0.277	3.368	1.50	1.55	4.41
64	3.389 ± 0.282	3.359	1.50	1.52	2.94
256	3.338 ± 0.280	3.311	1.27	1.31	2.81
1024	3.256 ± 0.270	3.232	1.08	1.10	2.63

Beta-posterior success distributions for full-attention and recurrent memory encoding on RMBench and RoboCasa — Recurrent encoding matches full attention on RMBench and improves the RoboCasa overall average.

Per-task success rate by memory encoder (%)
Task	Benchmark	Full attention	Recurrent
Obs. / Pick	RMBench	9	6
Rearrange	RMBench	100	99
Put Back	RMBench	100	100
Swap Blocks	RMBench	100	100
Swap T	RMBench	100	100
Battery	RMBench	41	35
Ranking	RMBench	83	84
Cover	RMBench	100	100
Press	RMBench	21	31
Retrieve fruit	RoboCasa	84	90
Retrieve oil	RoboCasa	76	80
Wash and return left/right	RoboCasa	86	96
Wash and return same location	RoboCasa	62	58
Rinse cutting board	RoboCasa	0	4
Sweeten hot chocolate	RoboCasa	58	64
Pan transfer	RoboCasa	76	72
Frying pan adjustment	RoboCasa	72	70
Organize condiments	RoboCasa	62	74
Place equal ice cubes	RoboCasa	6	20
Scrub cutting board	RoboCasa	66	76
Move from counter to sink	RoboCasa	98	96
RMBench average	RMBench	72.7	72.8
RoboCasa memory average	RoboCasa	61.0	65.3
RoboCasa non-memory average	RoboCasa	63.3	68.0
RoboCasa overall average	RoboCasa	62.2	66.7

Ablations & analysis

Memory Attention Analysis

Cover Blocks analysis using the unsupervised 50–100% history projection and stream attention at step 750.

Paper figure comparing E2E, frozen, and adapted memory through t-SNE embeddings, linear-probe and task success, attention over history streams at step 750, and Cover Blocks outcomes — The paper figure compares representation structure, linear-probe and task success, attention over five history streams at step 750, and the resulting Cover Blocks action for E2E, Frozen, and Ours.

Cover Blocks analysis — 50–100% episode progress; attention query at step 750
Training	t-SNE silhouette S	Linear probe (%)	Task success (%)
E2E	−0.08	19.9	15
Frozen bootstrapped memory	+0.23	83.3	35
Ours (finetuned)	+0.36	91.5	100

Result analysis Across end-to-end, frozen, and jointly adapted memory, probe accuracy rises 19.9% → 83.3% → 91.5% and task success 15% → 35% → 100%, linking structured memory to better control.

Ablations & analysis

Memory Learning and VLA Flexibility

Training-recipe ablations and results across three VLA backbones.

Posterior violin comparison of end-to-end training, frozen bootstrapped memory, and jointly adapted memory — Multi-stream memory supervision improves over end-to-end action supervision, and joint adaptation improves performance further.

Training-recipe ablation — per-task success rate (%)
Task	E2E	Ours (frozen)	Ours
Obs. / Pick	12	7	9
Rearrange	90	99	100
Put Back	100	100	100
Swap Blocks	90	99	100
Swap T	45	99	100
Battery	27	20	41
Ranking	73	68	83
Cover	15	35	100
Press	5	2	21
Average	50.8	58.8	72.7

Posterior violin comparison of success with and without memory for pi zero point five, pi zero, and MolmoAct2 — The same memory pipeline improves average success for π_0.5, π₀, and MolmoAct2.

Per-task success across VLA backbones (%)
Task	π_0.5: No memory	π_0.5: With memory	π₀: No memory	π₀: With memory	MolmoAct2: No memory	MolmoAct2: With memory
Obs. / Pick	8	9	8	6	4	4
Rearrange	63	100	13	99	52	100
Put Back	20	100	49	100	0	100
Swap Blocks	16	100	4	100	11	65
Swap T	8	100	0	98	15	100
Battery	30	41	27	30	7	5
Ranking	66	83	54	56	62	74
Cover	15	100	0	78	18	83
Press	3	21	2	0	0	6
Average	25.4	72.7	17.4	63.0	18.8	59.7

Memory supervision ablations

Which inputs and objectives make memory useful?

Full-attention encoder with a fixed 256-token memory bottleneck.

Side-by-side posterior violin plots comparing historical memory inputs and memory bootstrap objectives — The left panel ablates which history streams enter memory; the right panel ablates which reconstruction and progress objectives supervise memory.

Historical input-stream ablation on RMBench (%)
Historical inputs	Obs. / Pick	Rearrange	Put Back	Swap Blocks	Swap T	Battery	Ranking	Cover	Press	Average
Vision only	9	93	100	81	45	30	80	100	7	60.6
Vision + Action	6	100	100	100	99	38	72	83	18	68.4
Vision + State	9	94	100	98	99	28	46	70	4	60.9
Vision + Action + State (IMPRINT)	9	100	100	100	100	41	83	100	21	72.7

Memory bootstrap-objective ablation on RMBench (%)
Bootstrap objective	Obs. / Pick	Rearrange	Put Back	Swap Blocks	Swap T	Battery	Ranking	Cover	Press	Average
All objectives (IMPRINT)	9	100	100	100	100	41	83	100	21	72.7
Without progress	6	91	100	90	97	35	73	100	26	68.7
Visual only	6	78	100	89	99	30	85	100	15	66.9
Action + state	5	100	100	89	98	36	61	52	23	62.7

Result analysis Visual-only memory loses 12.1 points, and visual-only reconstruction loses 5.8 points. Combining visual, action, and state histories with reconstruction and progress supervision performs best at 72.7%.

Memory Without Curation:
Learning Multi-Stream Latent Memory for Robot Policies

Method

Multi-stream history

Memory encoder

Teach memory what to retain

Read memory inside the action expert

All objectives (IMPRINT)

Choose an in-context folding example

Orange towel · Style 1 executions

How memory attends over time

Head camera

Left wrist

Right wrist

Action history

State history

Result Highlights

Contributions

Real-World Evaluation

Dust Table

Orange towel · Style 1 demonstration

IMPRINT

Trial 01

Select a rollout

MemER keyframes and predicted subtask

RMBench Simulation Results

Representative RMBench trials

RoboCasa: Memory and Non-Memory Tasks

Browse RoboCasa rollouts

Retrieve Fruit · Rollout 01

Select a trial

Success vs. Inference Efficiency

Memory Encoding

Memory Capacity

Full Attention vs. Recurrent Encoding

Memory Attention Analysis

Memory Learning and VLA Flexibility

Training Recipes

VLA Flexibility

Which inputs and objectives make memory useful?

Memory Without Curation: Learning Multi-Stream Latent Memory for Robot Policies

Method

Interactive IMPRINT method

Multi-stream history

Memory encoder

Teach memory what to retain

Read memory inside the action expert

All objectives (IMPRINT)

Choose an in-context folding example

Orange towel · Style 1 executions

How memory attends over time

Head camera

Left wrist

Right wrist

Action history

State history

Result Highlights

Contributions

Real-World Evaluation

Dust Table

Orange towel · Style 1 demonstration

IMPRINT

Trial 01

Select a rollout

MemER keyframes and predicted subtask

RMBench Simulation Results

Representative RMBench trials

RoboCasa: Memory and Non-Memory Tasks

Browse RoboCasa rollouts

Retrieve Fruit · Rollout 01

Select a trial

Success vs. Inference Efficiency

Memory Encoding

Memory Capacity

Full Attention vs. Recurrent Encoding

Memory Attention Analysis

Memory Learning and VLA Flexibility

Training Recipes

VLA Flexibility

Which inputs and objectives make memory useful?

Memory Without Curation:
Learning Multi-Stream Latent Memory for Robot Policies