HeLM: Hierarchical Explicit Language Memory for Non-Markovian Manipulation

Gyeonghun KimDonghoon KimMinsoo ChoiSuyun Lee

Seoul National University

Code arXiv Video

Abstract

HeLM introduces explicit language memory for VLM-guided non-Markovian manipulation. A high-level vision-language model maintains working memory within an episode and episodic context across episodes. Memory is updated when a task-relevant event occurs, and action commands guide a low-level robot policy.

We evaluate HeLM on 10 real-world tasks with 56 variants, covering counting, ordered actions and object history, including five inter-episode tasks that require persistent context across instruction boundaries. HeLM improves intra-episode subtask success by 12.8 percentage points over the strongest baseline and achieves 78% subtask success on compositional generalization tasks. The paper reports up to 4.5× less memory and 2.5× faster inference than the keyframe baseline at long horizons.

01 / THE PROBLEM

Why memory is needed

After a button is released, the scene can look almost unchanged. A policy that sees only the current observation can repeat a completed action.

A vanilla policy repeats the red-button press, while HeLM remembers completed presses and follows the red, green, blue order.
THE MOTIVATING EXAMPLE A sequence needs a record of progress. This is the non-Markovian part: the correct action depends on history.
Interactive example: button-order memory

Progress determines the next action

The instruction stays the same: press red, green, then blue. Select a step to see how the memory records completed presses and changes the next command.

Photographs illustrate the selected button action. Progress and commands follow the task specification; photos are not synchronized rollout frames.

TASK-SPECIFICATION EXAMPLEA2 / RGB
“Press red, green, then blue.”
WHAT HeLM REMEMBERS

Progress: red

Next actionPress the green button

Select a step. The memory changes the next action.

02 / THE METHOD

HeLM framework

HeLM separates deciding what comes next from executing the movement. The high-level vision-language model (HLP) manages memory; the low-level action policy (LLP) controls the robot.

  1. 01

    Notice an event

    Detect a meaningful change, such as a button press. If nothing happens, keep the memory.

    DETECT
  2. 02

    Update the memory

    Record the new progress and decide which action command should come next.

    UPDATE
  3. 03

    Execute the action

    The low-level policy turns the command and current observations into robot actions.

    ACT
HeLM architecture from the paper: the high-level policy reads the instruction, current observation and language memory, then sends action commands to the low-level VLA policy. Working memory tracks events; episodic context carries information across episodes. The shared VLM uses detect mode to watch for events and update mode to revise memory and the next command.
FIGURE 2 / THE HeLM ARCHITECTURE

The high-level policy maintains language memory and chooses an action command. The low-level policy turns that command into robot actions. Detect and update are two modes of the same VLM.

Scroll horizontally to read the full figure.

Open full-size figure
a.

Working memory

Progress inside the current instruction, updated when a task event occurs.

Count: 2 (Goal: 3)
b.

Episodic context

Information retained across episodes, available when a new instruction arrives.

Previous_Count: 3
Go deeper: event-triggered updates

Detect mode emits a task-relevant event or none. Update mode runs when an event is detected or the global instruction changes. Working memory tracks the current task; episodic context carries information across task boundaries.

Detect and update calls across episodes, with working memory and episodic context updated at task events.
The evaluation loop across intra-episode and inter-episode tasks.

03 / TASKS

Real-world tasks

Select a task to view its description and available recording. Inter-episode video is provided for Human-Order Press. The suite contains 10 tasks and 56 variants, covering both intra-episode and inter-episode memory.

Inspect task-specification memory states

Select “Inspect memory” on a task to see its instructions, progress and memory states. These are task-specification examples, separate from the recorded videos. This explorer contains 51 repository variants; the paper evaluates 56 variants.

Choose a task and variant. Play the sequence or select a step; use ← → when focused.

Reference photographs switch to matching actions or outcomes when available; they are not synchronized rollout frames. Window regions use the robot’s frame. Missing drawer-specific photographs are marked as setup references. Progress and memory follow the task specifications. Playback timing is illustrative. For cross-episode memory, try E1 and follow the new-instruction marker.

See the real-world task setups
Real-world setups for five intra-episode tasks and five inter-episode tasks.
Top: Intra-episode tasks (A1–A5). Bottom: Inter-episode tasks (E1–E5).

04 / THE EVIDENCE

Evaluation results

Real-world evaluation across 10 tasks. The evaluation protocol uses 15–45 trials per task (3–9 variants, 5 trials each). Intra-episode results in Figure 5 use 15–30 trials per task. Subtask success and strict task success are reported separately.

85.7%

Intra-episode

Mean subtask-level success.
Keyframe-HLP: 72.9%.

89.2%

Inter-episode

Mean subtask-level success on E1–E5.
Keyframe-HLP: 16.6%.

78%

Compositional generalization

Subtask success on held-out variants.
Keyframe-HLP: 65.7%.

INTRA-EPISODE COMPARISON

Intra-episode subtask and strict task success

HeLMKeyframe-HLPPCMB-π₀Naïve-π₀

All intra-episode scores
Subtask-level success rate (%), paper Fig. 5
MethodA1A2A3A4A5Avg.
HeLM10092.296.666.673.385.7
Keyframe-HLP90★91.16079.2★44.472.9
PCMB-π₀10024.433.332.96.739.5
Naïve-π₀31.16.633.38.34.416.7

★ Keyframe-HLP fails to detect termination. Subtask scores count partial progress; use the chart’s strict metric for ordered completion and stopping.

Inter-episode scores
MethodE1E2E3E4E5Avg.
HeLM (Ours)90.587.7908097.789.2
Keyframe-HLP27.6★033.3022.216.6

★ Keyframe-HLP fails to detect termination. PCMB-π0 and Naïve-π0 are excluded here: they reset internal state at episode boundaries and have no mechanism for cross-episode context.

Compositional generalization

The LLP is fine-tuned to execute specific action commands, but the HLP receives no task-specific finetuning for these compositions — e.g. counting to five when training only covered up to three, or grounding an object never seen during training.

MethodA1′A2′A3′A4′A5′Avg.
HeLM (Ours)1009567676178
Keyframe-HLP100★63.977.838.9★48.1★65.7

★ Keyframe-HLP fails to detect termination in A1′, A4′, and A5′.

Memory and HLP fine-tuning ablations
Table 2 · LIBERO-Mem task success (%)
MethodT1–T2T3–T8T9–T10Avg.
π₀.₅651087.536.5
π₀.₅ + HLP (w/o memory)27.5032.512
π₀.₅ + HLP + memory (w/o FT)5037.58.5
HeLM (Ours)67.545.887.558.5

All configurations use π₀.₅ as LLP. T3–T8 require persistent temporal state tracking. A shared Qwen2.5-VL-7B HLP is trained using 100 demonstrations per task; evaluation uses 20 held-out rollouts per task.

Two-stage inference ablation
Table 3 · Real-world intra-episode subtask success (%)
MethodA1A2A3A4A5Avg.
HeLM w/ one-stage inference10081.522.286.848.167.7
HeLM (Ours)10092.296.666.673.385.7
Failure analysis and limitations

Among HeLM failures, Detection accounts for 64.1%, Low-Level Execution for 31.2%, and Order for 4.7%; no Termination failures were observed. Detection errors can cause incorrect memory updates that the framework cannot self-correct.

The evaluation uses predefined task-relevant events and focuses on discrete state changes. Synchronous HLP invocation may miss events within an LLP chunk. The benefit of separating working and episodic memory, relative to a single persistent memory, remains to be evaluated.

Task-length robustness and memory scaling

The paper reports up to 4.5× less memory and 2.5× faster inference than Keyframe-HLP at long horizons. HeLM has additional HLP inference overhead and is initially slower at short horizons.

Paper Figure 6: success across longer task sequences, inference latency and GPU memory as stored events increase.
Figure 6 from the latest manuscript. A1 remains at 100% through seven subtasks; A2 ranges from 77% to 100%. The figure reports memory scaling against the same Qwen2.5-VL-7B HLP backbone.
Additional simulation results: LIBERO-Mem

As a complement to the real-robot experiments, we ran HeLM on the LIBERO-Mem benchmark against π₀.₅ and published memory-augmented baselines. Full per-task results are reported in Table 11; memory and HLP fine-tuning ablations are reported in Table 2.

MethodT1T2T3T4T5 T6T7T8T9T10Avg.
HeLM (Ours)75.060.075.055.065.025.045.010.085.090.058.5
π0.550.080.015.020.015.05.00.05.085.090.036.5
Naive E-SlotSSM50.00.033.30.014.30.00.00.030.020.014.8
SlotVLA (h=1)0.00.00.00.00.00.00.00.00.00.00.0
SlotVLA (h=8)50.00.00.00.00.00.00.00.00.00.05.0
π0 (h=1)50.00.00.00.00.00.00.00.00.00.05.0

Baseline rows for π0, SlotVLA, and Naive E-SlotSSM are taken from the LIBERO-Mem paper. In T9–T10, the wrist camera can reveal basket contents, weakening the intended memory requirement (Appendix F.3).

05 / EXPLORE THE PROJECT

Code and resources

Task specifications, the data pipeline, HLP training and robot evaluation code are available in the repository.

Explore the code
Paper PDF ↗arXiv ↗
01
Get started

Installation and minimal validation

02
Build the data

Task specs → event and memory labels

03
Train the policies

High-level memory and low-level actions

BibTeX

Author order and title follow the manuscript. Available as arXiv:2611.02660.

@misc{helm2026,
  author = {Kim, Gyeonghun and Kim, Donghoon and Choi, Minsoo and Lee, Suyun},
  title = {HeLM: Hierarchical Explicit Language Memory for Non-Markovian Manipulation},
  year  = {2026},
  eprint = {2611.02660},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.02660}
}