Chain-of-Goals: latent chain-of-thought for long-horizon offline RL

Created on July 26, 2026

2026   ·   chain-of-thought   latent-reasoning   offline-rl   goal-conditioned-rl   hierarchical-rl   mlp-mixer   paper-review   vertical   ·   reading   papers   reinforcement-learning   offline-gcrl

\[\newcommand{\st}{s} \newcommand{\goal}{g} \newcommand{\act}{a} \newcommand{\adv}{A}\]

Tone: clean-academic body + mechanism-heavy method sections + professor-cautious limitations. Numbers are quoted from the paper’s Table 1 (verified against the arXiv full text, not a secondary review).

0. The big problem — what is a “chain of thought,” really?

Chain-of-thought (Wei et al., 2022) made one idea famous: a model solves a hard problem better when it generates intermediate steps before the answer, instead of emitting the answer in one shot. But that framing hides a fork:

  • Explicit CoT — the steps are human-readable tokens (words, equations). Legible, but not necessarily faithful to the computation.
  • Internal / latent CoT — the steps live in a latent space, never verbalized. Less interpretable, but the “reasoning” is a real intermediate structure the model conditions on.

The big problem. Is a chain of thought a narrative the model tells after the fact, or the actual computation that produces the answer? And if it is real computation, does it have to be words at all?

CoGHP (Choi et al., 2026) is a sharp instance of the latent side of that fork — in a domain far from language. It reformulates a long-horizon control policy so that it first generates a chain of latent subgoals, then an action. The subgoals are not language; they are points in a learned goal space. But their function is exactly a chain of thought: each intermediate structure conditions the next prediction. Reading CoGHP is a way to ask the big problem in a setting where “reasoning” can be measured by task success, not vibes.

TL;DR. CoGHP turns hierarchical RL into one autoregressive sequence — state, final goal, several latent subgoals, action — generated by a single MLP-Mixer network, and it wins on exactly the tasks where you’d expect a reasoning chain to help: hard, multi-step, long-horizon ones.

1. Problem

1-1. Offline goal-conditioned RL, long horizon

In goal-conditioned RL (GCRL) the agent is given a goal $\goal$ and must reach it. Offline means it learns only from a fixed dataset of trajectories — no new environment interaction (crucial where real interaction is costly or unsafe: robot manipulation, navigation, industrial control). The headache is the horizon: the farther $\goal$ is from the current state $\st$, the sparser the reward signal, the more Bellman-backup value error accumulates, and the harder it is to attribute a single action to a distant outcome.

Hierarchy is the natural response: split “reach the far goal” into high-level subgoal selection + low-level control, so each policy faces a shorter horizon.

1-2. Why prior hierarchical methods fall short

CoGHP names three structural limits of existing offline hierarchical RL (e.g. HIQL (Park et al., 2023)):

  1. Single subgoal. Most methods emit one intermediate goal at a time; long-horizon tasks need several coordinated intermediate decisions (open → move → place → close).
  2. Loss of final-goal awareness. If the high level’s subgoal is wrong, a low level that only sees the subgoal drifts from the true final goal.
  3. Fragmented optimization. Separate high/low networks block the error signal from flowing through the whole hierarchy.

2. Core idea

2-1. Main contribution

Redefine the hierarchy as one unified autoregressive sequence model. Input: current state + final goal. Output: a sequence of latent subgoals followed by the primitive action. Each subgoal conditions the next prediction — one network, one computational graph.

The analogy to language CoT is structural, not literal: in both, the final output is not produced in one shot — an intermediate structure is built first to condition what comes next. The difference: CoGHP’s subgoals are latent future-state points, not words, and the final goal is kept as a condition across the entire sequence, so awareness can’t leak away. Generation is far-to-near: the farthest subgoal first, a coarse plan, then refined toward the nearest control target.

3. Architecture / method

Piece Choice
Token sequence [state] [goal] [subgoal₁ … subgoal_k] [action], each token a fixed semantic role
Backbone MLP-Mixer (Tolstikhin et al., 2021) (token-mixing + channel-mixing)
Autoregression a causal mixer — a learnable lower-triangular matrix over stacked tokens
Value goal-conditioned IQL (Kostrikov et al., 2022)
Policy objective AWR (Peng et al., 2019) (advantage-weighted regression)

Why MLP-Mixer over a Transformer? The token roles are fixed (position = meaning) and the sequence is short, so feed-forward token-mixing is stable where attention’s dynamic interaction is overkill. The shared value function is the quiet key: because latent subgoals and the final goal live in the same embedded goal space, one IQL value gives training signal to both subgoals and actions.

4. Training / data / recipe

  • Data. Offline only; evaluated on OGBench (Park et al., 2025) (pointmaze/antmaze navigation; cube/scene manipulation). Subgoal targets = future states sampled at a fixed interval from dataset trajectories.
  • Two-part training. (i) goal-conditioned IQL value (TD error, no OOD-action query); (ii) hierarchical policy via AWR on both subgoal and action, merged into one end-to-end loss.
  • Teacher forcing during training (ground-truth subgoal embeddings) → stable, but a train/eval exposure mismatch remains (rollout uses predicted subgoals).
  • Knobs that matter: subgoal interval (hierarchy granularity) and subgoal count (hierarchy depth).

5. Evaluation

Reported success rates (OGBench, Table 1 — quoted from the paper):

Task CoGHP HIQL OTA SAW note
pointmaze-giant 79 ± 8 46 ± 9 72 ± 6 68 ± 8 long nav
antmaze-giant 78 ± 8 65 ± 5 77 ± 4 73 ± 4 long nav
cube-single 97 ± 3 41 ± 6 33 ± 4 77 ± 4 GCIQL 99 ± 1 edges it
cube-triple 42 ± 3 2 ± 1 2 ± 1 17 ± 3 hard multi-object
scene 78 ± 7 38 ± 3 20 ± 4 63 ± 6 sequential (unlock→open→place→close)

(Professor-cautious read.) The honest signal is complexity scaling, not the average. On easy tasks the architecture barely matters — on cube-single a flat GCIQL (99) actually beats CoGHP (97). The chain pays off precisely where reasoning should: cube-triple (42 vs HIQL’s 2) and scene (78 vs 38), the tasks that demand several coordinated intermediate decisions. That is the claim doing honest work: CoGHP is not a universal upgrade; it is a long-horizon, multi-decision upgrade.

6. Limitations

  • Latent subgoals aren’t legible. They approximate future states, not a human-readable plan — so this is internal CoT: you cannot inspect the reasoning the way you can read a language chain.
  • Offline-data sensitivity. Subgoal supervision is future states from the dataset; poor trajectory coverage weakens both subgoal and generalization.
  • Teacher-forcing mismatch. Train on ground-truth subgoals, roll out on predicted ones — error can re-accumulate at evaluation.
  • Backbone conclusion is task-dependent. MLP-Mixer wins here (fixed roles, short sequence); longer sequences, image tokens, or language conditioning may flip it.
  • Hyperparameters. Subgoal interval/count need tuning per task; auto-selecting them in a real robot setting is unsolved.

7. My take

Read against §0’s big problem, CoGHP is evidence for a strong claim: a chain of thought does not have to be words to be real. Its subgoals are latent, un-verbalized, and yet they measurably improve long-horizon control — the “reasoning” is validated by task success, not by how convincing the narrative reads. That is the cleanest rebuttal I know to “CoT is just post-hoc storytelling”: here the chain is the computation, because there is no language layer to narrate anything.

Reusable ideas: (1) collapse planner + controller into one goal→subgoal→action sequence model; (2) for fixed-role control tokens, a simple MLP-Mixer can beat a Transformer; (3) treat intermediate latent states as reasoning tokens — a bridge to multi-step agent planning, where plan/tool/action need not be separate networks.

8. Summary

CoGHP recasts long-horizon offline GCRL as autoregressive generation of latent subgoals

  • action in one MLP-Mixer policy, sharing an IQL value across the hierarchy and keeping the final goal as a global condition. It shines as task difficulty rises — the signature of a method whose latent “chain of thought” earns its keep only when the horizon is long and the decisions are many.

Reference map

Clustered + cross-linked — follow the → edges. [arXiv] = formal, [code]/[blog] = informal.

① The paper

② Benchmark & closest baseline

  • OGBench · arXiv · code — the offline-GCRL benchmark CoGHP runs on.
  • HIQL · arXiv — the single-subgoal hierarchical baseline CoGHP most directly improves on. → value method IQL

③ Method primitives

  • IQL · arXiv — implicit Q-learning; the shared value learner (no OOD-action query).
  • AWR · arXiv — advantage-weighted regression; the policy objective.
  • MLP-Mixer · arXiv — the fixed-role token backbone (vs a Transformer).

④ The CoT lineage (why this is a reasoning paper)

  • Chain-of-Thought · arXiv — the explicit, verbalized ancestor. CoGHP is its latent cousin: reasoning steps that are never words. (Latent-space reasoning à la “Coconut” is the closest LLM analog — named in prose, not formally cited pending verification.)

⑤ Informal — another review

  • DimensionSTP review · blog — an independent write-up of the same paper (structure inspiration; read for a second angle).
Formal cited bibliography (auto-generated)

2026

  1. Chain-of-Goals Hierarchical Policy for Long-Horizon Offline Goal-Conditioned RL
    Jinwoo Choi, Sang-Hyun Lee, and Seung-Woo Seo
    arXiv preprint arXiv:2602.03389, 2026

2025

  1. OGBench: Benchmarking Offline Goal-Conditioned RL
    Seohong Park, Kevin Frans, Benjamin Eysenbach, and 1 more author
    In International Conference on Learning Representations (ICLR), 2025
    arXiv:2410.20092

2023

  1. HIQL: Offline Goal-Conditioned RL with Latent States as Actions
    Seohong Park, Dibya Ghosh, Benjamin Eysenbach, and 1 more author
    In Advances in Neural Information Processing Systems (NeurIPS), 2023
    arXiv:2307.11949

2022

  1. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
    Jason Wei, Xuezhi Wang, Dale Schuurmans, and 2 more authors
    In Advances in Neural Information Processing Systems (NeurIPS), 2022
    arXiv:2201.11903
  2. Offline Reinforcement Learning with Implicit Q-Learning
    Ilya Kostrikov, Ashvin Nair, and Sergey Levine
    In International Conference on Learning Representations (ICLR), 2022
    IQL; arXiv:2110.06169

2021

  1. MLP-Mixer: An All-MLP Architecture for Vision
    Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, and 2 more authors
    In Advances in Neural Information Processing Systems (NeurIPS), 2021
    arXiv:2105.01601

2019

  1. Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
    Xue Bin Peng, Aviral Kumar, Grace Zhang, and 1 more author
    arXiv preprint arXiv:1910.00177, 2019

Numbers quoted from arXiv:2602.03389 (v2) Table 1. All cited works have verified arXiv identifiers.

Related on this blog