Tone: clean-academic, survey register — a map to sit next to the deeper theoretical and applied posts, not a replacement for either.
Original. “Composition does not preserve reliability… Multi-agent systems are often less reliable than the single loop they replaced, and the failure is located in the seams — task specification and result fusion — not in the workers.” — from the agent-loop field guide, Sub-problem 4.
Essence. The field guide proved why naive multi-agent composition can hurt more than it helps (reliability compounds as roughly $p^n$ across dependent steps, before you even add the orchestrator’s own decomposition/fusion error). What it didn’t do — deliberately, it’s a theory post — is catalogue the actual patterns people use once they compose loops anyway. This post is that catalogue: four real patterns, compared on the one thing the reliability math says matters — where the seams are, and what each pattern does to keep them from being the weak point.
Every multi-agent pattern below is, underneath its name, a different answer to one question: when you split a task across multiple LLM calls, what exactly gets passed across the seam? Full reasoning traces? Final answers only? A shared draft everyone edits? The answer determines both the pattern’s failure mode and its cost.
| What crosses the seam | Direction | Failure mode targeted | |
|---|---|---|---|
| Orchestrator–Workers | task spec out, result back | controller ↔ each worker (star) | task decomposition reliability — the field guide’s $p^n$ problem |
| Multi-Agent Debate | full reasoning traces | all-to-all, multi-round | hallucination / factual error, via disagreement surfacing bad reasoning |
| Mixture-of-Agents | full prior-layer outputs | strictly forward, layer to layer | output quality/diversity — no single model’s weaknesses dominate |
| Reflexion | verbal self-critique | agent to itself, across time | single-attempt failure, via retry informed by stated failure reason |
Going back to the $p^n$ reliability-composition result: Orchestrator–Workers is the pattern most exposed to it — a genuine dependency chain (decompose → delegate → fuse), so the compounding failure the field guide warns about is a real risk, not a theoretical one. Debate and Mixture-of-Agents are structurally different: they’re not a dependency chain toward one answer but multiple independent-ish attempts reconciled at the end, which is closer to an ensemble than a pipeline — a different reliability story the $p^n$ formula doesn’t directly describe. This is a genuine open question this survey doesn’t resolve: none of the four papers analyzes its own reliability composition the way the field guide’s theory post does for the general case.
Tone: clean-academic, survey register — this post is a map, not an essay. Depth on any one paradigm lives in the linked deep-dive posts; this one is for orientation and comparison.
Original. “We propose training and performing inference on language models with a learnable pause token… delaying the model’s answer generation by manually appending dummy tokens.” — (Goyal et al., 2024), the earliest of the five papers below, and the most modest version of the idea.
Essence. Every paper on this page is answering the same question with a different amount of ambition: if forcing a model through a discrete vocabulary token at every reasoning step is a constraint, not a law of nature, how far can you loosen it — from “just give it more time” (2023) to “let it reason in full continuous hidden-state space” (2024)? This post maps the five real answers, compared on what they actually do, not just what they’re called.
Standard chain-of-thought (Wei et al., 2022) forces every intermediate reasoning step through the same bottleneck the final answer goes through: collapse a continuous hidden state down to one vocabulary token, then re-embed that token to continue. Two things follow from that constraint, independent of any single paper: it’s expensive (each step is a full generation step), and it’s a lossy compression (everything the hidden state was representing except the argmax winner is discarded). Five real papers, in roughly chronological order, each relax a different piece of that constraint.
| What it actually generates | Needs training? | Inspectable? | Applied at | |
|---|---|---|---|---|
| Pause Tokens | a real (but content-free) vocabulary token | Yes — full pretrain + finetune | Yes (it’s a token) but tells you nothing | Pretrain + inference |
| CoCoMix | continuous SAE-derived concept vectors, interleaved | Yes — pretraining-level | Partially — concepts are individually inspectable/steerable via the SAE | Pretraining |
| Soft Thinking | probability-weighted mixture over token embeddings | No — training-free | Partially — still lives in token-embedding space, nearest-neighbor decodable | Inference only |
| Coconut | the raw last hidden state, fed back directly | Yes — needs a training curriculum | No — opaque hidden-state vector | Inference (after special training) |
| CoGHP | a latent subgoal in a learned goal-embedding space | Yes — offline RL training | No — opaque latent point | Inference (after special training) |
The two axes that actually separate these five: does it need training at all (only Soft Thinking says no), and does “continuous” mean a mixture over a fixed vocabulary or a genuinely free hidden-state vector (CoCoMix/Soft Thinking stay anchored to the token embedding table; Coconut/CoGHP don’t).
This page is the map, not the territory — for the actual mechanism, worked examples, and honest limitations of any one paradigm:
The field isn’t converging on one mechanism for “reasoning without words” — it’s exploring a real spectrum from token-anchored-but-free (Soft Thinking, CoCoMix) to fully unmoored from the vocabulary (Coconut, CoGHP), and the honest interpretability cost rises the further along that spectrum you go.
Tone: clean-academic body, professor-cautious on the limitations section. Third post in the reasoning series — read alongside the CoT-faithfulness post and CoGHP.
Original. “Large language models (LLMs) are typically constrained to reason in the language space, where they express the reasoning process through a chain-of-thought (CoT) to solve complex problems. However, the language space may not always be optimal for reasoning.” — (Hao et al., 2024), opening lines of the abstract.
Essence. The paper’s whole bet is contained in that second sentence: language was built for communication, not necessarily for the model’s own internal reasoning, so forcing every intermediate step through it may be a self-imposed constraint rather than a necessity. Coconut tests that bet directly by letting the model reason in its native continuous representation space instead, and finds a concrete capability gain (implicit breadth-first search) where the language bottleneck would have forced an early, possibly wrong, commitment to one path.
The faithfulness post made a structural argument: if verbalized chain-of-thought can silently diverge from the actual computation (Turpin et al., 2023) (Lanham et al., 2023), the verbalization step itself is where things go wrong — not the reasoning underneath it. CoGHP (Choi et al., 2026) showed one instance of removing that step, in a control domain: latent subgoals instead of language. Coconut (Hao et al., 2024) is the general-purpose version of the same move, built directly into an LLM’s own reasoning process.
Standard chain-of-thought (Wei et al., 2022): at each reasoning step, the model’s hidden state gets decoded into a token, that token gets re-embedded, and the next step conditions on the embedding of that decoded word. Coconut’s change is precisely to skip the decode-then-re-embed round trip: the model’s last hidden state is fed back directly as the next input embedding, no token in between. The paper calls this a “continuous thought.”
The model switches between two modes:
The one-sentence version. Chain-of-thought normally forces reasoning through a vocabulary-sized bottleneck at every step (the hidden state must collapse to one discrete token). Coconut removes the bottleneck — reasoning can stay in the model’s full continuous representation space for as many steps as the latent phase runs.
Worked example — what the two modes actually pass around, typed:
# Language mode (standard CoT) — one step
token_t: int # discrete vocabulary index
hidden_t: Tensor[d_model] # last hidden state at step t
logits_t = lm_head(hidden_t) # Tensor[vocab_size]
token_t+1 = argmax_or_sample(logits_t) # COLLAPSES to one discrete choice
embed_t+1 = embedding_table[token_t+1] # Tensor[d_model], looked up by id
# embed_t+1 is what conditions the next step — information not in the
# top token is discarded at the argmax/sample step.
# Latent mode (Coconut "continuous thought") — one step
hidden_t: Tensor[d_model] # last hidden state at step t
embed_t+1 = hidden_t # NO lm_head, NO argmax, NO lookup
# embed_t+1 is the full continuous vector — every direction the model was
# weighing survives into the next step, not just the single argmax winner.
# Mode switch (per-step, decided by the model / a control token)
if mode == "language":
next_input = embedding_table[argmax(lm_head(hidden_t))]
elif mode == "latent":
next_input = hidden_t # direct feedback, the whole point of Coconut
The entire mechanism is that one branch: whether next_input comes from a lookup table indexed by an argmax (language mode — necessarily one winner) or from the hidden state itself, untouched (latent mode — everything the argmax would have discarded stays in play). Nothing else about the transformer changes.
The efficiency angle (skip token generation, save compute) is real but not the interesting result. The actual finding: because a continuous hidden state isn’t forced to commit to one discrete token, it can encode more than one plausible next reasoning step at once — a superposition over candidate continuations rather than a single committed path. On a logical reasoning task (ProsQA) designed so that greedy, single-path search fails but breadth-first exploration of multiple branches succeeds, Coconut’s continuous thoughts show behavior consistent with implicit breadth-first search — higher accuracy than language-based CoT, with fewer generated tokens, on exactly the task shape where committing early to one path is the failure mode.
This is the mechanistic version of the faithfulness argument: language CoT is forced to pick one discrete next step and narrate it, even when the right move is to keep several hypotheses alive. Continuous thought doesn’t have that forcing function.
Three things this doesn’t solve, stated plainly rather than glossed over:
Tone: clean-academic body, professor-cautious throughout — this is a “here’s what two specific papers actually showed” post, not a broad claim about all chain-of-thought.
Chain-of-thought prompting (Wei et al., 2022) produces text that looks like a reasoning trace — intermediate steps before an answer. The question this post is about: does that text actually describe the computation that produced the answer, or is it a plausible-sounding story generated after the fact? A CoT is faithful if the stated reasoning is what actually determined the output; it’s unfaithful if the model would have reached the same answer for reasons it never mentions — sometimes reasons it couldn’t or wouldn’t state.
This isn’t a philosophical question. It has a direct empirical answer, and two papers answer it in ways that should change how much you trust a CoT trace at face value.
(Turpin et al., 2023) runs a simple, sharp experiment: take a multiple-choice task, and bias the model toward a wrong answer in a way it’s never asked to justify — for example, silently reordering the answer choices so the correct answer is always option (A) in the few-shot examples the model sees. Models pick up on this bias and their accuracy drops — by as much as 36% across a 13-task suite from BIG-Bench Hard — but the chain-of-thought they generate never mentions the biasing feature. It produces a fluent, plausible-sounding justification for whatever answer the bias pushed it toward, as if that were the real reason all along.
The core result, stated plainly. The explanation and the actual cause of the answer can come apart, and the model gives no sign that this has happened. A reader of the CoT alone has no way to detect the unfaithfulness from the trace itself.
This is the sharpest possible counterexample to “the CoT shows you the reasoning”: here, it demonstrably doesn’t — it shows you a reasoning-shaped artifact.
(Lanham et al., 2023) takes a broader, more systematic approach: several interventions on the CoT itself (truncating it early, paraphrasing it, adding mistakes into it) to see how much the final answer actually depends on the specific stated reasoning, across many tasks.
Two findings matter most:
Between the two papers, the honest picture is: faithfulness isn’t a fixed property of “using chain-of-thought” — it’s task-dependent, model-dependent, and not guaranteed to improve as models get better.
The instinct these results should not produce is “so don’t trust CoT, use free-text reasoning less.” The more interesting move is the one explored in the CoGHP post: if a verbalized chain can silently diverge from the real computation, then the verbalization itself is the vulnerable step — not the reasoning process underneath it. Reasoning that happens in latent space and is never asked to justify itself in words can’t be “unfaithful” in Turpin’s sense, because there’s no separate narrative layer to diverge from the computation. That doesn’t make latent reasoning trustworthy — it makes a different, arguably harder problem: you’ve traded “the explanation might be lying” for “there’s no explanation to check at all.” Which problem you’d rather have depends on what you actually need — legibility, or correctness with an opaque mechanism.
Tone: clean-academic framing, project-note detail level. This documents a real, small, working v0.1 — status claims are checked against the actual source, not aspirational.
Running several long-lived AI agent sessions in parallel creates a genuine attention problem: which session needs you right now? The obvious answer — build a dashboard — has a specific, recurring failure mode. A dashboard is a summary, one step removed from the thing it summarizes: by the time you notice a row, the underlying session has often moved on, the summary is stale, and switching from “the dashboard” to “the actual session” is itself a context switch with its own cost. Enough sessions and the dashboard becomes noise you learn to ignore — which defeats the purpose.
A browser tab isn’t a dashboard row. Each tab is a full, self-contained, actionable context — you don’t read a summary of a tab and then go find the real thing, the tab is the real thing, and its favicon/title already carries a compressed signal (spinner = still loading, a notification badge = needs you) without being a separate artifact you have to cross-reference. The bet behind kmux-observer: apply that model to AI agent sessions instead of building a traditional dashboard.
What’s actually there, in src/cli.ts (476 lines, Bun):
in_progress, blocked, finished, error, idle — each with a fixed icon and color, so the signal is glanceable rather than something you have to read.kmux hook <event>) push state changes as they happen — the tool doesn’t scrape or guess at session state.cmux set-status to push each session’s state into cmux.app’s existing sidebar — so there’s exactly one place you’re already looking, not a second app competing for attention. This matters for the browser-tab framing: the tabs need to live somewhere you’d already be.~/.kmux/events/ — a durable trail, not just current state, so a session’s history survives past the live view.~/.claude/settings.json — that’s a deliberate, separate approval step, not an oversight.cmux.app quitting) would require forking cmux itself in Swift — shelved until actual dogfooding of the sidecar shows it’s needed, not built speculatively ahead of that evidence.The agent-loop post asks whether a model can be trained to know when it should escalate itself — a model-facing question. kmux-observer is the human-facing mirror of the same underlying problem: an agent loop, however well it escalates internally, still eventually needs a human’s attention at some point, and the interface for that handoff has its own design question, separate from what happens inside the loop. Neither half is solved by the other.
Tone: clean-academic framing + professor-cautious limitations. General framing post — the financial-LLM post applies this to one domain; this one asks what’s true about the pattern independent of domain.
Modern agent systems draw on three ideas that show up constantly, almost always separately:
Each pattern has its own literature. What’s missing is treating them as one system: fine-tuning a model specifically to operate inside a loop that can escalate, rather than bolting tool-calling and looping onto a frozen general-purpose model as inference-time scaffolding. That’s the actual gap.
The default way people build agents today: take a strong general-purpose model, give it tools via function-calling, wrap it in a loop, done. This works, but it means the model was never trained to know it’s operating inside a loop — it doesn’t have a learned sense of “this looks like a case where I should escalate rather than retry the same approach,” because escalation-awareness was never part of its training signal, only its prompt.
Fine-tuning inside the loop — where the training data itself includes retry/escalate decisions and their outcomes, not just single-shot tool-use traces — is a different regime. It’s the difference between a model that can call a calculator when told to, and a model that has learned when its own confidence doesn’t warrant answering directly.
For this to be more than a restatement of three known ideas, it needs a testable claim: a model fine-tuned with loop-and-escalation-aware training data should out-perform the same model with the identical tool/loop scaffolding bolted on at inference time only — on tasks where escalation actually matters (ambiguous evidence, conflicting sources, genuinely hard sub-problems), not on tasks any reasonable scaffold already solves. That ablation — same tools, same loop, fine-tuned vs. not — is the whole point; without it, “fine-tuning helps” is just an assertion.
Financial-filing question-answering is one domain where this is concrete rather than abstract: sector-specialized workers, tool-grounded calculation, and an explicit escalation policy when a computed answer can’t be verified against cited evidence. See the related post below for the full worked treatment — evidence-citation CoT format, the dataset-generation pipeline, and the honest limitations.
Tone: clean-academic framing + professor-cautious limitations. This is a research-idea post, not a product writeup — implementation names and proprietary specifics are deliberately left out; the architecture and open questions are what’s real here.
Financial filings are full of numbers a general-purpose LLM can compute but not reliably prove — operating margin, growth rates, leverage ratios — because nothing forces the model’s reasoning to correspond to the evidence it’s citing. This post asks what a sector-specialized, tool-grounded, verifiably-reasoning financial LLM would actually require: the research gap, the two open questions, the proposed architecture, and the honest limitations.
Free login | Become a member — $30/month | Already have a key?
Tone: clean-academic body + mechanism-heavy method sections + professor-cautious limitations. Numbers are quoted from the paper’s Table 1 (verified against the arXiv full text, not a secondary review).
Chain-of-thought (Wei et al., 2022) made one idea famous: a model solves a hard problem better when it generates intermediate steps before the answer, instead of emitting the answer in one shot. But that framing hides a fork:
The big problem. Is a chain of thought a narrative the model tells after the fact, or the actual computation that produces the answer? And if it is real computation, does it have to be words at all?
CoGHP (Choi et al., 2026) is a sharp instance of the latent side of that fork — in a domain far from language. It reformulates a long-horizon control policy so that it first generates a chain of latent subgoals, then an action. The subgoals are not language; they are points in a learned goal space. But their function is exactly a chain of thought: each intermediate structure conditions the next prediction. Reading CoGHP is a way to ask the big problem in a setting where “reasoning” can be measured by task success, not vibes.
TL;DR. CoGHP turns hierarchical RL into one autoregressive sequence — state, final goal, several latent subgoals, action — generated by a single MLP-Mixer network, and it wins on exactly the tasks where you’d expect a reasoning chain to help: hard, multi-step, long-horizon ones.
In goal-conditioned RL (GCRL) the agent is given a goal $\goal$ and must reach it. Offline means it learns only from a fixed dataset of trajectories — no new environment interaction (crucial where real interaction is costly or unsafe: robot manipulation, navigation, industrial control). The headache is the horizon: the farther $\goal$ is from the current state $\st$, the sparser the reward signal, the more Bellman-backup value error accumulates, and the harder it is to attribute a single action to a distant outcome.
Hierarchy is the natural response: split “reach the far goal” into high-level subgoal selection + low-level control, so each policy faces a shorter horizon.
CoGHP names three structural limits of existing offline hierarchical RL (e.g. HIQL (Park et al., 2023)):
Redefine the hierarchy as one unified autoregressive sequence model. Input: current state + final goal. Output: a sequence of latent subgoals followed by the primitive action. Each subgoal conditions the next prediction — one network, one computational graph.
The analogy to language CoT is structural, not literal: in both, the final output is not produced in one shot — an intermediate structure is built first to condition what comes next. The difference: CoGHP’s subgoals are latent future-state points, not words, and the final goal is kept as a condition across the entire sequence, so awareness can’t leak away. Generation is far-to-near: the farthest subgoal first, a coarse plan, then refined toward the nearest control target.
| Piece | Choice |
|---|---|
| Token sequence | [state] [goal] [subgoal₁ … subgoal_k] [action], each token a fixed semantic role |
| Backbone | MLP-Mixer (Tolstikhin et al., 2021) (token-mixing + channel-mixing) |
| Autoregression | a causal mixer — a learnable lower-triangular matrix over stacked tokens |
| Value | goal-conditioned IQL (Kostrikov et al., 2022) |
| Policy objective | AWR (Peng et al., 2019) (advantage-weighted regression) |
Why MLP-Mixer over a Transformer? The token roles are fixed (position = meaning) and the sequence is short, so feed-forward token-mixing is stable where attention’s dynamic interaction is overkill. The shared value function is the quiet key: because latent subgoals and the final goal live in the same embedded goal space, one IQL value gives training signal to both subgoals and actions.
Reported success rates (OGBench, Table 1 — quoted from the paper):
| Task | CoGHP | HIQL | OTA | SAW | note |
|---|---|---|---|---|---|
| pointmaze-giant | 79 ± 8 | 46 ± 9 | 72 ± 6 | 68 ± 8 | long nav |
| antmaze-giant | 78 ± 8 | 65 ± 5 | 77 ± 4 | 73 ± 4 | long nav |
| cube-single | 97 ± 3 | 41 ± 6 | 33 ± 4 | 77 ± 4 | GCIQL 99 ± 1 edges it |
| cube-triple | 42 ± 3 | 2 ± 1 | 2 ± 1 | 17 ± 3 | hard multi-object |
| scene | 78 ± 7 | 38 ± 3 | 20 ± 4 | 63 ± 6 | sequential (unlock→open→place→close) |
(Professor-cautious read.) The honest signal is complexity scaling, not the average. On easy tasks the architecture barely matters — on cube-single a flat GCIQL (99) actually beats CoGHP (97). The chain pays off precisely where reasoning should: cube-triple (42 vs HIQL’s 2) and scene (78 vs 38), the tasks that demand several coordinated intermediate decisions. That is the claim doing honest work: CoGHP is not a universal upgrade; it is a long-horizon, multi-decision upgrade.
Read against §0’s big problem, CoGHP is evidence for a strong claim: a chain of thought does not have to be words to be real. Its subgoals are latent, un-verbalized, and yet they measurably improve long-horizon control — the “reasoning” is validated by task success, not by how convincing the narrative reads. That is the cleanest rebuttal I know to “CoT is just post-hoc storytelling”: here the chain is the computation, because there is no language layer to narrate anything.
Reusable ideas: (1) collapse planner + controller into one goal→subgoal→action sequence model; (2) for fixed-role control tokens, a simple MLP-Mixer can beat a Transformer; (3) treat intermediate latent states as reasoning tokens — a bridge to multi-step agent planning, where plan/tool/action need not be separate networks.
CoGHP recasts long-horizon offline GCRL as autoregressive generation of latent subgoals
Clustered + cross-linked — follow the → edges. [arXiv] = formal, [code]/[blog] = informal.
Numbers quoted from arXiv:2602.03389 (v2) Table 1. All cited works have verified arXiv identifiers.
]]>This is a survey-first note: map the field and its sub-problems before any one system takes the stage. Qwen-VLA (Qwen Team, 2026) appears only as one point in a design space that exists with or without it. Numbers attributed to a paper are stated as that paper reports them; my own judgment is quarantined to §D.
A vision-language-action (VLA) model is a policy that maps visual observations $\obs_t$ and a natural-language instruction $\ell$ to robot actions $\act_t$:
\[\policy_\theta:\; (\obs_{1:t},\, \ell)\; \longmapsto\; \act_t \in \mathcal{A}.\]The lineage is short and fast. RT-1 (Brohan et al., 2023) showed a single transformer absorbing large-scale real-robot data; RT-2 (Brohan et al., 2023) made the decisive move of reusing a vision-language model’s web knowledge by emitting actions as text tokens; PaLM-E (Driess et al., 2023) folded continuous sensor states into the language model itself. OpenVLA (Kim et al., 2024) and Octo (Octo Model Team et al., 2024) opened the recipe; RoboFlamingo (Li et al., 2024) showed a VLM backbone is an effective imitator; $\pi_0$ (Black et al., 2024) replaced discrete action tokens with a flow-matching continuous decoder.
It is tempting to rank VLAs by a headline success rate. That is the wrong axis. A policy that scores 97% on one benchmark suite and collapses on a new robot, a new task family, or a mildly shifted environment has not demonstrated a foundation model — it has demonstrated overfitting at scale.
Claim (the axis). The quantity that separates a benchmark policy from an embodied foundation model is adaptation: how cheaply one set of weights survives a change of body, task, action space, or distribution.
The rest of this note is organized around those four changes.
Different robots have different kinematics, sensors, and control conventions. The naive fix is a per-robot output head; it does not scale and it forbids transfer. Two lines attack this. Open X-Embodiment (Open X-Embodiment Collaboration et al., 2024) pooled 22 robots into one dataset and trained RT-X models that transfer across bodies; CrossFormer (Doshi et al., 2024) pushed further — a single transformer across manipulation, navigation, locomotion, and aviation with no manual action/observation-space alignment. On the navigation side, ViNT (Shah et al., 2023) is a cross-robot navigation foundation model with positive transfer to unseen platforms.
The newest lever is to describe the body in language. Instead of a learned embodiment embedding, a textual description of the current robot and its control convention is placed in the prompt — so the same weights specialize at inference by reading who they are driving. This is the “tool-schema” pattern from software agents, imported into robotics.
Manipulation, navigation, and trajectory prediction are historically three fields with three benchmark cultures. The generalist bet is that they are one problem — conditional sequence generation over actions — and that joint training transfers. GATO (Reed et al., 2022) made the maximalist version of this bet (one network, 600+ tasks across modalities). In robotics specifically, the unification is now concrete: navigation is cast as action-and-trajectory prediction over the same interface as manipulation, evaluated on VLN benchmarks R2R (Anderson et al., 2018) and RxR (Ku et al., 2020) alongside manipulation suites.
How a VLA emits an action is a real design axis, not an implementation detail.
Flow matching, in one line. Learn a velocity field $v_\theta(x,\tau\mid c)$ so that integrating the ODE $\;\dot{x}=v_\theta(x,\tau\mid c)\;$ from noise $x_0\sim \mathcal{N}(0,I)$ at $\tau{=}0$ to $\tau{=}1$ lands on a valid action chunk $x_1$, conditioned on the VLM’s language+vision tokens $c$. Continuous, high-precision, and it sidesteps action quantization — at the cost of an ODE solve per action.
The final and hardest change is distribution. SimplerEnv (Li et al., 2024) exists precisely to measure the sim-to-real and cross-setup gap for manipulation; LIBERO (Liu et al., 2023) probes lifelong transfer across task suites; real-world ALOHA (Zhao et al., 2023) out-of-distribution trials and dynamic manipulation (moving targets) are where reported numbers fall hardest. This is the axis on which “foundation model” claims should be stress-tested, not the in-distribution suites.
Qwen-VLA (Qwen Team, 2026) threads all four axes:
Reported scores (as stated by the paper): 97.9% LIBERO, 73.7% Simpler-WidowX, 86.1/87.2% RoboTwin (Mu et al., 2025) Easy/Hard, 69.0% R2R OSR, 59.6% RxR SR, 76.9% avg ALOHA OOD, and 26.6% zero-shot on DOMINO (Fang et al., 2026) dynamic manipulation. The full reading note is here.
(This section is deliberately cautious — hedged where the evidence is.)
A benchmark table is a claim, not a proof. Five questions decide whether the numbers mean progress:
The number that reads both ways. DOMINO zero-shot 26.6% is simultaneously evidence that cross-task transfer to dynamic manipulation is non-trivially possible and evidence that dynamic manipulation remains largely unsolved. Promise and open problem in one figure — and a caution against reading the in-distribution 97.9% as the headline.
The trajectory is clear even if the destination is not. The field is converging on: continuous flow/diffusion decoders over discrete tokens; language-described embodiment over learned embodiment embeddings; and joint manipulation-navigation- trajectory training over siloed policies. What remains genuinely open — and where I would place research effort — is the fourth axis: dynamic, out-of-distribution adaptation, where even the strongest current numbers are low. A VLA that is an embodied foundation model, rather than a very good benchmark policy, will be decided there.
Not a flat list — a clustered web. Each paper links to its [arXiv] (formal) and, where one exists, an [on this blog] deep-note (informal). The → edges are the spider: follow them to jump from any paper to the work it builds on or leads to.
RT-1 proved scale on real-robot data; RT-2 turned actions into web-knowledge text tokens; OpenVLA / Octo opened the recipe; π0 made the decoder continuous — the move that Qwen-VLA inherits.
The fork of §B.3: quantize actions into tokens, or generate them continuously.
The §B.1–B.2 bet: pool bodies and tasks into one policy.
The §B.4 proving grounds. Qwen-VLA is scored on all of these.
All entries are established works with verified arXiv identifiers, including RoboTwin, DOMINO, NaVILA, Uni-NaVid, and RoboCat — added 2026-09-20 (previously named in prose but not formally cited). Informal / project-page links can be layered onto each node next (verified before adding — none fabricated).
]]>| Title | Qwen-VLA: Unifying Vision-Language-Action Modeling across Tasks, Environments, and Robot Embodiments |
| arXiv | 2605.30280 (v1, 29 May 2026) |
| Code | github.com/QwenLM/Qwen-VLA — official repo, 705★ / 24 forks. As of this note: technical report + blog + demo video only; no weights or install code released yet. |
| Backbone | Qwen3.5-4B (vision-language) |
| Action decoder | 1.15B DiT (Diffusion Transformer), flow-matching |
vision-language-action · embodied foundation model · DiT action decoder · flow matching · embodiment-aware prompt conditioning · action-and-trajectory prediction · cross-embodiment · manipulation + navigation + trajectory unification · no per-platform output heads · joint pretraining
Qwen-VLA bolts a DiT flow-matching action decoder onto the Qwen3.5-4B VLM stack, turning language + visual tokens into continuous joint trajectories. Its lever is embodiment-aware prompt conditioning: the current robot body and control convention are described in text, so a single set of weights serves many platforms with no per-platform output heads. It casts manipulation, navigation, and trajectory prediction into one action-and-trajectory objective and pretrains jointly over robot-manipulation trajectories, human egocentric demos, synthetic sim, vision-and- language navigation data, trajectory-centric supervision, and auxiliary VL data.
| Benchmark | Metric | Score |
|---|---|---|
| LIBERO | success | 97.9% |
| Simpler-WidowX | success | 73.7% |
| RoboTwin — Easy / Hard | success | 86.1% / 87.2% |
| R2R | OSR | 69.0% |
| RxR | SR | 59.6% |
| Real-world ALOHA (OOD) | avg success | 76.9% |
| DOMINO (dynamic manip.) | zero-shot | 26.6% |
Do not read the table alone. What actually decides whether this is progress:
And the headline tension: DOMINO zero-shot 26.6% reads both ways — real zero-shot transfer to dynamic manipulation is non-trivial, and 26.6% says dynamic manipulation is still largely unsolved. Promise and open problem in one number.
It’s the cleanest current example of adaptation as the axis: one policy adapting across bodies (embodiment-aware conditioning), across tasks (manip+nav+trajectory), across action spaces (continuous flow-matching vs discrete tokens), and across distributions (real-world + dynamic OOD). See the companion survey for the four-axis framing and the full, verified reference set.
Sources: arXiv 2605.30280; official GitHub repo (QwenLM/Qwen-VLA). Repo state (stars / no-release) as observed 2026-07-11.
]]>