VLA models through the lens of adaptation
This is a survey-first note: map the field and its sub-problems before any one system takes the stage. Qwen-VLA (Qwen Team, 2026) appears only as one point in a design space that exists with or without it. Numbers attributed to a paper are stated as that paper reports them; my own judgment is quarantined to §D.
A. The field first
A.1 What a vision-language-action model is
A vision-language-action (VLA) model is a policy that maps visual observations $\obs_t$ and a natural-language instruction $\ell$ to robot actions $\act_t$:
\[\policy_\theta:\; (\obs_{1:t},\, \ell)\; \longmapsto\; \act_t \in \mathcal{A}.\]The lineage is short and fast. RT-1 (Brohan et al., 2023) showed a single transformer absorbing large-scale real-robot data; RT-2 (Brohan et al., 2023) made the decisive move of reusing a vision-language model’s web knowledge by emitting actions as text tokens; PaLM-E (Driess et al., 2023) folded continuous sensor states into the language model itself. OpenVLA (Kim et al., 2024) and Octo (Octo Model Team et al., 2024) opened the recipe; RoboFlamingo (Li et al., 2024) showed a VLM backbone is an effective imitator; $\pi_0$ (Black et al., 2024) replaced discrete action tokens with a flow-matching continuous decoder.
A.2 Why adaptation is the right axis
It is tempting to rank VLAs by a headline success rate. That is the wrong axis. A policy that scores 97% on one benchmark suite and collapses on a new robot, a new task family, or a mildly shifted environment has not demonstrated a foundation model — it has demonstrated overfitting at scale.
Claim (the axis). The quantity that separates a benchmark policy from an embodied foundation model is adaptation: how cheaply one set of weights survives a change of body, task, action space, or distribution.
The rest of this note is organized around those four changes.
B. The four adaptations
B.1 Embodiment adaptation — one policy, many bodies
Different robots have different kinematics, sensors, and control conventions. The naive fix is a per-robot output head; it does not scale and it forbids transfer. Two lines attack this. Open X-Embodiment (Open X-Embodiment Collaboration et al., 2024) pooled 22 robots into one dataset and trained RT-X models that transfer across bodies; CrossFormer (Doshi et al., 2024) pushed further — a single transformer across manipulation, navigation, locomotion, and aviation with no manual action/observation-space alignment. On the navigation side, ViNT (Shah et al., 2023) is a cross-robot navigation foundation model with positive transfer to unseen platforms.
The newest lever is to describe the body in language. Instead of a learned embodiment embedding, a textual description of the current robot and its control convention is placed in the prompt — so the same weights specialize at inference by reading who they are driving. This is the “tool-schema” pattern from software agents, imported into robotics.
B.2 Task adaptation — manipulation, navigation, and trajectory in one model
Manipulation, navigation, and trajectory prediction are historically three fields with three benchmark cultures. The generalist bet is that they are one problem — conditional sequence generation over actions — and that joint training transfers. GATO (Reed et al., 2022) made the maximalist version of this bet (one network, 600+ tasks across modalities). In robotics specifically, the unification is now concrete: navigation is cast as action-and-trajectory prediction over the same interface as manipulation, evaluated on VLN benchmarks R2R (Anderson et al., 2018) and RxR (Ku et al., 2020) alongside manipulation suites.
B.3 Action-space adaptation — discrete tokens vs continuous flow
How a VLA emits an action is a real design axis, not an implementation detail.
- Discrete tokens. RT-2 (Brohan et al., 2023) bins each action dimension into integer tokens in the VLM vocabulary. Simple, reuses the LM head — but quantization caps precision and control rate. FAST (Pertsch et al., 2025) improves this with frequency-space (DCT) tokenization for autoregressive VLAs.
- Continuous generative decoding. Diffusion Policy (Chi et al., 2023) models an action chunk as a conditional denoising process; ACT (Zhao et al., 2023) predicts a chunk with a CVAE/transformer. The architectural enabler is the Diffusion Transformer (Peebles & Xie, 2023), and the training objective that $\pi_0$ (Black et al., 2024) and Qwen-VLA adopt is flow matching (Lipman et al., 2023).
Flow matching, in one line. Learn a velocity field $v_\theta(x,\tau\mid c)$ so that integrating the ODE $\;\dot{x}=v_\theta(x,\tau\mid c)\;$ from noise $x_0\sim \mathcal{N}(0,I)$ at $\tau{=}0$ to $\tau{=}1$ lands on a valid action chunk $x_1$, conditioned on the VLM’s language+vision tokens $c$. Continuous, high-precision, and it sidesteps action quantization — at the cost of an ODE solve per action.
B.4 Distribution / OOD adaptation — the honest frontier
The final and hardest change is distribution. SimplerEnv (Li et al., 2024) exists precisely to measure the sim-to-real and cross-setup gap for manipulation; LIBERO (Liu et al., 2023) probes lifelong transfer across task suites; real-world ALOHA (Zhao et al., 2023) out-of-distribution trials and dynamic manipulation (moving targets) are where reported numbers fall hardest. This is the axis on which “foundation model” claims should be stress-tested, not the in-distribution suites.
C. Qwen-VLA as the worked case
Qwen-VLA (Qwen Team, 2026) threads all four axes:
- Action space — a 1.15B DiT flow-matching action decoder on a Qwen3.5-4B VLM backbone (continuous, §B.3).
- Embodiment — embodiment-aware prompt conditioning: the body and control convention are described in text, no per-platform head (§B.1).
- Task — manipulation, navigation, and trajectory folded into one action-and-trajectory objective (§B.2).
- Distribution — evaluated out to real-world ALOHA OOD and dynamic manipulation (§B.4).
Reported scores (as stated by the paper): 97.9% LIBERO, 73.7% Simpler-WidowX, 86.1/87.2% RoboTwin (Mu et al., 2025) Easy/Hard, 69.0% R2R OSR, 59.6% RxR SR, 76.9% avg ALOHA OOD, and 26.6% zero-shot on DOMINO (Fang et al., 2026) dynamic manipulation. The full reading note is here.
D. Reading it critically
(This section is deliberately cautious — hedged where the evidence is.)
A benchmark table is a claim, not a proof. Five questions decide whether the numbers mean progress:
- Baselines. Is each score against a specialist SOTA or a weak generalist? A high number against the wrong baseline measures nothing.
- Train–test split. How much of LIBERO/R2R/RxR distribution entered the joint pretraining mix? Generalist scores are only credible if the evaluation split was genuinely held out.
- Embodiment-OOD definition. “New embodiment” is meaningful only if the control convention was truly unseen — otherwise it is in-distribution wearing a costume.
- Action normalization. Cross-embodiment results hinge on how joint/action spaces are normalized; a favorable normalization can manufacture transfer.
- Inference latency. A flow-matching ODE solve is not free. Real-robot control rate — not just success rate — determines whether the policy is deployable.
The number that reads both ways. DOMINO zero-shot 26.6% is simultaneously evidence that cross-task transfer to dynamic manipulation is non-trivially possible and evidence that dynamic manipulation remains largely unsolved. Promise and open problem in one figure — and a caution against reading the in-distribution 97.9% as the headline.
E. Where it points
The trajectory is clear even if the destination is not. The field is converging on: continuous flow/diffusion decoders over discrete tokens; language-described embodiment over learned embodiment embeddings; and joint manipulation-navigation- trajectory training over siloed policies. What remains genuinely open — and where I would place research effort — is the fourth axis: dynamic, out-of-distribution adaptation, where even the strongest current numbers are low. A VLA that is an embodied foundation model, rather than a very good benchmark policy, will be decided there.
Reference map
Not a flat list — a clustered web. Each paper links to its [arXiv] (formal) and, where one exists, an [on this blog] deep-note (informal). The → edges are the spider: follow them to jump from any paper to the work it builds on or leads to.
① The VLA lineage — scale → web-knowledge → open → continuous
RT-1 proved scale on real-robot data; RT-2 turned actions into web-knowledge text tokens; OpenVLA / Octo opened the recipe; π0 made the decoder continuous — the move that Qwen-VLA inherits.
- RT-1 · arXiv · project — transformer at real-robot scale. → leads to RT-2
- RT-2 · arXiv · project — actions as text tokens (web knowledge). → open version OpenVLA; continuous successor π0; token axis §B.3
- PaLM-E · arXiv — sensor states inside the LM. → sibling RT-2
- OpenVLA · arXiv · project — open VLA recipe. → generalist cousin Octo
- Octo · arXiv · project — open generalist policy. → cross-body CrossFormer
- RoboFlamingo · arXiv — VLM backbone as imitator. → backbone idea in Qwen-VLA
- π0 · arXiv · project — flow-matching continuous decoder. → same decoder family as Qwen-VLA; theory Flow Matching
② How actions are emitted — discrete tokens vs continuous flow
The fork of §B.3: quantize actions into tokens, or generate them continuously.
- FAST · arXiv — frequency-space action tokens. → discrete counterpart to Diffusion Policy; pairs with RT-2
- Diffusion Policy · arXiv · project — action chunk as denoising. → architecture DiT; chunking ACT
- ACT / ALOHA · arXiv · project — action chunking + low-cost bimanual. → also a benchmark, see ④
- DiT · arXiv — transformer diffusion backbone. → the decoder in π0 and Qwen-VLA
- Flow Matching · arXiv — the training objective. → used by π0, Qwen-VLA
③ One policy, many bodies — embodiment & task unification
The §B.1–B.2 bet: pool bodies and tasks into one policy.
- Open X-Embodiment · arXiv · project — 22-robot pooled dataset + RT-X. → scaled by CrossFormer
- CrossFormer · arXiv · project — one transformer across 20 embodiments. → language-described bodies in Qwen-VLA
- ViNT · arXiv · project — cross-robot navigation foundation. → navigation benchmarks R2R/RxR
- NaVILA (Cheng et al., 2024) · arXiv — VLA for legged-robot navigation; mid-level language actions bridge to a low-level locomotion policy. → sibling unification attempt to Uni-NaVid
- Uni-NaVid (Zhang et al., 2024) · arXiv — one video-based VLA unifying four navigation sub-tasks (instruction-following, object search, QA, tracking) instead of one model per task.
- GATO · arXiv · blog — one net, 600+ tasks. → maximalist ancestor of Qwen-VLA
- RoboCat (Bousmalis et al., 2023) · arXiv · blog — self-improving generalist: few-shot adapt to a new embodiment, then generate more of its own training data. → same multi-embodiment bet as CrossFormer
④ Where we measure — benchmarks
The §B.4 proving grounds. Qwen-VLA is scored on all of these.
- LIBERO · arXiv · project — lifelong-transfer manipulation.
- SimplerEnv · arXiv · project — real-to-sim manipulation eval.
- R2R · arXiv — vision-and-language navigation. → multilingual scale-up RxR
- RxR · arXiv — multilingual, denser VLN.
- ALOHA (real-world OOD) · arXiv · project — the OOD stress test. → method side ②
⑤ The anchor
- Qwen-VLA · arXiv · on this blog — pulls the flow-matching decoder from ②, language-described embodiment from ③, and is scored on ④. The worked case of §C.
Formal cited bibliography (auto-generated)
2026
- Qwen-VLA: Unifying Vision-Language-Action Modeling across Tasks, Environments, and Robot EmbodimentsarXiv preprint arXiv:2605.30280, 2026
- Towards Generalizable Robotic Manipulation in Dynamic EnvironmentsIn European Conference on Computer Vision (ECCV), 2026DOMINO; verify fields before relying on them
2025
- FAST: Efficient Action Tokenization for Vision-Language-Action ModelsarXiv preprint arXiv:2501.09747, 2025
- RoboTwin: Dual-Arm Robot Benchmark with Generative Digital TwinsarXiv preprint, 2025Verify fields before relying on them
2024
- OpenVLA: An Open-Source Vision-Language-Action ModelIn Conference on Robot Learning (CoRL), 2024arXiv:2406.09246
- Octo: An Open-Source Generalist Robot PolicyIn Robotics: Science and Systems (RSS), 2024arXiv:2405.12213
- Vision-Language Foundation Models as Effective Robot ImitatorsIn International Conference on Learning Representations (ICLR), 2024arXiv:2311.01378
- \pi_0: A Vision-Language-Action Flow Model for General Robot ControlarXiv preprint arXiv:2410.24164, 2024Physical Intelligence technical report
- Open X-Embodiment: Robotic Learning Datasets and RT-X ModelsIn IEEE International Conference on Robotics and Automation (ICRA), 2024arXiv:2310.08864
- Scaling Cross-Embodied Learning: One Policy for Manipulation, Navigation, Locomotion and AviationIn Conference on Robot Learning (CoRL), 2024arXiv:2408.11812
- Evaluating Real-World Robot Manipulation Policies in SimulationIn Conference on Robot Learning (CoRL), 2024SimplerEnv; arXiv:2405.05941
-
-
2023
- RT-1: Robotics Transformer for Real-World Control at ScaleIn Robotics: Science and Systems (RSS), 2023arXiv:2212.06817
- RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic ControlIn Conference on Robot Learning (CoRL), 2023arXiv:2307.15818; proceedings lead author Zitkovich et al.
- PaLM-E: An Embodied Multimodal Language ModelIn International Conference on Machine Learning (ICML), 2023arXiv:2303.03378
- ViNT: A Foundation Model for Visual NavigationIn Conference on Robot Learning (CoRL), 2023arXiv:2306.14846
- Diffusion Policy: Visuomotor Policy Learning via Action DiffusionIn Robotics: Science and Systems (RSS), 2023arXiv:2303.04137
- Learning Fine-Grained Bimanual Manipulation with Low-Cost HardwareIn Robotics: Science and Systems (RSS), 2023ALOHA/ACT; arXiv:2304.13705
- Scalable Diffusion Models with TransformersIn International Conference on Computer Vision (ICCV), 2023arXiv:2212.09748
- Flow Matching for Generative ModelingIn International Conference on Learning Representations (ICLR), 2023arXiv:2210.02747
- LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot LearningIn NeurIPS Datasets and Benchmarks Track, 2023arXiv:2306.03310
- RoboCat: A Self-Improving Generalist Agent for Robotic ManipulationarXiv preprint, 2023Verify fields before relying on them
2022
- A Generalist AgentTransactions on Machine Learning Research (TMLR), 2022arXiv:2205.06175
2020
- Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal GroundingIn Empirical Methods in Natural Language Processing (EMNLP), 2020RxR; arXiv:2010.07954
2018
- Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real EnvironmentsIn IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018R2R; arXiv:1711.07280
All entries are established works with verified arXiv identifiers, including RoboTwin, DOMINO, NaVILA, Uni-NaVid, and RoboCat — added 2026-09-20 (previously named in prose but not formally cited). Informal / project-page links can be layered onto each node next (verified before adding — none fabricated).