Large language models have shown promise for autonomous vehicle motion planning by reformulating trajectory prediction as a language generation problem. However, deploying capable LLMs on resource-constrained onboard systems remains challenging due to computational requirements.

The task is to generate safe, comfortable, and goal-directed vehicle trajectories. Input consists of observations O (natural language descriptions of detected objects and their predicted positions) and ego states S (current velocity, acceleration, heading angular velocity, and historical trajectory). Output is a planned trajectory T = {(x₁, y₁), …, (x₆, y₆)} consisting of six waypoints at 0.5-second intervals over a 3-second horizon in ego-centric coordinates. The model must also generate a chain-of-thought reasoning trace including Notable Objects, Potential Effects, Meta Action, and the final Trajectory.

The objective is to minimize displacement from ground-truth trajectories while avoiding collisions. Success is measured by:

1. L2 displacement error (meters) at 1s, 2s, and 3s horizons under STP-3 and UniAD averaging conventions
2. collision rate (percentage of frames where ego vehicle overlaps with ground-truth object bounding boxes), and

3. format error rate (fraction of unparseable outputs).

   The study uses the nuScenes autonomous driving dataset as processed by GPT-Driver, containing 1,000 driving scenarios with 5,119 validation frames for evaluation.

1. Teacher model: Qwen3-8B fine-tuned with supervised fine-tuning (SFT) on GPT-Driver nuScenes dataset using qwen3_nothink chat template for deterministic structured outputs.

2. Student model: Qwen3-1.7B (5× parameter reduction) trained using on-policy Generalized Knowledge Distillation (GKD).

3. GKD training procedure: Student samples full responses ŷ ~ p_θ_S(·

   x), then trained to match teacher’s token-level distributions along on-policy trajectories using objective:

   \[L_{GKD}(θ) = E_x[E_{ŷ~p_θ_S(·|x)}[D(p_T ∥ p_θ_S)(ŷ | x)]]\]

4. Token-averaged divergence:

   \[D(p_T ∥ p_θ_S)(ŷ | x) = \frac{1}{|ŷ|} \sum_{n=1}^{|ŷ|} D(p_T(· | ŷ_{<n}, x) ∥ p_θ_S(· | ŷ_{<n}, x))\]

5. Generalized Jensen-Shannon divergence with β = 0.5:

   \[D_{JSD}^{(β)}(p_T ∥ p_S) = βD_{KL}(p_T ∥ m) + (1-β)D_{KL}(p_S ∥ m)\]

   where m = βp_T + (1-β)p_S.

6. Baseline: Dense-feedback RL using teacher log-probabilities as per-token rewards in policy gradient framework with advantage:

   \[A_n(x, ŷ) = \text{sg}[\log p_T(ŷ_n | x, ŷ_{<n}) - \log p_S(ŷ_n | x, ŷ_{<n})]\]

   The core contribution is applying on-policy distillation to address train-inference distribution mismatch in autoregressive motion planning, where coordinate errors can cascade through the trajectory sequence.

1. Teacher training: Qwen3-8B fine-tuned on GPT-Driver nuScenes training split using LLaMA-Factory with DeepSpeed ZeRO-3, learning rate 10⁻⁴, batch size 4 per device with 2 gradient accumulation steps (effective batch size 8), qwen3_nothink chat template.

2. GKD student training: Qwen3-1.7B trained using TRL GKDTrainer with learning rate 5×10⁻⁵, batch size 2 per device with 4 gradient accumulation steps (effective batch size 8), maximum 512 new tokens per rollout, β = 0.5, λ = 0.5 (50% on-policy student sequences, 50% ground-truth sequences).

3. RL baseline training: Qwen3-1.7B with dense-feedback policy gradient, learning rate 5×10⁻⁵, batch size 1 per device with 8 gradient accumulation steps (effective batch size 8), temperature 0.7 for student rollouts.

4. Hardware: Single node with 8 NVIDIA H200 GPUs for all experiments.

5. Training duration: All models trained for 5 epochs with checkpoints after each epoch. Best checkpoint selected via validation sweep: epoch 3 for teacher, epoch 3 for GKD student, epoch 1 for RL student.

   Data details: nuScenes dataset with 1,000 driving scenarios. Wall-clock time and data filtering details not reported.

Prior work: GPT-Driver (Mao et al., 2023) reformulated motion planning as language generation using GPT-3.5 on nuScenes, achieving strong imitation learning performance. Generalized Knowledge Distillation (Agarwal et al., 2024) addressed distribution mismatch in language model distillation through on-policy training. Dense-feedback RL approaches (Zhao et al., 2026) used teacher log-probabilities as per-token rewards for distillation.

Delta: This work applies GKD specifically to autonomous driving motion planning, comparing it systematically against dense-feedback RL baseline under controlled conditions. It demonstrates 5× model compression with minimal performance degradation.

Applied-specific assessment:

• Architectural idea: Standard application of existing GKD framework to new domain - not architecturally novel
• Benchmark gains: GKD achieves only 5-6% degradation vs teacher (significant for compression), but 55-41% improvement over RL baseline (meaningful margin)
• Fair comparison: Same models (Qwen3), same data, same evaluation protocol between GKD and RL baseline, but comparison to prior work limited
• Generalization: Results limited to nuScenes dataset; unclear if gains hold without specific teacher-student capacity ratio or domain

The work demonstrates solid engineering application of existing techniques rather than fundamental innovation. The systematic comparison between distillation approaches is valuable but the core methods are incremental extensions.

Verdict: INCREMENTAL — competent application of existing on-policy distillation to autonomous driving with systematic comparison, but no significant methodological advances.

1. L2 displacement error (STP-3 average): GKD 0.373m, Teacher 0.355m, RL 0.579m - GKD achieves 5% degradation from teacher, 55% improvement over RL.

2. L2 displacement error (UniAD average): GKD 0.772m, Teacher 0.730m, RL 1.092m - GKD achieves 6% degradation from teacher, 41% improvement over RL.

3. L2 error at 1s horizon: GKD 0.151m, Teacher 0.145m, RL 0.282m - GKD nearly matches teacher performance.

4. L2 error at 2s horizon: GKD 0.334m, Teacher 0.319m, RL 0.540m - error gap increases with time horizon.

5. L2 error at 3s horizon: GKD 0.634m, Teacher 0.600m, RL 0.916m - largest gap at longest horizon, showing error compounding in RL.

6. Collision rate (STP-3 average): GKD 0.138%, Teacher 0.101%, RL 0.363% - GKD maintains low collision rate, 2.6× better than RL.

7. Format error rate: GKD 0%, RL 0%, Teacher ~0.08% - both students achieve perfect parsing reliability.

   Results show consistent ordering Teacher ≥ GKD ≫ RL across all metrics. Missing comparisons to other motion planning baselines beyond RL alternative.

1. Model size: Teacher 8B parameters, Student 1.7B parameters (5× compression ratio).

2. Training compute: 8 NVIDIA H200 GPUs per experiment, 5 epochs training duration. Specific GPU hours not reported.

3. Inference speed/latency: Greedy decoding with maximum 512 new tokens. Specific inference times not reported.

4. Memory footprint: Uses DeepSpeed ZeRO-3 for teacher training, gradient accumulation for memory efficiency. Exact memory requirements not specified.

5. Deployment practicality: 5× parameter reduction makes deployment more feasible on onboard automotive hardware, but still requires significant computational resources for real-time operation. No actual deployment benchmarks or hardware-specific latency measurements provided.

1. Evaluation limited to nuScenes benchmark dataset - no real-world deployment results reported.

2. No hardware experiments on actual vehicles or automotive computing platforms.

3. No production integration or sim-to-real validation discussed.

4. Experiments use offline dataset evaluation rather than closed-loop simulation or real-world testing.

5. Missing analysis of real-time performance requirements for practical autonomous driving deployment.

1. EVALUATION: Limited to single dataset (nuScenes) without cross-dataset generalization testing.

2. EVALUATION: No closed-loop simulation or real-world validation beyond offline metrics.

3. ENGINEERING: Still requires significant computational resources despite 5× compression - unclear if sufficient for real-time onboard deployment.

4. FUNDAMENTAL: Inherits limitations of language-based trajectory representation - potential precision issues for safety-critical applications.

5. EVALUATION: Missing comparison to specialized neural motion planning baselines beyond the RL alternative.

6. ENGINEERING: Training stability issues observed in RL baseline (early stopping at epoch 1) indicate potential reliability concerns.

   Failure modes:

7. Coordinate generation errors can cascade through trajectory sequence, as shown in qualitative example where both students miss the turn despite teacher success.
8. Model may generate syntactically correct but physically implausible trajectories not caught by format parsing.

This work addresses the problem of “thinking with images” in Multimodal Large Language Models (MLLMs), where models need to actively invoke visual tools during multi-turn reasoning. Current approaches suffer from a reasoning-action gap: models may generate plausible textual reasoning while executing imprecise or irrelevant visual actions, leading to training instability and poor performance.

Input: Original image with question prompt (e.g., “What is the color of the helmet?”) Output: Final answer through multi-turn reasoning trajectory involving textual reasoning, visual actions (crop, zoom), and visual observations

The task follows a Markov Decision Process formulation where at each step t, the state st is defined as:

where (X₀, I₀) is the original image-question pair, and subsequent pairs represent reasoning-observation sequences.

Evaluation uses accuracy metrics on fine-grained visual reasoning benchmarks: V* (Avg@8), HR-Bench (Avg@8), and MME-RealWorld-Lite (Avg@1). These datasets feature ultra-high-resolution images (2K-8K) requiring precise localization of small visual details.

1. Base Architecture: Built on Ovis2.5-9B MLLM with visual tool integration for crop/zoom operations

2. Semantic Scoring Mechanism: At each reasoning step t, the model generates both an action at and explicit descriptive label yt describing expected visual content. Raw semantic score calculated as:

   \[z_t = \text{CLIP}(y_t, I_t)\]

3. Trajectory-Aware Discount: Apply temporal discount to semantic scores to prevent reward hacking and encourage efficient exploration:

   \[R_{sem} = \frac{1}{T} \sum_{t=1}^T \lambda^{T-t} z_t\]

   where λ = 0.95 and T is trajectory length.

4. Group-Based Advantage Estimation: Generate G responses per query, compute advantage as:

   \[\tilde{A}^{(i)} = \tilde{A}^{(i)}_{out} + \beta \cdot \tilde{A}^{(i)}_{sem}\]

   where both components are group-normalized (β = 0.4).

5. MAPO Objective: Optimize policy using clipped importance sampling with combined semantic-outcome rewards:

   \[J_{MAPO}(\theta) = \mathbb{E}_{q\sim D, \{\tau_i\}_{i=1}^G \sim \pi_{\theta_{old}}} \left[\frac{1}{G} \sum_{i=1}^G \frac{1}{|\tau_i|} \sum_{I=1}^{|\tau_i|} \min\left(w_I^{(i)}(\theta) e^{\tilde{A}_I^{(i)}}, \text{clip}(w_I^{(i)}(\theta), \epsilon) e^{\tilde{A}_I^{(i)}}\right)\right]\]

1. Cold-Start SFT: Initial supervised fine-tuning to equip model with tool invocation and multi-turn reasoning capabilities using open-source datasets. Specific data scale and training details not reported.

2. MAPO RL Phase: - Data: Uses open-source datasets for “thinking with images” capability - Optimizer: SGD with decaying step size (specific learning rate not reported) - Hardware: Not reported - Training details: Group size G not specified, clipping parameter ε not reported - Wall-clock time: Not reported

3. Hyperparameters: λ = 0.95 (discount factor), β = 0.4 (semantic weight balance), length constraints on generated labels to prevent reward hacking

   Most training recipe details are not reported, limiting reproducibility.

Prior work:

1. DeepEyes (2025) - pioneered agentic “thinking with images” using RL for visual tool integration
2. GRPO/GSPO (2024-2025) - group-based policy optimization methods for language model alignment

3. Standard multimodal CoT approaches that rely on outcome-based rewards without intermediate supervision

   Delta: MAPO adds explicit semantic consistency enforcement between textual reasoning and visual actions. Key innovations:

4. mandatory generation of descriptive labels alongside actions
5. CLIP-based semantic scoring of action-observation alignment
6. trajectory-aware discount factor

7. integration of semantic scores into advantage estimation.

   Applied-specific assessment:

   • Architectural novelty: Limited - combines existing CLIP scoring with established group-based RL. The “talk then walk” paradigm is intuitive but not groundbreaking.
   • Benchmark gains: Modest improvements (1-2% absolute) on most metrics, within potential noise margin for some benchmarks.
   • Fair comparisons: Uses same base model (Ovis2.5-9B) for RL method comparisons, but many implementation details missing.
   • Scale dependence: Improvements appear consistent but small, unclear if gains would hold without specific hyperparameter tuning.

   The theoretical analysis provides some justification but is relatively standard variance reduction theory applied to this setting.

   Verdict: INCREMENTAL — solid engineering contribution that combines existing techniques (CLIP scoring + group RL) for a specific problem, but lacks substantial conceptual novelty.

1. V* benchmark: MAPO achieves 89.5% vs DeepEyes 88.0% vs Mini-o3 89.3%, improvement of 0.2-1.5%
2. HR-Bench overall: MAPO 79.8% vs GRPO 77.8% vs DeepEyes 73.0%, improvement of 2.0-6.8%
3. HR-Bench attribute: MAPO 81.0% vs GRPO 78.5% vs DeepEyes 74.9%, improvement of 2.5-6.1%
4. HR-Bench relative: MAPO 78.6% vs GRPO 77.0% vs DeepEyes 71.2%, improvement of 1.6-7.4%
5. MME-RealWorld-Lite 4K: MAPO 55.8% vs GRPO 55.5% vs DeepEyes 48.4%, marginal 0.3% improvement over GRPO

6. MME-RealWorld-Lite 8K: Results mixed, some baselines not reported

   Results show consistent but modest improvements. Largest gains are on HR-Bench overall metrics. Performance on MME-RealWorld-Lite is close to baseline methods, suggesting limited advantage on some benchmark types.

1. Model size: 9B parameters (Ovis2.5-9B base model)
2. Training compute: Not reported (GPU hours, hardware details missing)
3. Inference speed: Negligible overhead claimed for CLIP semantic scoring, but specific latency numbers not provided
4. Memory footprint: Described as “negligible memory overhead” from CLIP verification, but quantitative analysis missing
5. Deployment practicality: Method requires pre-trained CLIP model alongside main MLLM, adds modest computational overhead but should be practical for deployment

1. Evaluation limited to curated benchmarks (V*, HR-Bench, MME-RealWorld-Lite) - no production deployment results reported
2. No hardware experiments on actual robotic systems or autonomous vehicles
3. No sim-to-real transfer evaluation
4. Authors claim method is extensible to web browsing agents and code interpreters, but provide no experimental validation
5. Visual tool integration limited to basic crop/zoom operations - unclear how method scales to more complex tool ecosystems

1. FUNDAMENTAL: Method requires explicit label generation which may not capture all aspects of visual reasoning intent
2. ENGINEERING: Training stability improvements shown but limited to specific hyperparameter settings (λ=0.95, β=0.4)
3. EVALUATION: Most training details not reported, limiting reproducibility and fair comparison
4. FUNDAMENTAL: CLIP-based semantic scoring may have biases or fail for complex spatial relationships

5. ENGINEERING: Method adds computational overhead and complexity compared to standard outcome-based RL

   Likely failure modes:

6. Generated descriptive labels may be gaming the CLIP scoring rather than reflecting genuine reasoning intent
7. Performance may degrade on visual tasks requiring complex spatial or temporal reasoning beyond CLIP’s capabilities

Vision-Language-Action (VLA) models that integrate Large Language Models (LLMs) achieve state-of-the-art performance in autonomous driving by leveraging world knowledge for complex scenarios. However, their massive parameter counts (7B+) create severe computational bottlenecks for deployment, consuming 31GB GPU memory and introducing high latency in real-time driving applications.

This work addresses LLM knowledge distillation for end-to-end autonomous driving. The input is multi-view camera streams (6 cameras at 640×640 resolution) with navigation commands. The output is trajectory waypoints for vehicle control. The distillation objective combines feature matching loss:

and ground truth supervision:

Success is measured on the challenging Bench2Drive closed-loop benchmark with Driving Score (DS), Success Rate (SR), and Multi-Ability scores across 5 driving skills (merging, overtaking, emergency braking, giving way, traffic signs). The benchmark contains 44 interactive scenarios, 23 weather conditions, and 12 towns with 220 evaluation routes.

1. Teacher model: ORION architecture with EVA-02-L vision encoder, QT-Former for temporal aggregation, Vicuna-v1.5 7B LLM, and VAE-based generative planner

2. Vision encoder extracts multi-view features Fm, QT-Former processes scene/perception/history queries via self-attention and cross-attention with image features

3. Student model: Replace 7B LLM with lightweight 6-layer transformer decoder (0.1B parameters)

4. Student architecture: input projection layer → learnable planning query Qplan → 6-layer transformer decoder with cross-attention → output projection to match teacher’s planning token dimension Cp

5. Distillation strategy: Freeze vision encoder and QT-Former (initialized from teacher), only train student decoder and VAE planner

6. Joint training with L1 feature distillation loss and ground truth trajectory supervision including collision, boundary, regression, and VAE losses

   The core contribution is demonstrating that a shallow transformer decoder can replace massive LLMs for driving reasoning without performance loss.

1. Pretraining: Vision encoder, QT-Former, and VAE planner initialized with pretrained ORION weights; 7B LLM replaced with randomly initialized 6-layer decoder (0.1B params)

2. Distillation training: Vision encoder and QT-Former frozen, only student decoder and VAE planner updated; 950 training clips, 50 validation clips from Bench2Drive (1K total)

3. Optimizer: AdamW with learning rate 5×10⁻⁵, weight decay 1×10⁻⁴, trained for 20 epochs at 640×640 resolution

4. Hardware: Single RTX A6000 (48GB) GPU, ~20 hours training time

5. Batch size: Not explicitly reported

6. Data: Real driving data from CARLA V2 simulator, multi-view camera streams (6 cameras), no synthetic data augmentation mentioned

Prior work:

1. VERDI (2025) - distills VLM knowledge for autonomous driving but uses complex progressive projectors and focuses on text output alignment
2. DiMA (2024) - explores VLM feature distillation via KL-divergence but limited to open-loop evaluation

3. ORION (2024) - the teacher VLA model combining vision encoder, LLM, and generative planner

   Delta: This work demonstrates that simple L1 feature distillation from LLM latents to a shallow decoder can outperform the teacher model in closed-loop evaluation, which prior distillation works haven’t shown.

   Applied-specific assessment:

   • Architectural idea: Using shallow transformer decoder to replace LLM is straightforward, but showing it can exceed teacher performance is non-obvious
   • Benchmark gains: +2.9 DS improvement over 77.7 baseline is meaningful (3.7% relative gain) and consistent across multiple metrics
   • Fair comparisons: Uses same vision encoder, same evaluation protocol, same compute budget for training
   • Scale dependency: Gains likely hold since approach doesn’t require proprietary data or massive compute scaling

   The finding that vision-only models can exceed VLA performance challenges the field’s assumption that massive LLMs are necessary for driving reasoning.

   Verdict: SIGNIFICANT — demonstrates that expensive LLM inference may be unnecessary for autonomous driving, with clear performance gains and major efficiency improvements.

1. Bench2Drive Driving Score: 80.6 vs ORION teacher 77.7 (+2.9 improvement, new SOTA)

2. Bench2Drive Success Rate: 55.5% vs ORION teacher 54.6% (+0.9% improvement)

3. Multi-Ability Mean Score: 60.5% vs ORION teacher 54.7% (+5.8% improvement across 5 driving skills)

4. Efficiency metric: 157.7 vs ORION teacher 151.5

5. Comfortness metric: 10.3 vs ORION teacher 17.4 (lower is better)

6. Open-loop L2 error: 0.79m vs ORION teacher 0.68m (slight degradation)

7. Inference latency: 267ms vs ORION teacher 806ms (3× overall speedup, 150× reasoning module speedup)

   Results are consistently strong across closed-loop metrics. The method also outperforms other SOTA methods like MindDrive (78.0 DS), UniDrive-WM (79.2 DS), and DriveTransformer-Large (63.5 DS).

1. Model size: 0.1B parameters (reasoning module) vs 7B for teacher LLM, full model size not reported

2. Training compute: ~20 hours on single RTX A6000 (48GB) GPU

3. Inference speed: 267ms total latency vs 806ms teacher (3× speedup), reasoning module 150× faster (3.5ms vs 524.6ms)

4. Memory footprint: 8GB GPU memory vs 31GB for teacher (74% reduction)

5. Deployment practicality: Highly practical - dramatic memory and latency reductions make real-time deployment feasible, while vision encoder remains the primary bottleneck

1. Closed-loop evaluation: Comprehensive testing on Bench2Drive with 220 routes across 44 interactive scenarios, 23 weather conditions, 12 towns

2. CARLA V2 simulator: Realistic physics and environmental conditions for autonomous driving evaluation

3. Interactive scenarios: Cut-ins, overtaking, detours, emergency braking, traffic sign recognition

4. No real vehicle deployment: Testing limited to simulation environment

5. Production considerations: 3× latency reduction and 74% memory savings address key deployment constraints, though vision encoder remains computational bottleneck

1. FUNDAMENTAL: Requires expensive teacher model training phase to obtain distillation targets

2. ENGINEERING: Vision encoder (EVA-02-L) remains primary computational bottleneck during inference

3. EVALUATION: Validation limited to Bench2Drive benchmark; needs verification across diverse driving datasets

4. ENGINEERING: Joint training paradigm requires careful balancing of distillation and trajectory supervision losses

5. FUNDAMENTAL: Approach inherently limited by teacher model’s capabilities - cannot exceed teacher’s knowledge boundaries

   Likely failure modes:

6. Performance degradation in scenarios not well-represented in teacher model’s training data
7. Potential overfitting to specific simulator dynamics (CARLA V2) that may not transfer to real-world conditions

Real-world context: Embodied agents must interact with complex environments through vision and actions to accomplish tasks like “rinse off something for serving soup and move it to the table.” This requires multi-turn interactions, long-horizon planning, and managing extensive contextual dependencies—capabilities that current Vision-Language Models (VLMs) struggle with despite their impressive multimodal understanding.

Task definition: Input is a natural language instruction I and egocentric RGB image observations ot at each timestep t. Output is a sequence of atomic actions at from predefined action set A. The environment evolves as st+1 = E(st, at). Success is measured by checking if final state sT satisfies goal conditions {ri}^Ngoal_i=1 where ri(sT) = 1/0.

Evaluation criteria: Success Rate (SR) for ALFWorld and EB-ALFRED benchmarks, Subgoal Success Rate (SSR) for LoTa-WAH. Tasks involve household activities like cleaning, heating, cooling objects across multiple rooms.

Benchmarks: ALFRED training split (6,374 tasks, 20k instructions), evaluated on ALFWorld, EB-ALFRED, EB-Habitat, LoTa-WAH test sets with strict generalization requirements to unseen scenes and instructions.

1. Single VLM (Qwen2.5-VL-3B) implements both scheduler and 5 specialized capabilities: Exploration Guidance (EG), Object Grounding (OG), Scene Description (SD), Action Decoding (AD), Experience Summarization (ES).

2. Scheduler generates queries to invoke capabilities: M(I, pS, c^S_i) = [(g^j_i, q^j_i)]^ni_j=1, where pS is scheduler prompt, c^S_i is maintained context.

3. Each capability processes query and optional image: M(p_g^j_i, q^j_i, o^j_i) = (a^j_i, f^j_i), producing actions a^j_i or feedback f^j_i.

4. EG predicts exploration direction using commonsense knowledge, OG performs open-vocabulary object detection, SD describes target object states, AD translates commands to atomic actions, ES summarizes execution outcomes.

5. Expert trajectories decomposed into exploration (EG→AD→OG) and manipulation (SD→AD→ES) sub-plans with template-based CoT traces.

1. Stage 1 (SFT on expert data): 640k samples from ALFRED training tasks, 2 epochs, lr=1e-5, batch size 32, using privileged simulator information (scene graphs, segmentation masks, environment messages) for capability supervision.

2. Stage 2 (DAgger-style SFT): Model generates trajectories on training tasks, corrective supervision constructed using semantic similarity matching for queries, 690k augmented samples, 1 epoch, same hyperparameters.

3. Stage 3 (Expert-Induced Policy Optimization): Novel RL algorithm optimizing J(π) = E[r(a,s)Â^π*(s,a)] with expert advantage, 25k synthetic trajectories, lr=5e-6, batch size 512, 120 iterations.

   Hardware: 4 NVIDIA H800 (80GB) GPUs. Wall-clock time not reported.

Prior work:

1. SEEA-R1 (2024): RL-based VLM training for embodied planning, achieved 36.0% on ALFWorld
2. WAP (2024): CoT-enhanced planning with behavior cloning, achieved 62.7% on EB-ALFRED

3. Various RL methods like RL4VLM, GFlowVLM achieved ~26% on ALFWorld

   Delta: This paper decomposes planning into explicit capability invocations rather than free-form CoT, uses privileged simulator information for intermediate supervision, and introduces Expert-Induced Policy Optimization.

   Applied-specific assessment:

   • Architectural idea is incremental: capability decomposition is sensible but not fundamentally novel
   • Benchmark gains are substantial: 77.6% vs 36.0% on ALFWorld, 67.0% vs 62.7% on EB-ALFRED
   • Comparisons appear fair within same evaluation protocols
   • Success likely depends on privileged simulator supervision during training
   • Cross-domain transfer results (EB-Habitat, LoTa-WAH) show significant gaps vs closed-source models

   Verdict: SIGNIFICANT — Clear performance improvements through systematic capability decomposition and multi-stage training, though architectural novelty is limited.

1. EB-ALFRED: 67.0% SR vs 62.7% WAP (previous best open-source), 67.7% Claude-3.7-Sonnet (best overall)
2. ALFWorld (visual): 77.6% SR vs 36.0% SEEA-R1 (previous best), 24.0% GPT-4o
3. ALFWorld (text): 92.1% seen, 94.0% unseen vs 92.5%/89.1% DynaMind (previous best)
4. EB-Habitat (OOD): 22.3% SR vs 22.0% RoboGPT-R1, 59.0% GPT-4o

5. LoTa-WAH (OOD): 22.1% SSR vs 10.4% LLaMA-30B, 37.4% GPT-4

   Results are strong on in-domain benchmarks but show significant gaps on out-of-domain evaluation. Performance particularly strong on ALFWorld visual tasks.

1. Model size: Qwen2.5-VL-3B parameters (significantly smaller than compared baselines using 7B+ models)
2. Training compute: 4 NVIDIA H800 (80GB) GPUs across 3 training stages, specific hours not reported
3. Inference speed/latency: Not reported
4. Memory footprint: Not reported
5. Deployment practicality: End-to-end single model approach eliminates external tool dependencies, but requires environment simulator access for optimal training supervision

1. Evaluation conducted entirely in simulation environments (AI2-THOR, Habitat, VirtualHome)
2. No real robot deployment results reported
3. No hardware experiments on actual robotic platforms
4. Limited sim-to-real discussion beyond noting domain gaps in out-of-distribution results
5. Training heavily relies on privileged simulator information (scene graphs, segmentation masks) that may not be available in real-world deployment

1. FUNDAMENTAL: Heavy dependence on privileged simulator information during training limits real-world applicability
2. FUNDAMENTAL: Capability set is fixed and manually designed, may not generalize to novel task types
3. ENGINEERING: Significant performance gaps on out-of-domain evaluation suggest limited cross-simulator generalization
4. EVALUATION: All results from simulation, no validation on real robotic systems

5. ENGINEERING: Training requires multi-stage pipeline with different supervision sources

   Failure modes: 1) Performance degrades substantially when transferred to different simulators/domains, 2) May struggle with novel tasks requiring capabilities not in the predefined set of 5.

Real-world context: Vision-Language Models (VLMs) have shown strong performance on static visual understanding tasks, but their deployment in complex 3D embodied environments remains severely limited. Existing benchmarks either use passive tasks that avoid interactive dynamics, simplified 2D environments that fail to assess depth perception, or rely on privileged state information that bypasses genuine visual processing.

Task definition: The paper introduces PokeGym, a visually-driven long-horizon benchmark set in Pokémon Legends: Z-A, a complex 3D open-world RPG. Agents receive only raw RGB observations (current frame, optionally previous frame, left/right peripheral views) and must complete tasks through keyboard/mouse actions. Tasks span 30-220 environment steps across three categories: navigation (moving to locations), interaction (object manipulation), and mixed scenarios. Each task has three instruction granularities: Visual-Guided (procedural steps with visual anchors), Step-Guided (procedural steps without visual anchors), and Goal-Only (ultimate objective only).

Evaluation criteria: Success is measured by task completion rate (percentage of episodes completing within step budget) and average environment steps for successful episodes. Task success is verified automatically via Array of Bytes (AOB) memory scanning to extract game state variables like coordinates and quest flags.

The benchmark contains 30 tasks derived from 10 quests, with step budgets ranging from 180-360 environment steps based on human demonstrations.

1. Environment: Pokémon Legends: Z-A game running on Ryujinx C# emulator with GPU texture extraction for RGB observations
2. Observation Interface: Raw RGB frames (1920x1080) extracted directly from GPU textures, optionally including previous frame and left/right peripheral views
3. VLM Decision Module: Standard VLMs (GPT-5.2, Gemini-3-Pro, Claude-Sonnet-4.6, Qwen series, GLM-4.6V) process visual inputs and generate action decisions
4. Optional Self-Reflection: Every k=5 steps, VLM analyzes recent history and updates short-term memory and experience library through ADD/DEL/MOD/KEEP operations
5. Action Interface: Two paradigms - (a) Defined high-level actions (MoveForward, RotateRight) with fixed durations, or (b) Parametric control with continuous joystick values X,Y ∈ [-1.0, 1.0]
6. Evaluation Interface: Independent AOB memory scanning extracts game state (map ID, coordinates, quest flags) for automated success verification without exposing privileged information to agent

7. Auxiliary Design: Adaptive pause mechanism during VLM inference for time-sensitive combat scenarios

   The core technical contribution is enforcing strict code-level isolation between agent (vision-only) and evaluator (memory scanning) to enable automated assessment without state leakage.

No model training is performed - this is purely an evaluation benchmark.

1. Evaluation Setup: Pre-trained VLMs evaluated directly without fine-tuning
2. Data: 30 tasks with pre-configured save files as initial states, 5 trials per task per model
3. Hardware: Not reported for evaluation infrastructure
4. Inference: Models generate actions autoregressively based on visual context and interaction history
5. Budget: Fixed environment step budgets (180-360 steps) per task based on human demonstrations

Step 1 - Prior work:

• MineDojo (2022): 3D embodied agents in Minecraft with privileged state access and symbolic representations
• Cradle (2024): VLM agents in AAA games but requires expensive human evaluation
• Lumine (2024): Similar game-based evaluation but relies on human assessment for scalability

Step 2 - Delta: PokeGym adds automated evaluation via memory scanning while maintaining pure visual input, resolving the trade-off between environmental realism and scalable assessment. It introduces systematic instruction granularity probes and detailed failure mode analysis.

Step 3 - Applied-specific assessment:

• Architectural idea: The combination of pure-pixel input with automated memory-based evaluation is a solid engineering contribution, not architecturally novel
• Benchmark gains: The paper focuses on diagnostic analysis rather than model improvements; shows meaningful performance differences across instruction granularities
• Fair comparisons: All models evaluated under identical conditions with same budgets and initial states
• Scalability: The approach requires game emulation and memory scanning setup but is more scalable than human evaluation

The diagnostic framework revealing deadlock types (Unaware vs Aware) and correlation with task failure (r=-0.57 to -0.65) provides actionable insights, but the core contribution is primarily a well-engineered benchmark rather than a fundamental advance.

Verdict: INCREMENTAL — Well-executed benchmark engineering that resolves practical evaluation challenges, but represents expected extension of existing game-based evaluation rather than breakthrough methodology.

1. Navigation Tasks: Best model Qwen3.5-122B (Visual-Guided) achieves 60.00% success rate, Gemini-3-Pro (Step-Guided) 70.00%
2. Interaction Tasks: GPT-5.2 and Gemini-3-Pro achieve 100.00% success rate in Goal-Only setting, GPT-5.2 93.33% in Visual-Guided
3. Mixed Tasks: GPT-5.2 reaches 60.00% success rate in Visual-Guided, but performance drops significantly in Goal-Only (6.67% for Claude-Sonnet-4.6)
4. Overall Performance: Gemini-3-Pro leads in Step-Guided (74.44% average), GPT-5.2 strongest in Visual-Guided (59.44%)
5. Physical Metrics: Strong negative correlation between success rate and Ineffective Moves (r=-0.57 to -0.65, p<0.001)

6. Failure Analysis: Execution Failure is universal bottleneck across all models, Unaware Deadlocks dominate weaker models, Aware Deadlocks more common in stronger models

   Results show mixed performance with clear capability gaps between proprietary and open-weight models, and systematic degradation from Visual-Guided to Goal-Only instruction granularities.

1. Model sizes: Range from GLM-4.6V to Qwen3.5-122B parameters, exact parameter counts not specified for all models
2. Training compute: Not applicable - evaluation-only benchmark
3. Inference speed: Not reported, but adaptive pause mechanism implemented to normalize VLM inference latency differences
4. Memory footprint: Game emulator + VLM inference requirements not quantified
5. Deployment practicality: Requires legal ROM acquisition, emulator setup, and memory scanning infrastructure - moderately complex deployment but more scalable than human evaluation

1. Game Environment: Complex 3D open-world RPG with realistic lighting, occlusion, and viewpoint changes that mirrors real-world visual challenges
2. Visual Complexity: Dense, cluttered scenes with multiple depth layers, dynamic elements, and UI overlays similar to real environments
3. Skill Transfer: Navigation, object interaction, and spatial reasoning skills tested are directly relevant to real-world robotics
4. Limitations: Game-based evaluation may not capture all real-world physics and interaction dynamics
5. Sim-to-Real Gap: Paper acknowledges this as a limitation but argues game mechanics provide valuable embodied reasoning testbed
6. No Hardware Results: Pure simulation-based evaluation without physical robot deployment

1. Game-specific mechanics - FUNDAMENTAL: Evaluation tied to specific game rules and physics that may not generalize
2. Legal ROM requirements - ENGINEERING: Requires users to obtain game copies legally, limiting accessibility
3. Memory scanning brittleness - ENGINEERING: AOB signatures may break across game versions or updates
4. Limited action space - FUNDAMENTAL: Discrete high-level actions may not capture continuous control nuances
5. Emulator dependency - ENGINEERING: Relies on specific emulator implementation and performance

6. Task scope - EVALUATION: 30 tasks may not comprehensively cover all embodied capabilities

   Failure modes:

7. Physical deadlocks: Agents get trapped in collision states with high correlation to task failure
8. Metacognitive breakdown: Weaker models suffer “Unaware Deadlocks” (oblivious to being stuck) while stronger models have “Aware Deadlocks” (recognize problem but can’t recover)