Long video understanding is crucial for applications like surveillance, content analysis, and video question answering, but current Multimodal Large Language Models (MLLMs) face severe bottlenecks when processing hour-long videos. The dense visual streams quickly saturate token budgets (e.g., 8K-12K tokens) and cause the lost-in-the-middle phenomenon where critical evidence gets buried in extensive contexts.

The task takes as input a long video V (up to hour-long, 4101s in extreme cases) and a user query Q, then generates a textual response. Videos are uniformly partitioned into N temporal segments S = {S1, …, SN}, with each segment containing 4-8 frames sampled at 2 FPS. The objective is to compress each segment Si into compact video memory tokens while respecting a global token budget Bmax, formalized as:

Success is measured on benchmarks including LongVideoBench (473s avg), MLVU (651s avg), Video-MME (1010s avg), and the extreme-long LVBench (4101s avg), using accuracy metrics for video question answering tasks.

1. Small Vision-Language Model (SVLM) Local Compressor: Uses Qwen3-VL-2B-Instruct as base, processes each video segment Si with visual tokens Xi, user query Q, and learnable memory tokens M under causal attention to generate compressed representations Hi ∈ R^(kmax×ds) with kmax = 128 tokens.

2. Query-Conditioned Cross-Modal Distillation: The SVLM constructs causal sequences with system prompt, visual tokens Xi, user query Q, and memory tokens M (placed last so they attend to all preceding context), performing early cross-modal semantic distillation.

3. Zero-Shot Relevance Prior Extraction: During the same forward pass, intercepts hidden state h_rel_i to compute relevance score using logit difference: si = σ((w_yes - w_no)^T h_rel_i) between “Yes”/”No” vocabulary tokens.

4. Adaptive Token Allocation (ATA): Training-free O(1) dynamic router that allocates per-segment budgets ki ∈ [kmin, kmax] based on normalized relevance scores, using contrastive linear allocation and capacity-aware protection.

5. Global LLM Decoder: Qwen3-LM-4B processes compressed memory tokens with explicit temporal tags (e.g., <t=2.0s>) via standard auto-regressive generation.

   The core contribution is casting visual token reduction as query-aware cross-modal distillation rather than query-agnostic compression.

1. Stage 0 - Modality Alignment: Freeze SVLM and LLM, train only linear projector on LCS-558K dataset, standard supervised learning setup, specific optimizer/hardware not reported.

2. Stage 1 - Pre-training: Unfreeze entire architecture, train on ~2M images + ~1.38M videos + ~143K text samples, videos sparsely sampled at 8 frames, specific training details not reported.

3. Stage 2 - Broad Supervised Fine-Tuning: Train on ~0.93M images + ~2.25M videos + ~71K text samples, maximum 128 frames per video, follows VideoChat-Flash and LLaVA-OneVision data mixtures.

4. Stage 3 - Long-Context SFT: Freeze SVLM, fine-tune only global LLM on ~384K high-quality samples from Stage 2, extend maximum frame limit to 384 frames for context extrapolation.

   Training uses 64-GPU cluster with FSDP, all datasets are publicly accessible. Wall-clock time, specific learning rates, and detailed hyperparameters not reported.

Prior work:

1. VideoChat-Flash
2. applies hierarchical token compression and visual redundancy reduction but remains query-agnostic.
3. LongVU
4. introduces query-aware spatiotemporal compression but relies on disjoint auxiliary modules decoupled from end-to-end training.
5. LongVA

6. extends context windows algorithmically but requires processing dense visual streams with prohibitive computational costs.

   Delta: This paper introduces query-aware video compression as an early cross-modal distillation process, using an SVLM to simultaneously extract relevance scores and compressed representations in a single forward pass. The ATA mechanism exploits zero-shot relevance priors without auxiliary supervision.

   Applied-specific assessment:

   • Architectural idea: The unified SVLM+LLM architecture with single-pass relevance extraction is novel, moving beyond separate routing modules
   • Benchmark gains: Substantial improvements on extreme-long videos (52.3 vs 30.8 for GPT-4o on LVBench), but more modest on standard benchmarks
   • Fair comparisons: Uses much smaller model (6B vs 7-8B) with extreme compression (2.9-4.3 avg tokens/frame vs 16-91 for others)
   • Generalizability: Strong results across diverse benchmarks suggest robustness, though relies on specific base model priors

   Verdict: SIGNIFICANT — The single-pass query-aware compression with zero-shot relevance routing represents a clear architectural advance for long video understanding that most engineers should read.

1. LongVideoBench (473s avg): Tempo 65.1% vs VideoChat-Flash 64.7% (previous best specialized model), modest 0.4% improvement
2. MLVU (651s avg): Tempo 75.6% vs VideoChat-Flash 74.7%, 0.9% improvement
3. Video-MME Overall (1010s): Tempo 67.8% vs VideoChat-Flash 65.3%, 2.5% improvement
4. Video-MME Long (2386s): Tempo 57.8% vs VideoChat-Flash 55.4%, 2.4% improvement
5. LVBench (4101s extreme-long): Tempo 52.3% vs GPT-4o 30.8% and Gemini 1.5 Pro 33.1%, massive 19-22% improvements over proprietary baselines

6. LVBench scaling: Tempo reaches 53.7% with 2048 frames and 12K budget

   Results show modest improvements on standard benchmarks but substantial gains on extreme-long videos, validating the approach for hour-long content.

1. Model size: 6B parameters (2B SVLM + 4B LLM)
2. Training compute: 64-GPU cluster with FSDP, specific GPU hours not reported
3. Inference speed: Single forward pass design, O(1) head truncation, no autoregressive decoding overhead for routing
4. Memory footprint: Extremely efficient with 0.5-16 tokens/frame dynamic range, actual usage 2.9-4.3 tokens/frame on average, strict budget enforcement (4K-12K total visual tokens)
5. Deployment practicality: Highly practical due to predictable memory footprint, compact model size, and training-free inference strategy, suitable for resource-constrained deployment

1. Evaluation on diverse real-world video content: Benchmarks include natural videos from LongVideoBench, MLVU spanning different domains and temporal scales
2. No reported hardware experiments or robot deployment
3. No production integration details provided
4. Sim-to-real gap not discussed as this focuses on video understanding rather than embodied AI
5. Uses publicly accessible datasets ensuring reproducibility, but no mention of deployment in production systems or edge devices

1. FUNDAMENTAL: Relies on base model’s zero-shot relevance prior - effectiveness tied to quality of foundation model pretraining
2. FUNDAMENTAL: Fixed segment-based processing may miss cross-segment temporal dependencies in complex narratives
3. ENGINEERING: Progressive training curriculum requires careful stage design and may not transfer to other base models
4. ENGINEERING: Head truncation assumes semantic front-loading which may not hold for all video types or base architectures

5. EVALUATION: Limited analysis on failure cases where zero-shot relevance scoring produces poor segment rankings

   Failure modes:

6. Queries requiring global temporal reasoning across many segments may suffer from aggressive local compression
7. Videos with uniformly relevant content may struggle under strict budget constraints leading to information loss.

Visual spatial reasoning faces a critical faithfulness problem. While multimodal language models (MRMs) trained with reinforcement learning achieve higher accuracy on spatial reasoning benchmarks, they often generate Chain-of-Thought (CoT) reasoning that contradicts the final answer or contains visually ungrounded claims about objects and spatial relationships.

The task is multimodal spatial reasoning where models receive an image-question pair and must generate a structured response with reasoning trace and final answer: Input: Image I and question Q (typically multiple choice) Output: CoT trace T within tags and answer A within tags Format: reasoning stepsanswer_choice

Success is measured along three axes:

1. Answer accuracy: exact match with ground truth after stripping punctuation
2. Logical consistency: does the reasoning trace logically entail the final answer?

3. Visual grounding: do reasoning steps accurately describe visible objects, attributes, and spatial relationships?

   The paper evaluates on seven spatial reasoning benchmarks: CVBench-2D, CVBench-3D, MindCube, MMVP, OmniSpatial, RealWorldQA, and SAT-Real, spanning both in-distribution and out-of-distribution settings.

The method builds on Qwen2.5-VL backbones (7B and 3B) with a two-stage training pipeline:

1. Supervised Fine-tuning: Train on 45K CoT traces generated via MCTS from Qwen2.5-VL-72B teacher, covering SAT, VGR, and VisCoT datasets. Uses bounding-box grounded reasoning format with [x1,y1,x2,y2] tags.

2. Faithful GRPO (FGRPO): Core contribution - constrained policy optimization treating consistency and grounding as hard constraints via Lagrangian dual ascent:

   \[\max_\theta \mathbb{E}_{x,o \sim \pi_\theta}[R_{task}(o)]\]

   subject to:

   \[\mathbb{E}[R_C(o)] \geq \tau_C\] \[\mathbb{E}[R_S(o)] \geq \tau_S\] \[\mathbb{E}[R_G(o)] \geq \tau_G\]
3. Reward Formulation:
   • Task reward:
   \[R_{task}(o) = 0.5 \cdot R_{acc}(o) + 0.5 \cdot R_{fmt}(o)\]
   • Consistency reward:
   \[R_C(o) \in \{0,1\}\]

   via LLM judge

   • Semantic grounding:
   \[R_S(o) = \frac{1}{K'} \sum_{k=1}^{K'} \hat{s}_k\]

   via VLM judge per sentence

   • Spatial grounding:
   \[R_G(o) = \frac{1}{\max(N,M)} \sum_{i=1}^N \text{CIoU}(\hat{b}_i, b_{\sigma(i)})\]

   via Hungarian matching

4. Lagrangian Implementation: Converts to unconstrained problem with adaptive multipliers updated via dual ascent:

   \[\lambda_k \leftarrow \text{clip}(0, \lambda_{max}, \lambda_k + \eta_\lambda \cdot (\tau_k - \bar{c}_k))\]
5. Decoupled Advantage: Key technical contribution - independently normalize each reward signal before combining to prevent signal cancellation in group normalization.

1. SFT Stage: - Data: 45K CoT traces from MCTS on 6K seed samples (SAT, VGR, VisCoT) - Optimizer: AdamW, lr=1e-6, weight decay=0.01, cosine schedule, warmup ratio=0.03 - Hardware: 4× A100 80GB, DeepSpeed ZeRO Stage 3, ~12 hours - Format: Bounding-box grounded reasoning with tags

2. RL Stage: - Data: 49K samples (36K from filtered seed datasets + 13K TreeVGR-RL-37K) - Filtering: Difficulty-based selection (intermediate difficulty, exclude trivially easy/hard) - Optimizer: GRPO with AdamW, lr=1e-6, G=5 rollouts per prompt - Batch: Rollout batch size=128, KL coefficient=0.01, clip ratio=0.28 - Hardware: 8× H100 GPUs, vLLM for rollout generation - Duration: not reported

3. FGRPO Specifics: - Constraint thresholds: τ_C=0.95, τ_G=0.7, τ_S=0.95 - Dual learning rate: η_λ=0.05, λ_max=5.0 - Judge models: Qwen3-VL-30B-A3B-Instruct (training), GPT-5.4 (evaluation) - Masking: Consistency/semantic rewards only on correct answers, spatial only on VGR/TreeVGR

Prior Work:

1. ViGoRL-Spatial (2025): Uses MCTS-generated point-grounded CoT traces with coordinate supervision
2. TreeVGR (2025): Supervises localization and reasoning with dual IoU-based rewards

3. Vision-R1 (2025): Progressive thinking suppression during RL training

   Delta: This paper adds two specific elements:

4. Constraint formulation: Treats consistency and grounding as hard constraints via Lagrangian dual ascent, rather than soft reward terms

5. Decoupled advantage normalization: Independently normalizes each reward signal to prevent cancellation in GRPO’s group normalization

   Applied-specific assessment:

   • Architecture novelty: The constrained optimization approach is well-established (CPO, RCPO). Applying Lagrangian methods to multimodal RL is incremental but sensible.
   • Benchmark gains: Modest improvements (+2% accuracy, 26.1%→1.7% inconsistency rate). The consistency improvement is substantial, accuracy gains are moderate.
   • Fair comparisons: Uses same backbone (Qwen2.5-VL) and similar training data as baselines. Evaluation uses independent judge (GPT-5.4) rather than training judge.
   • Generalizability: The dual ascent approach requires careful threshold tuning (τ_C=0.95, τ_G=0.7, τ_S=0.95). Results may be sensitive to these hyperparameters.

   The core insight—that RLVR sacrifices reasoning quality for accuracy—is valuable. However, the solution (constrained optimization) is a straightforward application of existing techniques. The decoupled normalization addresses a real technical issue but is not fundamentally novel.

   Verdict: INCREMENTAL — Solid engineering contribution that addresses a real problem, but applies well-known constrained optimization techniques without significant methodological innovation.

1. CVBench-2D: 82.38% (FGRPO) vs 79.97% (GRPO-T), +2.41pp improvement
2. CVBench-3D: 87.04% vs 85.92%, +1.12pp
3. MindCube: 49.28% vs 41.71%, +7.57pp (largest gain)
4. MMVP: 73.33% vs 74.00%, -0.67pp (slight decrease)
5. OmniSpatial: 44.78% vs 40.90%, +3.88pp
6. RealWorldQA: 67.64% vs 66.67%, +0.97pp

7. SAT-Real: 65.66% vs 67.00%, -1.34pp (slight decrease)

   Average: 67.16% vs 65.17%, +1.99pp improvement

   Reasoning Quality Metrics:

   • Inconsistency Rate: 1.7% (FGRPO) vs 26.1% (GRPO-T), 24.4pp reduction
   • Semantic Grounding: 86.0% vs 72.7%, +13.3pp improvement

   3B Scale Results: Similar pattern with 62.39% vs 61.33% average accuracy (+1.06pp)

   Mixed Results: While average accuracy improves, FGRPO shows slight decreases on MMVP and SAT-Real. The gains are concentrated on MindCube and OmniSpatial, suggesting the method works best on datasets requiring multi-step spatial reasoning. The massive improvement in consistency (26.1%→1.7%) is the most compelling result.

1. Model size: Qwen2.5-VL-7B (7 billion parameters) and Qwen2.5-VL-3B (3 billion parameters)

2. Training compute: - SFT: 4× A100 80GB GPUs, ~12 hours per variant - RL: 8× H100 GPUs, duration not reported - Additional: Online VLM judge (Qwen3-VL-30B) for reward computation during training

3. Inference speed/latency: Not reported, but requires online judge calls for constraint evaluation during training (significant overhead)

4. Memory footprint: Uses bf16 precision, DeepSpeed ZeRO Stage 3 for SFT, vLLM for rollout generation

5. Deployment practicality: - Training requires substantial compute overhead due to online judge calls - Inference matches standard Qwen2.5-VL since no architectural changes - Constraint evaluation (consistency/grounding scoring) only needed during training - Method adds complexity to training pipeline but maintains inference efficiency

1. Evaluation scope: Tested only on curated benchmarks (CVBench, MindCube, MMVP, etc.), no real-world deployment results reported

2. Data diversity: Training uses images from COCO, GQA, OpenImages, Flickr30k covering diverse real-world scenarios

3. Domain transfer: Shows out-of-distribution performance on RealWorldQA (real photos) and SAT-Real datasets

4. Production considerations: - No sim-to-real discussion - No hardware deployment experiments (robotics, autonomous vehicles)
   - No analysis of computational constraints in practical settings - Judge dependency during training limits practical adoption

5. Generalization: Method tested on two backbone sizes (3B, 7B) showing consistent improvements, suggesting scale-invariant benefits

   The work remains primarily academic with evaluation limited to standard benchmarks. While the datasets include real-world images, there’s no evidence of deployment in actual applications requiring spatial reasoning like robotics or autonomous navigation.

1. Judge dependency (ENGINEERING): Requires online VLM judge calls during training, adding computational overhead and potential bottlenecks

2. Hyperparameter sensitivity (FUNDAMENTAL): Success depends on carefully tuned constraint thresholds (τ_C=0.95, τ_G=0.7, τ_S=0.95) and dual learning rates

3. Dataset bias (EVALUATION): Spatial grounding constraint only applicable to VGR/TreeVGR samples with bounding box annotations, limiting constraint coverage

4. Mixed accuracy results (EVALUATION): Shows slight decreases on MMVP and SAT-Real, suggesting method doesn’t universally improve performance

5. Constraint masking (FUNDAMENTAL): Consistency and semantic grounding rewards masked to correct predictions only, potentially allowing poor reasoning on incorrect samples

6. Judge reliability (ENGINEERING): Training and evaluation judges may have different biases (Qwen3-VL vs GPT-5.4)

   Failure Modes:

7. Constraint satisfaction without accuracy: Model could learn to satisfy consistency/grounding constraints while sacrificing task performance
8. Judge gaming: Model might learn to generate reasoning that fools the judge rather than being genuinely faithful to visual content

The web presents billions of interactive interfaces that require multi-step navigation for tasks like booking flights, shopping, or accessing services. These tasks demand sustained attention, domain knowledge, and patience from users. Web agents—autonomous systems that navigate and execute web tasks—could transform digital interaction, especially for users with disabilities or limited digital skills.

The task is instruction-conditioned web browsing: given a natural language task instruction (e.g., “find cheapest flights Seattle to Tokyo”), a screenshot of the current webpage, and action history, predict the next browser action to complete the task. The agent operates purely from visual screenshots without HTML/DOM access. Actions include mouse clicks at normalized coordinates [0,100], typing text, scrolling, navigation, and task completion signals.

Success is measured on live website benchmarks: WebVoyager (multi-step web navigation), Online-Mind2Web (web QA and interaction), DeepShop (e-commerce tasks), and WebTailBench (browser automation). Evaluation uses VLM judges (GPT-4o/o1-mini) to assess task completion from final screenshots and trajectories.

The paper introduces MolmoWebMix: 130K+ task trajectories (100K synthetic from AxTree agents, 30K+ human demonstrations), 116K atomic skill segments, and 10.5M GUI perception examples (screenshot QA, grounding) across 2.6K domains.

1. Built on Molmo2 vision-language model with Qwen3 language model and SigLIP2 vision encoder
2. Input: task instruction, current screenshot, action history from last 10 steps, current URL/title
3. Output: structured JSON with natural language thought + browser action
4. Action space: 13 primitives including mouse_click(x,y), keyboard_type(text), scroll(dx,dy), goto(url), send_msg_to_user(msg)
5. Coordinates normalized to [0,100] with 2 decimal precision, converted to viewport pixels at execution
6. Training data mixture combining four sources: - Synthetic trajectories from AxTree agents (Gemini-3-Flash) and multi-agent pipeline - Human demonstrations via Chrome extension capture - Node traversal trajectories from deterministic website graph exploration
   - GUI perception data: 7M+ grounding pairs, 2.2M screenshot QA examples
7. Single-stage supervised fine-tuning on mixed data, no distillation from visual web agents
8. Models available in 4B and 8B parameter sizes

1. Data preparation: Mix trajectories (80%), atomic skills, and GUI perception (20%) with optimized ratios from ablation studies
2. Base model: Molmo2 checkpoint pretrained on image captioning and single-image QA
3. Training: Supervised fine-tuning with 64 H100 GPUs, global batch size 128, up to 50K steps (≈3.2 epochs)
4. Optimizer details: not reported beyond following Molmo2 best practices
5. Learning rate and schedule: not reported
6. Wall-clock time: not reported
7. Data filtering: WebVoyager LLM judge removes failed synthetic trajectories; human trajectories manually reviewed
8. All parameters tuned: language model, vision encoder, and adapter layers

Prior Work:

• Mind2Web (2024): First large-scale web agent benchmark with 2.4K trajectories, DOM-based agents
• Fara-7B (2024): 145K trajectory dataset, achieved 73.5% WebVoyager with distillation from visual agents
• UI-TARS (2024): Vision-language models for GUI interaction, 66.4% WebVoyager

Delta: This paper adds:

1. fully open 130K+ trajectory dataset with diverse generation pipelines
2. GUI perception data integration
3. vision-only agents without DOM/AxTree access
4. multi-agent synthetic generation

5. atomic skill decomposition.

   Assessment:

   • Architectural novelty: LOW - standard VLM fine-tuning on web interaction data
   • Benchmark gains: MODERATE - 4-5pt gains over open models, matches/exceeds some closed SoM agents
   • Fair comparisons: QUESTIONABLE - compares vision-only model to DOM-based agents, different input modalities
   • Scale dependence: HIGH - requires large synthetic data generation with frontier LLMs

   The main contribution is data engineering and thorough evaluation rather than architectural innovation. While the vision-only approach is principled, the performance gains largely come from data scale/quality rather than novel techniques.

   Verdict: INCREMENTAL — Solid data and engineering contribution with moderate gains, but primarily applies known VLM fine-tuning to web agents without fundamental innovation.

1. WebVoyager: MolmoWeb-8B 78.2%, previous open SOTA (Fara-7B) 73.5%, improvement +4.7pt
2. Online-Mind2Web: MolmoWeb-8B 35.3%, Fara-7B 34.1%, improvement +1.2pt
3. DeepShop: MolmoWeb-8B 42.3%, Fara-7B 26.2%, improvement +16.1pt
4. WebTailBench: MolmoWeb-8B 49.5%, Fara-7B 38.4%, improvement +11.1pt
5. Grounding benchmarks: ScreenSpot 87.2% (MolmoWeb-4B), ScreenSpot v2 89.5%
6. Pass@4 scaling: WebVoyager 94.7% vs 78.2% pass@1, Online-Mind2Web 60.5% vs 35.3% pass@1
7. Outperforms SoM GPT-4o on WebVoyager (78.2% vs 65.1%) and DeepShop (42.3% vs 16.0%)
8. Mixed results: strong on DeepShop/WebTailBench, modest on Online-Mind2Web, competitive on WebVoyager

1. Model size: 4B and 8B parameters (based on Molmo2 architecture)
2. Training compute: 64 H100 GPUs for up to 50K steps (wall-clock time not reported)
3. Inference speed/latency: Not reported, but notes browser action execution as bottleneck
4. Memory footprint: Not reported, likely standard for 4B/8B VLMs
5. Deployment practicality: Reasonably efficient for deployment, uses normalized coordinates avoiding DOM parsing, supports browser environments like Browserbase for scaling

1. Evaluated on live websites across WebVoyager, Online-Mind2Web, DeepShop, WebTailBench benchmarks
2. Uses Browserbase environment for large-scale parallel browser sessions with captcha-solving
3. Handles real e-commerce sites, news websites, travel booking, and general web browsing
4. Human demonstrations collected on real websites via Chrome extension
5. No simulation-to-real gap since training and evaluation both use real web interfaces
6. Public demo deployed for broad user testing with safety guardrails
7. Works across 2.6K domains in training data covering popular websites

1. FUNDAMENTAL: Vision-only approach struggles with fine-grained text reading and OCR of small text
2. ENGINEERING: Gets stuck repeating same actions (clicking same location, endless scrolling) without recovery
3. FUNDAMENTAL: Requires specific instructions mentioning website names/URLs for best performance
4. ENGINEERING: Inconsistent thought-action correlation, thoughts don’t always match predicted actions
5. EVALUATION: Human vs synthetic data shows limited benchmark gains despite collection effort
6. ENGINEERING: Struggles with infrequent actions like drag-and-drop, hover, element-specific scrolling

7. FUNDAMENTAL: Cannot handle ambiguous instructions or complex multi-constraint searches effectively

   Failure modes:

   • Action loops: Repeatedly clicking same location when element is not clickable
   • OCR failures: Missing or misreading small text, especially in complex layouts

Real-world context: Current agentic multimodal models exhibit “blind tool invocation” - reflexively calling external tools (code execution, web search, image operations) even when queries can be resolved from internal knowledge. This creates severe latency bottlenecks and injects noise that degrades reasoning. The problem is that models struggle with meta-cognitive arbitration: deciding when to use internal knowledge versus external utilities.

Task definition: Given multimodal prompts (image + text query), the model must generate multi-turn responses by interleaving chain-of-thought reasoning with selective tool invocations. The policy $\pi_\theta$ generates sequences ${y_1, y_2, …, y_G}$ where each response $y_i$ contains $T_i$ tool interactions before yielding a final answer. The objective combines accuracy and efficiency:

Evaluation criteria: Success measured by (1) task accuracy using LLM judges for correctness, (2) tool usage frequency (percentage of queries requiring tools), and (3) reasoning quality across multiple domains.

The paper evaluates on perception benchmarks (V*Bench, HRBench-4K/8K), document understanding (CharXiv), and mathematical reasoning (MathVista, WeMath, DynaMath).

1. Base model: Qwen3-VL-8B-Instruct multimodal language model with external tool environment supporting code execution, text search, and image search.

2. Hierarchical Decoupled Policy Optimization (HDPO): Replace scalarized reward $R^{\text{mix}}_i = R^{\text{acc}}_i + \alpha \cdot R^{\text{tool}}_i$ with two orthogonal channels.

3. Accuracy channel: Standard reward $R^{\text{acc}}_i = \lambda_a \cdot R^{\text{ans}}_i + \lambda_f \cdot R^{\text{fmt}}_i$ with GRPO advantage estimation over all rollouts:

   \[A^{\text{acc}}_i = \frac{R^{\text{acc}}_i - \text{mean}(\{R^{\text{acc}}_1, ..., R^{\text{acc}}_G\})}{\text{std}(\{R^{\text{acc}}_1, ..., R^{\text{acc}}_G\}) + \epsilon}\]

4. Efficiency channel: Conditional tool reward $R^{\text{tool}}_i = \frac{1}{T_i + 1}$ if correct, 0 otherwise. Advantage computed only over correct rollouts in qualifying set $Q = {j

   R^{\text{ans}}_j > 0}$:

   \[A^{\text{tool}}_i = \frac{R^{\text{tool}}_i - \text{mean}(\{R^{\text{tool}}_k\}_{k \in Q})}{\text{std}(\{R^{\text{tool}}_k\}_{k \in Q}) + \epsilon}\]

5. Joint loss: $L_{\text{HDPO}}(\theta) = w_{\text{acc}} \cdot L_{\text{GRPO}}(A^{\text{acc}}) + w_{\text{tool}} \cdot L_{\text{GRPO}}(A^{\text{tool}})$

   Core contribution: Eliminates gradient entanglement from coupled rewards by maintaining orthogonal optimization channels with conditional advantage estimation.

1. Data curation stage: Rigorously filter existing tool-augmented datasets (DeepEyesV2, V-Interaction, Thyme) by (i) executing all code to remove hallucinated environmental dynamics, (ii) filtering samples solvable by base model with pass@8=1 to isolate genuine tool necessity, (iii) multi-dimensional quality filtering using Gemini-3.1-Pro judge.

2. Supervised Fine-Tuning (SFT): Train for 2 epochs on curated data using AdamW optimizer, cosine learning rate decay, peak LR 1×10^-5, global batch size 128. Data source not fully specified but includes OpenMMReasoner for tool-free reasoning preservation.

3. Reinforcement Learning (RL): HDPO optimization with batch size 128, G=16 rollouts per prompt, learning rate 1×10^-6, KL penalty coefficient 0 for extensive exploration. Loss weights $w_{\text{acc}}=1.0$, $w_{\text{tool}}=0.15$. Maximum response length 16,384 tokens.

4. Hardware: 8 NVIDIA Blackwell B200 GPUs. Wall-clock time not reported.

5. RL training set: ~5K high-quality prompts with pass@8 ∈ (0,1) variance requirement, covering perception (45%), search (36%), mathematical reasoning (19%).

Prior work:

1. GRPO/PPO-style RL for LLMs (Shao et al. 2024): Standard reinforcement learning with scalarized rewards for task performance.
2. Agentic multimodal models (DeepEyes 2024, Thyme 2024): Tool-augmented MLLMs that interleave reasoning with external utilities but suffer from blind tool invocation.

3. Tool-use optimization (various 2024-2025): Existing methods penalize tool usage via scalarized reward $R = R_{\text{acc}} + \alpha \cdot R_{\text{tool}}$.

   Delta: This paper identifies mathematical pathology in coupled rewards: shared advantage normalization entangles objectives via covariance terms $\text{Var}(R^{\text{mix}}) = \sigma^2_{\text{acc}} + \alpha^2\sigma^2_{\text{tool}} + 2\alpha \text{Cov}(R^{\text{acc}}, R^{\text{tool}})$. Proposes orthogonal optimization channels with conditional advantage estimation exclusively over correct trajectories.

   Applied-specific assessment:

   • Architectural novelty: The conditional advantage mechanism is a clever technical contribution, though builds on established RL foundations.
   • Benchmark gains: Substantial improvements (e.g., 98%→2% tool usage while improving accuracy) are impressive and hold across diverse settings.
   • Fair comparisons: Uses same backbone (Qwen3-VL-8B) and appears to use comparable training resources, though some baselines use different scales.
   • Generalization: Results hold across multiple benchmark types (perception, math, document understanding).

   Verdict: SIGNIFICANT — The mathematical analysis of reward coupling is non-obvious, the conditional advantage mechanism is technically sound, and the empirical gains are substantial across diverse benchmarks.

1. V*Bench: 91.1% (Metis) vs 88.7% (Qwen3-VL+GRPO), +2.4% improvement
2. HRBench-4K: 83.5% vs 78.9% (base model), substantial improvement
3. HRBench-8K: 82.0% vs 74.6% (base model), +7.4% improvement
4. TreeBench: 45.2% vs 40.7% (base model), +4.5% improvement
5. CharXiv (Reasoning Questions): 54.1% vs 48.9% (DeepEyesV2), +5.2% improvement
6. MathVista: 78.0% vs 71.9% (DeepEyesV2), +6.1% improvement
7. MathVerse: 65.9% vs 52.7% (DeepEyesV2), +13.2% improvement
8. WeMath: 65.2% vs 38.1% (DeepEyesV2), massive +27.1% improvement
9. DynaMath: 69.2% vs 57.2% (DeepEyesV2), +12.0% improvement

10. LogicVista: 56.2% vs 48.7% (DeepEyesV2), +7.5% improvement

    Results are consistently strong across all benchmarks. Most impressive gains on mathematical reasoning tasks where code execution provides clear value. Tool usage dramatically reduced (98%→2% on HRBench, 92%→2% on V*Bench) while maintaining/improving accuracy.

1. Model size: 8B parameters (Qwen3-VL-8B backbone)
2. Training compute: 8 NVIDIA Blackwell B200 GPUs for both SFT and RL stages, wall-clock time not reported
3. Inference speed/latency: Dramatically improved due to 90%+ reduction in tool calls (from 98% to 2% tool usage), though absolute latency numbers not provided
4. Memory footprint: Standard for 8B model, not specifically reported
5. Deployment practicality assessment: Highly practical - the core contribution is reducing real-world latency bottlenecks from excessive tool invocation while maintaining accuracy. The model learns to use tools only when genuinely necessary rather than reflexively.

1. Tool environment integration: Deployed with Python code execution, text search, and image search APIs in controlled environment with persistent state across multi-turn interactions.

2. Execution validation: All training trajectories rigorously validated by executing code in sandboxed environment to eliminate hallucinated environmental dynamics.

3. Meta-cognitive decision making: Demonstrates practical arbitration between internal knowledge and external tool queries, addressing real deployment constraint of API latency costs.

4. No robot/vehicle hardware experiments: Work focuses on multimodal reasoning benchmarks rather than embodied deployment.

5. Production considerations: The dramatic reduction in tool usage (90%+ decrease) directly addresses real-world deployment costs and latency constraints in API-dependent agentic systems.

1. EVALUATION - Limited analysis of failure cases where tools are genuinely needed but model abstains due to over-conservative efficiency penalty.

2. ENGINEERING - Hyperparameter sensitivity: performance degrades with efficiency weight above 0.15, requiring careful tuning of $w_{\text{tool}}$.

3. FUNDAMENTAL - Conditional advantage mechanism requires at least 2 correct rollouts per group (

   Q

   ≥ 2), limiting training signal when task success rate is very low.

4. EVALUATION - No analysis of computational overhead from sampling G=16 rollouts per prompt during RL training.

5. ENGINEERING - Data curation pipeline is labor-intensive, requiring execution validation and multi-dimensional quality filtering.

   Failure modes:

6. Over-conservative tool abstention on genuinely difficult tasks requiring external computation or knowledge.
7. Training instability when qualifying set Q is consistently small, leading to sparse efficiency gradient signals.

Large language model (LLM) reinforcement learning pipelines are bottlenecked by rollout generation, which comprises ~70% of training time due to expensive autoregressive decoding. To accelerate training, practitioners run rollouts with quantization (e.g., W4A16, W8A8) while maintaining full-precision training. However, this creates a severe training-inference mismatch where responses are sampled from low-precision rollout engines but policy updates are computed using full-precision models.

The task is to perform stable RL training on LLMs under quantized rollouts. The input consists of mathematical problem queries, and the output is Chain-of-Thought reasoning responses that are evaluated using verifiable rewards. The method must handle long-form generations (up to 16,384 tokens) where quantization errors accumulate over autoregressive decoding steps. The core challenge is that quantized rollouts produce “error tokens” - repetitive, garbled outputs that occur when the model goes off-trajectory during long generations.

Success is measured on mathematical reasoning benchmarks including AIME2024/2025, AMC, Math-500, OlympiadBench, and Minerva using pass@1 accuracy. Out-of-distribution evaluation includes ARC-Challenge, GPQA-Diamond, LiveCodeBench, and MMLU-Pro. The paper uses OpenR1-Math-46K dataset containing 46,000 mathematical problems for training.

1. Quantized Rollout Engine: Deploy low-bit inference (W4A16 or W8A8) in rollout generation using vLLM to accelerate decoding by 1.3-1.4×

2. Rollout-Aligned Quantization-Aware Training (QaRL): On training side, maintain master weights θ_BF16 but perform forward pass using aligned low-bit GEMM operations that mirror rollout engine arithmetic exactly

3. Mismatch Correction via Importance Sampling: Apply mismatch weight to compensate for distribution shift:

   \[w_{\text{mismatch}} = \frac{\pi_{\text{learner}}(a|\theta_{\text{old}})}{\pi_{\text{sampler}}(a|\theta_{\text{old}})}\]

4. Trust-Band Policy Optimization (TBPO): Address error token instability through: - Dual clipping for negative advantage samples: clip(r, 1-δ_ℓ, 1+δ_h) instead of one-sided PPO clipping - Sequence-level objectives using geometric mean of token probabilities - Drop entire responses when sequence-level ratios exceed trust region bounds

5. Weight Synchronization: Directly publish quantized weights from training engine to rollout engine to ensure exact alignment

   The core technical contribution is identifying that quantized rollouts produce “error tokens” that break PPO’s trust region assumptions, and solving this via sequence-level dual clipping rather than token-level approaches.

1. Pre-trained Base Models: Start from Qwen2.5-Math-1.5B/7B, Qwen3-8B-Base, Qwen3-30B-A3B-Base

2. RL Training Stage: - Data: OpenR1-Math-46K (46,000 mathematical problems) - Optimizer: Muon (faster convergence than AdamW) - Learning rate: 1e-6 with weight decay 0.01 - Batch size: 512 with 8 rollouts per query (G=8) - Sequence lengths: 2048 prompt + 16384 response tokens - Hardware: 8× NVIDIA H800 GPUs - Framework: Verl for training, vLLM for inference - Wall-clock time: Not reported

3. RL Algorithm: - GRPO (Group Relative Policy Optimization) for dense models - GSPO (Group Sequence Policy Optimization) for MoE models - Temperature 1.0 for rollout, 0.6 for evaluation - Sequence-level clipping ratios: ε_h=0.0004, δ_h=0.0007, δ_ℓ=0.0003 - TIS cap: c=2 for mismatch weight truncation

4. Quantization Settings: W4A16 or W8A8 quantization schemes with straight-through estimator for gradients

Prior work:

1. Decoupled PPO (Hilton et al., 2022): Introduced importance sampling correction for behavior-policy mismatch in RL, using w_mismatch ratios
2. GRPO/GSPO (Shao et al., 2024): PPO-style policy gradient methods for LLM RL with group relative rewards, established current SOTA for mathematical reasoning

3. Quantization-Aware Training literature: Standard QAT uses fake quantization for forward pass simulation while maintaining full-precision arithmetic

   Delta: This paper adds rollout-aligned quantized training that executes actual low-bit GEMM operations (not just fake quant simulation) and identifies “error tokens” as a fundamental failure mode in quantized rollouts. Introduces TBPO with sequence-level dual clipping specifically designed for this failure mode.

   Applied-specific assessment:

   • Architectural idea: The alignment between rollout and training engines via exact low-bit GEMM is novel, though the underlying QAT concepts are established
   • Benchmark gains: +5.5 points improvement over quantized rollout training is substantial and holds across multiple model sizes (1.5B to 30B MoE)
   • Fair comparisons: Yes - same data, models, and hardware. Compares against proper baselines (BF16 GRPO, quantized rollout GRPO)
   • Scale dependence: The approach works across different model scales and doesn’t require proprietary data

   The identification of error tokens and sequence-level dual clipping represents a non-obvious insight specific to quantized RL that wouldn’t be apparent from standard QAT literature.

   Verdict: SIGNIFICANT — Clear advance in making quantized RL training stable; most practitioners doing LLM RL should read this to understand quantization-induced instabilities.

1. AIME 2024: QaRL 27.5% vs quantized rollout 22.0% vs BF16 27.9% (Qwen3-30B-A3B)
2. AIME 2025: QaRL 22.0% vs quantized rollout 18.7% vs BF16 21.6% (Qwen3-30B-A3B)
3. AMC: QaRL 62.9% vs quantized rollout 55.4% vs BF16 63.2% (Qwen3-30B-A3B)
4. Math-500: QaRL 87.2% vs quantized rollout 84.0% vs BF16 88.8% (Qwen3-30B-A3B)
5. Minerva: QaRL 51.4% vs quantized rollout 47.4% vs BF16 54.7% (Qwen3-30B-A3B)
6. OlympiadBench: QaRL 56.1% vs quantized rollout 47.1% vs BF16 56.7% (Qwen3-30B-A3B)
7. ARC-Challenge: QaRL 96.6% vs quantized rollout 89.3% vs BF16 95.2% (Qwen3-30B-A3B)
8. GPQA-Diamond: QaRL 48.2% vs quantized rollout 42.4% vs BF16 50.1% (Qwen3-30B-A3B)
9. MMLU-Pro: QaRL 68.0% vs quantized rollout 65.3% vs BF16 70.3% (Qwen3-30B-A3B)

10. LiveCodeBench: QaRL 55.4% vs quantized rollout 47.9% vs BF16 55.8% (pass@4, Qwen3-30B-A3B)

    Pattern: QaRL consistently outperforms quantized rollout training by 2-6 points across all benchmarks and approaches BF16 performance. Results hold across 1.5B, 7B, 8B, and 30B MoE model scales. Improvements are consistent rather than cherry-picked on specific benchmarks.

1. Model size: Experiments on 1.5B, 7B, 8B, and 30B-A3B MoE parameters
2. Training compute: 8× NVIDIA H800 GPUs per experiment, specific GPU hours not reported
3. Inference speed: 1.3× speedup over BF16 training for QaRL, 1.4× speedup for quantized rollout training (MoE models)
4. Memory footprint: W4A16 quantization reduces memory requirements significantly, enabling larger batch sizes and reduced GPU count for MoE models
5. Deployment practicality: High - method integrates with standard RL frameworks (Verl) and inference engines (vLLM), requires no specialized hardware beyond quantization kernel support

1. Framework Integration: Successfully deployed in hybrid RL systems using Verl training framework + vLLM inference engine, demonstrating practical integration paths
2. Hardware Compatibility: Works with standard NVIDIA H800 GPUs using existing quantization kernels (W4A16, W8A8), no custom hardware required
3. Production Considerations: Method addresses real bottleneck (70% of RL training time in rollouts) and provides meaningful speedup while maintaining quality
4. Scale Validation: Tested across model sizes from 1.5B to 30B MoE, showing approach scales to production-relevant model sizes
5. Benchmark Diversity: Evaluation spans both in-distribution (math) and out-of-distribution tasks, suggesting generalizability beyond specific domains

1. FUNDAMENTAL: Approach still requires careful hyperparameter tuning (clipping bounds, TIS caps) and may not generalize to all quantization schemes or model architectures
2. ENGINEERING: Implementation complexity due to hybrid architecture requiring synchronization between training and inference engines; requires specialized kernel support for exact quantization alignment
3. EVALUATION: Limited to mathematical reasoning domain primarily; unclear performance on other RL applications like dialogue, creative writing, or code generation
4. ENGINEERING: Still incurs 1.3× vs 1.4× speedup trade-off compared to naive quantized rollout due to alignment overhead

5. FUNDAMENTAL: Sequence-level clipping may be overly conservative, potentially rejecting valid exploration sequences that contain few error tokens

   Failure modes:

   • Error token clustering: When multiple consecutive error tokens occur, sequence-level rejection may be too aggressive
   • Domain transfer: Method tuned for math reasoning may not transfer to other domains where “error tokens” have different characteristics