Applied AI Digest — Apr 8, 2026

Multimodal Large Language Models (MLLMs) require high-resolution visual inputs for fine-grained tasks like document understanding, OCR, and dense scene perception. However, current approaches scale entire images uniformly, flooding the quadratic self-attention mechanism with thousands of visually redundant tokens and severely bottlenecking inference throughput.

The task is to develop query-aware adaptive high-resolution perception for MLLMs. Input consists of:

1. source image at variable resolution, and

2. user text query. Output is the MLLM’s textual response. The method must dynamically decide whether coarse global features suffice or if high-resolution Region-of-Interest (RoI) extraction is needed.

   The core optimization objective balances perceptual accuracy with computational efficiency:

   \[\min_{\theta} \mathbb{E}_{(x_v, x_t)} [\mathcal{L}_{task}(f_{\theta}(x_v, x_t), y) + \lambda \cdot C(x_v, x_t)]\]

   where $C$ represents computational cost and $\lambda$ controls the accuracy-efficiency trade-off.

   Success is measured by:

3. standard VQA accuracy on Document & OCR benchmarks (DocVQA, InfoVQA, ChartQA, OCRBench, TextVQA), and
4. High-Resolution benchmarks (V*, MME-RealWorld, HR-Bench), while

5. tracking inference throughput (samples/second) and visual token usage reduction.

   The paper evaluates on established benchmarks without introducing new datasets.

1. Dynamic Gating Network: Lightweight router using pre-trained LLM layers B+1 to B+R, predicts refinement probability $Y_{pred} = \sigma(LP_{gate}(H^{B+R}_{gate}[-1]))$ where the final query token aggregates semantic intent.

2. Self-Distilled Region Proposal Network (SD-RPN): Operates on intermediate features $H^B_{context}$, repurposes self-attention to predict spatial heatmap via:

   \[Q_{RoI} = LP_q(Norm(H^{B+R-1}_u[-1]))\] \[K_v = LP_k(Norm(H^{B+R-1}_v))\] \[\hat{M}_{RoI} = Q_{RoI}K_v^T\]

3. Spatio-temporal Positional Encoding: For RoI tokens, applies temporal shift $t_{roi} = t_{src} + \delta$ and spatial interpolation to preserve global coordinate context:

   \[p^{(i,j)}_{roi} = Embed(t_{src} + \delta, y_1 + \frac{i}{H'-1}(y_2-y_1), x_1 + \frac{j}{W'-1}(x_2-x_1))\]

4. Training Strategy: - Gating network uses consistency-aware sample generation with BCE loss: $\mathcal{L}_{gate} = BCE(Y_{pred}, Y_{label})$ - SD-RPN trained via self-distillation using filtered cross-attention maps with tri-state labels - Post-SFT on hard-mined failure cases where base model succeeds but RoI integration fails

   The core contribution is decoupling visual resolution from computational cost through query-aware conditional routing and precise RoI extraction in a single prefilling pass.

1. Consistency-aware Sample Generation: Generate training data by evaluating MLLM across resolution trajectory R = {r₁, r₂, …, rₖ}, filter for valid transitions (low-res fail → high-res succeed), assign binary labels based on resolution-dependent correctness.

2. SD-RPN Training: - Data: 185K self-distilled pseudo-labels from VQA datasets, exclude extreme resolution for LLaVA variants - Loss: Selective BCE on tri-state labels ($\mathcal{L}_{RPN} = BCE(\hat{M}_{RoI}, \bar{M}_{RoI})$) - Optimizer: Not specified - Hardware: Not reported

3. Dynamic Gating Training: - Data: Filtered samples from standard VQA datasets using consistency-aware generation
   - Loss: Binary cross-entropy against deterministic routing labels - Freeze base MLLM, optimize only gating network parameters - Training details: Not reported

4. Post-Supervised Fine-tuning (Post-SFT): - Data: ~7K hard samples mined via LLM-as-a-Judge (base model correct, RoI model incorrect) - Freeze vision encoder and projector, fine-tune only LLM backbone - Applied only to Qwen variants (LLaVA lacks MRoPE) - Optimizer, schedule, hardware: Not reported

   Wall-clock training time not specified. All stages use frozen base MLLM except Post-SFT.

Prior Work:

1. ViCrop (ICLR 2025): Training-free method using contrastive cross-attention maps between generic and task-specific prompts, requires multiple prefilling passes
2. Thyme/DeepEyes (ICLR 2026): RL-based “Think-with-Image” paradigm using Chain-of-Thought decoding to locate RoIs, computationally expensive with lengthy inference latency

3. AnyRes strategy: Spatial partitioning of high-resolution inputs into multiple patches, processed independently then concatenated

   Delta: This paper adds:

4. Query-aware dynamic gating to bypass high-res processing when unnecessary
5. Single-pass RoI extraction using self-distilled intermediate features

6. Spatio-temporal alignment scheme for global-local feature fusion.

   Applied-specific Assessment:

   • Architectural novelty: The combination is incremental - gating networks and attention-based RoI extraction exist separately. The self-distillation from cross-attention is a reasonable engineering choice but not fundamentally novel.
   • Benchmark gains: Meaningful improvements (2.52×-4.39× speedup, 3-8% accuracy gains) but achieved through better engineering rather than breakthrough insights.
   • Fair comparisons: Generally fair, though some concurrent methods missing wall-clock comparisons. Efficiency gains partly from implementation optimizations (KV cache reuse).
   • Scale dependence: Likely requires substantial base model capacity; gains may not transfer to smaller models.

   The consistency-aware sample generation and tri-state pseudo-labeling are solid engineering contributions but represent incremental refinements of known techniques.

   Verdict: INCREMENTAL — Well-executed combination of existing techniques with meaningful practical improvements but no fundamental algorithmic breakthrough.

1. DocVQA (val): Q-Zoom 94.3% vs baseline 92.0%, +2.3% improvement
2. ChartQA (test): Q-Zoom 85.6% vs baseline 83.0%, +2.6% improvement
3. OCRBench (test): Q-Zoom 85.4% vs baseline 82.8%, +2.6% improvement
4. InfoVQA (val): Q-Zoom 79.4% vs baseline 70.1%, +9.3% improvement
5. TextVQA (val): Q-Zoom 83.5% vs baseline 81.1%, +2.4% improvement
6. V* Bench (overall): Q-Zoom 85.3% vs baseline 78.0%, +7.3% improvement
7. MME-RealWorld (overall): Q-Zoom 78.5% vs baseline 72.5%, +6.0% improvement

8. HR-Bench 4K (overall): Q-Zoom 77.3% vs baseline 63.6%, +13.7% improvement

   Results show consistent improvements across document understanding and high-resolution tasks. Performance gains are larger on challenging high-resolution benchmarks (HR-Bench 4K: +13.7%) compared to document tasks (+2-9%).

   Throughput improvements: 2.52× speedup on Document & OCR, 4.39× on High-Resolution benchmarks while maintaining accuracy.

   Missing: Some concurrent methods lack complete benchmark coverage, wall-clock training time comparisons absent.

1. Model size: Uses existing base models (Qwen2.5-VL-7B primary testbed), adds lightweight gating network and SD-RPN using R=3 transformer layers each, minimal parameter overhead

2. Training compute: Training details not fully specified - mentions filtering 185K samples for SD-RPN, ~7K samples for Post-SFT, but no GPU hours or wall-clock time reported

3. Inference speed: 2.52× acceleration on Document & OCR benchmarks, 4.39× on High-Resolution tasks. Throughput measured on single NVIDIA RTX A6000 GPU

4. Memory footprint: Achieves 53.0% visual token reduction (Document & OCR) and 73.2% reduction (High-Resolution) compared to 4096-token baseline. Uses KV-cache reuse for efficiency

5. Deployment practicality: High - operates as plug-and-play module, single prefilling pass eliminates multiple forward passes of training-free methods, significantly more practical than RL-based approaches requiring expensive Chain-of-Thought decoding

1. Benchmark vs Real-world: Evaluated primarily on standard benchmarks (DocVQA, TextVQA, etc.) rather than deployment scenarios, though these benchmarks use real-world document images

2. Hardware experiments: Tested only on NVIDIA RTX A6000 GPU, no mobile or edge device evaluation reported

3. Production integration: No production deployment results mentioned, though plug-and-play design suggests practical integration potential

4. Generalization: Demonstrates transfer across multiple base architectures (LLaVA, Qwen2.5-VL, Qwen3-VL) and integrates with RL-trained models (ZwZ-Qwen variants), showing broad applicability

   Limited real-world deployment evidence, but strong cross-architecture transferability suggests practical utility.

1. FUNDAMENTAL: Method requires base MLLM to have reasonable cross-attention localization capability for self-distillation - fails if base model attention is too noisy

2. FUNDAMENTAL: Spatio-temporal alignment only works with models supporting Multimodal Rotary Positional Embeddings (MRoPE) - LLaVA variants cannot benefit from Post-SFT

3. ENGINEERING: Consistency-aware sample generation requires multiple resolution evaluations, increasing dataset preparation cost

4. ENGINEERING: Performance depends on careful threshold tuning (τgate, τroi, τfg, τbg) which may require per-model calibration

5. EVALUATION: Limited evaluation on models smaller than 3B parameters - scalability to resource-constrained settings unclear

6. EVALUATION: No analysis of failure cases where RoI extraction misses relevant regions or includes distractors

   Failure Modes:

7. Query ambiguity: Dynamic gating may misclassify queries requiring fine-grained perception as simple, leading to incorrect bypass of RoI extraction
8. Spatial fragmentation: SD-RPN may predict fragmented or incomplete bounding boxes for complex multi-object scenarios, missing critical visual evidence

Knowledge-Based Visual Question Answering (KB-VQA) addresses situations where answering visual questions requires external encyclopedic knowledge not present in the image alone. This is challenging because VQA models must both understand visual content and retrieve relevant external information from large knowledge bases.

The task takes as input a query image and textual question, requiring retrieval of relevant context from a knowledge base (typically Wikipedia articles with images) to produce accurate answers. The formal objective is to maximize answer accuracy given:

where $A$ is the answer, $I$ is the query image, $Q$ is the question, $KB$ is the knowledge base, and $C$ is retrieved context.

Success is measured using dataset-specific metrics: BEM score for EVQA, VQA Accuracy (standard and relaxed) for InfoSeek, and retrieval metrics include Recall@K and Pseudo Recall@K.

The paper evaluates on three benchmarks: EVQA (1M samples, 16.7K entities), InfoSeek (1.3M training, 8.9K evaluation samples, 11K entities), and M2KR (unified framework across 9 vision-language datasets).

1. Multi-modal knowledge base construction using aligned <image, section> pairs, indexed via concatenated visual and textual embeddings:

   \[v_i = \text{Concat}[\Phi_{vis}(I_{kb}), \Phi_{text}(T_{kb})]\]

2. VLM Refiner (Qwen2.5-VL-3B) trained with GRPO to rewrite queries using visual cues, optimizing reward function:

   \[r_i = r_{retrieval}(o_i) + r_{format}(o_i)\]

3. Weighted multi-modal dense retrieval with hyperparameter $\alpha$ controlling visual/textual balance:

   \[v_q = \text{Concat}[\alpha \cdot \Phi_{vis}(I_q), (1-\alpha) \cdot \Phi_{text}(T_q)]\]

4. Multi-modal reranker (from EchoSight) filters top candidates using cosine similarity

5. VLM Inspector (Qwen3-VL-8B) validates retrieved context sufficiency and routes to either: - LLM Generator (Qwen2.5-7B) for reliable context - VLM internal knowledge for insufficient context

   The core contribution is repositioning VLMs from generic answer generators to specialized agents (Refiner + Inspector) while using LLMs for text comprehension tasks.

1. Multi-modal retriever training:
   • Encoders: EVA-CLIP-8B (visual), Qwen3-Embedding-0.6B (textual), frozen weights
   • Index construction using FAISS with cosine similarity
   • Hardware: not reported
2. Refiner training via GRPO:
   • Data: 7K samples per dataset (EVQA, InfoSeek) based on retrieval hit rank sampling
   • Optimizer: AdamW, learning rate 1×10⁻⁶, batch size 32
   • Rollout temperature 0.7, group size 5, 600 training steps
   • Vision tower frozen, only language components updated
   • Hardware: 4 NVIDIA A800 40GB GPUs
3. Inspector fine-tuning:
   • Data: 38K mixed samples (18K PASS, 20K FAIL from EVQA + InfoSeek)
   • Supervised fine-tuning for binary classification with structured JSON outputs
   • Training details: not fully reported
4. Answer Generator fine-tuning:
   • LLM: Qwen2.5-7B-Instruct fine-tuned per dataset
   • EVQA: 10K samples with LLM-expanded answers
   • InfoSeek: 13.6K samples with ground-truth supervision
   • Training framework: LlamaFactory
   • Other details: not reported

Prior Work:

1. Wiki-LLaVA (2024): First hierarchical retrieval pipeline for multimodal RAG, using VLMs as answer generators
2. EchoSight (2024): Enhanced multimodal rerankers for KB-VQA, visual-only retrieval with VLM generation

3. OMGM (2025): Multiple granularities and modalities for retrieval, still using traditional VLM-as-generator paradigm

   Delta: This paper repositions VLMs from answer generators to specialized agents (Refiner + Inspector), introduces multi-modal dense retrieval with weighted concatenation, and employs RL-based query refinement.

   Applied-specific Assessment:

   • Architectural idea: The agent-based VLM repositioning is somewhat novel but builds incrementally on existing RAG patterns
   • Benchmark gains: Substantial improvements (5.45pp on EVQA, consistent gains across InfoSeek/M2KR) across multiple metrics
   • Fair comparisons: Uses same reranker (EchoSight) and evaluates on standard benchmarks with consistent protocols
   • Scalability: Relies on substantial compute for multi-model training and RL optimization; gains may not hold without this scale

   The core insight that VLMs are poor at text comprehension but good at visual reasoning is empirically validated and leads to meaningful architectural changes.

   Verdict: SIGNIFICANT — Clear architectural insight with consistent large gains across multiple benchmarks, though building incrementally on established RAG patterns.

1. EVQA retrieval: R@1 44.1% vs OMGM 42.8% (+1.3pp), R@5 59.9% vs 55.7% (+4.2pp)
2. InfoSeek retrieval: R@1 67.0% vs OMGM 64.0% (+3.0pp), R@5 83.7% vs 80.8% (+2.9pp)
3. EVQA VQA accuracy: 55.62% vs OMGM 50.17% (+5.45pp), substantial improvement
4. InfoSeek VQA accuracy: 44.72% vs OMGM 43.49% (+1.23pp), modest improvement
5. M2KR EVQA split: R@1 43.1% vs PreFLMR 40.4% (+2.7pp)
6. M2KR OKVQA split: R@1 20.7% vs 13.8% (+6.9pp), large improvement

7. M2KR OVEN split: R@1 42.8% vs 31.1% (+11.7pp), very large improvement

   Results show consistent improvements across all benchmarks with particularly strong gains on EVQA VQA accuracy and M2KR entity recognition tasks. No major benchmarks are conspicuously absent.

1. Model size: Refiner 3B parameters (Qwen2.5-VL), Inspector 8B parameters (Qwen3-VL), Generator 7B parameters (Qwen2.5), total ~18B parameters across specialized models
2. Training compute: 4 NVIDIA A800 40GB GPUs, RL training 600 steps, total wall-clock time not reported
3. Inference speed: Marginally slower than PreFLMR baseline due to multi-agent architecture but faster retrieval than PreFLMR
4. Memory footprint: Requires loading multiple specialized models simultaneously, significantly higher than single-model approaches
5. Deployment practicality: Challenging due to multiple model requirements, complex multi-stage pipeline, and RL training dependencies; production deployment would require substantial infrastructure

1. Evaluation limited to academic benchmarks (EVQA, InfoSeek, M2KR) using curated Wikipedia knowledge bases
2. No deployment results or production integration reported
3. No hardware experiments on real robots or autonomous systems
4. No sim-to-real transfer evaluation
5. Knowledge base construction requires pre-processed Wikipedia articles with aligned image-text sections
6. Framework assumes access to structured encyclopedic knowledge bases which may not exist for specialized domains

1. Hard routing mechanism between LLM and VLM is suboptimal - ENGINEERING (authors acknowledge this limitation)
2. Single-pass retrieval only, cannot handle multi-hop questions - FUNDAMENTAL (inherent to current architecture)
3. Requires multiple specialized models increasing deployment complexity - ENGINEERING (could be addressed with model unification)
4. RL training adds significant complexity and compute requirements - ENGINEERING (could use simpler fine-tuning approaches)
5. Limited to Wikipedia-style structured knowledge bases - FUNDAMENTAL (approach may not generalize to other knowledge formats)

6. Inspector routing accuracy only 82.1% leading to suboptimal decisions - ENGINEERING (better training data/methods could improve)

   Failure modes:

7. Poor performance when visual content is ambiguous and textual query is insufficient for retrieval
8. Inspector misrouting leading to wrong generation path (VLM vs LLM) degrading final answers

3D Visual Grounding (3D-VG) aims to localize target objects in 3D indoor scenes using natural language descriptions. This addresses critical needs in human-robot interaction, embodied AI navigation, and AR/VR applications, where natural language is the most intuitive way for humans to specify spatial targets. The challenge lies in bridging semantic understanding of language with precise 3D spatial reasoning, especially when 3D training data is scarce and expensive to collect.

The task takes as input a natural language query Q and sequential RGB-D video streams V = {(Ii, Di)}^T_{i=1} with camera intrinsics K and extrinsics T_{c2w}. The output is a 3D bounding box B ∈ R^6 precisely localizing the target object. The formal objective is to minimize localization error:

Success is measured using Intersection-over-Union (IoU) metrics: Acc@0.25 and Acc@0.5 represent the fraction of predictions with IoU > 0.25 and 0.5 respectively against ground truth bounding boxes.

The paper evaluates on ScanRefer and Nr3D benchmarks built on ScanNet indoor scenes. ScanRefer contains “Unique” and “Multiple” query categories depending on presence of same-class distractors. Nr3D divides queries into “Easy”/”Hard” and “View-Dependent”/”Independent” subsets.

1. VLM Agent Core: Qwen3-VL-32B serves as the primary reasoning agent, guided by an expert “3D-VG Skill” that defines the execution blueprint for Think-Act-Build loops.

2. Query Analysis Tool: Parses natural language queries into structured JSON format extracting target_class, visual attributes, spatial conditions, and global scene features.

3. Reference Target Localization: Uses Grounding DINO for coarse filtering to detect target class, followed by VLM-based fine filtering for scene verification. Score&Rank tool evaluates candidates, Seg&Marker tool uses SAM3 for segmentation with numeric IDs.

4. Semantic Temporal Expansion: Bidirectional tracking from reference frame using VLM verification and SAM segmentation, building video context V_sem = {(It, Dt, Mt)}_{t∈T_local}.

5. Centroid Extraction: Inverse-projects masked pixels to 3D using camera parameters:

   \[P_c = D_t(u,v) \cdot K^{-1} \begin{bmatrix} u \\ v \\ 1 \end{bmatrix}\] \[P_w = T_{c2w} P_c\] \[P_{centroid} = \frac{1}{N} \sum_{k=1}^N P_w^k\]

6. Multi-View Geometric Expansion: Projects 3D centroid back to 2D frames using visibility checks including FoV boundaries, depth validity, and Z-buffer occlusion tests.

7. Final Reconstruction: Aggregates multi-view masks via Statistical Outlier Removal and DBSCAN clustering to compute axis-aligned 3D bounding box.

   The core contribution is the Semantic-Anchored Geometric Expansion mechanism that overcomes VLM tracking brittleness by using deterministic geometry to propagate targets across unobserved frames.

1. No Training Required: TAB operates entirely with pre-trained, off-the-shelf models without any task-specific fine-tuning.

2. Foundation Models Used: - VLM Agent: Qwen3-VL-32B for reasoning and visual analysis - Object Detection: Grounding DINO for coarse target class filtering
   - Segmentation: SAM3 (Segment Anything Model 3) for precise instance masks - All models used in zero-shot inference mode

3. Data Processing: 300 frames sampled per video from ScanNet RGB-D sequences with aligned depth maps and camera parameters.

4. Expansion Limits: Both Semantic Temporal Expansion and Multi-View Geometric Expansion capped at maximum 32 frames for computational efficiency.

5. Hyperparameters: Depth noise tolerance ε = 0.4 for Z-buffer occlusion checks, Statistical Outlier Removal and DBSCAN clustering parameters not reported.

Prior work includes:

1. LLM-Grounder (Yang et al., 2023) pioneered zero-shot 3D-VG using LLMs but relied heavily on pre-scanned 3D point clouds, degrading the task to proposal matching.
2. VLM-Grounder (Xu et al., 2025) attempted direct 2D-to-3D grounding but used brittle semantic tracking vulnerable to viewpoint variations.

3. SPAZER (Jin et al., 2025) and SeeGround (Li et al., 2025) also depend on preprocessed 3D point clouds as static inputs.

   Delta: This paper introduces the Semantic-Anchored Geometric Expansion mechanism that decouples semantic reasoning (handled by 2D VLMs) from 3D structure instantiation (handled by deterministic multi-view geometry). The key insight is using a 2D→3D→2D mapping: semantic tracking establishes initial 3D centroid, then geometric projection expands to unobserved frames.

   Applied-specific assessment:

   • Architectural novelty: The semantic-anchored geometric expansion is a reasonable engineering solution but not fundamentally novel - it combines standard multi-view geometry with VLM tracking.
   • Benchmark gains: Improvements are meaningful (71.2% vs 57.2% Acc@0.25 on ScanRefer) but gains are conditional on having RGB-D streams with camera parameters.
   • Fair comparisons: Comparisons mix methods with/without 3D point cloud access, making direct comparison difficult. The “3D-assisted” results show the method benefits significantly from proposal refinement.
   • Scalability: Gains likely depend on high-quality depth maps and camera calibration, limiting real-world deployment.

   The work addresses a real limitation (point cloud dependency) but the solution is incremental engineering rather than fundamental innovation.

   Verdict: INCREMENTAL — solid engineering solution combining existing VLM capabilities with standard multi-view geometry, but lacks fundamental novelty.

1. ScanRefer Overall: TAB achieves 71.2% Acc@0.25 and 46.4% Acc@0.5 vs previous best zero-shot SPAZER at 57.2% and 48.8% respectively. With 3D assistance: 71.6% and 61.6%.

2. ScanRefer Unique subset: 90.2% Acc@0.25 and 57.6% Acc@0.5, significantly outperforming prior zero-shot methods.

3. ScanRefer Multiple subset: 60.1% Acc@0.25 and 39.9% Acc@0.5, showing good disambiguation of same-class distractors.

4. Nr3D Overall: 68.0% accuracy vs previous best zero-shot SPAZER at 63.8% and fully-supervised SceneVerse at 64.9%.

5. Nr3D Hard queries: 63.2% accuracy, demonstrating robustness to challenging spatial descriptions.

6. Nr3D View-Dependent: 62.5% accuracy on perspective-sensitive queries.

   Results show consistent but modest improvements across benchmarks. Notable that performance varies significantly between “native” reconstruction (46.4% Acc@0.5) and 3D-assisted refinement (61.6% Acc@0.5), suggesting the core geometric reconstruction has limitations.

1. Model Size: Qwen3-VL-32B (32 billion parameters) plus Grounding DINO and SAM3 - total parameters not reported but likely 35B+.

2. Training Compute: Zero - no training required, purely inference-based framework.

3. Inference Speed: Not reported. Framework requires multiple VLM calls per query (parsing, filtering, verification, tracking), plus geometric projections across up to 64 frames total.

4. Memory Footprint: Not reported, but must load and process up to 300 RGB-D frames simultaneously with multiple large vision models.

5. Deployment Practicality: Requires high-quality RGB-D sensors with accurate camera calibration, plus significant compute for running 32B parameter VLM. Practical deployment limited to well-equipped robotic systems or research environments.

1. Input Requirements: Operates on raw RGB-D video streams, which is more practical than requiring pre-scanned point clouds, but still needs calibrated depth sensors and camera parameters.

2. Hardware Dependencies: Requires RGB-D sensors (e.g., Intel RealSense, Kinect) with accurate depth maps and camera calibration - common in robotics but not ubiquitous.

3. Environmental Constraints: Tested only on ScanNet indoor scenes with controlled lighting and static environments. No evaluation on dynamic scenes, outdoor environments, or varying lighting conditions.

4. No Deployment Results: Paper provides no evidence of real robotic deployment, sim-to-real transfer, or integration with actual robotic systems beyond controlled dataset evaluation.

5. Scalability Issues: Framework requires processing hundreds of frames with multiple foundation models, limiting real-time applicability for interactive robotic applications.

1. Depth Map Dependency (FUNDAMENTAL): Framework fundamentally relies on high-quality aligned depth maps and accurate camera calibration, limiting applicability to scenarios with poor depth sensing.

2. Computational Overhead (ENGINEERING): Multiple VLM inference calls and processing 300+ frames per query creates significant latency unsuitable for real-time applications.

3. Indoor Scene Bias (EVALUATION): Evaluation limited to controlled indoor scenes from ScanNet - no testing on outdoor environments, dynamic scenes, or challenging lighting.

4. Camera Parameter Requirement (FUNDAMENTAL): Geometric expansion critically depends on accurate camera intrinsics and extrinsics, failing when calibration is poor.

5. Static Scene Assumption (FUNDAMENTAL): Framework assumes static scenes where objects don’t move during video capture, breaking down in dynamic environments.

   Likely failure modes:

6. Dense crowds or cluttered scenes where occlusions break semantic tracking
7. Reflective or transparent surfaces that confuse depth sensing and geometric projection.

This paper addresses the problem of agentic coding - enabling AI models to autonomously plan, execute, and verify multi-step software engineering tasks in real-world development environments. Traditional code generation focuses on single-turn responses, but real development requires long-horizon reasoning, tool orchestration, environment interaction, and execution feedback integration.

The task spans five domains:

1. SWE - software engineering with issue resolution in authentic repositories
2. WebCoding - frontend generation with aesthetic quality from colloquial inputs
3. Terminal - command-line reasoning and system operations
4. WebSearch - multi-hop information synthesis via tool invocation, and

5. General - instruction following and code-math reasoning. Input modalities include natural language instructions, code repositories, and execution environments. Output includes code edits, complete applications, terminal commands, and structured responses.

   Success is measured across multiple benchmarks: SWE-bench Verified (software engineering repair), PinchBench (agent task execution), Terminal-Bench Hard (CLI operations), frontend aesthetics evaluation (10-dimensional scoring for UI quality), and τ²-Bench (general reasoning). The paper introduces systematic aesthetic evaluation for reference-free Text-to-UI generation.

   The training dataset encompasses 2M+ Issue-PR pairs from 100K+ GitHub repositories, 30K verified tasks from AutoBuilder pipeline, 100K+ WebSearch trajectories, and expert-annotated terminal tasks across 12 technical domains.

1. Base model: KAT-Coder-V1 with continued post-training following a “Specialize-then-Unify” paradigm

2. Domain decomposition: Five expert domains (SWE, WebCoding, Terminal, WebSearch, General) each undergo independent supervised fine-tuning and reinforcement learning

3. SWE Expert: Issue-PR pairing with semantic association scoring:

   \[s(i,p) = \cos(e_i, e_p)\]

   AutoBuilder pipeline with F2P (Fail-to-Pass) + P2P (Pass-to-Pass) verification:

   \[\forall t \in T_{fail}: t(\hat{c}) = \text{Pass} \wedge \forall t \in T_{pass}: t(\hat{c}) = \text{Pass}\]

4. WebCoding Expert: Tri-perspective label system with hierarchical mapping $L: V_{user} \rightarrow V_{design} \rightarrow V_{impl}$ and autoregressive level inference:

   \[\hat{L}_k = f_\theta(L_1, \hat{L}_2, \ldots, \hat{L}_{k-1})\]

5. Reinforcement learning with Modified Turn-level Policy Optimization using turn-based importance ratios:

   \[r_{turn}^{(n)}(\theta) = \prod_{i \in T_n} \frac{\pi_\theta(y_i|x,y_{<i})}{\pi_{\theta_{old}}(y_i|x,y_{<i})}\]

6. MCLA (Monte-Carlo Log-probability Averaging) for MoE stability:

   \[\bar{\log}\pi(a) = \frac{1}{K}\sum_{k=1}^K \log\pi^{(k)}(a)\]

7. On-Policy Distillation (OPD) for expert fusion combining RL environment rewards with dense expert supervision

1. Supervised Fine-Tuning stage:
   • Data: 2M+ Issue-PR pairs, 30K AutoBuilder tasks, 100K+ WebSearch trajectories, expert terminal annotations
   • Five parallel expert training pipelines with domain-specific data construction
   • Optimizer: not reported; hardware and timing: not reported
2. Reinforcement Learning stage:
   • Data: 100K+ samples from Agentic Scaling across task complexity, intent alignment, and scaffold generalization
   • Modified Turn-level Policy Optimization with MCLA (K=8 forward passes for variance reduction)
   • Tree Training for 6.2x speedup on tree-structured trajectories
   • High-concurrency sandbox training via KRL framework
   • Optimizer, learning rates, batch sizes: not reported
3. On-Policy Distillation stage:
   • Joint optimization combining RL environment rewards with expert log-probability supervision
   • Dynamic expert selection per task domain
   • Training details: not reported

   Infrastructure: KwaiEnv supporting tens of thousands of concurrent sandbox instances, integrated with multiple scaffolds (Claude Code, OpenCode, Kilo Code, OpenClaw)

Prior work:

1. SWE-bench (Jimenez et al., 2024) established software engineering benchmarks for code repair
2. Claude Code and similar agent scaffolds demonstrated multi-turn coding capabilities

3. Standard RL approaches like GRPO for code generation.

   Delta: This paper adds three key components:

4. systematic domain decomposition with independent expert training across five orthogonal domains
5. infrastructure-level innovations including KwaiEnv’s modular architecture and Tree Training for tree-structured RL

6. On-Policy Distillation for lossless expert fusion.

   Applied-specific assessment:

   • Architectural idea: The “Specialize-then-Unify” paradigm is a reasonable extension of mixture-of-experts concepts, but the domain decomposition is somewhat arbitrary and the necessity of separate training unclear
   • Benchmark gains: Results show marginal improvements (79.6% vs 80.8% on SWE-bench Verified), within noise margin of Claude Opus 4.6
   • Comparisons: Fair comparison methodology with same scaffolds, but heavily relies on proprietary infrastructure and datasets that may not be reproducible
   • Scale dependency: The approach clearly depends on massive compute (tens of thousands of concurrent sandboxes) and proprietary data that smaller teams cannot replicate

   The core technical contributions (MCLA, Tree Training) address real engineering problems but are incremental optimizations rather than fundamental advances. The domain expert approach is sensible but lacks theoretical justification for why independent training should outperform joint training.

   Verdict: INCREMENTAL — solid engineering with useful optimizations, but represents expected scaling of known approaches rather than fundamental innovation.

1. SWE-bench Verified: 79.6% (KAT-Coder-V2) vs 80.8% (Claude Opus 4.6), -1.2pp gap
2. SWE-bench Multilingual: 75.4% vs 77.8% (Claude), -2.4pp gap
3. PinchBench: 88.7% vs 87.4% (Claude), +1.3pp improvement, surpassing GLM-5 (86.4) and MiniMax M2.7 (87.1)
4. Frontend aesthetics - Landing Page: 59.8% vs GLM-5 (57.6%), +2.2pp improvement
5. Frontend aesthetics - Slides: 57.6% vs GLM-5 (42.8%), +14.8pp improvement
6. Frontend aesthetics - Data Visualization: 67.6% vs GLM-5 (42.4%), +25.2pp improvement
7. Terminal-Bench Hard: 46.8% vs Claude (46.2%), marginal +0.6pp
8. τ²-Bench: 93.9% vs Claude (92.1%), +1.8pp improvement

9. Claw-Eval: 55.6% vs Claude (66.3%), -10.7pp gap

   Results are mixed - strong performance on frontend tasks and competitive on most benchmarks, but trailing Claude on key SWE and agent execution metrics. The frontend results show the largest improvements but lack comparison to specialized UI generation models.

1. Model size: Based on KAT-Coder-V1, specific parameter count not reported
2. Training compute: Tens of thousands of concurrent sandbox instances, specific GPU hours not reported
3. Inference speed: Tree Training provides 6.2x speedup for tree-structured trajectories
4. Memory footprint: MCLA requires K=8 forward passes during training, increasing memory usage
5. Deployment practicality: Requires KwaiEnv infrastructure with massive sandbox orchestration; publicly available at streamlake.com but deployment complexity likely prohibitive for most users

1. Production integration: Model publicly deployed at streamlake.com/product/kat-coder with real user access
2. Multi-scaffold compatibility: Tested across 10+ mainstream AI coding scaffolds (Claude Code, OpenCode, Kilo Code, OpenClaw) using native interaction protocols
3. Real repository testing: Evaluated on authentic GitHub repositories with actual issue-PR pairs and code dependencies
4. Sandbox environments: Docker-based isolated execution environments mirroring real development setups
5. Frontend generation: Produces complete HTML/CSS/JS applications with professional-grade visual quality assessed by UI/UX designers
6. No explicit sim-to-real discussion or hardware deployment results beyond web-based code generation

1. Infrastructure dependency - FUNDAMENTAL: Requires proprietary KwaiEnv infrastructure with massive sandbox orchestration that most users cannot replicate
2. Scale dependency - FUNDAMENTAL: Performance likely depends on training at massive scale (100K+ RL samples, tens of thousands of concurrent sandboxes) that smaller teams cannot afford
3. Domain decomposition justification - EVALUATION: No theoretical or empirical justification provided for why five specific domains should be trained independently rather than jointly
4. Expert fusion overhead - ENGINEERING: On-Policy Distillation introduces additional training complexity and potential performance degradation during fusion

5. Benchmark gaps - EVALUATION: Trailing performance on key benchmarks (SWE-bench Verified, Claw-Eval) suggests fundamental limitations remain

   Likely failure modes:

6. Performance degradation without massive computational resources
7. Poor generalization to domains not covered by the five expert categories

Egocentric video understanding requires models to reason over temporal dependencies in first-person perspective videos, where camera viewpoints change rapidly and causal relationships between frames are critical. Existing multimodal large language models (MLLMs) often fail at temporal reasoning, instead relying on spatial shortcuts from single frames.

Task definition: Given an egocentric video $c$ and a question $s$, the model must generate a response $y$ that demonstrates temporal awareness. The input consists of 32 uniformly sampled video frames and multiple-choice or yes/no questions. The objective is to maximize temporally calibrated rewards:

where $r(y)$ is the reward for the response to the original video and $r(\hat{y})$ is the reward for a response to the same video with shuffled frames.

Evaluation criteria: Performance is measured by accuracy on multiple-choice questions across five benchmarks. Success requires correctly answering questions that cannot be solved from single frames alone, requiring temporal coherence and causal understanding.

The paper evaluates on existing benchmarks: EgoSchema (temporal split), EgoPlan/EgoPlan2 (action planning), VLM4D (motion understanding), and EgoTempo (holistic temporal reasoning).

1. Base architecture: Qwen2.5-VL-3B multimodal large language model as the foundation

2. Temporal Global Policy Optimization (TGPO): Novel RL algorithm that generates two types of responses for each video-question pair - one from temporally ordered frames using temperature sampling, and one baseline from shuffled frames using greedy decoding

3. Temporally calibrated reward computation:

   \[r_T(y) = r(y) - r(\hat{y})\]

4. Global advantage estimation for TGPO-GRPO integration:

   \[\hat{A}(B; \theta) = \frac{1}{|B|} \sum_{j=1}^{|B|} \frac{1}{|G|} \sum_{i=1}^{|G|} \frac{1}{|y_{j,i}|} \sum_{t=1}^{|y_{j,i}|} \frac{\pi_\theta(y_{j,i,t} | s_j, c_j)}{\pi_{\theta_{old}}(y_{j,i,t} | s_j, c_j)} \cdot \frac{r_T(y_{j,i}) - \mu_B}{\sigma_B}\]

5. Composite reward function combining accuracy and format compliance:

   \[r(a) = r_{Accu}(a) + \lambda r_{Form}(a)\]

   The core contribution is the temporal calibration mechanism that explicitly rewards improvements from temporal ordering while suppressing spatial shortcuts through global normalization across training batches.

1. Cold-start reinforcement learning regime: Direct RL optimization without supervised fine-tuning stage, following DeepSeek-R1-Zero paradigm

2. Training data: EgoIT99K dataset, restricted to multiple-choice and yes/no questions suitable for verifiable rewards

3. Optimization setup: Learning rate 1×10^-6 with constant scheduler, KL regularization coefficient 1×10^-4, weight decay 0.01, number of rollouts 8, sampling temperature 1.0

4. Batch configuration: Micro-batch size 4 per GPU, mini-batch size 64, trained on 8 nodes with 8 NVIDIA A100 40GB GPUs each

5. Video processing: 32 uniformly sampled frames per video sequence

6. Format reward weight λ = 0.1 to encourage proper response structure with thinking and answer tags

7. Training duration: 3000 training steps as reported in learning curves

8. Framework: VERL reinforcement learning framework extended to support video input processing

Prior work:

1. GRPO (Shao et al., 2024) - Group Relative Policy Optimization that normalizes rewards within sample groups for policy gradient estimation
2. GSPO (Zheng et al., 2025) - Group Sequence Policy Optimization that uses sequence-level likelihood ratios instead of token-level ratios

3. EgoVLM (Vinod et al., 2025) - Applied GRPO to egocentric video understanding with keyframe-based rewards

   Delta: This paper adds temporal calibration through contrastive rewards between temporally ordered vs shuffled video frames, plus global normalization across training batches instead of within-group normalization.

   Applied-specific assessment:

   • Architectural novelty: The temporal calibration mechanism is conceptually straightforward but non-obvious - using shuffled frames as a baseline to isolate temporal reasoning is a clever idea
   • Benchmark gains: Consistent but modest improvements (2-5 points) across all benchmarks; largest gains on EgoSchema (+3.1 over GRPO) and EgoPlan2 (+5.2 over GRPO)
   • Fair comparisons: Uses same base model, training data, and hyperparameters as GRPO/GSPO baselines, ensuring fair evaluation
   • Scale dependency: The approach should work without massive compute since it’s applied to 3B model, though the cold-start RL paradigm itself requires substantial resources

   Verdict: INCREMENTAL — Solid extension of existing RL methods with a sensible temporal awareness mechanism, but gains are modest and the core insight is fairly straightforward.

1. EgoSchema (temporal split): TGPO achieves 49.6-49.7%, GRPO 46.5%, GSPO 45.4% - improvement of +3.1-3.2 points

2. EgoPlan: TGPO achieves 36.7-36.8%, GRPO 36.5%, GSPO 36.3% - minimal improvement of +0.2-0.3 points

3. EgoPlan2: TGPO achieves 41.1-42.3%, GRPO 37.1%, GSPO 32.7% - notable improvement of +4.0-5.2 points

4. VLM4D: TGPO achieves 48.6-49.6%, GRPO 46.8%, GSPO 47.4% - improvement of +1.8-2.8 points

5. EgoTempo: TGPO achieves 42.6-45.2%, GRPO 40.0%, GSPO 42.0% - improvement of +2.6-5.2 points

   Results are consistently positive but modest across all benchmarks. The largest improvements are on EgoSchema and EgoPlan2, suggesting the method is most effective on tasks requiring sequential reasoning over extended time horizons.

1. Model size: 3B parameters (Qwen2.5-VL-3B backbone)

2. Training compute: 8 nodes × 8 NVIDIA A100 40GB GPUs, 3000 training steps duration, specific wall-clock time not reported

3. Inference speed/latency: Not reported - likely similar to base Qwen2.5-VL model since no architectural changes

4. Memory footprint: Not explicitly reported, but requires additional memory during training for sampling both original and shuffled video responses

5. Deployment practicality: Should be deployable at same scale as base 3B model since no inference-time architectural changes, though training requires substantial compute for RL optimization

1. Evaluation limited to curated benchmark datasets - no deployment results or real-world testing reported

2. No hardware experiments, robot integration, or production deployment discussed

3. No sim-to-real transfer analysis provided

4. Method evaluated only on multiple-choice question answering format, which may not reflect real-world egocentric reasoning scenarios

5. Training data (EgoIT99K) appears to be derived from existing egocentric video datasets rather than direct real-world capture

   The work remains primarily in the research benchmark evaluation stage without demonstrated real-world applicability.

1. FUNDAMENTAL: Reliance on multiple-choice format limits applicability to open-ended real-world scenarios where temporal reasoning is needed

2. FUNDAMENTAL: Shuffled frame baseline may not capture all forms of non-temporal reasoning, potentially missing other shortcut behaviors

3. ENGINEERING: Cold-start RL training requires substantial computational resources and careful hyperparameter tuning

4. EVALUATION: Limited to five specific benchmarks, may not generalize to other forms of temporal reasoning or video understanding tasks

5. ENGINEERING: Global normalization across batches could be sensitive to batch composition and dataset characteristics

   Failure modes:

   • Method likely fails when temporal dependencies are very subtle or when spatial cues are genuinely informative
   • May struggle with very long video sequences due to 32-frame sampling limitation