Applied AI Digest — Apr 11, 2026

Multimodal generation faces fundamental limitations in current text-driven pipelines where language controls vision but cannot reason within the visual space. Cross-modal alignment bottlenecks and task-specific architectural branches fragment the representation space, making unified understanding and generation difficult.

This paper addresses unified multimodal generation through an image-in, image-out paradigm. The input consists of visual prompt canvases $I_v$ containing rendered text instructions, spatial layouts, arrows, and doodles combined with optional source images $I_{src}$. The output is a target image $I^*$ that follows the visual instructions. The formal objective is flow matching loss:

\[L_{FM} = E_{(z_0,z_1),t} \|v_\theta(z_t, t) - (z_1 - z_0)\|_2^2\]

where $z_0$ is the latent encoding of visual prompts and $z_1$ is the target image latent.

Success is measured by four criteria:

1. Instruction Faithfulness - adherence to rendered text commands
2. Content Consistency - preservation of source image content when applicable
3. Visual Realism - photorealistic quality via CLIP-IQA, and

4. Spatial Precision - accurate geometric placement measured by DINOv3 directional similarity.

   The paper introduces VisPrompt-5M, a dataset of 5 million visual prompt pairs spanning text-to-image generation (2.86M), text-in-image editing (1.6M), text bounding box editing (24K), visual marker editing (250K), doodles editing (1K), and physics-aware tasks including force dynamics and trajectory prediction (1.5K).

1. Visual instruction encoding: Input images $I_v$ with rendered text are processed by Janus-Pro-1B SigLIP vision transformer, then mapped via MLP projector to unified features $X_{fuse} = MLP(SigLIP(I_v)) \in \mathbb{R}^{N \times D}$

2. Latent space mapping: Visual features are encoded to source latent distribution via text-image VAE: $z_0 \sim \mathcal{N}(\bar{\mu}_{z_0}, \text{diag}(\bar{\sigma}^2_{z_0}))$. Target images encoded via frozen LDM VAE: $z_1 = Enc_{img}(I^*)$

3. Dual-Path Spatially-Adaptive Modulation: Novel mechanism with task-specific computational paths. For text-to-image tasks ($I_{edit} = 0$), bypasses cross-attention to follow pure semantic evolution. For image editing ($I_{edit} = 1$), uses cross-attention with source image latents and adaptive gating network predicting token-level weights $\Lambda = \sigma(MLP_\theta([\tilde{H}^{(l)} | \Delta H_{struct}]))$

4. Flow matching backbone: DiT-variant transformer learns continuous velocity field $v_\theta(z_t, t)$ for deterministic transport from visual instruction latents to target image latents, eliminating noise scheduling and diffusion sampling

   The core contribution is unifying all modalities as visual representations and learning intra-modal transport via flow matching, replacing cross-modal alignment with pure visual flows.

1. Pretraining: FlowInOne 1.2B parameters initialized from CrossFlow, trained on VisPrompt-5M at 256×256 resolution for 240k steps using balanced WebDataset sampling

2. Data: 5M visual prompt pairs - 2.86M text-to-image from text-to-image-2M and ImageNet, 1.6M text-in-image editing from GPT-Image-Edit and UnicEdit, 315K structured pairs from PixWizard, 24K text bounding box editing, 250K visual marker editing, 1K doodles editing, 1.5K physics-aware tasks

3. Optimizer: Not specified, likely AdamW based on standard practice

4. Learning rate and schedule: Not reported

5. Batch size: 512 for ablation studies, full training batch size not reported

6. Hardware: Not specified

7. Wall-clock time: Not reported

8. Loss function: Combined Flow matching MSE loss, KL divergence for VAE regularization, and CLIP contrastive loss for semantic alignment

9. Training strategy: Joint training across all 5M samples proves superior to two-stage training (47.8% vs 29.1% pass rate in ablations)

Prior work:

1. Flow Matching (Lipman et al. 2023) - Introduced continuous transport between distributions via velocity field learning, avoiding complex noise scheduling of diffusion models
2. CrossFlow (Liu et al. 2025) - Applied flow matching to cross-modal generation, providing the backbone architecture that FlowInOne builds upon

3. Vision-centric models (Salesky et al. 2021, Xiao et al. 2024) - Showed text can be processed visually by rendering to pixel space, but remained perception-focused

   Delta: This paper reformulates multimodal generation as pure intra-modal visual flow by:

4. rendering all instructions as visual prompts on input canvas
5. learning continuous transport within unified visual latent space

6. introducing Dual-Path Spatially-Adaptive Modulation for task-specific computation paths.

   Applied-specific assessment:

   • Architectural novelty: The visual instruction rendering + flow matching combination is non-obvious. Most prior work uses cross-modal conditioning rather than converting everything to visual space.
   • Benchmark gains: Mixed results - achieves SOTA on own VP-Bench (54.0% vs competitors’ <56.0%) but evaluation is primarily on custom benchmark rather than established datasets like MS-COCO.
   • Fair comparisons: Comparisons include both open-source and commercial models, but evaluation heavily relies on custom metrics and VLM judges which may favor the method’s design philosophy.
   • Scale dependence: Gains likely depend on large-scale VisPrompt-5M dataset construction and 1.2B parameter model - unclear if approach works at smaller scales.

   Verdict: INCREMENTAL — Solid engineering combining existing flow matching with visual instruction rendering, but core ideas are natural extensions of known techniques rather than breakthrough insights.

1. VP-Bench overall pass rate: FlowInOne 54.0% (Gemini-3), 39.2% (GPT-5.2), 50.3% (Qwen3.5), 44.9% (Human) vs Nano Banana 56.0%/30.2%/46.9%/40.6%, open-source models <7.6%

2. Class-to-image generation: FlowInOne 85.0-89.0% vs Nano Banana 60.0-68.0%, open-source models 2.0-27.0%

3. Text-to-image generation: FlowInOne 70.0-80.0% vs Nano Banana 90.4-98.0%, open-source models 2.0-12.0%

4. Text-in-image editing: FlowInOne 7.9-35.5% vs Nano Banana 15.2-42.3%, open-source models 0.0-8.0%

5. Force understanding: FlowInOne 50.0-72.7% vs Nano Banana 12.7-52.0%, open-source models 0.0-2.0%

6. Text bounding box editing: FlowInOne 11.6-30.2% vs Nano Banana 2.3-61.4%, open-source models 0.0-4.7%

7. CLIP-IQA (visual realism): FlowInOne 0.684 vs Nano Banana 0.688, open-source 0.603-0.646

8. DINOv3 Similarity (spatial precision): FlowInOne 48.7% vs Nano Banana 47.3%, open-source 12.5-20.1%

   Results are mixed - FlowInOne leads in most categories but trails commercial baseline on some metrics. Performance on editing tasks remains limited across all methods.

1. Model size: 1.2B parameters for FlowInOne, built on CrossFlow backbone with additional cross-attention layers and gating networks

2. Training compute: Not reported - trained for 240k steps at 256×256 resolution but hardware specifications and total GPU hours not provided

3. Inference speed/latency: Not reported - flow matching typically faster than diffusion sampling but specific timing benchmarks not provided

4. Memory footprint: Not specified for training or inference

5. Deployment practicality: Moderate - 1.2B parameters manageable for deployment but requires frozen LDM VAE encoder/decoder and Janus-Pro-1B visual encoder, increasing overall system complexity. 256×256 resolution limits practical applications requiring higher resolution outputs

1. Evaluation limited to curated VP-Bench and synthetic dataset: no evidence of deployment on real-world production systems or user-generated content

2. Physics-aware capabilities demonstrated on controlled scenarios: force dynamics and trajectory prediction tested on synthetic Blender-rendered videos and controlled force prompting dataset, not real physics simulations

3. No hardware experiments reported: no testing on actual robots, autonomous vehicles, or physical systems despite claims of physics understanding

4. No production integration discussed: paper focuses on benchmark performance rather than real deployment considerations like latency, memory usage, or integration complexity

5. Sim-to-real gap not addressed: physics understanding validated only on synthetic/rendered content without verification on real-world physical systems

   The work remains primarily academic with limited evidence of real-world performance beyond controlled benchmark scenarios.

1. FUNDAMENTAL: Resolution limited to 256×256 during training, requiring significant retraining for higher resolutions needed in practical applications

2. FUNDAMENTAL: Heavy dependence on visual text rendering quality - poorly rendered or illegible text instructions lead to complete task failure

3. EVALUATION: Evaluation primarily on custom VP-Bench with VLM judges that may favor the visual instruction paradigm over traditional text-conditional methods

4. ENGINEERING: Complex pipeline requiring multiple frozen components (LDM VAE, Janus-Pro-1B encoder) increases deployment complexity and memory requirements

5. FUNDAMENTAL: Physics understanding limited to controlled synthetic scenarios - no validation on real-world physical systems or complex dynamics

6. ENGINEERING: Performance gaps in fine-grained editing tasks (7.9-35.5% success rate) indicate need for better spatial control mechanisms

   Failure modes:

7. Visual instruction parsing failures when text is occluded, rotated, or rendered in complex backgrounds leading to complete misunderstanding of task requirements
8. Spatial precision breakdown in multi-object scenes where bounding boxes or arrows overlap, causing confused editing targets and unintended modifications

3D Visual Grounding (3D-VG) addresses the practical need for AI systems to precisely localize objects in 3D environments based on natural language descriptions - critical for human-robot interaction, embodied AI navigation, and AR/VR applications. This is challenging because it requires bridging complex spatial semantics with precise 3D geometry understanding.

The task takes as input a natural language query Q and sequential RGB-D video streams V = {(Ii, Di)}T_i=1 consisting of T frames (where Ii is RGB image and Di is aligned depth map) with camera intrinsics K and extrinsics Tc2w. The output is a 3D bounding box B ∈ R^6 localizing the target object. The objective is to reconstruct the target object without relying on preprocessed 3D point clouds:

\[P_c = D_t(u,v) \cdot K^{-1} \begin{bmatrix} u \\ v \\ 1 \end{bmatrix}\] \[P_{centroid} = \frac{1}{N} \sum_{k=1}^N P_k^w\]

Success is measured by Acc@0.25 and Acc@0.5 (fraction of predicted 3D bounding boxes with IoU > 0.25 and 0.5 against ground truth) on ScanRefer, and top-1 selection accuracy on Nr3D.

The paper evaluates on ScanRefer (categorized as “Unique”/”Multiple” based on same-class distractors) and Nr3D (divided into “Easy”/”Hard” and “View-Dependent”/”Independent” subsets), both built on ScanNet indoor scenes.

The TAB framework follows an agentic ReAct-style paradigm with three main phases:

1. Reference Target Localization: VLM agent parses query into structured JSON format, performs coarse filtering using GroundingDINO to retain frames with target class, then fine filtering via VLM to verify scene constraints, followed by scoring/ranking to select Reference Frame

2. Semantic-Anchored Geometric Expansion: Two-stage expansion mechanism to overcome VLM tracking brittleness - Semantic Temporal Expansion: bidirectional tracking from reference frame using VLM verification and SAM segmentation - Multi-View Geometric Expansion: projects 3D centroid across unobserved frames using camera parameters with visibility checks

3. 2D to 3D Reconstruction: Inverse-projects masked pixels into 3D world coordinates, applies Statistical Outlier Removal and DBSCAN clustering, computes axis-aligned 3D bounding box

   The core architecture uses Qwen3-VL-32B as the primary VLM agent, GroundingDINO for object detection, and SAM3 for instance segmentation. The key technical contribution is the Semantic-Anchored Geometric Expansion mechanism that mathematically projects 3D centroids to acquire complete multi-view coverage, bypassing semantic tracking failures through deterministic geometry.

1. No training required: Framework operates entirely on pre-trained foundation models without any scene-specific training or fine-tuning

2. Foundation model deployment: - VLM: Qwen3-VL-32B (open-source) - Object detector: GroundingDINO - Segmentation: SAM3 - Processing: 300 frames per video from ScanNet sequences - Maximum 32 frames for both Semantic Temporal Expansion and Multi-View Geometric Expansion

3. Hardware requirements: not reported

4. Inference configuration: Visibility check threshold ε set to 0.4 to accommodate depth sensor noise and object thickness

Prior work:

1. SeeGround (Li et al., 2025): Zero-shot 3D grounding using preprocessed point clouds, achieving 44.1% Acc@0.25 on ScanRefer through proposal matching
2. VLM-Grounder (Xu et al., 2025b): Direct 2D image operation but relies on heuristic semantic matching, achieving 51.6% Acc@0.25 on ScanRefer

3. SPAZER (Jin et al., 2025): Static workflow on preprocessed 3D point clouds, achieving 57.2% Acc@0.25 on ScanRefer

   Delta: This paper adds the Semantic-Anchored Geometric Expansion mechanism that projects 3D centroids mathematically to overcome multi-view coverage deficits, and reformulates the task as dynamic agentic reasoning rather than static proposal matching.

   Applied-specific assessment:

   • Architectural idea: The 2D→3D→2D projection strategy is a reasonable engineering solution but not fundamentally novel
   • Benchmark gains: Substantial improvements (71.2% vs 57.2% previous SOTA on ScanRefer) but achieved through better engineering of existing components
   • Comparisons appear fair: same evaluation protocols, though framework uses newer VLM (Qwen3-VL-32B vs older models in baselines)
   • Gains likely depend on powerful VLM reasoning capabilities and careful geometric projection implementation

   Verdict: INCREMENTAL — solid engineering advance combining existing techniques (VLM reasoning + multi-view geometry) with good experimental validation, but no fundamental breakthrough in approach.

1. ScanRefer Overall: Acc@0.25: 71.2% (vs SPAZER 57.2%), Acc@0.5: 46.4% (vs SPAZER 48.8%), improvement of +14.0% and -2.4%

2. ScanRefer Unique: Acc@0.25: 90.2% (vs SPAZER 80.9%), Acc@0.5: 57.6% (vs SPAZER 72.3%), improvement of +9.3% and -14.7%

3. ScanRefer Multiple: Acc@0.25: 60.1% (vs SPAZER 51.7%), Acc@0.5: 39.9% (vs SPAZER 43.4%), improvement of +8.4% and -3.5%

4. ScanRefer with 3D assistance: Overall Acc@0.25: 71.6%, Acc@0.5: 61.6% (substantial boost in Acc@0.5 when using Mask3D proposals)

5. Nr3D Overall: 68.0% accuracy (vs SPAZER 63.8%), improvement of +4.2%

6. Nr3D Easy: 72.1% (vs SPAZER 68.0%), improvement of +4.1%

7. Nr3D Hard: 63.2% (vs SPAZER 58.8%), improvement of +4.4%

8. Nr3D View-Dependent: 62.5% (vs SPAZER 59.9%), improvement of +2.6%

9. Nr3D View-Independent: 71.4% (vs SPAZER 66.2%), improvement of +5.2%

   Results show consistent improvements across most metrics, with particularly strong performance on “Unique” queries but mixed results on precision metrics (Acc@0.5).

1. Model size: Qwen3-VL-32B (32 billion parameters) plus GroundingDINO and SAM3 foundation models

2. Training compute: Zero - no training required, operates on pre-trained models

3. Inference speed/latency: Not reported, but processes 300 frames per video with maximum 32 frames for expansion phases

4. Memory footprint: Not reported, but requires loading large VLM (32B parameters) and multiple vision foundation models

5. Deployment practicality: Moderate - requires powerful hardware for 32B parameter VLM but avoids need for preprocessed 3D point clouds, making it more practical for real-world deployment than methods requiring 3D scene reconstruction

1. Direct RGB-D operation: Works on raw RGB-D video streams without requiring preprocessed 3D point clouds, making it more applicable to real-world scenarios

2. Benchmark data only: Experiments conducted exclusively on ScanNet indoor scenes - no deployment on real robots or vehicles reported

3. No sim-to-real validation: No discussion of transferring from curated datasets to real-world environments

4. Hardware requirements: Requires depth sensors (RGB-D cameras) and significant compute for 32B parameter VLM, which may limit deployment on mobile platforms

5. Indoor scene focus: Evaluation limited to indoor environments - outdoor or more diverse real-world settings not demonstrated

1. Dependency on VLM quality (FUNDAMENTAL): Framework heavily relies on VLM reasoning capabilities - failures in semantic understanding directly impact performance

2. Depth sensor requirements (ENGINEERING): Requires accurate aligned depth maps, limiting deployment to RGB-D equipped systems

3. Computational overhead (ENGINEERING): 32B parameter VLM creates significant inference costs compared to lighter alternatives

4. Indoor scene limitation (EVALUATION): Only validated on indoor ScanNet scenes, generalization to outdoor/diverse environments unclear

5. Camera pose dependency (FUNDAMENTAL): Relies on accurate camera extrinsics for geometric projection - pose estimation errors propagate through system

   Failure modes:

6. Semantic tracking breakdown: When VLM fails to track objects across viewpoint changes, geometric expansion may anchor incorrect centroids
7. Depth noise corruption: Poor depth quality can corrupt initial 3D centroid calculation, leading to failed geometric projections across frames

IGLOSS addresses zero-shot open-vocabulary semantic segmentation (OVSS) of 3D automotive LiDAR point clouds. In autonomous driving, manual 3D annotation is expensive and time-consuming, yet flexibility to segment arbitrary classes (not just predefined ones) is valuable for specialized scenarios, auto-labeling, and data retrieval.

The task takes as input:

1. a 3D LiDAR point cloud from automotive sensors, and

2. free-form text prompts defining classes of interest (e.g., “car”, “pedestrian”, “traffic sign”). The output is a semantic label assignment for each 3D point, where labels correspond to the text-specified classes. The method must work zero-shot, requiring no labeled examples of the target classes.

   The objective is to learn a mapping from 3D point features to text-defined semantic concepts:

   \[\text{point } p \rightarrow \text{class } c \in \{C_1, C_2, \ldots, C_N\}\]

   Success is measured by mean Intersection-over-Union (mIoU) on standard automotive datasets. Evaluation uses nuScenes (16 classes, multi-camera + LiDAR) and SemanticKITTI (19 classes, single camera + LiDAR). The method must handle both “thing” classes (countable objects) and “stuff” classes (amorphous regions).

1. 2D-3D Visual Foundation Model Alignment (ScaLR+): Distill DINOv2 (2D Vision Foundation Model) into WaffleIron-48-768 (3D backbone). Key improvements over original ScaLR: add MLP projection head, drop path regularization, replace ReLUs with GELUs, increase image resolution to 448×896, extend training to 65 epochs.

2. Text-to-Image Prototype Generation: Use text-conditioned image generators (ChatGPT-5, Gemini 2.5) to create 2-3 prototype images per class. For “thing” classes, prompt: “Generate an image of [CLASS] with a white background”. For “stuff” classes: “[CLASS] covering the whole image”. Apply tight cropping to remove white borders.

3. 2D Feature Extraction: Process prototype images through DINOv2, extract patch features at layer used for 2D-3D distillation, normalize individual patches, compute average, normalize resulting 1024-d feature vector per prototype.

4. 3D Point Classification: Extract normalized 3D features from LiDAR points using distilled 3D network. Fit multinomial logistic regression on 2D prototype features, classify 3D points using:

   \[\text{class}(p) = \arg\max_i (\mathbf{W}^\top \mathbf{f}_p^{3D} + \mathbf{b})_i\]

   where $\mathbf{W}$ and $\mathbf{b}$ are learned from prototype features ${\mathbf{f}_j^{2D}}$.

   The core contribution is avoiding the image-text modality gap in CLIP-like models by using image generation as the text-to-visual bridge, combined with stronger 2D vision features from foundation models.

Stage 1 - 2D-3D Distillation (ScaLR+):

1. Data: Multi-dataset training on nuScenes, SemanticKITTI, Panda64, PandaGT with 2D-3D correspondences
2. Optimizer: Not specified, learning rate 10^-4 for finetuning stage, layer-wise decay 0.99
3. Training: 65 epochs (vs 25 in original ScaLR), image resolution 448×896
4. Loss: Similarity loss between 2D DINOv2 features and projected 3D features

5. Hardware/time: Not reported

   Stage 2 - Zero-shot Inference (No Training):

6. Generate 2-3 prototype images per class using text-to-image models
7. Extract DINOv2 features from prototypes (forward pass only)
8. Fit logistic regression on prototype features (~70ms overhead)

9. Classify 3D point features using fitted classifier

   Optional Stage 3 - Self-training (IGLOSSclo):

10. Data: Pseudo-labels from IGLOSS on target dataset, 4D consistency via temporal aggregation
11. Optimizer: Same as Stage 1, 10^-4 learning rate, 10 epochs
12. Voxel-based consistency with 10cm voxels, majority voting within voxels

13. Hardware/time: Not reported

    The method requires no task-specific training data - only the pre-existing 2D-3D foundation model alignment.

Prior Work:

• OpenScene (Peng et al. 2023): Distills CLIP image encoder to 3D network, uses nearest neighbor matching between CLIP text features and 3D point features
• OVDiff (Karazija et al. 2024): 2D method using Stable Diffusion to generate prototype images, applies CutLER for background removal, uses complex part-level clustering
• ScaLR (Puy et al. 2024): 2D-3D distillation of DINOv2 into 3D networks for general representation learning

Delta: IGLOSS combines text-to-image generation with 2D-3D foundation model distillation in a novel way:

1. Uses image generation instead of CLIP text encoder to bridge text-visual gap
2. Applies logistic regression instead of nearest neighbor for classification

3. Improves 2D-3D distillation with ScaLR+.

   Assessment:

   • Architectural novelty: Moderate - combines existing components (text-to-image, 2D VFM distillation) but the specific combination avoiding CLIP’s modality gap is non-obvious
   • Benchmark gains: Significant on SemanticKITTI (+5.6 mIoU), modest on nuScenes (+1.4 mIoU without ensembling)
   • Fair comparisons: Mostly fair, though some baselines use different 2D-3D distillation methods and datasets
   • Scale dependence: Relies on strong foundation models (DINOv2, ChatGPT) but doesn’t require massive proprietary training

   Verdict: SIGNIFICANT — The insight to avoid CLIP’s text-image alignment issues via image generation is valuable and the results consistently improve over strong baselines across datasets.

1. nuScenes validation (open-vocabulary): IGLOSS achieves 47.5% mIoU vs previous SOTA GGSD 46.1% (+1.4%), IGLOSSmix with ensembling achieves 51.1% vs SAS 47.5% (+3.6%)

2. SemanticKITTI validation (open-vocabulary): IGLOSS achieves 34.3% mIoU vs previous best SAL 28.7% (+5.6% improvement), many methods don’t report SemanticKITTI results

3. nuScenes validation (closed-set after self-training): IGLOSSclo achieves 49.6% mIoU vs LOSC 49.3% (+0.3%), IGLOSSmix,clo achieves 54.1% vs AFOV 47.9% (+6.2%)

4. SemanticKITTI validation (closed-set after self-training): IGLOSSclo achieves 39.4% mIoU vs LOSC 35.2% (+4.2%), AdaCo 25.7%

   Results are mixed - strong improvements on SemanticKITTI but more modest gains on nuScenes. The method shows consistent improvements across both open-vocabulary and closed-set settings. Notably absent: results on other automotive datasets like KITTI-360 or A2D2, and computational efficiency comparisons during inference.

1. Model size: ScaLR+ uses WaffleIron-48-768 backbone (768-dim features, 48M parameters estimated), DINOv2 has ~300M parameters for feature extraction

2. Training compute: 2D-3D distillation training details not fully specified, extends original ScaLR from 25 to 65 epochs with higher resolution (448×896 vs 224×448)

3. Inference speed: Logistic regression fitting adds ~70ms overhead per query, becomes faster than nearest neighbor for repeated queries or many prototypes. Image generation time not quantified but noted as slower than text embedding

4. Memory footprint: 1024-d features per prototype, lightweight logistic regression classifier, full 3D point cloud processing

5. Deployment practicality: Requires access to text-to-image generation models (ChatGPT, Gemini), 2D foundation model (DINOv2), and 3D distilled network. Method designed for offline applications like auto-labeling rather than real-time vehicle deployment. Benefits from faster emerging image generators.

1. Automotive dataset evaluation: Tested on nuScenes (real urban driving in Boston/Singapore with 6 cameras + LiDAR) and SemanticKITTI (real highway/urban driving in Germany with single camera + LiDAR)

2. Cross-dataset generalization: Single ScaLR+ model works across both nuScenes and SemanticKITTI without dataset-specific retraining, unlike some baselines

3. Practical driving scenarios: Shows segmentation of typically ignored classes like wheelchair, stroller, crosswalk that aren’t in standard 16 nuScenes classes, demonstrating open-vocabulary flexibility

4. Production considerations: Method designed for offline applications (auto-labeling, data retrieval) rather than real-time autonomous driving. Requires cloud access to image generation models

5. Sim-to-real discussion: Not addressed - method focuses on real sensor data from automotive datasets

   No reported deployment on actual vehicles or robotics platforms, but demonstrates clear applicability to real-world automotive perception tasks through comprehensive evaluation on standard driving datasets.

1. Image generation dependency (ENGINEERING): Requires access to high-quality text-to-image models; performance varies significantly across generators (ChatGPT > Gemini > Web > Flux)

2. Generation time overhead (ENGINEERING): Image generation slower than forward pass for text embeddings, though mitigated by needing only 2-3 images per class

3. Limited evaluation scope (EVALUATION): Only tested on automotive datasets; unclear how method generalizes to indoor scenes, aerial imagery, or other 3D domains

4. Foundation model dependence (FUNDAMENTAL): Performance tied to quality of underlying 2D-3D distillation and image generation capabilities

5. Prompt sensitivity (ENGINEERING): Requires explicit handling of broad classes (manmade → building, wall, pole subclasses) and negative definitions (other flat → traffic island)

6. Single modality assumption (FUNDAMENTAL): Unlike some baselines, doesn’t leverage complementary camera images or temporal sequences at test time

   Failure modes:

   • Poor prototype generation for abstract or uncommon classes where text-to-image models struggle
   • Misalignment when 3D point features don’t match well with 2D prototype features due to viewpoint, lighting, or occlusion differences

This work addresses the challenge of training multimodal large language models (MLLMs) on diverse visual tasks with highly heterogeneous reward structures. The practical need arises from the fact that visual tasks exhibit extreme variance in reward topologies—from sparse binary rewards in math visual question answering to dense continuous scores in grounding tasks—creating severe training instabilities when optimizing jointly.

Task Definition: The input consists of visual-textual query pairs across six task categories: general VQA, math VQA, chart understanding, spatial reasoning, document understanding, and visual grounding. The model must produce text responses of varying lengths and formats depending on task requirements. The training objective combines the standard GRPO loss with novel advantage normalization:

\[\mathcal{L}_{G^2RPO} = \mathbb{E}_{q\sim\mathcal{D}, \{y_i\}^G_{i=1}\sim\pi_{\theta_{old}}} \left[ \frac{1}{G} \sum_{i=1}^G \frac{1}{|y_i|} \sum_{t=1}^{|y_i|} \min\left( r_{i,t}(\theta) \hat{A}_i, \text{clip}(r_{i,t}(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_i \right) \right]\]

where advantages are computed via optimal transport mapping:

\[\hat{A}_i = \Phi^{-1}(F_{R_\tau}(R_i))\]

Evaluation Criteria: Performance is measured across 18 benchmarks using task-specific metrics including accuracy (MMMU, MMBench), IoU scores (RefCOCO variants), and specialized metrics like OCRBench scores.

Dataset: The model is trained on a filtered subset of the OneThinker-600k dataset covering the six task categories mentioned above.

1. Base Architecture: Qwen3-VL-Instruct-8B serves as the foundation model with standard vision-language architecture.

2. Gaussian GRPO (G²RPO): Core contribution that replaces linear advantage normalization with non-linear distributional matching via 1D Optimal Transport. For each task τ with rewards R_τ = {R₁, …, R_N}, advantages are computed by mapping empirical reward distribution to N(0,1):

   \[\Psi(R_i, R_\tau) = \Phi^{-1}(F_{R_\tau}(R_i))\]

3. Task-level Response Length Shaping: Applies trapezoidal reward envelope based on task-specific length thresholds (L_min, L_low, L_high, L_max):

   \[R_{length}(y) = \begin{cases} 0, & |y| < L_{min} \text{ or } |y| > L_{max} \\ \frac{|y|-L_{min}}{L_{low}-L_{min}}, & L_{min} \leq |y| < L_{low} \\ 1, & L_{low} \leq |y| \leq L_{high} \\ \frac{L_{max}-|y|}{L_{max}-L_{high}}, & L_{high} < |y| \leq L_{max} \end{cases}\]

4. Task-level Entropy Shaping: Regularizes exploration via margin-based penalty:

   \[L_{ent\_reg} = \max(0, H_{task} - H_{max}) + \max(0, H_{min} - H_{task})\]

   The core technical contribution is using optimal transport to enforce Gaussian topology on advantage distributions, theoretically ensuring inter-task gradient equity and robustness to outliers.

1. Hardware: AWS Trainium instances (Trn1.32xlarge), approximately 3 days training time

2. Optimization: Single epoch training with AdamW optimizer, batch size 128, learning rate 2×10⁻⁶, maximum generation length 4096 tokens

3. Data: Filtered subset of OneThinker-600k dataset with dynamic filtering to discard uniformly correct/incorrect rollouts

4. Initialization: Qwen3-VL-Instruct-8B base model

5. Regularization: KL regularization disabled, following practices from Yu et al. (2025a)

6. Specific implementation details: G²RPO advantage computation uses closed-form CDF mapping with tie-breaking strategy for identical rewards

   Not reported: Exact dataset size after filtering, detailed hardware specifications beyond instance type, specific hyperparameter values for length and entropy shaping bounds per task.

Prior Work:

1. Group Relative Policy Optimization (GRPO) by Guo et al. (2025) - established the base RL objective for MLLM training but suffers from inter-task imbalance due to reward variance differences
2. EMA-GRPO by Feng et al. (2025b) - attempted to address multi-task imbalance using exponential moving averages of task-specific variance but still relies on linear scaling

3. Dr.GRPO by Liu et al. (2025b) - removed sample-wise normalization but created inter-task dominance issues

   Delta: This paper replaces linear moment-matching with non-linear distributional matching via 1D Optimal Transport, mathematically forcing each task’s advantage distribution to converge to N(0,1).

   Applied-Specific Assessment:

   • Architectural novelty: The core idea of using optimal transport for advantage normalization is genuinely novel and non-obvious. While OT has been used in LLMs for preference alignment, applying it specifically to multi-task RL advantage computation is new.
   • Benchmark gains: Improvements are substantial across diverse tasks (e.g., +18.9% on MMMU relative improvement), though absolute gains vary. The consistency across 18 benchmarks is impressive.
   • Fair comparisons: Controlled experiments using same base model, training data, and compute resources strengthen the claims. Comparisons include both reproduced baselines and external models.
   • Generalizability: The method addresses a fundamental statistical issue that should generalize beyond the specific training setup, though scalability to larger models remains unverified.

   Verdict: SIGNIFICANT — The optimal transport approach to advantage normalization is a clear non-obvious advance that addresses a real bottleneck in multi-task MLLM training, with substantial empirical validation.

1. MMMU: 71.6% (vs 60.2% Qwen3-VL baseline, 70.7% GPT-4o) - surpasses GPT-4o
2. MMBench: 88.2% (vs 85.1% baseline, 84.3% GPT-4o) - new SOTA among compared models
3. MMStar: 73.8% (vs 68.5% baseline, 65.1% GPT-4o) - significant improvement
4. MathVista: 79.5% (vs 74.2% baseline, 63.8% GPT-4o) - large margin over GPT-4o
5. MathVerse: 65.8% (vs 58.1% baseline, 41.2% GPT-4o) - substantial gain
6. MathVision: 53.4% (vs 45.4% baseline, 30.4% GPT-4o) - notable improvement
7. AI2D: 87.5% (vs 82.3% baseline, 84.9% GPT-4o) - competitive performance
8. ChartQA: 87.4% (vs 82.8% baseline, 86.7% GPT-4o) - slight improvement
9. CharXiv: 53.0% (vs 44.5% baseline, 47.1% GPT-4o) - good improvement
10. OCRBench: 911 (vs 819 baseline, 810 GPT-5) - exceeds frontier models
11. DocVQA: 96.7% (vs 95.3% baseline, 91.5% GPT-5) - strong document understanding
12. InfoVQA: 86.4% (vs 83.1% baseline, 79.0% GPT-5) - consistent improvement
13. EmbSpatial: 83.1% (vs 78.2% baseline, 82.9% GPT-5) - competitive spatial reasoning
14. RefSpatial: 44.6% (vs 43.9% baseline, 23.8% GPT-5) - solid performance
15. RoboSpatial: 63.2% (vs 60.6% baseline, 53.5% GPT-5) - good spatial skills
16. RefCOCO: 93.4% (vs 89.9% baseline) - excellent grounding performance
17. RefCOCO+: 88.2% (vs 84.5% baseline) - strong localization

18. RefCOCOg: 90.4% (vs 86.8% baseline) - consistent grounding gains

    Results show consistent improvements across all benchmarks with particularly strong performance in math reasoning and document understanding tasks.

1. Model size: 8B parameters (Qwen3-VL-Instruct-8B base)

2. Training compute: AWS Trainium Trn1.32xlarge instances, approximately 3 days total training time, single epoch with batch size 128

3. Inference speed/latency: Not reported

4. Memory footprint: Not reported beyond maximum generation length of 4096 tokens

5. Deployment practicality: Model appears deployable given 8B parameter size, though specific inference requirements not detailed. Training efficiency seems reasonable at 3 days on cloud hardware.

1. Benchmark-focused evaluation: The paper primarily evaluates on standard academic benchmarks rather than real-world deployment scenarios.

2. Task diversity: Covers practically relevant tasks including document understanding (OCR, InfoVQA, DocVQA), visual grounding (RefCOCO variants), and spatial reasoning that could apply to robotics.

3. No deployment results: No reported integration into production systems or real-world applications.

4. Limited sim-to-real discussion: While spatial reasoning benchmarks (EmbSpatial, RoboSpatial) suggest robotics applicability, no actual robot experiments or sim-to-real transfer analysis.

5. Academic benchmark focus: Strong performance on established benchmarks suggests utility for researchers and developers, but practical deployment remains unverified.

1. Hyperparameter sensitivity - ENGINEERING: Method introduces task-specific hyperparameters (L_min, L_low, L_high, L_max, H_min, H_max) that require manual tuning per task category.

2. Scalability questions - ENGINEERING: Training limited to 8B model on filtered dataset subset; unclear if benefits hold at larger scales or with full datasets.

3. Limited theoretical analysis - EVALUATION: While claiming theoretical guarantees of inter-task gradient equity, lacks rigorous mathematical proof of convergence properties.

4. Task categorization dependency - FUNDAMENTAL: Approach requires a priori knowledge of task types to apply appropriate length/entropy shaping, limiting generalizability.

5. Computational overhead - ENGINEERING: Optimal transport computation and task-specific shaping may add training overhead not fully quantified.

   Failure Modes:

   • Incorrect task categorization: If tasks are misclassified (e.g., treating reasoning-heavy task as vision-centric), inappropriate length/entropy bounds could hurt performance
   • Distribution shift: G²RPO’s Gaussian assumption may break down with significantly different reward distributions during deployment vs. training

Multi-turn agentic reasoning in long-horizon, stochastic environments presents a critical challenge where agents must continuously interact with environments that provide noisy feedback. This is particularly problematic in search-augmented question answering and embodied tasks where suboptimal actions lead to misleading observations, creating a vicious cycle that degrades subsequent reasoning performance.

The task involves agents taking sequential actions $a_t$ in response to observed states $s_t$, where each action generates environmental feedback that becomes part of the cumulative context. The environment provides terminal rewards $r(\tau) = 1$ for successful completion and $r_t = 0$ for intermediate steps. Input modalities include textual instructions and environmental observations, while outputs are textual actions (search queries, navigation commands).

Success is measured by task completion accuracy on benchmarks including MuSiQue, WebWalkerQA, GAIA, Frames, Bamboogle (for search tasks), and ALFWorld (for embodied reasoning). The paper evaluates both in-distribution and out-of-distribution performance using LLM-as-judge for QA tasks and environment-determined success for embodied tasks.

Evaluation uses curated training sets (4000 HotPotQA samples for search tasks, 3553 ALFWorld training configurations) with test sets ranging from 125 questions (Bamboogle) to 2500 questions (MuSiQue).

1. Process-Level Critic: A critic model $\phi$ evaluates each action step, outputting integer score $l_t$ and textual critique $c_t$ based on trajectory history $\tau_t$, action $a_t$, and subsequent observation $s_{t+1}$: $l_t, c_t = \phi(\tau_t, a_t, s_{t+1})$.

2. Adverse Step Detection: Actions are deemed suboptimal when critic score falls below threshold: $l_t \leq l_{th}$, triggering intervention to break error propagation cycles.

3. Refined Demonstration Generation: Upon detecting adverse steps, a refining policy $\mu$ generates improved actions: $a’_t = \mu(\tau_{t-1}, a_t, l_t, c_t)$, replacing original actions to prevent contextual contamination.

4. Model-Agnostic Framework: Both critic and refiner can use external models or the policy model itself ($\pi_\theta$), enabling self-contained operation without external dependencies.

5. Real-Time Intervention: The system rewinds to the problematic step and replaces it with refined demonstrations before continuing rollout, actively intervening rather than passively filtering completed trajectories.

1. Data Collection: Groups collected via ProCeed rollouts (50% of each group) combined with direct policy sampling (50% of group) to maintain on-policy references for group-based RLVR.

2. Training Algorithm: DAPO optimization with batch size 32, group size 8, learning rate 1e-6, temperature 0.7. For some experiments, additional SFT on correct ProCeed samples.

3. Policy Optimization: Uses chord-φ weighting to handle distributional shift between refined demonstrations and on-policy samples:

   \[\sigma(d_{j,t}) = \pi_\theta(d_{j,t}|x, d_{j,<t}) \cdot (1 - \pi_\theta(d_{j,t}|x, d_{j,<t}))\]

4. Hardware: Nvidia H100 and A100-80G GPUs, approximately 10 GPU-months total compute budget.

5. Masking Strategy: Demonstration steps in failed trajectories are masked during optimization to prevent off-policy instability.

Prior Work:

1. “Let’s verify step by step” (Lightman et al., 2024) - introduced process reward models for step-level supervision in mathematical reasoning
2. “DeepSeekMath: Pushing the limits of mathematical reasoning” (Shao et al., 2024) - established RLVR framework for single-turn reasoning tasks

3. “Search-R1” (Jin et al., 2025) - applied RLVR to agentic reasoning with outcome rewards

   Delta: This paper specifically addresses the “vicious cycle” problem in multi-turn agentic tasks where suboptimal actions compound through environmental feedback. The key addition is real-time intervention during rollout rather than post-hoc filtering.

   Applied-Specific Assessment:

   • Architectural novelty: The real-time critic-intervention mechanism is a reasonable extension of existing process supervision, but the specific application to breaking error propagation cycles is somewhat novel
   • Benchmark gains: Improvements are meaningful (3.72% average on deep search, 10%+ on ALFWorld) and consistent across multiple tasks
   • Fair comparisons: Comparisons control for model architecture and training data, though computational overhead makes direct comparison complex
   • Scale dependence: Method works with self-critic, suggesting gains don’t purely depend on larger external models

   Verdict: INCREMENTAL — Solid engineering contribution applying process supervision to multi-turn agentic tasks with meaningful gains, but builds incrementally on established RLVR and process supervision paradigms.

1. Bamboogle: ProCeedRL 73.87%, previous baseline (DAPO) 70.83%, improvement +3.04%
2. MuSiQue: ProCeedRL 29.52%, previous baseline (DAPO) 23.60%, improvement +5.92%
3. Frames: ProCeedRL 46.42%, previous baseline (DAPO) 43.59%, improvement +2.83%
4. GAIA: ProCeedRL 13.79%, previous baseline (DAPO) 10.10%, improvement +3.69%
5. WebWalkerQA: ProCeedRL 23.01%, previous baseline (DAPO) 19.51%, improvement +3.50%
6. ALFWorld (in-distribution): ProCeedRL 51.43%, DAPO baseline 45.23%, improvement +6.20%
7. ALFWorld (out-of-distribution): ProCeedRL 55.22%, DAPO baseline 53.24%, improvement +1.98%

8. ALFWorld with SFT: ProCeedSFT achieves 57.14% (in-dist) and 58.95% (out-of-dist), representing 10%+ improvements over base models

   Results show consistent improvements across benchmarks, with particularly strong gains on complex multi-hop reasoning tasks like MuSiQue.

1. Model size: Experiments conducted on Qwen3-1.7B and Qwen3-8B parameter models
2. Training compute: Approximately 10 GPU-months on Nvidia H100 and A100-80G GPUs (2 GPUs for 1.7B, 8 GPUs for 8B models)
3. Inference overhead: ProCeed trajectory costs ~2.5x vanilla samples for 8B model, ~1.8x for 1.7B model due to critic generation
4. Memory footprint: Not explicitly reported, but uses standard transformer architectures
5. Deployment practicality: Method can operate in self-contained mode using policy model as critic, eliminating external model dependencies at test time

1. Search environment integration: Tested with commercial search engine (You.com) and local Wikipedia retriever, showing robustness across different information retrieval systems
2. Embodied simulation: ALFWorld provides household environment simulation with PDDL solver verification, though not real robot deployment
3. Environmental noise analysis: Empirically demonstrates sensitivity to environmental feedback quality, validating real-world applicability concerns
4. No production deployment: Paper lacks discussion of deployment in production systems or real-world robotic platforms
5. Simulation-to-reality gap: While ALFWorld provides structured environments, transfer to unstructured real-world scenarios remains untested

1. ENGINEERING: Computational overhead (~2x generation cost) compared to vanilla exploration, though offset by improved efficiency
2. FUNDAMENTAL: No theoretical guarantee of improvement - relies on LLM’s internal knowledge to identify and fix suboptimal actions
3. ENGINEERING: Requires careful threshold calibration ($l_{th}$) across different tasks and critic capabilities
4. EVALUATION: Limited to simulated environments and structured benchmarks rather than open-ended real-world deployment

5. FUNDAMENTAL: Excessive rewinding can disrupt reasoning flow and replace adequate actions with suboptimal ones

   Failure modes:

   • Critic misjudgment: Low-quality critic scores may trigger unnecessary refinements or miss genuine errors
   • Refinement degradation: Refined actions may be worse than originals, particularly for already high-quality steps