Applied AI Digest — Mar 23, 2026

Autonomous driving requires reliable reasoning over fine-grained 3D scene facts, including object identities, motion states, and spatial relations. Fine-grained question answering over multi-modal driving observations provides a natural way to evaluate this capability, yet existing perception pipelines and driving-oriented large language model methods still suffer from unreliable scene facts, hallucinations, opaque reasoning, and heavy reliance on task-specific training.

1. Task definition: Given a temporal window of multi-modal sensor observations (multi-view cameras and LiDAR point clouds), the system must answer factual queries about the current scene. Input includes synchronized sensor streams and natural language questions. Output is structured answers (yes/no, counts, categorical labels) or sentence-form responses.

2. Questions span five categories: existence (“Are there any parked trucks?”), counting (“How many vehicles are approaching?”), object query (“What is the thing to the front of the pedestrian?”), status query, and comparison (“Does the car have the same status as the bus?”).

3. Evaluation criteria: Exact match accuracy for structured answers on NuScenes-QA, and SPICE/METEOR/CIDEr metrics for sentence-level responses on GVQA.

4. NuScenes-QA contains 39,704 factual reasoning questions across five categories. GVQA contains 104,030 QA tasks with richer dimensional coverage beyond basic facts.

KLDrive consists of two tightly coupled components:

1. Energy-based scene fact construction module that consolidates multi-source evidence (camera-only detector RayDN, LiDAR-only FocalFormer3D, fusion-based IS-Fusion) into a reliable scene knowledge graph through consensus-aware pooling and temporal recovery.

2. Energy-based refinement using structured inference over candidate sets with energy function:

   \[E(\tilde{Y}_\tau, z_\tau) = \sum_i z_\tau^i E_{keep}(i) + \sum_{i<j} z_\tau^i z_\tau^j E_{pair}(i,j) + \sum_i z_\tau^i E_{attr}(i) + \sum_i z_\tau^i E_{sup}(i)\]

3. Scene knowledge graph generation from refined entities with compact relational operators.

4. LLM agent (Qwen3-7B) that performs fact-grounded reasoning over constrained action space (Resolve, RelSelect, Count, Exists, GetType, etc.) under Plan-Execute-Observe loop.

5. Structured prompting with few-shot in-context exemplars enables adaptation without task-specific fine-tuning.

1. Multi-source detector training: RayDN, FocalFormer3D, and IS-Fusion trained on NuScenes-QA/GVQA training splits using standard protocols (not detailed in paper).

2. Energy-based model parameters: Estimated using binary supervision on pooled candidates from training data. Specific optimizer, learning rate, batch size not reported.

3. LLM reasoning agent: Uses frozen Qwen3-7B backbone with no gradient updates. Controlled solely through few-shot in-context learning with structured system prompt and paradigmatic exemplars.

4. No task-specific fine-tuning of the LLM component - adaptation achieved through prompt engineering and in-context learning only.

5. Training data split: 8:2 ratio for training/test on both benchmarks. Hardware: AMD Ryzen Threadripper 9985WX CPU, 768GB RAM, NVIDIA RTX A6000 GPUs. Wall-clock time not reported.

Step 1 — Prior work: DriveLM (2024) introduced driving-oriented LLM frameworks with task-specific encoders and large-scale driving data training. LiDAR-LLM (2024) and MAPLM (2024) represent recent SOTA for multimodal driving scene understanding with similar LLM-based approaches. These methods achieved ~55-60% accuracy on driving QA benchmarks.

Step 2 — Delta: This paper adds (1) energy-based multi-source evidence consolidation into scene knowledge graphs, (2) constrained action space for LLM reasoning with explicit tool calls, and (3) fact-grounded interpretable reasoning without task-specific training.

Step 3 — Applied-specific assessment:

• Architectural idea: Energy-based refinement for multi-source perception fusion is known technique; constraining LLM action space is incremental application of tool-use paradigms
• Benchmark gains: 65.04% vs 60.17% overall (+4.87 points) is meaningful; counting improvement of +46.01 points is substantial but may reflect baseline weaknesses on this specific task
• Comparisons appear fair with same evaluation protocols, though baselines use task-specific training while this method does not
• Gains likely depend on quality of underlying detectors and specific prompt engineering

Verdict: INCREMENTAL — Solid engineering of known techniques (energy models, tool-constrained LLMs) for driving scene reasoning with meaningful but not transformative improvements.

1. NuScenes-QA overall accuracy: KLDrive 65.04%, previous SOTA (MAPLM) 60.17%, improvement +4.87 points

2. NuScenes-QA counting accuracy: KLDrive 64.46%, previous best 18.45%, improvement +46.01 points

3. NuScenes-QA existence accuracy: KLDrive 74.67%, MAPLM 79.29%, KLDrive underperforms by 4.62 points

4. GVQA SPICE score: KLDrive 42.45%, previous best (DriveLM) 42.16%, improvement +0.29 points

5. GVQA METEOR score: KLDrive 0.41, DriveLM 0.42, KLDrive slightly underperforms

6. GVQA CIDEr score: KLDrive 3.48, DriveLM 3.53, KLDrive slightly underperforms

7. KLDrive-GT (ground truth perception): 84.49% overall accuracy on NuScenes-QA, showing reasoning capability ceiling

   Results are mixed - strong on counting, competitive overall, but not consistently superior across all metrics.

1. Model size: Qwen3-7B backbone (7 billion parameters) plus three perception models (RayDN, FocalFormer3D, IS-Fusion) - total parameters not specified

2. Training compute: Not reported for energy model training. No training required for LLM component (frozen backbone).

3. Inference speed: Average 1.26s per question on NVIDIA RTX A6000, ranging from 0.71s (existence) to 2.95s (comparison). Edge deployment: 57.05s average on NVIDIA Jetson Orin NX.

4. Memory footprint: Not reported, but requires loading multiple perception models plus 7B LLM backbone.

5. Deployment practicality: Current latency makes it suitable for offline analysis rather than real-time autonomous driving. Authors acknowledge this limitation and position work for offline fine-grained reasoning rather than millisecond-level online response.

1. Evaluation on real-world NuScenes dataset with actual multi-view camera and LiDAR data from autonomous vehicles.

2. No deployment results or integration with actual autonomous driving systems reported.

3. No hardware experiments beyond inference speed measurements on RTX A6000 and Jetson Orin NX.

4. Authors explicitly focus on offline analysis setting rather than real-time deployment, stating “efficiency optimization for real-time deployment can be pursued as important direction for future work.”

5. System tested on curated benchmark data rather than continuous real-world deployment scenarios.

1. FUNDAMENTAL: System limited to offline analysis due to latency (1.26s average, up to 57s on edge devices) - cannot support real-time autonomous driving decisions.

2. ENGINEERING: Performance heavily dependent on quality of underlying perception models - ground truth analysis shows accuracy jumps from 65.04% to 84.49% with perfect perception.

3. ENGINEERING: Multi-entity compositional reasoning remains challenging, especially counting tasks under complex spatial relations.

4. EVALUATION: Evaluation limited to curated benchmarks rather than continuous real-world scenarios.

5. FUNDAMENTAL: Constrained action space may limit handling of novel query types not covered in prompt engineering.

   Failure modes:

6. System will fail when underlying perception models miss critical objects
7. Complex multi-hop reasoning chains may accumulate errors despite structured approach.

Repository-level code understanding is essential for automating complex software engineering tasks, yet current benchmarks fail to evaluate genuine codebase exploration capabilities. Most existing evaluations rely on popular repositories where Large Language Models (LLMs) can answer questions through memorized knowledge rather than actual code navigation.

The task is repository-level question answering: given a code repository and a natural language question about its structure, functionality, or implementation details, the system must explore the codebase interactively to provide accurate, grounded answers. Input consists of executable Python repositories and questions requiring multi-file reasoning. Output is structured answers with explicit file path citations and line number references.

Success is measured using LLM-as-a-Judge evaluation across five dimensions: correctness, completeness, relevance, clarity, and reasoning quality. Each dimension is scored 1-10, with the objective function being:

\[\text{Score} = w_1 \cdot \text{Correctness} + w_2 \cdot \text{Completeness} + w_3 \cdot \text{Relevance} + w_4 \cdot \text{Clarity} + w_5 \cdot \text{Reasoning}\]

where $w = (0.3, 0.2, 0.2, 0.1, 0.2)$ emphasizes correctness while downweighting clarity to discourage fluent but incorrect answers.

SWE-QA-Pro introduces a benchmark with 260 questions across 26 long-tail repositories, constructed through issue-driven clustering to ensure topical balance across 48 semantic categories, with rigorous difficulty calibration to filter out questions solvable by direct answering without codebase interaction.

1. Benchmark construction pipeline with four stages: (i) hierarchical K-means clustering of 1.6M GitHub issues using Qwen3-8B embeddings to create 48 semantic task categories, (ii) Claude Code synthesis of QA pairs from clustered issues across 1,484 repositories, (iii) difficulty calibration using cross-model agreement to filter questions solvable without repository interaction, (iv) human validation and answer refinement.

2. SWE-QA-Pro Agent: ReAct-based workflow abandoning RAG retrieval for direct repository exploration using three tools: Search (keyword-based file location), View (scoped file inspection), CommandLine (read-only structural analysis). Agent iteratively reasons, acts, and observes until sufficient evidence is gathered.

3. Two-stage training recipe: Supervised Fine-Tuning on 1,000 tool-invocation trajectories generated by Claude Sonnet 4.5, followed by Reinforcement Learning from AI Feedback (RLAIF) using 464 additional questions. RLAIF reward computed as:

   \[r = w^T s / 10\]

   where $s \in [1,10]^5$ represents LLM-as-Judge scores across five evaluation dimensions and $w = (0.3, 0.2, 0.2, 0.1, 0.2)$. Policy optimization uses GRPO algorithm with normalized rewards.

4. Difficulty calibration methodology using standardized cross-model agreement:

   \[z_m(q) = \frac{\bar{s}_m(q) - \mu_m}{\sigma_m}\] \[\text{Difficulty}(q) = -\frac{1}{|M|} \sum_{m \in M} z_m(q)\]

   Questions with high direct-answer consensus are filtered out to isolate cases requiring genuine codebase interaction.

1. Supervised Fine-Tuning stage: 1,000 multi-turn tool-invocation trajectories generated by Claude Sonnet 4.5, trained on Qwen3-8B using AdamW optimizer, learning rate 5×10⁻⁶, cosine schedule with 0.05 warmup ratio, batch size 1 per device with 2 gradient accumulation steps, 4 epochs, bfloat16 precision, 32K context window.

2. Reinforcement Learning stage: 464 questions with Claude Code reference answers, GRPO algorithm, actor learning rate 1×10⁻⁶, 8 rollouts per question, temperature 1.0, KL coefficient 0.02, batch size 8, 32K max model length, FSDP strategy.

3. Data construction: Deduplicated 1,464 questions from benchmark construction pipeline, randomly split into 1K for SFT and 464 for RL. SFT data consists of tool-augmented conversation trajectories. RL uses LLM-as-Judge reward distinct from evaluation judge to mitigate reward hacking.

4. Hardware and training time: Not explicitly reported for training duration. Inference conducted on NVIDIA A100 80GB GPUs with temperature 0, maximum 25 turns, 32K context window. Training implemented using SWIFT framework for SFT and Verl-Tool for RL.

Step 1 — Prior work:

• SWE-QA (Peng et al., 2025): Repository-level QA benchmark with 576 questions, but lacks semantic diversity and includes many questions solvable without codebase interaction
• LongCodeQA (Rando et al., 2025): 443 questions using long context windows for large codebase ingestion, but no explicit tool usage or agentic exploration
• SWE-Bench (Jimenez et al., 2023): Focuses on code generation and bug fixing rather than understanding, uses popular repositories vulnerable to memorization

Step 2 — Delta: This paper adds (1) systematic difficulty calibration using cross-model agreement to filter memorization-solvable questions, (2) issue-driven clustering for balanced semantic coverage across 48 categories, (3) focus on long-tail repositories with executable environments, (4) explicit agentic training recipe combining SFT and RLAIF for tool usage learning.

Step 3 — Applied-specific assessment:

• Architectural novelty: The difficulty calibration methodology is a solid engineering contribution but not architecturally novel—filtering easy questions is a reasonable validation step
• Benchmark gains: Shows meaningful 13-point gap between direct vs agentic approaches, and trained Qwen3-8B surpasses GPT-4o by 2.3 points, indicating genuine improvement
• Fair comparisons: Uses consistent evaluation protocol across models with proper anonymization and multiple runs for variance reduction
• Generalizability concerns: Limited to 260 questions from 26 repositories, Python-only due to executable environment requirements, and RLAIF training objectives mirror evaluation metrics creating potential reward hacking

The semantic clustering and difficulty calibration represent solid engineering advances, but the core insight—that repository QA requires filtering memorizable questions—is somewhat obvious in hindsight. The agentic training recipe shows clear empirical benefits but follows established SFT→RL patterns.

Verdict: INCREMENTAL — Solid benchmark contribution with reasonable training recipe, but the core innovations are expected extensions of known approaches rather than breakthrough insights.

1. SWE-QA-Pro overall scores (agent setting): Claude Sonnet 4.5 (40.67), DeepSeek V3.2 + Agent (38.69), Gemini 2.5 Pro + Agent (39.46), GPT-4.1 + Agent (38.47), Devstral-Small-2-24B + Agent (37.30), SWE-QA-Pro-8B (SFT+RL) + Agent (35.39), SWE-QA-Pro-8B (SFT) + Agent (34.34), GPT-4o + Agent (33.08), Qwen3-32B + Agent (32.08), Qwen3-8B + Agent (30.03), LLaMA-3.3-70B + Agent (23.73).

2. Direct answer vs agent gaps demonstrate benchmark difficulty: Claude Sonnet 4.5 shows 12.98 point improvement (27.69→40.67), GPT-4o shows 6.50 point improvement (26.58→33.08), confirming questions require genuine codebase interaction.

3. Correctness dimension (agent setting): Claude Sonnet 4.5 (7.34), DeepSeek V3.2 (6.49), GPT-4.1 (6.86), Gemini 2.5 Pro (7.12), showing SWE-QA-Pro-8B (SFT+RL) achieves 5.96, competitive with much larger models.

4. Training strategy comparison on same backbone: SFT-1000 + RL-464 outperforms SFT-1464 alone, demonstrating RL provides complementary supervision beyond scaling supervised data.

5. Tool usage analysis shows positive correlation between tool call frequency and performance, with Claude Sonnet 4.5 using highest volume of tool calls while maintaining efficiency.

   Results are consistently strong across evaluation dimensions, with clear separation between models that can and cannot effectively use tools for repository exploration.

1. Model size: Qwen3-8B (8 billion parameters) for trained model, comparison baselines range from 8B to 70B parameters.

2. Training compute: Training conducted on NVIDIA A100 80GB GPUs, specific GPU hours and wall-clock time not reported. Uses FSDP strategy for distributed training.

3. Inference speed/latency: Maximum 25 turns per question, 32K context window, temperature 0 for deterministic results. Specific latency numbers not provided.

4. Memory footprint: 32K maximum context window, bfloat16 precision for efficiency. Tool observations limited to 28,000 tokens to control memory usage.

5. Deployment practicality: Reasonably practical - 8B model can run on single A100, ReAct workflow is straightforward to implement, read-only tool constraints reduce safety concerns. However, requires executable repository environments which may limit real-world deployment scenarios.

1. Repository environments: Uses executable Python repositories from SWE-Rebench with working build systems, enabling end-to-end verification of agent actions in realistic development environments.

2. Long-tail repository focus: Deliberately selects less-studied repositories (1,484 for training, 26 for evaluation) to avoid memorization and better represent real-world diversity beyond popular GitHub projects.

3. Tool constraints: Read-only operations only (Search, View, CommandLine) mirror realistic code review and analysis scenarios where modification is not permitted.

4. Evaluation on production-style questions: Questions derived from actual GitHub issues reflect genuine developer information needs rather than synthetic academic problems.

5. Scalable data pipeline: Synthetic data generation approach can extend to other programming languages and repository types given appropriate executable environments.

   However, current limitation to Python ecosystem and requirement for pre-built executable environments may restrict immediate applicability to diverse real-world codebases that lack proper build configurations.

1. Scale limitation (EVALUATION): Only 260 questions from 26 repositories may not capture full diversity of software engineering tasks despite semantic clustering efforts.

2. Language restriction (FUNDAMENTAL): Limited to Python ecosystem due to executable environment requirements, though methodology is language-agnostic.

3. Reward hacking risk (FUNDAMENTAL): RLAIF training objectives closely mirror evaluation metrics since both use LLM-as-Judge frameworks, potentially causing models to optimize for judge preferences rather than objective correctness.

4. Embedding bias (ENGINEERING): Semantic clustering relies on Qwen3-8B embeddings which may introduce latent biases into task taxonomy construction.

5. Human annotation bottleneck (ENGINEERING): High cost of expert verification constrains benchmark scale and update frequency.

6. Repository availability (ENGINEERING): Requires executable sandbox environments which may not exist for many real-world repositories with complex dependencies.

   Failure modes:

7. Tool usage inefficiency: Models may make excessive tool calls without strategic planning, leading to context overflow and poor performance on complex multi-file reasoning tasks.

8. Memorization leakage: Despite filtering efforts, some questions may still be answerable through pretraining knowledge, particularly for repositories with extensive documentation or educational materials online.

Large Language Models (LLMs) have shown promise in automating system optimization and heuristic design, but struggle with complex multi-step design problems. Existing approaches fail due to evolutionary neighborhood bias (getting trapped in local optima by scalar scores) and the coherence ceiling (context degradation in long-horizon exploration).

The task is to design an AI agent architecture that can discover high-performance system heuristics across diverse domains. Input consists of problem specifications, baseline implementations, and evaluation environments (simulators/testbeds). Output is optimized code implementing novel heuristics. The agent must iteratively design, implement, test, and analyze mechanisms while maintaining coherence across long exploration horizons.

Success is measured by benchmark performance on diverse system optimization problems including multi-cloud multicast routing (cost minimization), LLM inference request routing (response time minimization), and database KV-cache optimization (cache hit rate maximization). Each domain has specific metrics and comparison against human state-of-the-art and existing LLM-based methods.

The paper evaluates on 9 problems from the ADRS benchmark suite plus custom testbeds, comparing against human experts and 4 LLM-based baselines across 100 evaluation runs per method.

1. Sequential Agent Architecture: Engram organizes exploration into a sequence of LLM agents, each operating with a fresh context window to avoid long-context degradation

2. Single Agent Exploration: Each agent follows a structured research agenda - review problem → formulate hypothesis → implement solution → run experiments → analyze results → iterate using scientific method principles

3. Persistent Archive: At conclusion of each agent run, all artifacts are stored including code snapshots, execution logs, experimental results, and performance metrics

4. Research Digest Handoff: The core technical contribution is a structured knowledge transfer mechanism where each agent distills high-level insights, modeling findings, and failure diagnoses into a compact Research Digest that subsequent agents read to build on prior discoveries

5. Context Management: Fresh agents inherit the Research Digest and can selectively access Archive contents through tool calls, ensuring prior history doesn’t consume context tokens unless explicitly needed

6. Tool Integration: Agents have access to evaluation playgrounds, shell commands, file operations, and can run simulations to gather experimental data for analysis

   The key innovation is decoupling long-horizon persistence from single-context constraints through structured handoff while preserving coherence and flexibility.

1. No model training involved - uses pre-trained LLMs (OpenAI o3, gpt-5.2) as reasoning engines in agent framework

2. Implementation: Built using deepagents library on LangChain and LangGraph for agent orchestration and tool integration

3. Agent Configuration: Each agent operates with system prompts that enforce structured research methodology and experimental discipline

4. Evaluation Budget: Each run allocated 100 evaluation simulations across agent sequence, with agents deciding when to terminate their individual explorations

5. Hardware: Not reported for the agent framework itself, but evaluation environments include GPU simulators and cloud topology testbeds

6. Hyperparameters: Not reported for Engram’s internal operation, focuses on comparison with baselines using their published configurations

Prior work falls into two categories:

1. Evolutionary code mutation approaches like OpenEvolve
2. , FunSearch
3. , Evolution of Heuristics
4. that use scalar fitness scores to guide LLM-generated code variants
5. Iterative reasoning-based approaches like Glia

6. that use flexible tool access and hypothesis-driven experimentation but are limited to single context windows.

   Delta: This paper introduces structured knowledge persistence across multiple agent contexts through the Research Digest mechanism. The key innovation is enabling cumulative progress without single-context degradation by having agents distill insights into compact external artifacts.

   Applied-specific assessment:

   • The architectural idea of structured handoff with Research Digest is novel and addresses a real limitation of existing approaches
   • Benchmark gains are substantial: beats human SOTA on 8/9 problems, with meaningful improvements (e.g., $625 vs $626 cost in multicast, 23.9s vs 25.7s response time in routing)
   • Comparisons appear fair - same evaluation environments, budgets, and multiple runs with confidence intervals
   • The gains appear to generalize across diverse domains and don’t obviously depend on proprietary advantages

   However, the core insight about context refresh and structured knowledge transfer, while effective, feels like an engineering solution to known LLM limitations rather than a fundamental breakthrough.

   Verdict: SIGNIFICANT — clear advance in agentic AI for systems optimization that most engineers working on LLM-based design should understand.

1. Multi-cloud multicast: Total cost ($), Engram $662, Human SOTA $626, best single run $625 (beats SOTA)
2. LLM request routing: Mean response time (s), Engram 23.9s, Glia 25.7s, Expert heuristic higher, improvement ~7%
3. KV-cache reuse: Combined score, Engram 0.721, Glia 0.719, OpenEvolve 0.714, GGR SOTA comparable
4. CBL: Score 103.6, Human SOTA 101.7 (slight regression)
5. CBL-Multi: Score 79.9, Human SOTA 92.3 (significant improvement)
6. EPLB: Score 0.273, Human SOTA 0.251 (improvement)
7. Prism: Score 27.94, Human SOTA 21.89 (improvement)
8. Telemetry: Score 0.954, Human SOTA 0.822 (significant improvement)

9. TXN: Score 3918.6, Human SOTA 2724.8 (improvement)

   Results show Engram exceeds human SOTA on 5/6 additional ADRS tasks and outperforms OpenEvolve on 4/6. Mixed results with one regression (CBL) but overall strong performance across diverse domains.

1. Model size: Uses pre-trained OpenAI o3 and gpt-5.2 (parameters not reported, likely 100B+ range)
2. Training compute: No training required, uses inference-only API calls to LLMs
3. Inference speed: Not explicitly reported, but each agent run completes within budget of 100 evaluations
4. Memory footprint: Minimal for framework itself, stores Archive and Research Digest as external files rather than in context
5. Deployment practicality: High - requires only API access to LLMs and can run on standard hardware, with structured artifact storage making it practical for real system optimization workflows

1. Multi-cloud evaluation uses realistic 71-node topology with real egress pricing and throughput measurements from cloud providers
2. LLM routing tested on ShareGPT workload simulation with realistic bursty arrival patterns on actual GPU configurations (4x NVIDIA A10)
3. Database workload uses real datasets (movies, beer, bird, pdmx, products) with natural language query patterns
4. No actual production deployment reported, but evaluation environments mirror real system conditions closely
5. Simulator-based testing (vidur for LLM serving) uses validated models of real system behavior rather than toy problems

1. FUNDAMENTAL: Depends on quality of LLM reasoning - cannot overcome fundamental model limitations in understanding complex system interactions
2. ENGINEERING: Research Digest compression may lose important nuanced insights that could be useful for future agents
3. ENGINEERING: Sequential agent design doesn’t exploit parallelism that could accelerate exploration
4. EVALUATION: Limited to simulation environments rather than production systems with real workloads and constraints
5. FUNDAMENTAL: Still susceptible to LLM hallucinations and may confidently implement incorrect solutions

6. ENGINEERING: Archive storage grows without bounds, could become unwieldy for very long explorations

   Likely failure modes:

7. Research Digest summarization loses critical technical details needed for breakthroughs
8. Agents may converge to locally optimal solution families and fail to explore radically different approaches despite structured handoff.

This paper addresses visual fine-tuning (VFT) in multimodal large language models (MLLMs), where a fundamental question remains unresolved: should the vision encoder be fine-tuned or frozen during instruction tuning?

Real-world context: Current MLLM construction follows a two-stage pipeline - pre-training for vision-language alignment, then instruction tuning for cross-modal understanding. However, influential models make inconsistent choices: LLaVA/InstructBLIP freeze the vision encoder while InternVL/Qwen-VL fine-tune it. This divergence creates instability when adapting pre-trained vision encoders to multimodal tasks, with existing VFT methods failing to consistently outperform frozen baselines.

Task definition: Given an image I ∈ R^(H×W×3) and textual instruction Q, the vision encoder V extracts visual embeddings z = V(I) ∈ R^(L_v×D_v), which are projected to z̃ = P(z) ∈ R^(L_v×D_t) and processed with text embeddings by the LLM. The training objective is:

\[\mathcal{L}_{\text{inst}} = -\sum_{t=1}^{T} \log p_{\boldsymbol{\theta}}(\mathbf{a}_t \mid \mathbf{a}_{<t}, \mathbf{Q}, \mathbf{I})\]

The key question is whether to update vision encoder parameters θ_v (∇{θ_v}\mathcal{L}{\text{inst}} ≠ 0) or freeze them.

Evaluation criteria: Performance measured on 12 multimodal benchmarks spanning:

1. General VQA (MME-Perception, MMBench, GQA)
2. Knowledge & OCR (ScienceQA-Image, AI2D, TextVQA)

3. Vision-centric tasks (MMVP, RealWorldQA, CV-Bench). Success measured by accuracy improvements and consistency across tasks.

   Dataset: Uses 558K image-caption pairs for pre-training and 665K image-instruction-answer triplets for instruction tuning, following LLaVA-1.5 protocol.

The method addresses “visual preference conflicts” - where context-agnostic vision encoders receive conflicting optimization signals from diverse multimodal tasks, causing instability.

1. Context-aware Visual Fine-tuning (CoVFT) Framework: Reformulates learning from context-agnostic p_{θ_v}(z

   I) to context-aware posterior:

   \[p_{\boldsymbol{\theta}_v}(\mathbf{z}|\mathbf{I},\mathbf{c})\]

   where c encodes multimodal context from instruction triplet (I,Q,A).

2. Contextual Vector Extraction (CVE): Uses frozen BERT text encoder to generate textual embedding t, applies residual blocks to refine visual tokens z and text t, then extracts contextual vector via cross-attention:

   \[\mathbf{c}_i = \text{CrossAttn}\big(\text{norm}(\hat{\mathbf{t}})_{\text{q}}, [\text{norm}(\hat{\mathbf{z}}),\text{norm}(\hat{\mathbf{t}})]_{\text{k,v}}\big)\]

3. Contextual Mixture-of-Experts (CoMoE): Replaces FFN layers in latter half of ViT with N parallel expert networks. Context-conditioned routing weights:

   \[\mathbf{g}(\mathbf{c}) = \text{softmax}(\mathbf{W}\mathbf{c} + \mathbf{b}) \in \mathbb{R}^N\]

   Dense aggregation of expert outputs:

   \[\tilde{\mathbf{z}} = \sum_{n=1}^{N} g^n(\mathbf{c})\,\mathcal{E}^n(\mathbf{z})\]

4. Architecture: CLIP-ViT-L/14-336 vision encoder, 2-layer MLP projector, Vicuna-1.5-7B/13B LLM. CoMoE applied to layers 11-22 of ViT with 4 experts per layer by default.

   Core technical contribution: First systematic framework addressing visual preference conflicts in MLLM training through explicit context conditioning and expert routing based on multimodal cues.

1. Pre-training stage: Only projector optimized on 558K image-caption pairs with learning rate 1×10^-3, batch size 256. Vision encoder and LLM frozen.

2. Instruction-tuning stage: - Data: 665K image-instruction-answer triplets (LLaVA-1.5 protocol) - LLM + projector: learning rate 2×10^-5, batch size 128 - Vision encoder: Updated according to each VFT method with default hyperparameters - For CoVFT: CVE + CoMoE modules + LayerNorm statistics optimized jointly, other vision parameters frozen - Text encoder (BERT): Frozen throughout training

3. Hardware: 8×NVIDIA H100 GPUs for training, single H100 for inference

4. Training time: CoVFT adds ~18 minutes total training overhead (3.8% vs Full FT)

5. Hyperparameters: 4 experts per CoMoE layer, CoMoE applied to layers 11-22 of ViT-L/14

   Wall-clock times and other specific training details not reported.

Prior work:

1. LLaVA/InstructBLIP (2023-2024): Freeze vision encoder during instruction tuning, achieving strong performance with simplified MLP projection
2. InternVL/Qwen-VL (2024): Joint optimization of vision encoder with other components, reporting superior results on some benchmarks

3. Standard VFT methods (LoRA, BitFit, VPT): Parameter-efficient approaches showing inconsistent results in MLLM settings

   Delta: This paper provides the first systematic analysis of why existing VFT methods fail in MLLMs, identifying “visual preference conflicts” where different instructions induce conflicting gradient directions. The core innovation is explicit context conditioning through CVE+CoMoE framework that decomposes conflicting signals into specialized expert subspaces.

   Applied-specific assessment:

   • Architectural novelty: Context-aware MoE routing in vision encoder is novel for MLLMs, though MoE itself is established
   • Benchmark gains: Consistent 2+ point improvements across 12 benchmarks is meaningful, with CoVFT achieving SOTA on 9/12 tasks
   • Fair comparisons: Uses unified LLaVA-1.5 protocol ensuring fair comparison; improvements hold across different scales (7B vs 13B)
   • Generalization: Results show 7B model with CoVFT outperforms 13B frozen baseline, suggesting gains aren’t just from scale
   • The insight about visual preference conflicts is non-obvious and the solution is architecturally principled

   Verdict: SIGNIFICANT — Identifies a fundamental problem (visual preference conflicts) in an important area and provides both principled analysis and effective solution with consistent gains across diverse benchmarks.

1. MME-Perception: CoVFT 1525.2 vs Freeze 1473.7 vs LLaVA-1.5 1510.7 (+51.5 improvement)
2. MMBench: CoVFT 68.13% vs Freeze 67.87% vs LLaVA-1.5 64.30% (+0.26 improvement)
3. MMBench-CN: CoVFT 60.40% vs Freeze 60.30% vs LLaVA-1.5 58.30% (+0.10 improvement)
4. GQA: CoVFT 63.37% vs Freeze 63.07% vs LLaVA-1.5 62.00% (+0.30 improvement)
5. ScienceQA-Image: CoVFT 69.51% vs Freeze 69.31% vs LLaVA-1.5 66.80% (+0.20 improvement)
6. AI2D: CoVFT 56.64% vs Freeze 55.76% vs LLaVA-1.5 55.21% (+0.88 improvement)
7. TextVQA: CoVFT 59.64% vs Freeze 58.53% vs LLaVA-1.5 58.20% (+1.11 improvement)
8. MMVP: CoVFT 36.67% vs Freeze 28.00% vs LLaVA-1.5 27.33% (+8.67 improvement)
9. RealWorldQA: CoVFT 57.52% vs Freeze 56.73% vs LLaVA-1.5 56.21% (+0.79 improvement)
10. CV-Bench COCO: CoVFT 66.96% vs Freeze 63.73% vs LLaVA-1.5 67.45% (+3.23 improvement)
11. CV-Bench ADE: CoVFT 56.08% vs Freeze 49.61% vs LLaVA-1.5 54.98% (+6.47 improvement)

12. CV-Bench Omni3D: CoVFT 61.83% vs Freeze 60.50% vs LLaVA-1.5 63.25% (+1.33 improvement)

    Overall Average: CoVFT 61.08% vs Freeze 58.93% vs LLaVA-1.5 59.13% (+2.15 point improvement)

    CoVFT achieves improvements on ALL 12 benchmarks compared to Freeze baseline and obtains SOTA performance on 9/12 individual tasks. Results are notably strong on vision-centric tasks (55.81% vs 51.71% frozen).

1. Model size: 7B total parameters (CLIP-ViT-L/14 vision encoder ~300M, Vicuna-7B LLM ~6.7B). Vision encoder represents <5% of total parameters.

2. Training compute: 8×NVIDIA H100 GPUs. CoVFT adds ~18 minutes total training time (3.8% overhead vs Full FT). Peak GPU memory increases 13.5% over Full FT.

3. Inference speed: ~10ms additional latency per sample on single H100. Frozen BERT encoder adds ~1.14ms per sample.

4. Memory footprint: CoVFT introduces moderate memory increase due to MoE experts and contextual processing overhead. Specific numbers not provided.

5. Deployment practicality: Method shows good scalability - works on both 7B and 13B models. Context extraction via frozen BERT is computationally lightweight. Dense routing avoids sparse activation complexity but increases compute vs sparse alternatives. The 3.8% training overhead for 2+ point accuracy gains represents favorable efficiency trade-off.

1. Evaluation scope: All experiments conducted on standard multimodal benchmarks covering diverse real-world scenarios including visual question answering, knowledge reasoning, OCR tasks, and vision-centric evaluation.

2. Architectural generalization: Method tested across different vision encoders (CLIP, SigLIP, DINOv3) and LLM backbones (Vicuna, Phi-3-mini), showing consistent improvements beyond LLaVA paradigm.

3. Scale validation: Demonstrates effectiveness on both 7B and 13B model scales with 7B+CoVFT outperforming 13B frozen baseline, suggesting practical deployment value.

4. Data efficiency: CoVFT surpasses full-data LLaVA-1.5-7B baseline when trained on only 75% of data, indicating strong data efficiency for resource-constrained scenarios.

5. Production considerations: The framework adds minimal inference overhead (~10ms) and uses standard components (BERT text encoder, MoE routing) that are deployment-friendly. However, no explicit production deployment results or sim-to-real validation provided.

1. Scale limitations (ENGINEERING): Evaluation limited to 13B parameters and million-scale training data due to computational constraints, though scalability trends suggest promise at larger scales.

2. Training overhead (ENGINEERING): MoE-based adaptation introduces 4% training overhead vs frozen baseline and 13.5% memory increase, requiring optimization for large-scale deployment.

3. Architecture dependency (FUNDAMENTAL): Method requires text encoder (BERT) for context extraction, adding architectural complexity and potential failure point.

4. Expert capacity tuning (ENGINEERING): Optimal number of experts depends on data diversity and scale - requires task-specific hyperparameter tuning.

5. Evaluation scope (EVALUATION): Testing limited to standard benchmarks without real-world deployment validation or adversarial robustness assessment.

   Failure modes:

6. Context extraction failure: If BERT text encoder produces poor representations or fails to capture task-specific cues, the entire CoVFT framework degrades.
7. Expert collapse: Dense routing might lead to expert homogenization under certain training conditions, reducing the benefit of specialized subspaces.

1. Real-world context: As AI systems evolve from text-based assistants to embodied agents in robots and autonomous vehicles, they must make morally consequential decisions based on visual inputs. However, current safety alignment techniques focus on text-based interactions, creating potential vulnerabilities when these systems process visual information in safety-critical scenarios.

2. Task definition: The paper evaluates moral reasoning consistency across modalities in Vision-Language Models (VLMs). Input modalities include: (a) structured text descriptions, (b) visual captions generated by the model, and (c) rendered images with embedded text in sandbox-style aesthetic. The task requires models to make binary moral decisions (act/don’t act) in dilemmas grounded in Moral Foundation Theory across five dimensions: Care, Fairness, Loyalty, Authority, and Purity. The formal objective evaluates decision consistency across modalities while systematically controlling for conceptual variables (personal force, intention of harm, self-benefit) and character variables (demographics, relationships).

3. Evaluation criteria: Success is measured by:
4. utilitarian sensitivity (S-shaped response curves to quantity ratios)
5. cross-modal consistency in moral foundation preferences
6. preservation of deontological constraints, and

7. maintenance of social value hierarchies. Metrics include action probability, log odds ratios, preference strength, and SHAP interaction values.

   1. Dataset: MDS benchmark contains 84,240 controlled samples across three subsets: Quantity (2,105 samples testing utilitarian sensitivity), Single Feature (71,895 samples for single-variable perturbation), and Interaction (10,240 samples for multi-dimensional effects).

1. Benchmark Architecture: MDS uses a controllable generation engine grounded in Moral Foundation Theory that creates moral dilemmas through orthogonal manipulation of variables. Each dilemma instantiates conflicts within or across MFT dimensions.

2. Visual Rendering: Sandbox-style aesthetic minimizes artistic confounds while depicting scenario elements. Images embed both visual scenes and textual descriptions for modal consistency.

3. Evaluation Protocol: Tri-modal assessment comparing Text Mode (structured descriptions), Caption Mode (model-generated captions with OCR), and Image Mode (direct visual input) to isolate visual processing effects from informational complexity.

4. Analysis Methods: Hierarchical logistic regression quantifies marginal effects of moral variables. Gradient Boosting Decision Trees with SHAP analysis decomposes decision drivers into Quantity (utilitarian calculus), Character (demographic bias), and Action Bias components.

5. Core Technical Contribution: First systematic framework enabling causal-level analysis of multimodal moral reasoning through controlled variable manipulation, revealing that visual inputs bypass text-based safety mechanisms and activate intuition-like pathways that override deliberative reasoning.

Not reported - this paper evaluates existing pre-trained VLMs without additional training. Models tested include:

1. Open-weight models: LLaVA-v1.6-34B, Qwen3-VL-8B-Instruct, Qwen3-VL-32B-Instruct, LLaMA-3.2-90B

2. Proprietary models: GPT-4o-mini, Gemini-2.5-flash

   All models evaluated with temperature 0.0 for reproducibility. No model-specific training or fine-tuning performed for this evaluation.

Step 1 — Prior work:

• ETHICS (Hendrycks et al., 2020): Text-only moral evaluation benchmark focusing on commonsense moral judgments
• Social Chemistry (Forbes et al., 2020): Text-based assessment of social norms and moral rules
• Recent VLM moral evaluation (Yan et al., 2024): Used diffusion models to generate images for moral evaluation but lacked systematic variable control

Step 2 — Delta: This paper adds:

1. first systematic multimodal moral benchmark with orthogonal variable control
2. tri-modal evaluation protocol isolating visual vs. contextual effects
3. causal analysis framework using controlled experimentation

4. demonstration that visual inputs fundamentally bypass text-based safety mechanisms.

   Step 3 — Applied-specific assessment:

   • Architectural idea: The controlled generation engine with orthogonal variable manipulation is novel for moral evaluation, enabling causal rather than just descriptive analysis
   • Benchmark gains: Clear, consistent evidence across multiple SOTA models showing visual modality undermines moral reasoning - effect sizes are large and systematic
   • Fair comparisons: Same models across three modalities with identical underlying moral scenarios, proper controls for informational content
   • Generalizability: Results hold across diverse model architectures and scales, suggesting fundamental rather than model-specific issue

   Verdict: SIGNIFICANT — This reveals a critical and previously unrecognized vulnerability in multimodal AI safety that will require new alignment approaches.

1. Utilitarian Sensitivity: Text Mode shows proper S-shaped curves (e.g., LLaMA-3.2-90B: 0.1 at 1:10 ratio to 0.6 at 10:1). Image Mode flattens responses (LLaMA-3.2-90B: 0.30-0.35 across all ratios). LLaVA-v1.6-34B shows extreme collapse from ~0.1 to ~1.0.

2. Moral Foundation Preferences: Text Mode maintains balanced profiles across Care, Fairness, Loyalty, Authority, Purity. Image Mode shifts toward Care/Loyalty while abandoning Authority/Purity (LLaVA-v1.6-34B most extreme).

3. Conceptual Variable Sensitivity: “Intention of Harm” log odds shift from negative (Text) to positive (Image) across models. “Self-Benefit” shows similar pattern with LLaVA-v1.6-34B shifting from -0.17 to +0.46.

4. Social Value Hierarchies: Text Mode shows strong hierarchical preferences (human vs. non-human ≈0.9). Image Mode collapses preferences toward zero across demographic categories.

5. Decision Driver Analysis: Quantity contribution drops dramatically in Image Mode (Qwen3-VL-8B: 22% to <5%), while Character bias expands (58% to 95%).

   Notable exception: Gemini-2.5-flash maintains better cross-modal consistency in several measures.

1. Model size: Evaluated models range from 8B (Qwen3-VL-8B) to 90B parameters (LLaMA-3.2-90B), plus proprietary models

2. Training compute: Not reported (evaluation-only study using pre-trained models)

3. Inference speed/latency: Not reported, though evaluation conducted at temperature 0.0 for reproducibility

4. Memory footprint: Not reported

5. Deployment practicality: The benchmark reveals critical safety vulnerabilities that could affect real-world deployment of VLMs in embodied systems, autonomous vehicles, and safety-critical applications where visual moral reasoning is required

1. Real-world relevance: Paper directly addresses deployment scenarios for embodied agents, household robots, and autonomous vehicles that must make moral decisions from visual perception

2. Practical implications: Findings expose critical gaps in current VLM safety for applications like medical triage robots, autonomous driving moral decisions, and any embodied AI requiring visual moral reasoning

3. No direct hardware experiments: Evaluation conducted on benchmark scenarios rather than physical robot deployments

4. Safety-critical warning: Results indicate current text-based alignment approaches are insufficient for visual contexts, creating potential safety risks in real-world multimodal AI deployments

5. Benchmark utility: MDS provides diagnostic platform for evaluating multimodal moral consistency before real-world deployment

1. EVALUATION: Limited to six VLMs - broader model coverage needed to establish generalizability across architectures
2. EVALUATION: Sandbox visual style may not capture complexity of real-world visual scenarios
3. FUNDAMENTAL: Cultural bias in MFT framework may not generalize across all moral systems and societies
4. ENGINEERING: OCR dependency for Caption Mode could introduce artifacts in text extraction

5. EVALUATION: Binary action choices may oversimplify complex real-world moral decisions requiring nuanced responses

   Failure modes:

6. Visual distraction overwhelms abstract reasoning: Models become insensitive to quantitative stakes when processing images
7. Demographic bias amplification: Visual features trigger holistic, pixel-level correlations that are harder to detect and mitigate than text-based biases