1. Motivation: Behavior-regularized reinforcement learning is essential in offline RL (where data is fixed) and RLHF (where models shouldn’t deviate too far from base models). Current methods either use unstable reparameterized policy gradients or overly conservative reject sampling that remains trapped within the reference distribution support.

2. Mathematical setup: Consider an MDP $M = \langle S, A, P, d_0, r, \gamma \rangle$ with state space $S$, action space $A$, transition dynamics $P(s’

   s,a)$, initial state distribution $d_0$, reward function $r(s,a)$, and discount factor $\gamma$. Given a reference distribution $\mu(a

   s)$ (dataset distribution in offline RL or base model in RLHF), the goal is to solve:

   \[\pi^* = \arg\max_\pi \mathbb{E}_{s \sim D, a \sim \pi(\cdot|s)}[R(s,a)]\]

   subject to:

   \[\mathbb{E}_{s \sim D}[M(\pi(\cdot|s), \mu(\cdot|s))] \leq \epsilon\]

   where $R(s,a)$ is a differentiable reward/value function and $M$ is a distance measure.

   Assumptions:

   1. $R(s,a)$ is $c$-Lipschitz with respect to action $a$
   2. Access to samples from reference distribution $\mu$
   3. Differentiable reward model in RLHF setting

3. Toy example: Consider a 2D continuous control bandit with bimodal reward distribution. The offline dataset samples from suboptimal regions with $\mu = \mathcal{N}((-1,0), I_2)$, but optimal actions lie near $(2,2)$ and $(-2,-2)$. Traditional methods stay within $\text{supp}(\mu)$ and miss high-reward regions.

4. Formal objective: Find the optimal transport from $\mu$ to the Boltzmann policy distribution:

   \[\pi_R^*(a|s) = \frac{1}{Z_s} \exp(R(s,a)/\alpha)\]

VGF casts behavior-regularized RL as optimal transport from reference distribution $\mu$ to the value-induced Boltzmann distribution $\pi_R^*$. The method uses discrete gradient flow via the Jordan-Kinderlehrer-Otto (JKO) scheme:

\[q^{k+1} = \arg\min_q \text{KL}(q \| \pi_R^*) + \frac{1}{2h} W_2^2(q, q^k)\]

For practical implementation, approximate $q^k$ with $N$ particles ${a_i^{(k)}}_{i=1}^N$ and update via:

\[a_i^{(l+1)} = a_i^{(l)} + \epsilon \cdot \phi(a_i^{(l)})\]

where the velocity field is:

\[\phi(x) = \frac{1}{N} \sum_{j=1}^N k(a_j, x) \nabla_{a_j} R(s, a_j)\]

using kernel $k(\cdot,\cdot)$ and step size $\epsilon$.

For LLMs, apply VGF in continuous embedding space $u \in \mathbb{R}^{T \times d}$ with chain rule:

\[\nabla_{u_i} \log \pi_R^*(y_i^{(l)}|x) = \frac{1}{\alpha} J_i^\top \nabla_y R(x, y_i^{(l)})\]

where $J_i = \frac{\partial \text{Dec}(u_i^{(l)})}{\partial u_i^{(l)}}$.

Applied to toy example: Initialize 3 particles from $\mu = \mathcal{N}((-1,0), I_2)$, then iteratively move particles toward high-reward regions using value gradients. After $L=5$ steps with $\epsilon=0.1$, particles discover both optimal modes at $(2,2)$ and $(-2,-2)$, unlike methods constrained to $\text{supp}(\mu)$.

Step 1 — Prior work:

• “Conservative Q-Learning for Offline RL” (Kumar et al., 2020): Adds explicit KL penalties to constrain policies near behavior distribution
• “Diffusion Policies as Expressive Policy Class” (Wang et al., 2023): Uses diffusion models as policies but requires unstable backpropagation through sampling steps
• “Policy Gradient Methods with Natural Policy Gradients” (various): Reparameterized gradients that become unstable for complex generative models

Step 2 — Delta: This paper eliminates explicit policy parameterization and regularization penalties. Instead, it uses transport budget (number of gradient flow steps) as implicit regularization. The key insight is recasting behavior-regularized RL as optimal transport solved via particle-based gradient flow.

Step 3 — Theory-specific assessment:

• The main theorem (Theorem 1) bounds MMD distance between initial and final particle distributions, which is predictable given standard Lipschitz assumptions
• The proof technique combines known results from SVGD (Stein Variational Gradient Descent) and optimal transport - not fundamentally new
• Theorem 2 shows particles can move beyond reference support, but this follows naturally from the gradient flow formulation
• No comparison to known lower bounds provided

The theoretical contribution is modest - essentially applying existing SVGD machinery to RL with a transport interpretation. The practical benefits (stability, scalability) are more significant than theoretical novelty.

Verdict: INCREMENTAL — solid application of known techniques with useful practical improvements but limited theoretical depth.

The main proof strategy combines Lipschitz analysis with kernel properties from SVGD theory.

1. For Theorem 1 (MMD bound), the key steps are:

   Establish Lipschitz constants: Value function $R(s,a)$ is $c$-Lipschitz, and kernel satisfies:

   \[k(x + \delta, y) - k(x, y) \leq \frac{\|\delta\|_\infty}{\sigma\sqrt{e}}\]

   Bound the velocity field:

   \[\|\phi(x)\|_\infty \leq \frac{c}{\alpha} + \frac{1}{\sigma\sqrt{e}}\]

   Apply triangle inequality to MMD expansion:

   \[\text{MMD}^2(\pi_N^L, \pi_N^0) = \mathbb{E}_{x,y \sim \pi_N^0}[k(x,y)] + \mathbb{E}_{x,y \sim \pi_N^L}[k(x,y)] - 2\mathbb{E}_{x \sim \pi_N^0, y \sim \pi_N^L}[k(x,y)]\]

   Accumulate Lipschitz bounds over $L$ steps:

   \[\text{MMD}^2(\mu, \pi_N^L) \leq 2\epsilon L \frac{1}{\sigma\sqrt{e}}\left(\frac{c}{\alpha} + \frac{1}{\sigma\sqrt{e}}\right)\]

2. For Theorem 2 (support expansion), use proof by contradiction:

   In discrete case: If updated particle $a_i = a_1 + \epsilon h(a_1)$ remains in original support, this requires exact equality that is measure-zero.

   In continuous case: Assume Gaussian distributions $\pi_N^0 \sim \mathcal{N}(\mu_1, \sigma_1^2)$, $\pi_N^1 \sim \mathcal{N}(\mu_2, \sigma_2^2)$. For particles at boundary of $\epsilon$-support, the velocity field has non-zero component pointing outward due to asymmetric kernel weighting.

   The proofs are routine applications of standard concentration and kernel analysis - no novel technical insights.

Offline RL Datasets:

• D4RL: MuJoCo locomotion (halfcheetah, hopper, walker2d) and AntMaze navigation tasks with medium, medium-replay, medium-expert variants
• OGBench: 9 goal-conditioned tasks across locomotion and manipulation

Baselines:

• Gaussian policies: TD3+BC, IQL, IVR
• Diffusion/Flow policies: Diffusion-QL, SfBC, FQL
• Best-of-N sampling methods

Key Results:

• D4RL: VGF achieves 57.1 vs 55.6 (best baseline) on halfcheetah-m, 98.0 vs 96.0 on antmaze-u
• OGBench: Major improvements on hard tasks like cube-double (70 vs 29), puzzle-3x3 (75 vs 30)
• RLHF: 68.1% win rate vs 61.2% for DPO on TL;DR summarization

RLHF Setup:

• TL;DR: 116k instructions, 93k preference pairs, Pythia-2.8B base model
• Anthropic-HH: 112k preference pairs
• Evaluation: GPT-4 win rates against reference model

Ablation Studies:

• Transport budget $L_{\text{train}}$ most critical hyperparameter
• Test-time scaling: performance improves with larger $L_{\text{test}}$ when value function generalizes well
• Small particle count ($N=5$) sufficient across tasks

The experimental evaluation is comprehensive with strong empirical results, particularly on challenging long-horizon tasks.

Limitations:

1. TECHNICAL: Heavy reliance on value function quality - when value estimates have large extrapolation errors, method degrades to best-of-N sampling. This is technically addressable through better value learning but currently limits applicability.

2. TECHNICAL: Kernel choice and hyperparameter sensitivity - requires tuning transport budget $L_{\text{train}}$ per task. The optimal choice depends on dataset quality and value function generalization, making it technically demanding to set properly.

3. RESTRICTIVE: Performance on heavily skewed reference distributions - when reference policy is extremely suboptimal, the transport may not reach good regions efficiently. This significantly narrows applicability to high-quality offline datasets.

4. TECHNICAL: Particle approximation with small $N=5$ may be insufficient for very high-dimensional or complex multimodal action spaces. Scalability limits aren’t fully characterized.

   Open Problems:

5. Theoretical characterization: Derive conditions under which the transport budget provides optimal regularization strength. Current MMD bounds don’t directly translate to performance guarantees.

6. Adaptive transport budget selection: Develop principled methods to automatically set $L_{\text{train}}$ and $L_{\text{test}}$ based on dataset quality and value function uncertainty estimates, removing the need for manual hyperparameter tuning per task.

Motivation: Entropic optimal transport with mutual information constraints arises naturally in machine learning (regularization for efficient algorithms like Sinkhorn) and information theory. While the unconstrained case has sharp two-sided bounds via rate-distortion integrals, the information-constrained case lacks tight characterization.

Mathematical setup: Given standard Gaussian measure $γ = N(0, I_n)$ and probability measure $µ$ on $\mathbb{R}^n$ with finite second moments. For $R ≥ 0$, define the information-constrained optimal transport value:

\[w(γ, µ, R) := \sup_{P_{YZ} \in Π(γ,µ): I(Y;Z) ≤ R} \mathbb{E}[⟨Y, Z⟩]\]

For $β ≥ 0$, define the regularized version:

\[f(γ, µ, β) := \sup_{R > 0} \{w(γ, µ, R) - βR\}\]

The rate-distortion function is defined as:

\[R_µ(σ²) := \inf_{P_{UZ}: P_Z = µ, \mathbb{E}[∥Z-U∥²] ≤ σ²} I(U; Z)\]

with $i_µ(σ) := \inf_{P_{Z\hat{Z}} \in Π_σ(µ)} I(Z; \hat{Z})$ where $Π_σ(µ)$ contains couplings with $\mathbb{E}[d²(Z,\hat{Z})] ≤ σ²$.

Assumptions:

1. $µ$ has finite second moments
2. $γ$ is standard Gaussian

3. Universal constants from majorizing measure theorem exist

   Toy example: When $n = 1$, $µ = δ_1$ (point mass at 1), and $R$ large, the optimal coupling places all mass on $(Y,Z) = (1,1)$ giving $w(γ,µ,R) = 1$. The rate-distortion $i_µ(σ) = 0$ for $σ ≥ 1$, so the integral becomes $\int_0^1 \sqrt{R} dσ = \sqrt{R}$.

   Formal objective: Establish two-sided bounds:

   \[w(γ, µ, R) \asymp \int_0^∞ \sqrt{R ∧ i_µ(σ)} dσ\]

The method uses a “lifting technique” that constructs a Gaussian process indexed by a random subset of the type class, then applies majorizing measure theory.

Key steps:

1. Approximate arbitrary measures by rational distributions
2. For rational $µ$, construct type class $C_µ$ of sequences with empirical distribution $µ$
3. Randomly select $L = ⌊\exp(NR)⌋$ sequences from $C_µ$ to form random set $A$

4. Show equivalence:

   \[w(γ, µ, R) = \lim_{N→∞} \mathbb{E}\left[\max_{z^N ∈ A} \frac{1}{N}⟨Y^N, z^N⟩\right]\]

   where $Y^N$ is uniform on the type class of $γ$.

5. Apply majorizing measure theorem to bound the expectation via:

   \[\mathbb{E}\left[\max_{z^N ∈ A} \frac{1}{N}⟨Y^N, z^N⟩\right] \asymp \frac{1}{N} δ_2(A)\]

6. Compute $δ_2(A)$ using concentration inequalities on the random subset $A$

   The regularized version uses Legendre transform techniques to convert between constrained and regularized formulations.

   Toy example application: For $µ = δ_1$, $n=1$, the type class contains sequences like $(1,1,…,1)$. The random subset $A$ contains $≈ e^{NR}$ copies. The Gaussian process becomes $\max_{i ≤ e^{NR}} Y_i$ where $Y_i \sim N(0,1)$, giving expectation $≈ \sqrt{2\log(e^{NR})} = \sqrt{2NR}$, scaling as $\sqrt{R}$ per coordinate.

Prior work:

1. “Simple and sharp generalization bounds via lifting” (Liu, 2025): Established the unconstrained rate-distortion integral $\sup_{P_{YZ} \in Π(γ,µ)} \mathbb{E}[⟨Y,Z⟩] \asymp \int_0^∞ \sqrt{R_µ(σ²)} dσ$ with two-sided bounds.

2. “Information constrained optimal transport: From talagrand, to marton, to cover” (Bai et al., 2023): Obtained bounds for Gaussian case using different methods, but dimension-dependent and one-sided.

3. Various works on Cover’s problem and entropic optimal transport provided partial results but lacked sharp two-sided characterization.

   Delta: This paper extends Liu (2025) to the information-constrained setting by:

   • Proving two-sided bounds with truncated integral $\int_0^∞ \sqrt{R ∧ i_µ(σ)} dσ$
   • Handling mutual information constraint via random subset selection instead of full type class
   • Establishing regularized version with exact tensorization property

   Theory-specific assessment:

   • Main theorem is a natural but non-obvious extension requiring new technical machinery
   • Proof technique cleverly adapts lifting method with crucial innovation of random subset to control mutual information
   • Bounds appear tight given the truncation structure, though no explicit lower bounds are established
   • The connection to majorizing measures is elegant but builds incrementally on Liu (2025)

   The random subset construction is technically non-trivial since it must maintain near-stationarity while controlling information. However, the overall approach follows the established lifting paradigm.

   Verdict: INCREMENTAL — solid extension of existing lifting technique to information-constrained setting with reasonable technical innovation.

The proof employs several key techniques:

1. Lifting via random subsets: Core innovation replacing full type class with random subset $A$ of size $L = ⌊\exp(NR)⌋$. This controls mutual information while preserving near-stationarity.

2. Type class approximation: Uses Lemma 5 to show any coupling can be approximated by integer-valued (rational) couplings. Key inequality:

   \[\max_{x,y} |Q_{XY}(x,y) - P_{XY}(x,y)| ≤ \frac{1}{N}\]

3. Information-theoretic concentration: Critical bound from Lemma 1 on type class sizes:

   \[(N+1)^{-|X||Y|} \exp(-NI(X;Y)) ≤ \frac{|C(x^N)|}{|C|} ≤ (N+1)^{|Y|} \exp(-NI(X;Y))\]

4. Sharp concentration for random subsets: For $R > i_µ(σ)$, uses Chernoff bound (Lemma 3) to show:

   \[P\left[|\ln µ_N(B(t,\sqrt{Nσ})) + Ni_µ(σ)| > 2\right] ≤ \exp(-e^{cN}/2)\]

   This ensures the random subset $A$ has approximately the right density in each ball.

5. Majorizing measure connection: Links to $δ_2(A)$ via:

   \[δ_2(A) ≥ \int_0^∞ \min_{t∈A} \sqrt{\ln \frac{1}{µ_N(B(t,λ))}} dλ\]

6. Change of variables technique: In Theorem 2, uses Lemma 4 to convert between integral forms:

   \[\int_0^∞ \min_x \{α^{-2}x² + y²\} dα \asymp \int_0^∞ y dx\]

   The proof’s technical heart is showing that despite randomness, the subset $A$ behaves like a stationary process due to concentration.

Purely theoretical. The paper provides one concrete verification: for $µ = γ = N(0,1)$, both $f(γ,µ,β)$ and $\int_β^∞ φ(µ,α) dα$ converge to constants as $β ↓ 0$ and scale as $1/β$ as $β → ∞$, confirming the bounds up to constants.

Empirical validation would require:

1. Computing rate-distortion functions $i_µ(σ)$ for various distributions $µ$
2. Numerically solving the optimization problems defining $w(γ,µ,R)$ and $f(γ,µ,β)$
3. Evaluating the integral bounds and comparing constants
4. Testing on diverse distributions (discrete, continuous, high-dimensional)

Limitations:

1. Results require finite second moments for $µ$ - NATURAL (standard assumption in optimal transport)
2. Constants are not explicit, only universal - TECHNICAL (could potentially be computed)
3. Lifting argument requires rational approximation - TECHNICAL (approximation error vanishes)
4. Random subset construction adds complexity vs. deterministic type class - TECHNICAL (needed for information control)

5. Only Gaussian marginal $γ$ is considered - RESTRICTIVE (significantly limits applicability)

   Open problems:

6. Extend to non-Gaussian marginals: Can similar two-sided bounds be established when $γ$ is not Gaussian? The lifting technique heavily exploits Gaussian structure.

7. Explicit constants: Determine the precise multiplicative constants in the bounds, not just their existence from majorizing measure theory. This would make the results practically useful.

1. Motivation: This work studies the large population limit of non-exchangeable interacting particle systems on $\mathbb{R}^d$ where interactions are governed by a matrix converging to a graphon. Such systems arise in social science and economics where inhomogeneous interactions are natural, going beyond classical mean-field theory.

2. Mathematical setup: Consider $N$ particles $(X^{i,N}_t)_{1 \leq i \leq N}$ evolving according to:

   \[dX^{i,N}_t = \sum_{j=1}^N \xi^N_{i,j} b(X^{i,N}_t - X^{j,N}_t) dt + dB^{i,N}_t\]

   Assumptions:

   1. $b$ is smooth, bounded with bounded derivatives
   2. $(\xi^N_{i,j})_{1 \leq i,j \leq N}$ is an interaction matrix with $\max_i \sum_j \xi^N_{i,j} \leq 1$ and $\xi^N_{i,j} \geq 0$
   3. The matrix converges to a graphon $G: [0,1]^2 \to [0,1]$ via the representation $\xi^N_{i,j} = G^N(i/N, j/N)/N$

   The independent projection system is:

   \[dY^{i,N}_t = \sum_{j=1}^N \xi^N_{i,j} \langle b(X^{i,N}_t - \cdot), Q^{j,N}_t \rangle dt + dB^{i,N}_t\]

   where $Q^{i,N}_t = \text{Law}(Y^{j,N}_t)$.

3. Toy example: When $N=2$, $d=1$, $b(x) = x$, and $\xi^2_{i,j} = 1/2$ for all $i,j$, the system reduces to two particles with symmetric interactions, and the independent projection becomes two independent processes with feedback from their marginal laws.

4. Formal objective: Establish quantitative bounds on the relative Fisher information:

   \[I^v_t = I(P^{v,N}_t | Q^{v,N}_t)\]

   where $P^{v,N}_t = \text{Law}(X^{i,N}_t, i \in v)$ and $Q^{v,N}_t = \text{Law}(Y^{i,N}_t, i \in v)$.

The method proceeds in two stages:

Stage 1 - Finite population bounds: Adapt the BBGKY hierarchy method to prove quantitative bounds on Fisher information between the particle system and independent projection. The key differential inequality is:

\[\frac{d}{dt}I^v_t \leq C I^v_t + \mathcal{A}H^v_t + \mathcal{A}I^v_t + C(v)\]

where $\mathcal{A}$ is the generator:

\[\mathcal{A}F(v) = \sum_{i \in v} \sum_{k \notin v} \xi^N_{i,k}(F(v \cup \{k\}) - F(v))\]

and:

\[C(v) = C \sum_{i \in v} \left(\sum_{j \in v} \xi^N_{i,j}\right)^2\]

Stage 2 - Graphon stability: For two graphon mean-field systems with graphons $G_1, G_2$, prove stability estimates. The key evolution equation for relative entropy is:

\[H^u_t - H^u_0 = C \int_0^t \int P^{1,u}_s \left|\int_0^1 G_1(u,v)\langle b(x_u, \cdot), P^{1,v}_s \rangle dv - \int_0^1 G_2(u,v)\langle b(x_u, \cdot), P^{2,v}_s \rangle dv\right|^2\]

Applied to toy example: For $N=2$ with $\xi^2_{i,j} = G^2(i/2, j/2)/2$, the method gives bounds of order $1/N^2$ on both relative entropy and Fisher information between the finite system and its graphon limit.

Step 1 — Prior work:

• Lacker (2018): Developed BBGKY hierarchy method for exchangeable diffusions, proving sharp $O(k^2/N^2)$ propagation of chaos rates in relative entropy

• Lacker-Le Flem (2022): Extended BBGKY method to non-exchangeable systems, obtaining optimal bounds for relative entropy $H(P^{v,N}_t

  Q^{v,N}_t)$

• Grass (2024): Proved first Fisher information bounds for mean-field systems using BBGKY hierarchy with fine Hessian estimates

Step 2 — Delta: This paper extends Fisher information analysis to non-exchangeable diffusions and adds graphon stability estimates. Key additions are:

1. Fisher information bounds for independent projections via modified BBGKY hierarchy

2. stability estimates for graphon mean-field systems in both relative entropy and Fisher information.

   Step 3 — Theory-specific assessment:

   • Main theorem is predictable: combines known BBGKY techniques with existing Fisher information methods from [15]
   • Proof is mostly routine: assembles known lemmas from [22] and [15] without fundamentally new techniques
   • Bounds appear sharp for the finite-N case but no lower bounds are established for the graphon stability estimates
   • The connection to graphon theory is natural but the specific distance metric used differs from standard cut norm

   Verdict: INCREMENTAL — solid extension combining existing BBGKY methods with Fisher information techniques, but no breakthrough insights.

The proof uses a multi-stage BBGKY hierarchy approach:

1. Fisher information evolution: Apply Lemma 2.4 from [15] to get the fundamental inequality:

   \[\frac{d}{dt}I^v_t \leq -2\sum_{i,j \in v} \int P^v_t \left|\nabla^2_{x_i,x_j} \log \frac{P^v_t}{Q^v_t}\right|^2 + \text{coupling terms}\]

2. Coupling term bounds: The key technical step bounds:

   \[\sum_{(i,j) \in v^2} \int P^v_t \left|\nabla_{x_j}(b^{i,v}_2 - b^{i,v}_1)\right|^2\]

   using the Fisher information decomposition:

   \[I^{v \cup \{k\}}_t - I^v_t = \sum_{j \in v} \int P^{v \cup \{k\}}_t |\nabla_{x_j} \log P^{v \cup \{k\}|v}_t|^2 + \text{entropy term}\]

3. Differential inequality system: Combine with relative entropy bounds from [22] to get:

   \[\frac{d}{dt}Z^v_t \leq \mathcal{A}Z^v_t + C(v)\]

   where $Z^v_t = I^v_t + \alpha H^v_t$ and $\alpha$ is chosen to absorb negative Fisher information terms.

4. Graphon stability: Use Gronwall-type estimates with the exponential operator $e^{t\mathcal{A}}$ where $\mathcal{A}$ acts as:

   \[\mathcal{A}f(u) = \int_0^1 G(u,v)f(v)dv\]

   The proof that $e^{t\mathcal{A}}$ preserves positivity enables the Gronwall argument.

Purely theoretical. Empirical validation would require:

1. Numerical comparison of finite particle systems with their independent projections to verify the $O(

   v

   ^2/N^2)$ scaling

2. Testing graphon stability estimates on specific examples like Erdős-Rényi random graphs converging to constant graphons
3. Verification that the Fisher information bounds are tight through Gaussian examples as suggested in [15].

Limitations:

1. Smoothness assumption on $b$ with bounded derivatives - TECHNICAL (needed for integration by parts in Fisher information computations, likely removable with more careful analysis)
2. Uniform boundedness of $|\nabla^2 \log Q^{u,N}_t|$ - RESTRICTIVE (significantly narrows applicability, though Lemma 7.1 provides some framework)
3. Initial condition requirements in Hypothesis 2.1 - NATURAL (Gaussian bounds are standard in diffusion theory)

4. Graphon values bounded in $[0,1]$ - NATURAL (standard normalization in graphon theory)

   Open problems:

5. Establish lower bounds for graphon stability estimates to determine optimality of the $d(G_1,G_2)^2$ scaling
6. Extend to unbounded interaction functions $b$ or singular interactions as in [31]