Theory Digest — Apr 22, 2026

1. Motivation: Bandit algorithms are widely used in online decision-making, but their outputs can be sensitive to reward realizations, leading to inconsistent behavior across repeated experiments on the same underlying problem. This matters for scientific reproducibility and high-stakes applications where conclusions should not vary substantially due to sample-level randomness.

2. Mathematical setup:
   • Multi-Armed Bandits: $K$ arms with unknown means $\mu_a \in \mathbb{R}$ for $a \in [K]$. At round $t$, select arm $a_t$ and observe reward $r_t(a_t) = \mu_{a_t} + \eta_t$ where $\eta_t$ is zero-mean $R$-sub-gaussian. Let $\mu^* = \max_{a} \mu_a$ and gap $\Delta_a = \mu^* - \mu_a$.
   \[\text{RegMAB}(T) = \sum_{t=1}^T (\mu^* - \mu_{a_t})\]
   • Linear Bandits: Unknown parameter $\theta^* \in \mathbb{R}^d$ with $|\theta^*|_2 \leq S$. At round $t$, observe action set $\mathcal{A}_t \subset \mathbb{R}^d$, select $a_t \in \mathcal{A}_t$, observe reward $r_t(a_t) = \langle a_t, \theta^* \rangle + \eta_t$.
   \[\text{RegLB}(T) = \sum_{t=1}^T (\langle a_t^*, \theta^* \rangle - \langle a_t, \theta^* \rangle)\]

   Assumptions:

3. Rewards are $R$-sub-gaussian

4. For linear bandits: $|\theta^*|_2 \leq S$ and action sets are compact

5. Toy example: For MABs with $K=2$, $\mu_1 = 0.6$, $\mu_2 = 0.4$, so $\Delta_2 = 0.2$. Standard UCB might select different arms in early rounds due to noise, but replicable algorithm should select the same sequence across runs.

6. Formal objective: Design $\rho$-replicable algorithms where two runs with shared internal randomness but independent rewards produce identical action sequences with probability $\geq 1-\rho$.

RepUCB (Multi-Armed Bandits):

1. Use batched UCB with replicable mean estimation
2. Initialize by pulling each arm once, compute replicable estimates
3. Select arm $a_t \in \arg\max_a U_a$ where $U_a = \hat{\mu}_a + 2r_a$
4. Play selected arm for batch length $B = \min{N_{a_t}, T-t+1}$ rounds

5. Update only the played arm’s statistics using RepMean

   RepRidge (Ridge Regression): Given data ${(x_i, y_i)}_{i=1}^n$, compute standard ridge estimate:

   \[\hat{\theta}_n = V_n^{-1} \sum_{i=1}^n x_i y_i, \quad V_n = \lambda I + \sum_{i=1}^n x_i x_i^T\]

   Apply randomized grid rounding in whitened coordinates:

   \[z_n = V_n^{1/2} \hat{\theta}_n\] \[\tilde{\theta}_n = V_n^{-1/2} Q_{\alpha,u}(z_n)\]

   where $Q_{\alpha,u}$ rounds each coordinate to nearest grid point with width $\alpha = \frac{2\beta_n(\delta)\sqrt{d}}{\rho - 2\delta}$.

   RepLinUCB (Linear Bandits):

6. Use determinant-triggered batching with growth factor $q$
7. At batch start $t_b$, compute replicable ridge estimate $\tilde{\theta}_b$ using RepRidge
8. Throughout batch, select $a_t \in \arg\max_a {\langle a, \tilde{\theta}_b \rangle + \tilde{\beta}_b |a|_{V_{t_b}^{-1}}}$

9. Start new batch when $\det(V_{t+1}) > q \det(V_{t_b})$

   Toy example application: For $d=2$ linear bandit with $\theta^* = [1, 0]^T$, RepRidge would whiten the estimate, round to shared grid, then map back to preserve $V_n$-norm confidence guarantees.

Step 1 — Prior work:

• Esfandiari et al. (2023): First replicable bandits using elimination-based methods. For MABs: $O(\frac{K}{\rho}\sqrt{T\log T})$ regret. For linear bandits: $\tilde{O}(d^4\sqrt{T}/\rho^2)$ via discretization and G-optimal design.
• Impagliazzo et al. (2022): Introduced replicability framework with replicable mean estimation oracle.

Step 2 — Delta: This paper develops the first UCB-style (optimistic) replicable algorithms, replacing elimination approaches. Key technical contribution is RepRidge - direct replicable ridge regression via randomized grid rounding in whitened coordinates, avoiding discretization of action space.

Step 3 — Theory-specific assessment:

• Main theorem is somewhat predictable - combining existing techniques (batching + replicable estimation) with standard confidence analysis
• Proof technique for RepRidge (whitening + grid rounding) is routine coordinate-wise rounding, though application to matrix-weighted norms is reasonable
• Linear bandit bound $\tilde{O}((d + d^3/\rho)\sqrt{T})$ vs prior $\tilde{O}(d^4\sqrt{T}/\rho^2)$ improves by factor $O(d/\rho)$ - significant but not surprising given direct optimization vs elimination
• No matching lower bounds established; gap remains between upper bounds and known $\Omega(\sqrt{T}/\rho)$ MAB lower bound

Verdict: INCREMENTAL — Solid extension of replicable bandits to UCB framework with meaningful regret improvement, but uses predictable combination of existing techniques without fundamental algorithmic innovation.

RepUCB Analysis:

1. Union bound over $M = K(1 + \lceil\log_2 T\rceil)$ RepMean calls, each $\rho_1$-replicable with $\rho_1 = \rho/M$

2. Key inequality for suboptimal arm selection: if arm $a$ with gap $\Delta_a > 0$ is selected, then:

   \[3\tau(n) \geq \Delta_a \Rightarrow n \leq \frac{9C_{ME}\log(1/\delta_1)}{(\rho_1 - \delta_1)^2 \Delta_a^2}\]

3. Doubling batches limit pulls: $N_a(T) \leq 1 + 2H_a$ where $H_a$ bounds count when arm becomes suboptimal

   RepRidge Analysis:

4. Standard ridge confidence bound: $P(|\hat{\theta}_n - \theta^*|_{V_n} \leq \beta_n(\delta)) \geq 1-\delta$
5. Coordinate-wise rounding error bound: $|Q_{\alpha,u}(z) - z|_2 \leq \frac{\alpha\sqrt{d}}{2}$
6. Key replicability insight: if $|z^{(1)} - z^{(2)}|_2 \leq \frac{\alpha}{\sqrt{d}}$, then $Q_{\alpha,u}(z^{(1)}) = Q_{\alpha,u}(z^{(2)})$

7. Grid width chosen as $\alpha = \frac{2\beta_n(\delta)\sqrt{d}}{\rho - 2\delta}$ to balance accuracy vs replicability

   RepLinUCB Analysis:

8. Inductive replicability over batches: identical actions up to batch start $\Rightarrow$ same design matrix $\Rightarrow$ RepRidge produces identical estimates
9. Elliptic potential analysis for regret: $\sum_{t=1}^T |a_t|_{V_t^{-1}}^2 \leq 2d\log\left(1 + \frac{TL^2}{\lambda d}\right)$
10. Batch count bounded by $B = d\log(1 + TL^2/\lambda d)$ via determinant growth condition

Purely theoretical.

Empirical validation would require:

1. Comparison with elimination-based replicable algorithms on synthetic MAB/linear bandit instances
2. Replicability verification: measuring action sequence agreement across independent runs
3. Regret comparison with standard non-replicable UCB algorithms to quantify price of replicability
4. Sensitivity analysis showing how performance varies with replicability parameter $\rho$

Limitations:

1. MAB regret $O(K^2\log^2 T/\rho^2)$ vs known $\Omega(\sqrt{T}/\rho)$ lower bound - gap suggests room for improvement (TECHNICAL - likely improvable with refined analysis)
2. Linear bandit requires oblivious adversary for action sets, unlike standard LinUCB which handles adaptive adversaries (TECHNICAL - needed for replicability proof but could potentially be relaxed)
3. RepRidge requires $\rho > 3\delta$ constraint (NATURAL - standard requirement for replicable estimation)

4. Batching introduces delays in adapting to new information (NATURAL - fundamental tradeoff for replicability)

   Open problems:

5. Close regret gap for MABs: develop algorithms achieving $O(\sqrt{T}/\rho)$ regret or prove stronger lower bounds
6. Extend replicable optimistic methods to non-linear contextual bandits and broader structured bandits

1. Motivation: Fine-tuning has revolutionized machine learning for adapting pre-trained models to new tasks, but its theoretical properties in high-dimensional nonparametric settings with variable selection remain underdeveloped. This problem is crucial for transfer learning applications where target domains have limited data but can leverage knowledge from data-rich source domains.

2. Mathematical setup: Consider source data ${(x^P_i, y^P_i)}_{i=1}^{n_P} \stackrel{iid}{\sim} P$ and target data ${(x^Q_j, y^Q_j)}_{j=1}^{n_Q} \stackrel{iid}{\sim} Q$ where $(x,y) \in \mathbb{R}^p \times \mathbb{R}$. The covariates admit factor structures:

   \[x^P = B^P f^P + u^P\] \[x^Q = B^Q f^Q + u^Q\]

   where $B^P, B^Q \in \mathbb{R}^{p \times r}$ are loading matrices with $p \gg r$, $f^P, f^Q \in \mathbb{R}^r$ are latent factors, and $u^P, u^Q \in \mathbb{R}^p$ are idiosyncratic components. The regression functions are:

   \[y^P_i = g^P(x^P_{i,J^P}) + \epsilon^P_i\] \[y^Q_j = g^Q(f^Q_j, u^Q_{j,J^Q}) + \epsilon^Q_j\]

   The key transferability assumption decomposes the target function as:

   \[g^Q(f^Q, u^Q_{J^Q}) = h(f^Q, u^Q_J, g^P(x^Q_{J^P}))\]

   where $h$ is a residual fine-tuning function and $J^Q = J \cup J^P$.

   Assumptions:

   1. Factor loading similarity: $p^{-1}|B^P(B^P)^\top - B^Q(B^Q)^\top|_F \leq \epsilon$
   2. Bounded support and weak dependence of factors and idiosyncratic components
   3. Hierarchical composition structure for regression functions

3. Toy example: When $r=1$, $p=3$, and $J^P = J^Q = {1}$, we have $x^Q = B^Q f^Q + u^Q$ with $B^Q = [b_1, b_2, b_3]^\top$. The target function becomes $g^Q(f^Q, u^Q_1) = h(f^Q, g^P(x^Q_1))$ where $x^Q_1 = b_1 f^Q + u^Q_1$. This illustrates how the pre-trained source function $g^P$ serves as an additional feature for learning the simpler residual function $h$.

4. Formal objective: Minimize the excess risk on the target distribution:

   \[E^Q(\hat{m}) := \mathbb{E}[|\hat{m}(x^Q) - g^Q(f^Q, u^Q_{J^Q})|^2]\]

The Fine-tuning Factor Augmented Neural Lasso (FAN-Lasso) operates in three stages:

1. Factor Transfer: Use Transfer Around Boundary (TAB) to construct a model-selected covariance matrix:

   \[\hat{\Sigma}^{TL} = \begin{cases} \hat{\Sigma}^A & \text{if } p^{-1}\|\hat{\Sigma}^Q - \hat{\Sigma}^A\|_F \leq \delta \\ \hat{\Sigma}^Q & \text{otherwise} \end{cases}\]

   where $\hat{\Sigma}^A = \frac{n_P}{n_P+n_Q}\hat{\Sigma}^P + \frac{n_Q}{n_P+n_Q}\hat{\Sigma}^Q$ is the pooled covariance. Extract surrogate factors:

   \[\tilde{f}^Q = p^{-1}(\hat{W}^{TL})^\top x^Q\]

2. Source Function Training: Train FAST-NN on source data using clipped-$\ell_1$ penalty:

   \[(\hat{g}^P, \hat{\Theta}^P) = \arg\min_{g \in \mathcal{G}^P, \Theta^P} \frac{1}{n_P}\sum_{i=1}^{n_P} [y^P_i - g([\tilde{f}^P_i, \text{trun}_M((\Theta^P)^\top x^P_i)])]^2 + \lambda_P \sum_{i,j} \psi_\tau(\Theta^P_{i,j})\]

   where $\psi_\tau(x) = \frac{

   x

   }{\tau} \wedge 1$ is the clipped-$\ell_1$ function.

3. Fine-tuning: Include pre-trained source predictions as features and train target network:

   \[(\hat{h}, \hat{\Theta}) = \arg\min_{h \in \mathcal{G}, \Theta} \frac{1}{n_Q}\sum_{j=1}^{n_Q} [y^Q_j - h([\tilde{f}^Q_j, \text{trun}_M(\Theta^\top x^Q_j), \hat{s}_j])]^2 + \lambda \sum_{i,j} \psi_\tau(\Theta_{i,j})\]

   where $\hat{s}_j = \hat{g}^P(x^Q_j)$ are the transferred source predictions.

   Application to toy example: With $r=1$, $J^P = J^Q = {1}$, the method learns $\hat{g}^P(x^Q_1)$ from source data, then trains a 2-dimensional network $\hat{h}(\tilde{f}^Q, \hat{g}^P(x^Q_1))$ on target data, dramatically reducing complexity from learning the full function $g^Q$.

Step 1 — Prior work:

1. Fan & Gu (2024): “FAST-NN” established minimax rates for factor-augmented sparse neural networks in single-domain high-dimensional nonparametric regression
2. Li et al. (2022): Minimax rates for transfer learning under sparse parameter differences in linear models

3. Cai & Pu (2024): Nonparametric transfer learning bounds without factor structure

   Step 2 — Delta: This paper adds:

4. Novel residual fine-tuning decomposition $g^Q = h(f^Q, u^Q_J, g^P(x^Q))$ that freezes source function
5. Unified framework handling both covariate shift ($B^P \neq B^Q$) and posterior shift ($g^P \neq g^Q$) simultaneously

6. Minimax-optimal rates characterizing precise conditions for statistical acceleration.

   Step 3 — Theory-specific assessment:

   • Main theorem: The rate $\frac{\log(n_P+n_Q)}{n_P+n_Q}^{\frac{2\gamma_P}{2\gamma_P+1}} + \frac{\log n_Q}{n_Q}^{\frac{2\gamma}{2\gamma+1}}$ is somewhat predictable given the decomposition structure, but the precise characterization of when fine-tuning helps is non-obvious.
   • Proof technique: The residual fine-tuning decomposition is genuinely new, allowing frozen source functions while maintaining minimax optimality. The analysis cleverly separates source and target complexities.
   • Tightness: Authors provide matching minimax lower bounds, confirming rates are sharp up to logarithmic factors.

   The automatic robustness to negative transfer (graceful degradation when source data doesn’t help) emerges organically from the estimator structure without requiring oracle knowledge.

   Verdict: SIGNIFICANT — Clear advance in transfer learning theory with novel residual decomposition and unified treatment of distribution shifts.

The proof strategy has several key components:

1. Factor Transfer Analysis: Uses matrix perturbation theory to bound the estimation error of the diversified projection matrix. The key inequality is:

   \[\nu_{\min}(p^{-1}(\hat{W}^{TL})^\top B^Q) \geq c_1 - c_2(\delta \wedge \varepsilon_A(t))\]

   This ensures the signal from latent factors dominates over the idiosyncratic noise.

2. Source Function Approximation: Leverages deep ReLU network approximation theory for hierarchical composition models. The source function complexity is controlled by:

   \[\|g^P - \hat{g}^P\|_{L_2} \lesssim \left(\frac{\log(n_P + n_Q)}{n_P + n_Q}\right)^{\frac{\gamma_P}{2\gamma_P + 1}}\]

   using the adaptive capacity of neural networks to unknown composition structures.

3. Residual Function Estimation: The core technical insight is decomposing the target excess risk as:

   \[E^Q(\hat{m}^{FT}) \lesssim \underbrace{\|g^P - \hat{g}^P\|^2}_{\text{source error}} + \underbrace{\|h - \hat{h}\|^2}_{\text{residual error}}\]

4. Minimax Lower Bound: Constructs worst-case scenarios using Le Cam’s method and Fano’s inequality. The key step shows that any estimator must pay the cost:

   \[\inf_{\hat{m}} \sup_{g^P, h} E^Q(\hat{m}) \geq c \left[\frac{1}{n_P + n_Q}^{\frac{2\gamma_P}{2\gamma_P+1}} + \frac{1}{n_Q}^{\frac{2\gamma}{2\gamma+1}}\right]\]

5. Negative Transfer Protection: The automatic robustness emerges from the rate decomposition - when source data doesn’t help (e.g., $n_P \ll n_Q$ or $\gamma_P \geq \gamma$), the first term vanishes and the estimator achieves the optimal single-task rate.

The paper presents extensive numerical experiments across diverse scenarios:

Datasets and Settings:

• Synthetic data with controlled covariate and posterior shifts
• Various combinations of $(n_P, n_Q, p, r)$ with $p \in {100, 500}$, $n_Q \in {50, 100, 200}$
• Different shift magnitudes and function complexities

Baselines:

• Single-task FAST-NN (target-only)
• Standard transfer learning methods
• Oracle estimators with known factors

Key Results:

• FAN-Lasso consistently outperforms baselines across shift scenarios
• Achieves near-oracle performance even with severe target sample constraints ($n_Q = 50$)
• Demonstrates robustness to negative transfer when source-target similarity is low
• Empirically validates the derived convergence rates

The experiments span both covariate shift (varying $B^P, B^Q$) and posterior shift scenarios, confirming theoretical predictions about when fine-tuning provides statistical acceleration.

Limitations:

1. TECHNICAL: Requires knowledge of factor dimension $r$ and sparsity pattern structure, though the method is robust to moderate overestimation of $r$

2. TECHNICAL: Hierarchical composition assumption on regression functions, while general, may not capture all nonparametric structures of practical interest

3. RESTRICTIVE: Bounded support assumptions on factors and covariates limit applicability to heavy-tailed distributions commonly encountered in finance and other domains

4. NATURAL: Sub-Gaussian noise assumption is standard in the field

5. TECHNICAL: The TAB threshold $\delta$ requires tuning, though the method is not overly sensitive to its choice

   Open Problems:

6. Adaptive factor dimension selection: Develop theory for data-driven selection of working factor dimension $\bar{r}$ without requiring prior knowledge of true dimension $r$

7. Heavy-tailed extensions: Extend the framework to handle heavy-tailed distributions for factors and noise, which are prevalent in financial and other real-world applications

Motivation: In-context learning (ICL) in transformers has been studied under stationary regression settings, but real-world applications often involve non-stationary tasks where the underlying function evolves over time. Understanding ICL in such dynamic environments is crucial for applications like time-series forecasting and streaming data analysis.

Mathematical setup: Consider a sequence of regression tasks where the $i$-th task has target function:

\[y_i = f_i(x_i) = \langle w_i, x_i \rangle\]

The weight vectors evolve according to a first-order autoregressive process:

\[w_i = \gamma w_{i-1} + e_i, \quad i \in [n+1]\]

where:

1. Initial weight: $w_0 \stackrel{i.i.d.}{\sim} N(0, \sigma_w^2 I)$
2. Noise terms: $e_i \stackrel{i.i.d.}{\sim} N(0, \sigma_e^2 I)$
3. Input vectors: $x_i \stackrel{i.i.d.}{\sim} N(0, \Lambda)$
4. Autoregressive parameter: $0 < \gamma < 1$

5. All random variables $w_{i-1}, e_i, x_i$ are mutually independent

   Toy example: When $d=2$, $\Lambda = I_2$, $\gamma = 0.9$, and $\sigma_e^2 = 0.01$, the weight vector $w_2$ retains 90% of $w_1$ plus small random drift. This creates a scenario where recent examples are more predictive than older ones.

   Formal objective: Train a gated linear attention (GLA) model to minimize the population risk:

   \[L(\theta) = \frac{1}{2} E_{w_{n+1},x_{n+1}}[(\hat{y}_{n+1} - \langle w_{n+1}, x_{n+1} \rangle)^2]\]

The method uses Gated Linear Attention (GLA) which extends standard linear attention with a forgetting mechanism. The GLA output at position $i$ is:

\[o_i = S_i q_i \quad \text{and} \quad S_i = \lambda S_{i-1} + v_i k_i^T\]

where $\lambda \in (0,1]$ is the forgetting factor. Unrolling the recursion gives:

\[S_{n+1} = \sum_{i=1}^{n+1} \lambda^{n+1-i} v_i k_i^T\]

The prediction is:

\[\hat{y}_{n+1} = \left[w_{V,21}^T \quad w_{V,-1}\right] \left(\sum_{i=1}^{n+1} \lambda^{n+1-i} z_i z_i^T\right) \begin{bmatrix} W_{KQ,11} \\ w_{KQ,21}^T \end{bmatrix} x_{n+1}\]

Applied to toy example: With $d=2$, $n=3$, the model computes weighted sums $\sum_{i=1}^4 \lambda^{4-i} z_i z_i^T$ where recent examples get higher weights when $\lambda < 1$. For $\lambda = 0.9$ and $\gamma = 0.9$, the model automatically learns to emphasize recent observations that better predict the current target.

Prior work:

1. “What can transformers learn in-context?” (Garg et al., 2022) - showed ICL capabilities in stationary linear regression
2. “Transformers learn to implement preconditioned gradient descent” (Ahn et al., 2023) - analyzed single-step GD implementation in stationary settings

3. “In-context convergence of transformers” (Zhang et al., 2024) - proved convergence for linear attention under stationary Gaussian data

   Delta: This paper extends ICL theory to non-stationary regression by:

   • Introducing first-order autoregressive weight evolution model
   • Analyzing GLA’s advantage over standard linear attention in time-varying settings
   • Providing closed-form expressions for training/testing errors with explicit dependence on $\gamma$ (non-stationarity) and $\lambda$ (forgetting factor)

   Theory-specific assessment:

   • Main theorem: Convergence to global minimum is somewhat predictable given prior stationary results, but explicit characterization of optimal $\lambda \neq \gamma$ is non-obvious
   • Proof technique: Largely assembles known techniques (gradient flow analysis, Gaussian calculations) but requires careful tracking of autoregressive dependencies
   • Bound tightness: No lower bounds provided; expressions are exact for the Gaussian setting but may not be tight for general distributions

   Verdict: INCREMENTAL — solid extension of existing stationary ICL theory to an important but natural non-stationary setting.

The proof uses gradient flow analysis with several key components:

1. Quadratic reformulation: Express the GLA prediction as:

   \[\hat{y}_{n+1} = u^T H u\]

   where $H = \frac{1}{2}X \otimes \left(\sum_{i=1}^{n+1} \lambda^{n+1-i} z_i z_i^T\right)$

2. Gradient flow dynamics: The parameter evolution satisfies:

   \[\frac{dU_{11}(t)}{dt} = -u_{-1}^2 \tilde{\Lambda}\Lambda U_{11}\Lambda + D_1 u_{-1}\Lambda^2\] \[\frac{du_{-1}(t)}{dt} = -\text{trace}(u_{-1} \tilde{\Lambda}\Lambda U_{11}\Lambda U_{11}^T) + D_1 \text{trace}(\Lambda^2 U_{11}^T)\]

3. Key autoregressive calculation: Using the first-order AR model, the cross-covariance is:

   \[E[w_a w_b^T] = \begin{cases}\] \[(\sigma_w^2 + \min\{a,b\}\sigma_e^2)I & \gamma = 1 \\\] \[(\gamma^{a+b}\sigma_w^2 + \frac{\gamma^{a+b} - \gamma^{|a-b|}}{\gamma^2-1}\sigma_e^2)I & \gamma \neq 1\] \[\end{cases}\]

4. Convergence analysis: Shows convergence to global optimum:

   \[\lim_{t \to \infty} W_V(t) = \sqrt{\frac{D_1}{\|\tilde{\Lambda}^{-1}\|_F}} \begin{bmatrix} 0_{d \times d} \\ 0_d \\ 0_d^T \\ 1 \end{bmatrix}\]

   The technical insight is that the optimal $\lambda$ balances tracking recent changes ($\sigma_e^2$ terms) versus leveraging longer history ($\sigma_w^2$ terms).

Synthetic experiments:

• Setting: $d=10$, $n=100$, $\sigma_w^2=1$, $\sigma_e^2=0.01$, $B=10^7$ training tasks
• Optimizer: AdamW with learning rate $10^{-2}$, 2000 epochs, batch size 5000
• Key finding: Optimal $\lambda < 1$ minimizes both training/testing error, confirming theoretical prediction of inverse U-shaped loss

Multi-layer experiments:

• Deeper GLA models (2-4 layers) consistently outperform single layer
• Each layer captures different timescales of non-stationarity
• Linear convergence maintained across depths

Real-world validation:

• SST-2 sentiment classification: GatedLinearGPT2 vs LinearGPT2 with 20 in-context demonstrations
• MNLI natural language inference: 10 demonstrations, 3-class prediction
• GLA achieves higher accuracy and confidence across varying demonstration counts
• Results support non-stationarity hypothesis in real language tasks

Baseline comparison:

• LMS training errors: [0.264, 0.317, 0.606, 1.007, 1.476] for $\gamma \in [0.8, 0.85, 0.925, 0.95, 0.975]$
• RLS training errors: [0.256, 0.375, 0.666, 0.888, 1.292]
• GLA outperforms both due to cross-sequence learning capability

Limitations:

1. TECHNICAL: Gaussian assumptions throughout (initial weights, noise, inputs) - needed for closed-form analysis but restrictive for real applications

2. TECHNICAL: First-order autoregressive model - higher-order dynamics or more complex temporal patterns not captured

3. TECHNICAL: Single global forgetting factor $\lambda$ - original GLA uses data-dependent per-token gating which may be more powerful

4. RESTRICTIVE: Initialization constraint in Theorem 1 (though experiments suggest it’s not necessary in practice)

5. RESTRICTIVE: Analysis limited to linear regression - nonlinear function classes remain open

   Open problems:

6. Multi-layer theory: Provide rigorous convergence analysis for deep GLA networks, explaining why multiple layers capture different timescales

7. Optimal architecture design: Characterize the trade-off between number of layers, forgetting factors, and types of non-stationarity for optimal ICL performance