Do Nonlinear Transformers Implement Kernel Regression In-Context? An Empirical Investigation

Recent work has shown that linear transformers can perform in-context learning (ICL) by implicitly implementing ridge-regression-like algorithms. However, practical transformers use softmax attention and nonlinear MLPs, and it remains unclear to what extent their ICL behavior can still be interpreted through a kernel-regression lens. We study this question through a kernel surrogate hypothesis, which compares a trained nonlinear ICL predictor with kernel ridge regression (KRR) using a combined attention- and MLP-inspired kernel. Rather than claiming an exact equivalence, we evaluate this hypothesis through controlled synthetic experiments on polynomial regression, Gaussian process (GP) regression, and sparse linear regression. Across these tasks, we compare trained single-layer transformers with closed-form KRR reference predictors in terms of predictive error, context-length scaling, attention-weight alignment, and model-width sensitivity. The results show that the KRR surrogate closely follows the Bayes reference on GP tasks, whereas trained transformers achieve high correlation with KRR outputs on polynomial tasks (r > 0.90) but retain a persistent MSE gap. Attention analysis further shows that transformer attention becomes more aligned with KRR contribution weights during training, although the alignment remains moderate. Overall, these findings suggest that kernel regression provides a useful reference-level description of nonlinear ICL in controlled settings, but not a complete account of finite-width trained transformer behavior.