Task-Dependent Algorithmic Equivalence of In-Context Learning

In-Context Learning (ICL) enables large language models to solve new tasks from a few demonstrations, yet existing theoretical accounts—gradient descent, Bayesian inference, kernel methods—each explain ICL under restricted assumptions. We hypothesise that the dominant algorithmic behaviour of ICL is task-dependent and introduce three computable task features—linear predictability (L), prior concentration (P), and nonlinear smoothness (S)—that predict which classical algorithm ICL most closely approximates. Four experiments on synthetic and real-world tasks validate the framework. Extreme task regions exhibit the predicted algorithmic specialisation; the empirical phase diagram reveals smooth but broad transition zones between regimes; real-world mapping achieves 75% prediction accuracy; and the dominant algorithm is preserved in 73.3% of architecture variants. A simplified theoretical analysis provides consistent explanations for the observed patterns, while the empirical phase diagram uncovers richer structure than the theory alone predicts, pointing to further factors governing ICL’s algorithmic selection.