Singular learning theory and statistical inference
Group Leader: Roberto-Paul Homeuca
Study Team: Roberto-Paul Homeuca
Supervisor: Calin Lazaroiu
Scheme:
A. Basic theory of Bayesian learning
B. Measures of model fitness; the KL divergence.
C. The Watanabe free energy and its relation to the KL landscape
D. Asymptotic expansion of Watanabe's free energy (the Watanabe formula)
E. Connection to Polchinski RG flow and the optimal transport problem
F. Sampling through random processes
G. Numerical studies
Bonus: History of Bayesian inference and of Bayesian models of learning.
References:
Papers and reviews
S. Watanabe, Review and Prospect of Algebraic Research in Equivalent Framework between Statistical Mechanics and Machine Learning Theory, Rev. Math. Phys (2025).
S. Watanabe, Mathematical Theory of Bayesian Statistics for Unknown Information Source, Phil. Trans. R. Soc. A.38120220151.
S. Watanabe, Recent Advances in Algebraic Geometry and Bayesian Statistics, Information Geometry 7 (2024) 187–209.
S. Watanabe, Information criteria and cross validation for Bayesian inference in regular and singular cases, Jap. J. Stat. & Data Science 4 (2021) 1–19
Books
S. Watanabe, Mathematical theory of Bayesian statistics, CRC Press, 2020
S. Watanabe, Algebraic geometry and statistical learning theory, Cambridge U.P., 2009.
Some background
J.W. Lamperti, Probability: a survey of the mathematical theory, Wiley, 1996.
Karatzas & Schreve, Brownian motion and stochastic calculus, Springer, 1991.
Bernardo & Smith, Bayesian theory, Wiley, 2009.
More background references here and here.
Note. Familiarity with the cognitive context provided on the page General Culture in Computer Science and AI is required for participating in this project. One of our motivations for exploring applications of Statistical Physics ideas and methods to Singular Learning Theory (SLT) is their potential to address problems of explainability and interpretability in the field of AI safety. See:
- Bereska & Gavves, Mechanistic Interpretability for AI Safety -- A Review, arXiv:2404.14082 [cs.AI]
Participation in this project also requires some familiarity with the mathematical theory of Machine Learning & Artificial Neural Networks. For an introduction to that subject from the viewpoint of statistical physics see:
- Roberts, Yaida & Hanin, The principles of deep learning theory, Cambridge U.P., 2022 (free draft)