Machine learning & artificial neural networks

For cognitive context, see Computer Science and General culture in computer science & AI.

Also see the Student project on Statistical Physics applied to SLT and BL.

Machine learning

History

Timeline | Brief

Introductory

Deisenroth, Faisal, Ong, Mathematics for machine learning

Strang, Linear algebra and learning from data

MacKay, Information theory, inference and learning algorithms

James et al, An introduction to statistical learning

Kearns and Vazirani, An introduction to computational learning theory

Devroye et al, A probabilistic theory of pattern recognition

Intermediate

Mitchell, Machine learning

Wilmott, Machine learning

Shalev-Shwartz & Ben-David, Understanding machine learning

Zhang, Mathematical analysis of machine learning algorithms

Mohri, Rostamizadeh & Ameet Talwalkar, Foundations of machine learning

Murphy, Machine learning: a probabilistic perspective

Bishop, Pattern recognition and machine learning

Hastie, Tibschirani & Friedman, The elements of statistical learning

Vapnik, The nature of statistical learning theory

Kaplan, Notes on contemporary machine learning for physicists

Advanced

Mumford & Desolneux, Pattern theory

Grohs & Kutyniok (eds), Mathematical aspects of deep learning

Huang, Statistical mechanics of neural networks

Singular Learning Theory 

Watanabe, Review and Prospect of Algebraic Research in Equivalent Framework between Statistical Mechanics and Machine Learning Theory

Watanabe, Mathematical Theory of Bayesian Statistics for Unknown Information Source

Watanabe, Mathematical theory of Bayesian statistics

Watanabe, Algebraic geometry and statistical learning theory

• For more references, see the student project on Singular learning theory and statistical inference.

Artificial neural networks

History 

Domingo, The master algorithm

Philipp Schmitt's, Blueprints for intelligence: a visual history of ANNs

Anderson & Rosenfeld, Talking nets: an oral history of neural networks

Wang & Raj, On the origin of deep learning

Schmidhuber, Deep learning in neural networks: an overview

Juergen Schmidhuber's homepage

Introductory

Classical

Rojas, Neural networks: a systematic introduction

Minski & Papert, Perceptrons: an introduction to computational geometry

Anthony & Bartlett, Neural network learning: theoretical foundations

Murphy, Probabilistic machine learning: an introduction

Deep learning

Prince, Understanding deep learning

Calin, Deep learning architectures: a mathematical approach

Bishop & Bishop, Deep learning: Foundations and concepts

Goodfellow, Bengio & Courville, Deep learning

Roberts, Yaida & Hanin, The principles of deep learning theory (free draft)

Intermediate

Haykin, Neural networks and learning machines

Aggarwal, Neural networks and deep learning

Advanced

Bronstein, Bruna, Cohen & Velickovic, Geometric deep learning

Attention mechanism

IBM think | wiki | GfG | medium | adaline

Niu et al, A review of the attention mechanism of deep learning

Soydaner, Attention mechanism in neural networks: where it comes and where it goes

Guo, Attention mechanisms in computer vision: a survey

Sun et al, Efficient attention mechanisms for large language models: a survey

Hernandez & Amigo, Attention mechanisms and their applications to complex systems

Ruan & Zhang, Towards understanding how attention mechanism works in deep learning

...

Transformers

IBM think | wiki | eventum

Vaswami et al, Attention is all you need

Phuong and Hutter, Formal algorithms for transformers

Lin et al, A survey of transformers

Fnu et al, Understanding the architecture of vision transformer and its variants: A review

Gumaan, Universal Approximation Theorem for a Single-Layer Transformer

Ravindran, Adversarial Activation Patching: A Framework for Detecting and Mitigating Emergent Deception in Safety-Aligned Transformers

Omidi et al, Memory-augmented transformers

GPTs: 

...

MoE

IBM think | wiki

Zhang et al,  Mixture of Experts in Large Language Models

LLMs & RLLMs

Zhang et al,  Survey of Large Language Models in Extended Reality

Liu et al, A Comprehensive Evaluation on Quantization Techniques for Large Language Models

Zhang et al, A Survey of Reinforcement Learning for Large Reasoning Models

LLM alignment, explainability & interp 

Pan et al, A Survey on Training-free Alignment of Large Language Models

Palikhe et al, Towards Transparent AI: A Survey on Explainable Language Models

...

Multimodal LLMs

IBM think | GfG | medium

Jaegle et al, Perceiver: general perception with iterated attention

Jaegle et al, PerceiverIO: A general architecture for structured inputs and outputs

Song et al, How to bridge the gap between modalities

Bae et al, Graph perceiver IO: a general architecture for graph-structured data

Carolan et al, A review of multimodal large language and vision models

Kibria et al: Decoding the multimodal maze

Chen et al: A survey of multimodal hallucination evaluation and deception

Xu et al, MARS2025 challenge

...

Embodied

Sanghai & Brown, Advances in transformers for robotics applications: a review

Shao et al, Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Survey

Applications to science

Hu et al, A survey of scientific large language models

Wei et al, From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discovery

Ma et al, A Survey of Deep Learning for Geometry Problem Solving

...

 

General culture in computer science & AI

For structured references see here.

A. Computer science 

Overview

I. Basic notions on computer architecture and function

1. Computer architecture

a.  Instruction set architecture

-Overview

-CISC, RISC, VLIW, EPIC

-x86 and ARM

b. Microarchitecture

The von Neumann and Harvard models

2. Programming language

a. Low level programming language

-Machine code

-Assembly language; assembler; disassembler

b. High level programming language

-Interpreter 

-Compiler

II. Networking

TCP/IP and the OSI model

WAN: ISPs | FTTx | 5G

LAN: Ethernet (over fiber or twisted pair) | WIFI 

PAN: Bluetooth

IoT: Thread and Zigbee protocols | The Matter standard (wiki) from CSA (wiki)

III. Free and Open Source Software

FOSS

Free: FSF; FSFe; GNU

Open Source: OSI; OFE

Unix and Linux

GitHub

People and movement

Stallman; Raymond; Torvalds

The GNU Manifesto | The Cathedral and the Bazaar

IoT

Air Gradient | Home Assistant (wiki) and Nabu Casa

IV. Programming

1. The Church-Turing thesis

2. Classical models of deterministic computation

a. Turing machine

b. Lambda calculus

c. General recursive functions

3. The Curry-Howard correspondence

4. Formal grammar

5. Context-free grammars 

a. Backus-Naur form

b. Syntax diagram

5. Programming paradigms:

a. Declarative: logical, functional, AML, Query, Regexp

b. Imperative: procedural, object-oriented

6. System programming; system programming language

7. General purpose language

8. Domain-specific language

9. API

10. Automatic programming

11. Evolutionary algorithms

a. Evolutionary programming

b. Genetic programming

12. History

V. Concurrent and distributed computing

1. Concurrency

a. Concurrent computing

b. Process

c. Thread

d. Parallel computing

e. Process calculus

f. The actor model

2. Real time operating systems (RTOS)

3. Distributed computing

a. Grid computing

b. Cloud computing

c. Edge computing 

-HBM

-NPU

-AI PC

d. IoT

e. Fog computing

f. Ubiquitous computing

g. Distributed algorithms

4. High performance computing (supercomputers & clusters)

a. History | FLOPS | Orders | CSNET | NSFNET

b. Exascale (DOE | ECP) 

-Europe (Euro HPC | PRACE) | Japan | China

- Frontier | Aurora | El Capitan | Jupiter (wiki, JS @FZJ) | Fugaku

c. Resources: NESRC | ACM's SIGHPC | HPCwire | InsideHPC | Top500

d. Companies: HPE | IBM | NVIDIA | Intel | AMD  | Dell | Atos (Eviden) | Fujitsu 

e. Frontier: Zettascale | Neuromorphic | Superconducting | Beyond CMOS | Unconventional | Hypercomputation

VI. Influential people

1. Alan Turing

2. Alonzo Church

3. John von Neumann

4. Norbert Wiener

5. Claude Shannon

6. Richard Stallman

7. Bill Joy

VII. Resources

IEEE Computer Soc. | ACM 

Berkeley | MIT | Carnegie Mellon | Stanford | Cambridge | Oxford | Melbourne

B. Artificial intelligence

Outline 

For structured introductions to AI see:

Russell & Norvig: Artificial intelligence: a modern approach

MIT courseware

I. Basic notions and techniques

1. The Dartmouth workshop of 1956

2. The cognitive revolution

3. Expert systems

4. Automated theorem proving

5. Natural language processing

6. Computer vision

7. Speech recognition

8. The common sense knowledge problem

9. AI winter

10. Distributed AI

a. Software agents

b. Multi-agent systems

c. Swarms

d. Artificial life

e. Artificial society | Social simulation | Computational sociology

f. Intelligent agents and agentic AI 

-Communication, orchestration & protocols

g. Agentic Web (a review)

-Protocols: MCP | A2A | ACP  (IBM think) | ANP | Agora

-Agentic browsers: Open AI Operator (wiki) | Fellou |  Perplexity Comet | Opera Neon | Chrome add-on 

h. Agentic programming

11.  Neural networks

History

a. Neuronal models & computational neuroscience:

-Connectionism | Blue Brain (wiki) | Brain/MINDS (wiki) | MICrONS (wiki) | China Brain Project 

-Biological neuron models | Neural backpropagation

-Neural learning & computation: Hebbian | BCM | Generalized Hebbian (Oja's rule) | Synaptic plasticity (STDP, SNNs)

b. Artificial neurons | perceptrons 

c. Artificial neural networks

-Feedforward (MLP/FCN | Autoencoders | TDNN  | CNN & CapsNet | PNN | RBFN | ITNN)

-Recurrent (LSTM | BRNN)

-SVM

-GMDH-based (MNN, Polynomial, Abductive)

-Transformers | Attention is all you need (paper)

12. Machine learning

a. Paradigms (supervised | unsupervised | reinforcement | self-supervised)

b. Techniques: transfer | feature (geometric) | hyperparameter optimization | MTL | ZSL | OSL | EBL | auto | meta | NAS | neuroevolution

c. Backpropagation | Attention

d. Bayesian learning

e. History | Timeline

13. Deep Learning

-Deep neural networks

-Foundation models | Stanford CRFM

-Multimodal DL

-Feedback NNs

-Neural scaling laws

-Test time computation: intro for RLMs

14. Generative AI

-SLMs

-LLMs

-RLMs

-DMs (LVM | HMM)

-GANs

15. Affective computing

16. Hybrid AI (wiki)

16. Narrow AI

17. The AI boom

A list of LLMs | Criticism

-ML companies & labs: DeepMind (Gemini) | Open AI (ChatGPT) | Anthropic (Claude) | MetaAI (Llama) |  IBM (WatsonX, Granite) | HuggingFace | Mistral | DeepSeek (homepage) | Alibaba Cloud (Qwen) | Microsoft (365 Copilot & AI2)

-Hyperscale computing | Hyperscalers

-Wafer-scale integration | Cerebras (wiki)

18. History 

-Timeline 

-Online briefs: GfG | SCI | AI100 | IBM

-Nilsson, The quest for artificial intelligence

-Mitchell, Artificial intelligence: a guide for thinking humans

-Fradkov, Early history of machine learning

II. Ethics and safety 

Ethics of AI 

1. Explainabiliy

2. Mechanistic interpretability

3. Alignment

-General:

What is AI alignment ? 

Christian, The alignment problem: machine learning and human values

-Groups:

ARC (Berkeley) | Ada Lovelace Institute (UK) | The Future Society (US & EU)

4. Hallucinations

5. Deception

-Deceptive machines 

-AI deception: a survey

6. Machine ethics and friendly AI

7. Safety

The UK AI safety summit (UK, 2023)| The Bletchley Declaration

The AI Seoul Summit (South Korea, 2024) | The Seoul Declaration

The AI Action Summit (France, 2025) | The Paris Statement

The First International AI Safety Report

The AI Impact Summit (India, 2026)

AISI centers

US:  CAISI and AISIC (National, @NIST) | CHAN (UC Berkeley)| CAIS (San Francisco) | CAISER (Oak Ridge) | AI Action Plan

EU: European AI Office| AI Board

Some national AISIs: UK | Japan | South Korea | Singapore

China: CNAISDA | AIDSN

Australia

Other:

US: IAPS | CSET | CHAI | Redwood | Palisade | FMF | Epoch | ARC & METR | ARI | CivAI | FAR | Apart | LawAI | AIFutures

UK: CLTR  | GovAI | Apollo | CLTR | Forethought | MATS | ARENA

EU: CFG  | Talos | CSH

Global: TFS | SAIF | Constellation | 80000hours

Some papers:

TASRA: a Taxonomy and Analysis of Societal-Scale Risks from AI (2023)

Two Types of AI Existential Risk: Decisive and Accumulative (2024)

Mechanistic Interpretability for AI Safety -- A Review (2024)

Safety Cases: How to Justify the Safety of Advanced AI Systems (2024)

Gradual disempowerment: Systemic Existential Risks from Incremental AI Development (2025)

AI-Enabled Coups: How a Small Group Could Use AI to Seize Power (2025)

III. Governance and policy

Regulation of AI | Global governance [GAIGO]

EU: The EU AI Act (law, wiki) ] | AI Innovation Package [AI Factories, AI Gigafactories , Gen4AIEU, NoE]| The Continent Action Plan | European approach to AI | European AI Alliance | AI Forum

US: America's AI Action Plan | The Stargate project (wiki)

China: Global AI Governance Initiative | Global AI Governance Action Plan

UN: Independent international scientific panel on AI

IV. Frontier, speculative and philosophical

General interest books: 

Kissinger, Schmidt & Huttenlocher, The age of AI and our human future

Suleyman & Bhaksar, The coming wave

Russell, Human compatible

Ord: The precipice | revisted

Togelius, Artificial general intelligence

Bennett, A brief history of intelligence

Kurzweil, The singularity is nearer

1. Artificial General Intelligence (AGI)

a. The Turing test

b. Artificial consciousness

c. Computational neuroscience

d. Philosophy of AI and Philosophy of mind 

-The Chinese room argument and hard problem of consciousness

-Unified Theories of Cognition and The Conscious Mind

-The binding problem

-Situated cognition

-Self-awareness and NCC

2. Computational cognition

-The CRUM hypothesis

-Statistical learning and SLA

3. Cognitive models and architectures

- SoM, MDM, GWT, MPF and DCM, DC

-Conceptual spaces, cognitive maps, image schemas

-ACT-R, Soar, 4CAPS, DUAL, CLARION, LIDA, SDM

-MindModeling@home

4. Machine theory of mind

-A review

-ToM in cognitive science 

-Computational ToM (also here)

5. Superintelligence

6. Recursive self-improvement and Technological singularity

Some essays:

Situational Awareness | AI Futures

V. Socio-political

1. AI and elections

2. National Security

-The AI arms race  

-Hendricks, Schmidt & Wang's Superintelligence Strategy (expert version)

3.  Military applications

-JADC2

-CDAO (wiki)

-Project Maven

-5GW

4. Social

-Predictive analytics and policing

-Lawbots, computational law and legal informatics

-Automated journalism | Virtual assistants | Machine translation | AVs

-AI art: visual | music | film | architecture | games

-Digital twins  | digital cloning | deepfakes | companions

-UBI

5. Ontological security

-Overview

-Ontological security theory in International relations

6. Existential risk

-CSER (wiki, TERRA, LCFI) 

-FLI (wiki)

-MIRI (wiki)/Less Wrong

7. Ideologies and movements

a. Effective altruism (EA)

CEA | EV | Open Philantropy

b. Transhumanism

c. TESCREAL 

d. Luddism and neo-Luddism

-The precautionary principle and progress

-Movements: Post-development | Anarcho-primitivism | Degrowth | Collapsology | Survivalism

-Essays: Industrial Society and Its Future | Why the Future Doesn't Need Us

VI. Influential people

1. Alan Turing

2. John McCarthy

3. Marvin Minsky

4. Geoffrey Hinton

5. Yann Le Cun

6. Yoshua Bengio

7. Kunihiko Fukushima

8. Ray Kurzweil

9. Eric Schmidt

VII. Resources

The Stanford AI index

IEE Spectrum | IBM Think | ASCl's State of AI

CSAIL (MIT) | Alan Turing Institute (UK) | HAI & SAIL (Stanford)

 

 

 

 

 

Future Studies

Outline | Techniques | Technology forecasts | Foresight | Emerging technologies

Biases: optimism | scope | discounting | availability | conjunction | overconfidence

Global risks and future problems (current issues)

-Limits: LTG & 30y update (recent) | Degrowth movement | Green growth

-Catastrophic (scenarios):  BAS, FTL, FLI

-Existential: CSER, SERI, ERO, MIRI

Some resources

Organizations: WSF | IFTF | Millenium Project  

Think tanks: CHT | Roots of Progress | New Atlantis | Foresight

Journals: TFSC | Futures | JFS

Forecasts: Metaculus (wiki)

 

 

Persistent algebraic topology and topological data analysis

Introductory

Carlsson, Persistent Homology and Applied Homotopy Theory, arXiv:2004.00738 [math.AT]

Huber, Persistent homology in data science

Edelsbrunner and Harrer, Persistent homology – a survey

Zomorodian and Carlsson, Computing persistent homology

Comprehensive

Oudot, Persistence Theory: From Quiver Representations to Data Analysis

Edelsbrunner and Harrer, Computational topology: an introduction

Persistent homotopy

Jardine, Persistent homotopy theory

Some applications to physics

M. Biagetti, A. Cole, G. Shiu, The Persistence of Large Scale Structures I: Primordial non-Gaussianity, JCAP 04 (2021) 061

A. Cole, M. Biagetti, G. Shiu, Topological Echoes of Primordial Physics in the Universe at Large Scales, arXiv:2012.03616 [astro-ph.CO] 

J. H. T. Yip, A. Rouhianen, G. Shiu, Learning from Topology: Cosmological Parameter Estimation from the Large-scale Structure, arXiv:2308.02636 [astro-ph.CO]

J. H. T. Yip et al, Cosmology with Persistent Homology: a Fisher Forecast, JCAP 09 (2024) 034.