Scientific machine learning, or SciML, was broadly diffused and defined in a DOE report that appeared in February 2019–see my earlier post AI for science. The topic has been experiencing exponential growth since then, and there is a sore need today for “making some order” in the plethora of approaches and applications of machine learning to scientific and industrial problems–in particuler to Digital Twins.

Ben Moseley in his PhD thesis, mapped out a basis for classifying the methods and approaches for scientific machine learning, or as he calls it, “physics infused machine learning.” One part of this has already been addressed in my previous blog post on NN for PDEs.

In this post, I will revisit Moseley’s classification in the light of Digital Twins. This was the subject of a recent talk that I gave in SIAM’s CSE23 conference. Recall that a DT (in my sense) can be considered as an outer-loop, inverse problem (parameter estimation, shape optimization, boundary control, data assimilation, etc.), and that an outer-loop/inverse problem relies on an inner-loop/PDE model. This inner-loop model must be solved often, as fast as possible and as accurately as is feasible, or acceptable. Hence the need for surrogate models. But, and this is a very big BUT, how can we ensure that a surrogate, computed by machine learning, indeed respects the underlying physics (or biology, or chemistry, etc), and is not just another local solution? To answer this question, requires (some kind of) SciML. What and how???

The 3 Classes of SciML

  1. Architecture based methods.
  2. Loss function based methods.
  3. Integrated (hybrid) approaches.

Let us discuss each of these three in turn, and see which are the SciML tools that belong to each one.

Architecture based SciML

The idea here is to incorporate scientific constraints directly into the NN architecture, thus introducing a very strong inductive bias–see Karniadakis, et al, Nature Reviews Physics, 2021.

This can take the following forms:

  • assign physical meaning physical meaning to some sub-networks of neurons - eg. LSTM, fully-connected, CNN, etc.
  • encode symmetries, translational and rotational invariances, eg. by CNNs.
  • use Koopman theory [see Kutz, Brunton].
  • use physically constrained Gaussian processes.

Loss Function based SCiML

The idea here is to modify the ML loss function to include scientific/physical/theoretical constraints that “encourage” the ML model to respect the prior knowledge - “soft power” - acts as a regulariser, in fact.

  • Encode positivity, bounds, conservation laws.
  • PINN - encode the governing equations [Raissi, Perdikaris, Karniadakis].
  • PINO - learn the operator [Lu, Perdikaris, Karniadakis].
  • FNO - learn the operator based on Fourier modes [Li, Stuart, Anandkumar].

Integrated SciML

The idea: tightly integrate the ML algorithm with a traditional one, to create a “learnable system”, strongly informed by priors, but flexible enough to still learn from and adapt to data.

  • Residual modelling: traditional algorithm is a black box, eg. super-resolution
  • Differentiable physics, based on autodiff, where some parts are fixed, others are learnable, eg. in the form of an NN.
  • Neural DEs insert more complex ML models into traditional algorithms to learn the underlying equations [Rackackaus, 2020].
  • In-the-loop where we use ML inside an iterative loop for forward and inverse solvers, eg. subgrid parametrisation, regularisers, gradients.

Conclusions and Perspectives

  • Incorporating scientific knowledge (almost) always improves the performance of ML algorithms:
    • Restricts the space of ML models.
    • Stronger inductive bias.
    • Alleviates ML flaws (poor generalisation, optimisation difficulties, lack of interpretability, large amounts of training data).
  • Incorporating ML in the scientific workflow:
    • Enhances the performance (efficiency, accuracy, insights, noise).
    • Compensates for unknown/intractable physics.
    • Takes advantage of large reservoirs of untapped data.