Learning Green's Functions with Neural Operators

[jeremypa67]

Learning Green's Functions with Neural Operators

I. Context and Motivation

During my internship at the Institut de Recherche Mathématique Avancée (IRMA) at the University of Strasbourg, I embarked on a research project focused on learning Green's functions using neural networks, a cutting-edge technique in the field of Scientific Machine Learning (SciML). The project aimed to explore the potential of neural operators as replacements for traditional numerical methods in solving partial differential equations (PDEs).

II. Theoretical framework

II.1. The Green's Function

II.1.a. Nonhomogeneous Second-Order Differential Equation

The nonhomogeneous second-order differential equation can be written as:

$$\mathcal{L}u = f(x) \quad \text{in } \Omega = [a,b] \quad (1)$$

where $\mathcal{L}$ is a differential operator, $\Omega$ is the domain, and $u$ satisfies the boundary conditions $u(a) = \alpha$ and $u(b) = \beta$.

II.1.b. Green's Function Definition [1,2]

$$ \mathcal{L}[G(x, \xi)] = \delta(x - \xi) \quad \text{in} ; \Omega = [a,b] \quad (2) $$

The Green's function $G(x, \xi)$ is the solution to the differential equation with a Dirac delta function $\delta(x - \xi)$ as the source term, subject to zero boundary conditions $G(a, \xi) = 0$ and $G(b, \xi) = 0$ on $\partial \Omega$.

II.1.c. Remarks on Green's Functions

Key Idea: This formulation allows the solution $u(x)$ to be expressed as a convolution of $G(x, \xi)$ with the source term $f(\xi)$.

Boundary Conditions: The Dirichlet boundary conditions were the only ones used in the project.

II.2. Complete Formulation of a PDE Solution using Green's Function

Preliminary Problem: Suppose the PDE follows the form in equation (1)

Solution:

The solution to the equation is given by:

$$ u(x) = u_0(x) + \int_a^b G(x, \xi)f(\xi) , d\xi(3), $$

where $u_0(x)$ is the homogeneous solution and $G(x, \xi)$ is the Green's function.

Corresponding Green's Function:

The Green's function $G(x, \xi)$ is defined as:

$$ \mathcal{L}[G(x, \xi)] = \delta(x - \xi), \quad a < x, \xi < b, $$

subject to the boundary conditions $G(a, \xi) = G(b, \xi) = 0$.

Neural Operators for PDEs [3,4]

The goal is to learn a neural network $N_{\kappa}$ with a kernel $\kappa$ that approximates the Green's function, as described in [3], and train $N_{\kappa}$ to minimize the mean squared error between the predicted and true solutions.

Neural Network Equation:

$$ L_\mu \left( \kappa(x, y) \right) = \delta(x - y) $$

with appropriate boundary conditions.

PDE Solution (inspired from equation (3)):

$$ u_{\text{NO}}(x) = \int_\Omega \kappa(x, y) f(y) , dy + u_{\text{hom, NO}}(x) \quad (4) $$

where $u_{\text{hom, NO}}(x)$ is the homogeneous solution.

III. The architecture

The figure below, sourced from Nicolas Boullé's thesis [5], illustrates the architecture of the neural network used in the implementation:

Role of $N_G$ (Green's Function Network):

• The Green's Function Network, denoted as $N_G$, approximates the Green's function $G(x, y)$ using the kernel $\kappa(x, y)$ as described in equation (4).
• This network is responsible for learning the system's response to various forcing terms.

Role of $N_{\text{hom}}$ (Homogeneous Solution Network):

• The Homogeneous Solution Network, denoted as $N_{\text{hom}}$, is tasked with learning the homogeneous solution $u_{\text{hom,NO}}(x)$ as outlined in equation (4).
• This network ensures that the boundary conditions are satisfied.

IV. Key Results and Visualizations

The test case involved in this project was the Helmholtz equation, as written below:

$$ \Delta u(x) + k^2 u(x) = f(x), $$

where $k$ is the wavenumber and $f(x)$ is the source term.

The domain is $[0,1]$ and the boundary conditions are $u(0) = 0$ and $u(1) = 0$.

The parametric domain is $\mu = k \in [1.0, 2.0]$.

The neural network configuration features 70 Fourier components with a standard deviation of 2.0, dependence on a physical parameter $\mu = k$ ranging from 1.0 to 2.0, a layer format of [50, 50, 50, 50, 50] for both $N_G$ and $N_{\text{hom}}$, an optimizer combining Adam (100 iterations) and LBFGS (2000 iterations), a batch size of 200 for collocation and sensor points, Gaussian sampling with an RBF kernel $l = 0.03$, and a rational activation function.

IV.1. Predicted Green's Function for the Parametric Helmholtz Equation

Here is a visualization of the predicted Green's function for the Helmholtz equation with varying wavenumbers, demonstrating the effectiveness of neural operators in capturing the fundamental solutions to complex PDEs.

For $k = 1$:

Figure 1: Predicted Green's Function for the Helmholtz Equation with k = 1

For $k = 1.5$:

Figure 2: Predicted Green's Function for the Helmholtz Equation with k = 1.5

For $k = 2$:

Figure 3: Predicted Green's Function for the Helmholtz Equation with k = 2

IV.2. Error Analysis for the Parametric Helmholtz Equation

The figure shows the relative error between the predicted and reference solutions for the parametric Helmholtz equation across different source terms, analyzed as functions of the wavenumber ( k ) (1.0 to 2.0, step size 0.05).

Remark: Source terms are in the appendix, section.

IV.3. Loss Function Convergence for the Parametric Helmholtz Equation

The convergence of the loss function during the training process for the Parametric Helmholtz Equation demonstrates the optimization and effectiveness of the neural networks in learning the solution to PDEs.

IV.4. Performance Comparison: Neural Network vs Finite Differences

The figure below shows the comparison of the CPU time and error between the neural network approach and finite differences for the Laplace equation. The neural network approach achieved a lower error with comparable CPU time, demonstrating its effectiveness in solving the PDE.

Remark: in our case, the Laplace equation can be written as $\Delta u(x) = f(x)$ with homogeneous Dirichlet boundary conditions on spatial domain $[0,1]$.

Figure: Performance Comparison: Neural Network vs Finite Differences

IV.5. Learning the Parametric Helmholtz Equation in the vicinity of Eigenmodes

The homogeneous Helmholtz equation has solutions called eigenmodes, representing natural oscillation patterns at wavenumbers that are multiples of $\pi$. The Green's function, describing the response to a point source, has a singularity at the source point, requiring special techniques for analysis.

Indeed, the exact Green's function for the Helmholtz equation is given by:

$$ G(x, \xi; k) = \begin{cases} -\frac{\sin(k\xi)\sin(k(1-x))}{k\sin(k)}, & 0 \leq \xi \leq x, \\ -\frac{\sin(kx)\sin(k(1-\xi))}{k\sin(k)}, & x \leq \xi \leq 1, \end{cases} $$

where $k$ is the wavenumber.

Same training settings as for the previous results related to the Parametric Helmoltz above.

Only the parameter domain for the training changes:

Neural Network Configurations:

• First network: parameter ( k ) training domain ([2, 3]).
• Second network: parameter ( k ) training domain ([2, 3.1]).
• Third network: parameter ( k ) training domain ([2, 3.14]).

For $k_{\text{max}} = 3$:

Figure: Predicted Green's Function for the Helmholtz Equation with maximal k value at 3

For $k_{\text{max}} = 3.1$:

Figure: Predicted Green's Function for the Helmholtz Equation with maximal k value at 3.1

For $k_{\text{max}} = 3.14$:

Figure: Predicted Green's Function for the Helmholtz Equation with maximal k value at 3.14

V. Conclusion and Future Work

The project successfully demonstrated the application of neural operators in learning Green's functions for the Helmholtz equation, showcasing the potential of neural networks in solving PDEs.

Future research could explore improving the accuracy of neural networks near eigenmodes, particularly in the Helmholtz equation, and applying these methods to more complex and higher-dimensional PDEs.

Key References

1. Green's Functions with Applications

• Author: D. G. Duffy
• Year: 2001
• Details: This book is an essential resource for understanding Green's functions, offering applications across various fields of mathematics and physics.
• Publisher: CRC Press
• DOI: 10.1201/9781420034790
• ISBN: 9780429142697

2. Mathematical Physics: A Modern Introduction to Its Foundations

• Author: S. Hassani
• Year: 2013
• Details: A modern approach to the foundations of mathematical physics, bridging the gap between complex physical theories and their mathematical formulations.
• Publisher: Springer, Cham
• DOI: 10.1007/978-3-319-01195-0
• ISBN: 978-3-319-01194-3

3. Neural Operator: Learning Maps Between Function Spaces with Applications to PDEs

• Authors: Z. Li, N. B. Kovachki, K. Azizzadenesheli, B. Liu, A. Stuart, A. Anandkumar
• Year: 2024
• Details: This paper introduces neural operators designed to learn mappings between function spaces, providing a novel machine learning approach to solving PDEs.
• Link: Read the paper

4. Learning Green’s Functions Associated with Time-Dependent Partial Differential Equations

• Authors: N. Boullé, S. Kim, T. Shi, A. Townsend
• Year: 2023
• Details: The study presents methods for learning Green's functions for time-dependent PDEs using neural networks, advancing data-driven approaches to classical mathematical methods.
• Link: Access the paper

5. Data-Driven Discovery of Green’s Functions

• Author: N. Boullé
• Year: 2022
• Details: This doctoral thesis from the University of Oxford explores the data-driven discovery of Green's functions, demonstrating the potential of computational approaches to uncover fundamental solutions in complex systems.
• Link: Read the thesis

Appendix: Source Terms

Source Term	Expression
$f_1(x; \mu)$	$x + \mu$
$f_2(x; \mu)$	$\mu x^2 + 2x + 1$
$f_3(x; \mu)$	$\mu x^3$
$f_4(x; \mu)$	$e^{-\mu x}$
$f_5(x; \mu)$	$e^{(x-\mu)^2}$
$f_6(x; \mu)$	$\sin(2\pi x)$
$f_7(x; \mu)$	$\frac{\mu x^2 + 2x + 1}{x^2 - 3}$
$f_8(x; \mu)$	$\sin(2\pi x) \cdot e^{\frac{(x-1)^2}{0.05\mu}}$
$f_9(x; \mu)$	$x \cdot \tan(x) \cdot \sin(2\pi\mu x) \cdot e^{-\mu x}$
$f_{10}(x; \mu)$	$\frac{e^{-\mu x} - 1}{x^2 - 4} \cdot \cos(2\pi x)$

Table: Source terms $f_{1}(x; \mu)$ to $f_{10}(x; \mu)$ used in the analysis.