feat(ppsci): support data_effient_nopt

Author: xiaoyewww

README.md

@@ -47,6 +47,7 @@ PaddleScience 是一个基于深度学习框架 PaddlePaddle 开发的科学计
 | 微分方程 | [若斯叻方程](https://paddlescience-docs.readthedocs.io/zh-cn/latest/zh/examples/rossler) | 数据驱动 | Transformer-Physx | 监督学习 | [Data](https://github.com/zabaras/transformer-physx) | [Paper](https://arxiv.org/abs/2010.03957) |
 | 算子学习 | [DeepONet](https://paddlescience-docs.readthedocs.io/zh-cn/latest/zh/examples/deeponet) | 数据驱动 | MLP | 监督学习 | [Data](https://deepxde.readthedocs.io/en/latest/demos/operator/antiderivative_unaligned.html) | [Paper](https://export.arxiv.org/pdf/1910.03193.pdf) |
 | 微分方程 | [梯度增强的物理知识融合 PDE 求解](https://github.com/PaddlePaddle/PaddleScience/blob/develop/examples/gpinn/poisson_1d.py) | 机理驱动 | gPINN | 无监督学习 | - |  [Paper](https://doi.org/10.1016/j.cma.2022.114823) |
+| 微分方程 | [PDE 求解](https://paddlescience-docs.readthedocs.io/zh-cn/latest/zh/examples/data_efficient_nopt) | 数据驱动 | FNO/Transformer | 无监督学习 | - |  [Paper](https://arxiv.org/abs/2402.15734) |
 | 积分方程 | [沃尔泰拉积分方程](https://paddlescience-docs.readthedocs.io/zh-cn/latest/zh/examples/volterra_ide) | 机理驱动 | MLP | 无监督学习 | - | [Project](https://github.com/lululxvi/deepxde/blob/master/examples/pinn_forward/Volterra_IDE.py) |
 | 微分方程 | [分数阶微分方程](https://github.com/PaddlePaddle/PaddleScience/blob/develop/examples/fpde/fractional_poisson_2d.py) | 机理驱动 | MLP | 无监督学习 | - | - |
 | 光孤子 | [Optical soliton](https://paddlescience-docs.readthedocs.io/zh-cn/latest/zh/examples/nlsmb) | 机理驱动 | MLP | 无监督学习 | - | [Paper](https://doi.org/10.1007/s11071-023-08824-w)|

docs/index.md

@@ -83,6 +83,7 @@
 | 微分方程 | [若斯叻方程](./zh/examples/rossler.md) | 数据驱动 | Transformer-Physx | 监督学习 | [Data](https://github.com/zabaras/transformer-physx) | [Paper](https://arxiv.org/abs/2010.03957) |
 | 算子学习 | [DeepONet](./zh/examples/deeponet.md) | 数据驱动 | MLP | 监督学习 | [Data](https://deepxde.readthedocs.io/en/latest/demos/operator/antiderivative_unaligned.html) | [Paper](https://export.arxiv.org/pdf/1910.03193.pdf) |
 | 微分方程 | [梯度增强的物理知识融合 PDE 求解](https://github.com/PaddlePaddle/PaddleScience/blob/develop/examples/gpinn/poisson_1d.py) | 机理驱动 | gPINN | 无监督学习 | - |  [Paper](https://doi.org/10.1016/j.cma.2022.114823) |
+| 微分方程 | [PDE 求解](./zh/examples/data_efficient_nopt.md) | 数据驱动 | FNO/Transformer | 无监督学习 | - |  [Paper](https://arxiv.org/abs/2402.15734) |
 | 积分方程 | [沃尔泰拉积分方程](./zh/examples/volterra_ide.md) | 机理驱动 | MLP | 无监督学习 | - | [Project](https://github.com/lululxvi/deepxde/blob/master/examples/pinn_forward/Volterra_IDE.py) |
 | 微分方程 | [分数阶微分方程](https://github.com/PaddlePaddle/PaddleScience/blob/develop/examples/fpde/fractional_poisson_2d.py) | 机理驱动 | MLP | 无监督学习 | - | - |
 | 光孤子 | [Optical soliton](./zh/examples/nlsmb.md) | 机理驱动 | MLP | 无监督学习 | - | [Paper](https://doi.org/10.1007/s11071-023-08824-w)|

docs/zh/examples/data_efficient_nopt.md

@@ -0,0 +1,80 @@
+# DataEfficientNopt
+
+=== "模型训练命令"
+
+    ``` sh
+    cd examples/data_efficient_nopt
+    # Download possion_64 data from https://drive.google.com/drive/folders/1crIsTZGxZULWhrXkwGDiWF33W6RHxJkf
+    # Download helmholtz_64 data from https://drive.google.com/drive/folders/1UjIaF6FsjmN_xlGGSUX-1K2V3EF2Zalw
+
+    # Update the file paths in `config/operators_possion.yaml` or `config/operators_helmholtz.yaml` to specify `train_path`, `val_path`, `test_path`, `scales_path`, and `train_rand_idx_path`.
+
+    # possion_64 pretrain
+    python pretrain_basic.py --run_name r0 --config pois-64-pretrain-e1_20_m3 --yaml_config ./config/operators_poisson.yaml
+
+    # possion_64 finetune
+    python pretrain_basic.py --run_name r0 --config pois-64-e5_15_b0 --yaml_config ./config/operators_poisson.yaml
+
+    # helmholtz_64 pretrain
+    python pretrain_basic.py --run_name r0 --config helm-64-pretrain-o1_20_m1 --yaml_config ./config/operators_helmholtz.yaml
+
+    # helmholtz_64 finetune
+    python pretrain_basic.py --run_name r0 --config helm-64-o5_15_ft5_r2 --yaml_config ./config/operators_helmholtz.yaml
+    ```
+
+=== "模型评估命令"
+
+    暂无
+
+=== "模型导出命令"
+
+    暂无
+
+=== "模型推理命令"
+
+    ``` sh
+    cd examples/data_efficient_nopt
+    # Update the file paths in `config/operators_possion.yaml` or `config/operators_helmholtz.yaml` to specify `train_path`, `test_path`, and `scales_path`.
+    # Use a fine-tuned model as the checkpoint in 'exp' or utilize `model_convert.py` to convert the official checkpoint.
+    python3 inference_fno_helmholtz_poisson.py --config ./config/inference_poisson.yaml --ckpt_path <ckpt_path> --num_demos 1
+    ```
+
+## 1. 背景简介
+
+data_efficient_nopt旨在提高偏微分方程（PDE）算子学习的数据效率，通过设计无监督预训练方法减少对高成本模拟数据的依赖。利用未标记的PDE数据（无需模拟解），并通过基于物理启发的重建代理任务对神经算子进行预训练。为了提升分布外（OOD）泛化性能，我们进一步引入了一种基于相似性的上下文学习方法，使神经算子能够灵活利用上下文示例，而无需额外的训练成本或设计。在多种PDE上的实验表明，该方法具有高度的数据效率、更强的泛化能力，甚至优于传统的视觉预训练模型。
+
+## 2. 模型原理
+
+这篇论文主要解决使用深度学习方法解决基于偏微分方程（PDEs）的科学问题时的数据效率问题。具体来说，作者指出当前的神经算子（Neural Operators）方法需要大量的高保真PDE数据，这导致了高昂的数值模拟成本。为了减少对这些昂贵数据的依赖，作者提出了一种无监督预训练方法，旨在通过使用无标签的PDE数据来提高模型的数据效率和泛化能力。
+
+论文通过以下方案解决上述提到的问题：
+
+1. 无监督预训练
+    - 无标签PDE数据定义：作者定义了无标签的PDE数据，这些数据不包含PDE的解，从而避免了昂贵的数值模拟。
+    - 物理启发的代理任务：作者提出了两个基于重构的代理任务，分别是Masked Autoencoder（MAE）和Super-resolution（SR）。MAE通过随机遮蔽部分输入并要求模型重建完整的输入来学习稀疏感知的不变性；SR通过应用高斯滤波器使输入模糊，然后要求模型重建高分辨率的输入来学习分辨率和模糊的不变性。
+    - 预训练过程：使用无标签的PDE数据和上述代理任务进行无监督预训练，从而获得更好的初始模型，减少后续监督训练所需的模拟数据量。
+
+2. 上下文学习
+    - 相似性挖掘：在推理时，通过计算查询输入与支持示例（demos）的输出距离来找到相似的示例。
+    - 聚合预测：对于每个查询的时空位置，找到相似的示例后，通过聚合这些示例的解来生成最终的预测。
+    - 方法优势：这种方法在推理时引入了零额外训练成本，而且可以无缝集成到现有的训练管道中，提高了模型在OOD数据上的泛化能力。
+
+## 5. 完整代码
+
+``` py linenums="1" title="examples/data_efficient_nopt/pretrain_basic.py"
+--8<--
+examples/data_efficient_nopt/pretrain_basic.py
+--8<--
+```
+
+``` py linenums="1" title="examples/data_efficient_nopt/inference_fno_helmholtz_poisson.py"
+--8<--
+examples/data_efficient_nopt/inference_fno_helmholtz_poisson.py
+--8<--
+```
+
+## 6. 结果展示
+
+## 7. 参考资料
+
+- [Data-Efficient Operator Learning via Unsupervised Pretraining and In-Context Learning](https://arxiv.org/abs/2402.15734)

examples/data_efficient_nopt/README.md

@@ -1,12 +1,7 @@
 default:
-  train_path: '/home/aistudio/data_efficient_nopt/data/helmholtz_64/helmholtz_64/helmholtz_64_o15_20_train.h5'
-  test_path: '/home/aistudio/data_efficient_nopt/data/helmholtz_64/helmholtz_64/helmholtz_64_o15_20_test.h5'
-  # scales_path: '/home/aistudio/data_efficient_nopt/data/helmholtz_64/helmholtz_64/helmholtz_64_o15_20_train_scale.npy'
-  # datapath: '/home/aistudio/data_efficient_nopt/data/helmholtz_64/helmholtz_64/helmholtz_64_o16_test.h5'
-  # datapath: '/home/aistudio/data_efficient_nopt/data/helmholtz_64/helmholtz_64/helmholtz_64_o15_test.h5'
-  # datapath: '/home/aistudio/data_efficient_nopt/data/helmholtz_64/helmholtz_64/helmholtz_64_o16_tensor15_test.h5'
-  # datapath: '/home/aistudio/data_efficient_nopt/data/helmholtz_64/helmholtz_64/helmholtz_64_o5_15_test.h5'
-  scales_path: '/home/aistudio/data_efficient_nopt/data/helmholtz_64/helmholtz_64/helmholtz_64_o5_15_train_scale.npy'
+  train_path: 'data/helmholtz_64/helmholtz_64/helmholtz_64_o15_20_train.h5'
+  test_path: 'data/helmholtz_64/helmholtz_64/helmholtz_64_o15_20_test.h5'
+  scales_path: 'data/helmholtz_64/helmholtz_64/helmholtz_64_o5_15_train_scale.npy'
 
   num_data_workers: 1
   subsample: 1

examples/data_efficient_nopt/config/inference_helmholtz.yaml

@@ -1,9 +1,7 @@
 default:
-  # datapath: '/home/aistudio/data_efficient_nopt/data/possion_64/poisson_64_e5_15_test.h5'
-  train_path: '/home/aistudio/data_efficient_nopt/data/possion_64/poisson_64_e15_50_train.h5' # pick demos
-  test_path: '/home/aistudio/data_efficient_nopt/data/possion_64/poisson_64_e15_50_test.h5'
-  # datapath: '/home/aistudio/data_efficient_nopt/data/possion_64/poisson_64_e20_test.h5'
-  scales_path: '/home/aistudio/data_efficient_nopt/data/possion_64/poisson_64_e5_15_train_scale.npy'
+  train_path: 'data/possion_64/poisson_64_e15_50_train.h5' # pick demos
+  test_path: 'data/possion_64/poisson_64_e15_50_test.h5'
+  scales_path: 'data/possion_64/poisson_64_e5_15_train_scale.npy'
 
   num_data_workers: 1
   subsample: 1

examples/data_efficient_nopt/config/inference_poisson.yaml

@@ -78,11 +78,10 @@ helmholtz: &helmholtz
 
 helm-64-scale-o5_15: &helm_64_o5_15
   <<: *helmholtz
-  train_path:   '/home/aistudio/data_efficient_nopt/data/helmholtz_64/helmholtz_64/helmholtz_64_o5_15_train.h5'
-  val_path:     '/home/aistudio/data_efficient_nopt/data/helmholtz_64/helmholtz_64/helmholtz_64_o5_15_val.h5'
-  test_path:    '/home/aistudio/data_efficient_nopt/data/helmholtz_64/helmholtz_64/helmholtz_64_o5_15_test.h5'
-  scales_path:  '/home/aistudio/data_efficient_nopt/data/helmholtz_64/helmholtz_64/helmholtz_64_o5_15_train_scale.npy'
-  # train_rand_idx_path: '/home/aistudio/data_efficient_nopt/data/helmholtz_64/helmholtz_64/old_gen/train_rand_idx.npy'
+  train_path:   'data/helmholtz_64/helmholtz_64/helmholtz_64_o5_15_train.h5'
+  val_path:     'data/helmholtz_64/helmholtz_64/helmholtz_64_o5_15_val.h5'
+  test_path:    'data/helmholtz_64/helmholtz_64/helmholtz_64_o5_15_test.h5'
+  scales_path:  'data/helmholtz_64/helmholtz_64/helmholtz_64_o5_15_train_scale.npy'
   batch_size: 128
   in_dim: 3
   out_dim: 1
@@ -101,11 +100,11 @@ helm-64-scale-o5_15: &helm_64_o5_15
 
 helm-64-pretrain-o1_20: &helm_64_o1_20_pt
   <<: *helmholtz
-  train_path:   '/home/aistudio/data_efficient_nopt/data/helmholtz_64/helmholtz_64/helmholtz_64_o1_20_train.h5'
-  val_path:     '/home/aistudio/data_efficient_nopt/data/helmholtz_64/helmholtz_64/helmholtz_64_o1_20_val.h5'
-  test_path:    '/home/aistudio/data_efficient_nopt/data/helmholtz_64/helmholtz_64/helmholtz_64_o1_20_test.h5'
-  scales_path:  '/home/aistudio/data_efficient_nopt/data/helmholtz_64/helmholtz_64/helmholtz_64_o1_20_train_scale.npy'
-  train_rand_idx_path: '/home/aistudio/data_efficient_nopt/data/helmholtz_64/helmholtz_64/train_rand_idx.npy'
+  train_path:   'data_efficient_nopt/data/helmholtz_64/helmholtz_64/helmholtz_64_o1_20_train.h5'
+  val_path:     'data_efficient_nopt/data/helmholtz_64/helmholtz_64/helmholtz_64_o1_20_val.h5'
+  test_path:    'data_efficient_nopt/data/helmholtz_64/helmholtz_64/helmholtz_64_o1_20_test.h5'
+  scales_path:  'data_efficient_nopt/data/helmholtz_64/helmholtz_64/helmholtz_64_o1_20_train_scale.npy'
+  train_rand_idx_path: 'data_efficient_nopt/data/helmholtz_64/helmholtz_64/train_rand_idx.npy'
   batch_size: 128
   in_dim: 3
   out_dim: 1
@@ -128,11 +127,11 @@ helm-64-pretrain-o1_20_ft: &helm_64_o1_20_ft
 
 helm-64-finetune-o5_15: &helm_64_o5_15_ft
   <<: *helmholtz
-  train_path:   '/home/aistudio/data_efficient_nopt/data/helmholtz_64/helmholtz_64/helmholtz_64_o5_15_train.h5'
-  val_path:     '/home/aistudio/data_efficient_nopt/data/helmholtz_64/helmholtz_64/helmholtz_64_o5_15_val.h5'
-  test_path:    '/home/aistudio/data_efficient_nopt/data/helmholtz_64/helmholtz_64/helmholtz_64_o5_15_test.h5'
-  scales_path:  '/home/aistudio/data_efficient_nopt/data/helmholtz_64/helmholtz_64/helmholtz_64_o5_15_train_scale.npy'
-  train_rand_idx_path: '/home/aistudio/data_efficient_nopt/data/helmholtz_64/helmholtz_64/train_rand_idx.npy'
+  train_path:   'data/helmholtz_64/helmholtz_64/helmholtz_64_o5_15_train.h5'
+  val_path:     'data/helmholtz_64/helmholtz_64/helmholtz_64_o5_15_val.h5'
+  test_path:    'data/helmholtz_64/helmholtz_64/helmholtz_64_o5_15_test.h5'
+  scales_path:  'data/helmholtz_64/helmholtz_64/helmholtz_64_o5_15_train_scale.npy'
+  train_rand_idx_path: 'data/helmholtz_64/helmholtz_64/train_rand_idx.npy'
   batch_size: 128
   in_dim: 3  #normal helmholtz has 3 dim, joint has 4
   out_dim: 1

examples/data_efficient_nopt/config/operators_helmholtz.yaml

@@ -78,11 +78,11 @@ poisson: &poisson
 
 poisson-64-scale-e5_15: &poisson_64_e5_15
   <<: *poisson
-  train_path:  '/home/aistudio/data_efficient_nopt/data/possion_64/poisson_64_e5_15_train.h5'
-  val_path:    '/home/aistudio/data_efficient_nopt/data/possion_64/poisson_64_e5_15_val.h5'
-  test_path:   '/home/aistudio/data_efficient_nopt/data/possion_64/poisson_64_e5_15_test.h5'
-  scales_path: '/home/aistudio/data_efficient_nopt/data/possion_64/poisson_64_e5_15_train_scale.npy'
-  train_rand_idx_path: '/home/aistudio/data_efficient_nopt/data/possion_64/train_rand_idx.npy'
+  train_path:  'data/possion_64/poisson_64_e5_15_train.h5'
+  val_path:    'data/possion_64/poisson_64_e5_15_val.h5'
+  test_path:   'data/possion_64/poisson_64_e5_15_test.h5'
+  scales_path: 'data/possion_64/poisson_64_e5_15_train_scale.npy'
+  train_rand_idx_path: 'data/possion_64/train_rand_idx.npy'
   batch_size: 128
   log_to_wandb: !!bool False
   learning_rate: 1E-3
@@ -101,11 +101,11 @@ poisson-64-scale-e5_15: &poisson_64_e5_15
 
 pois-64-pretrain-e1_20: &pois_64_e1_20_pt
   <<: *poisson
-  train_path:   '/home/aistudio/data_efficient_nopt/data/possion_64/poisson_64_e1_20_train.h5'
-  val_path:     '/home/aistudio/data_efficient_nopt/data/possion_64/poisson_64_e1_20_val.h5'
-  test_path:    '/home/aistudio/data_efficient_nopt/data/possion_64/poisson_64_e1_20_test.h5'
-  scales_path:  '/home/aistudio/data_efficient_nopt/data/possion_64/poisson_64_e1_20_train_scale.npy'
-  train_rand_idx_path: '/home/aistudio/data_efficient_nopt/data/possion_64/train_rand_idx.npy'
+  train_path:   'data/possion_64/poisson_64_e1_20_train.h5'
+  val_path:     'data/possion_64/poisson_64_e1_20_val.h5'
+  test_path:    'data/possion_64/poisson_64_e1_20_test.h5'
+  scales_path:  'data/possion_64/poisson_64_e1_20_train_scale.npy'
+  train_rand_idx_path: 'data/possion_64/train_rand_idx.npy'
   batch_size: 128
   log_to_wandb: !!bool False
   mode_cut: 32
@@ -122,11 +122,11 @@ pois-64-pretrain-e1_20: &pois_64_e1_20_pt
 
 pois-64-finetune-e5_15: &pois_64_e5_15_ft
   <<: *poisson
-  train_path:   '/home/aistudio/data_efficient_nopt/data/possion_64/poisson_64_e5_15_train.h5'
-  val_path:     '/home/aistudio/data_efficient_nopt/data/possion_64/poisson_64_e5_15_val.h5'
-  test_path:    '/home/aistudio/data_efficient_nopt/data/possion_64/poisson_64_e5_15_test.h5'
-  scales_path:  '/home/aistudio/data_efficient_nopt/data/possion_64/poisson_64_e5_15_train_scale.npy'
-  train_rand_idx_path: '/home/aistudio/data_efficient_nopt/data/possion_64/train_rand_idx.npy'
+  train_path:   'data/possion_64/poisson_64_e5_15_train.h5'
+  val_path:     'data/possion_64/poisson_64_e5_15_val.h5'
+  test_path:    'data/possion_64/poisson_64_e5_15_test.h5'
+  scales_path:  'data/possion_64/poisson_64_e5_15_train_scale.npy'
+  train_rand_idx_path: 'data/possion_64/train_rand_idx.npy'
   batch_size: 128
   log_to_wandb: !!bool False
   mode_cut: 32

examples/data_efficient_nopt/config/operators_poisson.yaml

@@ -11,17 +11,12 @@
 import paddle
 import paddle.distributed as dist
 import yaml
-
-# from utils.data_utils import get_data_loader
 from data_utils.pois_helm_datasets import get_data_loader
 from models.fno import build_fno
 from pretrain_basic import l2_err
 from scipy.stats import linregress
 from tqdm import tqdm
 
-# from utils.loss_utils import LossMSE
-# from utils.YParams import YParams
-
 
 @paddle.no_grad()
 def get_pred(args):
@@ -166,16 +161,11 @@ def get_pred(args):
 if __name__ == "__main__":
     parser = ArgumentParser()
     parser.add_argument("--config", type=str, default="config/inference_helmholtz.yaml")
-    # parser.add_argument('--ckpt_path', type=str, default='/pscratch/sd/p/puren93/neuralopt/expts/helm-64-o5_15_ft0/all_mask_m6/checkpoints/ckpt.tar')
     parser.add_argument(
         "--ckpt_path",
         type=str,
         default="/pscratch/sd/j/jsong/deff_archive/neuraloperators-foundation_/expts/helm-64-o5_15_ft0/b012_m6/checkpoints/ckpt.tar",
     )
-    # parser.add_argument('--ckpt_path', type=str, default='/pscratch/sd/j/jsong/deff_archive/neuraloperators-foundation_/expts/helm-64-o5_15_ft0/b01_m6/checkpoints/ckpt.tar')
-    # parser.add_argument('--ckpt_path', type=str, default='/pscratch/sd/j/jsong/neuraloperators-foundation/expts/pois-64-e5_15_ft9/b01_m0/checkpoints/ckpt.tar') # [X]
-    # parser.add_argument('--ckpt_path', type=str, default='/pscratch/sd/j/jsong/deff_archive/neuraloperators-foundation_/expts/pois-64-e5_15_ft9/b01_m0_/checkpoints/ckpt.tar')
-    # parser.add_argument('--ckpt_path', type=str, default='/pscratch/sd/j/jsong/deff_archive/neuraloperators-foundation_/expts/pois-64-e5_15_ft9/b01_m0_r0/checkpoints/ckpt.tar')
     parser.add_argument("--num_demos", type=int, default=None)
     parser.add_argument(
         "--tqdm", action="store_true", default=False, help="Turn on the tqdm"

examples/data_efficient_nopt/inference_fno_helmholtz_poisson.py

@@ -1,4 +1,3 @@
-# import numpy as np
 import paddle
 import paddle.nn as nn