Add icon-liskov DSL Preprocessor (#117)

Introduces icon_liskov a preprocessor which enables the generation of integration code between ICON and Gt4Py stencils in Icon4Py.

Author: samkellerhals

.flake8

@@ -3,25 +3,31 @@
 max-line-length = 100
 max-complexity = 15
 doctests = true
-ignore =
-    B008    # Do not perform function calls in argument defaults
-    D1      # Public code object needs docstring
-    DAR     # Disable dargling errors by default
-    E203    # Whitespace before ':' (black formatter breaks this sometimes)
-    E501    # Line too long (using Bugbear's B950 warning)
-    W503    # Line break occurred before a binary operator
+extend-ignore =
+    # Do not perform function calls in argument defaults
+    B008,
+    # Public code object needs docstring
+    D1,
+    # Disable dargling errors by default
+    DAR,
+    # Whitespace before ':' (black formatter breaks this sometimes)
+    E203,
+    # Line too long (using Bugbear's B950 warning)
+    E501,
+    # Line break occurred before a binary operator
+    W503
 
 exclude =
-    .eggs
-    .gt_cache
-    .ipynb_checkpoints
-    .tox
-    _local_
-    build
-    dist
-    docs
-    _external_src
-    tests/_disabled
+    .eggs,
+    .gt_cache,
+    .ipynb_checkpoints,
+    .tox,
+    _local_,
+    build,
+    dist,
+    docs,
+    _external_src,
+    tests/_disabled,
     setup.py
 
 rst-roles =

.pre-commit-config.yaml

@@ -37,6 +37,7 @@ repos:
           #   args: [--remove]
           - id: name-tests-test
             args: [--pytest-test-first]
+            exclude: ^liskov/tests/samples
 
     - repo: https://gitlab.com/bmares/check-json5
       rev: v1.0.0

liskov/README.md

@@ -0,0 +1,126 @@
+# icon4py-liskov
+
+A preprocessor that facilitates integration of gt4py code into the ICON model.
+
+## Installation
+
+To install the icon4py-liskov package, follow the instructions in the `README.md` file located in the root of the repository.
+
+## Description
+
+The icon4py-liskov package includes the `icon_liskov` CLI tool which takes a fortran file as input and processes it with the ICON-Liskov DSL Preprocessor. This preprocessor adds the necessary `USE` statements and generates OpenACC `DATA CREATE` statements and declares DSL input/output fields based on directives in the input file. The preprocessor also processes stencils defined in the input file using the `START STENCIL` and `END STENCIL` directives, inserting the necessary code to run the stencils and adding nvtx profile statements if specified with the `--profile` flag.
+
+### Usage
+
+To use the `icon_liskov` tool, run the following command:
+
+```bash
+icon_liskov <input_filepath> <output_filepath> [--profile]
+```
+
+Where `input_filepath` is the path to the input file to be processed, and `output_filepath` is the path to the output file. The optional `--profile` flag adds nvtx profile statements to the stencils.
+
+### Preprocessor directives
+
+The ICON-Liskov DSL Preprocessor supports the following directives:
+
+#### `!$DSL IMPORTS()`
+
+This directive generates the necessary `USE` statements to import the Fortran to C interfaces.
+
+#### `!$DSL START CREATE()`
+
+This directive generates an OpenACC `DATA CREATE` statement for all output fields used in each DSL (icon4py) stencil.
+
+#### `!$DSL END CREATE()`
+
+This directive generates an OpenACC `END DATA` statement which is neccessary to close the OpenACC data region.
+
+#### `!$DSL DECLARE()`
+
+This directive is used to declare all DSL input/output fields. The required arguments are the field name and its associated dimensions. For example:
+
+```fortran
+!$DSL DECLARE(vn=(nproma, p_patch%nlev, p_patch%nblks_e))
+```
+
+will generate the following code:
+
+```fortran
+! DSL INPUT / OUTPUT FIELDS
+REAL(wp), DIMENSION((nproma, p_patch%nlev, p_patch%nblks_e)) :: vn_before
+```
+
+Furthermore, this directive also takes two optional keyword arguments. `type` takes a string which will be used to fill in the type of the declared field, for example `type=LOGICAL`. `suffix` takes a string which will be used as the suffix of the field e.g. `suffix=dsl`, by default the suffix is `before`.
+
+#### `!$DSL START STENCIL()`
+
+This directive denotes the start of a stencil. Required arguments are `name`, `vertical_lower`, `vertical_upper`, `horizontal_lower`, `horizontal_upper`. The value for `name` must correspond to a stencil found in one of the stencil modules inside `icon4py`, and all fields defined in the directive must correspond to the fields defined in the respective icon4py stencil. Optionally, absolute and relative tolerances for the output fields can also be set using the `_tol` or `_abs` suffixes respectively. An example call looks like this:
+
+```fortran
+!$DSL START STENCIL(name=mo_nh_diffusion_stencil_06; &
+!$DSL       z_nabla2_e=z_nabla2_e(:,:,1); area_edge=p_patch%edges%area_edge(:,1); &
+!$DSL       fac_bdydiff_v=fac_bdydiff_v; vn=p_nh_prog%vn(:,:,1); vn_abs_tol=1e-21_wp; &
+!$DSL       vertical_lower=1; vertical_upper=nlev; &
+!$DSL       horizontal_lower=i_startidx; horizontal_upper=i_endidx)
+```
+
+In addition, other optional keyword arguments are the following:
+
+- `accpresent`: Takes a boolean string input, and controls the default data-sharing behavior for variables used in the OpenACC parallel region. Setting the flag to true will cause all variables to be assumed present on the device by default (`DEFAULT(PRESENT)`), and no explicit data-sharing attributes need to be specified. Setting it to false will require explicit data-sharing attributes for every variable used in the parallel region (`DEFAULT(NONE)`). By default it is set to false.<br><br>
+
+- `mergecopy`: Takes a boolean string input. When set to True consecutive before field copy regions of stencils that have the mergecopy flag set to True are combined into a single before field copy region with a new name created by concatenating the names of the merged stencil regions. This is useful when there are consecutive stencils. By default it is set to false.<br><br>
+
+- `copies`: Takes a boolean string input, and controls whether before field copies should be made or not. If set to False only the `#ifdef __DSL_VERIFY` directive is generated. Defaults to true.<br><br>
+
+#### `!$DSL END STENCIL()`
+
+This directive denotes the end of a stencil. The required argument is `name`, which must match the name of the preceding `START STENCIL` directive.
+
+Together, the `START STENCIL` and `END STENCIL` directives result in the following generated code at the start and end of a stencil respectively.
+
+```fortran
+#ifdef __DSL_VERIFY
+!$ACC PARALLEL IF( i_am_accel_node .AND. acc_on ) DEFAULT(NONE) ASYNC(1)
+vn_before(:, :, :) = vn(:, :, :)
+!$ACC END PARALLEL
+```
+
+```fortran
+call nvtxEndRange()
+#endif
+call wrap_run_mo_nh_diffusion_stencil_06( &
+z_nabla2_e=z_nabla2_e(:, :, 1), &
+area_edge=p_patch%edges%area_edge(:, 1), &
+fac_bdydiff_v=fac_bdydiff_v, &
+vn=p_nh_prog%vn(:, :, 1), &
+vn_before=vn_before(:, :, 1), &
+vn_abs_tol=1e-21_wp, &
+vertical_lower=1, &
+vertical_upper=nlev, &
+horizontal_lower=i_startidx, &
+horizontal_upper=i_endidx
+)
+```
+
+Additionally, there are the following keyword arguments:
+
+- `noendif`: Takes a boolean string input and controls whether an `#endif` is generated or not. Defaults to false.<br><br>
+
+- `noprofile`: Takes a boolean string input and controls whether a nvtx end profile directive is generated or not. Defaults to false.<br><br>
+
+#### `!$DSL INSERT()`
+
+This directive allows the user to generate any text that is placed between the parentheses. This is useful for situations where custom code generation is necessary.
+
+#### `!$DSL START PROFILE()`
+
+This directive allows generating an nvtx start profile data statement, and takes the stencil `name` as an argument.
+
+#### `!$DSL END PROFILE()`
+
+This directive allows generating an nvtx end profile statement.
+
+#### `!$DSL ENDIF()`
+
+This directive generates an `#endif` statement.

liskov/requirements-dev.txt

@@ -0,0 +1,3 @@
+-r ../base-requirements-dev.txt
+-e ../common
+-e .

liskov/requirements.txt

@@ -0,0 +1,3 @@
+-r ../base-requirements.txt
+../common
+.

liskov/setup.cfg

@@ -0,0 +1,55 @@
+# This file is mainly used to configure package creation with setuptools.
+# Documentation:
+# http://setuptools.readthedocs.io/en/latest/setuptools.html#configuring-setup-using-setup-cfg-files
+#
+[metadata]
+name = icon4py_liskov
+description = ICON preprocessor to integrate Gt4Py code.
+long_description = file: README.md
+long_description_content_type = text/markdown
+url = https://github.com/C2SM/icon4py
+author = ETH Zurich
+author_email = gridtools@cscs.ch
+license = gpl3
+license_files = LICENSE
+platforms = Linux, Mac
+classifiers =
+    Development Status :: 3 - Alpha
+    Intended Audience :: Science/Research
+    License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)
+    Operating System :: POSIX
+    Programming Language :: Python
+    Programming Language :: Python :: 3
+    Programming Language :: Python :: 3 :: Only
+    Programming Language :: Python :: 3.10
+    Programming Language :: Python :: Implementation :: CPython
+    Topic :: Scientific/Engineering :: Atmospheric Science
+    Topic :: Scientific/Engineering :: Mathematics
+    Topic :: Scientific/Engineering :: Physics
+project_urls =
+    Source Code = https://github.com/GridTools/gt4py
+
+[options]
+packages = find_namespace:
+install_requires =
+    icon4py-common
+
+python_requires = >=3.10
+package_dir =
+    = src
+zip_safe = False
+
+[options.package_data]
+# References:
+#  https://setuptools.pypa.io/en/latest/userguide/datafiles.html
+#  https://github.com/abravalheri/experiment-setuptools-package-data
+* = *.md, *.rst, *.toml, *.txt, py.typed
+
+[options.packages.find]
+where = src
+exclude =
+    tests
+
+[options.entry_points]
+console_scripts =
+    icon_liskov = icon4py.liskov.cli:main

liskov/setup.py

@@ -0,0 +1,18 @@
+# ICON4Py - ICON inspired code in Python and GT4Py
+#
+# Copyright (c) 2022, ETH Zurich and MeteoSwiss
+# All rights reserved.
+#
+# This file is free software: you can redistribute it and/or modify it under
+# the terms of the GNU General Public License as published by the
+# Free Software Foundation, either version 3 of the License, or any later
+# version. See the LICENSE.txt file at the top-level directory of this
+# distribution for a copy of the license or check <https://www.gnu.org/licenses/>.
+#
+# SPDX-License-Identifier: GPL-3.0-or-later
+
+from setuptools import setup
+
+
+if __name__ == "__main__":
+    setup()

liskov/src/icon4py/liskov/__init__.py

@@ -0,0 +1,12 @@
+# ICON4Py - ICON inspired code in Python and GT4Py
+#
+# Copyright (c) 2022, ETH Zurich and MeteoSwiss
+# All rights reserved.
+#
+# This file is free software: you can redistribute it and/or modify it under
+# the terms of the GNU General Public License as published by the
+# Free Software Foundation, either version 3 of the License, or any later
+# version. See the LICENSE.txt file at the top-level directory of this
+# distribution for a copy of the license or check <https://www.gnu.org/licenses/>.
+#
+# SPDX-License-Identifier: GPL-3.0-or-later

liskov/src/icon4py/liskov/cli.py

@@ -0,0 +1,64 @@
+# ICON4Py - ICON inspired code in Python and GT4Py
+#
+# Copyright (c) 2022, ETH Zurich and MeteoSwiss
+# All rights reserved.
+#
+# This file is free software: you can redistribute it and/or modify it under
+# the terms of the GNU General Public License as published by the
+# Free Software Foundation, either version 3 of the License, or any later
+# version. See the LICENSE.txt file at the top-level directory of this
+# distribution for a copy of the license or check <https://www.gnu.org/licenses/>.
+#
+# SPDX-License-Identifier: GPL-3.0-or-later
+
+import pathlib
+
+import click
+
+from icon4py.liskov.logger import setup_logger
+from icon4py.liskov.pipeline import (
+    load_gt4py_stencils,
+    parse_fortran_file,
+    run_code_generation,
+)
+
+
+logger = setup_logger(__name__)
+
+
+@click.command("icon_liskov")
+@click.argument(
+    "input_filepath",
+    type=click.Path(
+        exists=True, dir_okay=False, resolve_path=True, path_type=pathlib.Path
+    ),
+)
+@click.argument(
+    "output_filepath",
+    type=click.Path(dir_okay=False, resolve_path=True, path_type=pathlib.Path),
+)
+@click.option(
+    "--profile", "-p", is_flag=True, help="Add nvtx profile statements to stencils."
+)
+def main(
+    input_filepath: pathlib.Path, output_filepath: pathlib.Path, profile: bool
+) -> None:
+    """Command line interface for interacting with the ICON-Liskov DSL Preprocessor.
+
+    Usage:
+        icon_liskov <input_filepath> <output_filepath> [--profile]
+
+    Options:
+        -p --profile Add nvtx profile statements to stencils.
+
+    Arguments:
+        input_filepath Path to the input file to process.
+        output_filepath Path to the output file to generate.
+    """
+    parsed = parse_fortran_file(input_filepath)
+    parsed_checked = load_gt4py_stencils(parsed)
+    run_code_generation(parsed_checked, input_filepath, output_filepath, profile)
+
+
+if __name__ == "__main__":
+    main()

liskov/src/icon4py/liskov/codegen/__init__.py

@@ -0,0 +1,15 @@
+# ICON4Py - ICON inspired code in Python and GT4Py
+#
+# Copyright (c) 2022, ETH Zurich and MeteoSwiss
+# All rights reserved.
+#
+# This file is free software: you can redistribute it and/or modify it under
+# the terms of the GNU General Public License as published by the
+# Free Software Foundation, either version 3 of the License, or any later
+# version. See the LICENSE.txt file at the top-level directory of this
+# distribution for a copy of the license or check <https://www.gnu.org/licenses/>.
+#
+# SPDX-License-Identifier: GPL-3.0-or-later
+
+# required import for CheckForDirectiveClasses metaclass
+import icon4py.liskov.parsing.types as ts  # noqa: F401