pythonizing regression

Pythonizing: Regression

There exist two workflows for regression testing, interactive mode and global mode.

Intent

The intent is that Chapter Committees run this automated regression tool on their chapters and commit the resulting approvals.json file to the pull request branch. When we have a full/complete approvals.json (for every single binding), we'll know that the process is complete.

Interactive Workflow

• Get two git clones: one of the main repo, and one for the Pythonization PR.
  • For example:

$ git clone [email protected]:mpi-forum/mpi-standard.git mpi4
...git clone output...
$ git clone [email protected]:martinruefenacht/mpi-standard.git mpi-pythonization
...git clone output...

• Git checkout the mpi-4.x branch in the main repo clone.
• Git checkout the feat-modern-lang-bindings in the Pythonization PR repo clone.
  • For example:

$ cd mpi4
$ git checkout mpi-4.x
...some git output...
$ cd ../mpi-pythonization
$ git checkout feat-modern-lang-bindings
...some git output...

• In the Pythonization repo:
  1. Run make cleandoc
  2. Run ./binding-tool/regress.py --interactive <PATH_TO_MPI4_REPO> <PATH_TO_PYTHONIZATION_REPO>.
     • For example: ./binding-tool/regress.py --interactive ../mpi4 .

The interactive mode will iterate through all common bindings between the two repositories and show them to the user for approval.

When a non-approved binding is found, a diff is shown to the user with a likeness ratio and locations in files of both repositories with line of file, respectively.

This is an example of a difference:

################################################################################
################################################################################

likeness ratio 0.9048843187660668
original ../mpis/mpi3/chap-coll/coll.tex, 1154
modified chap-coll/coll-rendered.tex, 1103


- \mpifnewbind{MPI\_Scatter(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, root, comm, ierror) \fargs TYPE(*), DIMENSION(..), INTENT(IN) :: sendbuf
?                                                                               
+ \mpifnewbind{MPI\_Scatter(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, root, comm, ierror) \fargs TYPE(*), DIMENSION(..), INTENT(IN) :: sendbuf
?                                                                               

 \\ TYPE(*), DIMENSION(..) :: recvbuf \\ INTEGER, INTENT(IN) :: sendcount, recvcount, root \\ TYPE(MPI\_Datatype), INTENT(IN) :: sendtype, recvtype \\ TYPE(MPI\_Comm), INTENT(IN) :: comm \\ INTEGER, OPTIONAL, INTENT(OUT) :: ierror}
   -------------------------------------

 \\ INTEGER, INTENT(IN) :: sendcount, recvcount, root \\ TYPE(MPI\_Datatype), INTENT(IN) :: sendtype, recvtype \\ TYPE(*), DIMENSION(..) :: recvbuf \\ TYPE(MPI\_Comm), INTENT(IN) :: comm \\ INTEGER, OPTIONAL, INTENT(OUT) :: ierror}

                                 +++++++++++++++++++++++++++++++++++++

approve(a), skip(s), exit(x): 

A ###### marker is used for difference below 95% matching ratio. A <><><> is shown for less severe mismatches.

The prompt asks for approval for this binding. The user can exit or skip the binding instead of approving. When approved a binding, when encountered is automatically skipped.

Approvals serve as a mechanism to allow for changes between MPI-4 version bindings and the Pythonized version bindings. For example spacing changes, reordering of tildes (non-linebreaking spacing in LaTeX), reordered bindings, etc.

An example of a less severe mismatch is:

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

likeness ratio 0.9841269841269841
original ../mpis/mpi3/chap-io/io-2.tex, 3342
modified chap-io/io-2-rendered.tex, 3221


- \mpifbind{MPI\_FILE\_GET\_ATOMICITY(FH, FLAG, IERROR)\fargs INTEGER FH, IERROR
+ \mpifbind{MPI\_FILE\_GET\_ATOMICITY(FH, FLAG, IERROR) \fargs INTEGER FH, IERRO
?                                                      +                        

\\LOGICAL FLAG}
R \\ LOGICAL FLAG}
 +  +

approve(a), skip(s), exit(x):

Automatic approving is done when identical bindings are discovered between the two repositories.

Automatic skipping is performed for approved bindings, which are saved in the approvals.json file, stored in the working directory.