Error transforming mesh in parallel

[baagaard-usgs]

Describe the bug

[0]PETSC ERROR: Argument out of range
[0]PETSC ERROR: Replica number 1 should be in [0, 1) for subcell type segment in cell type segment
— snip —
[0]PETSC ERROR: #1 DMPlexTransformGetTargetPoint() at /Users/brad/software/unix/petsc-dev/src/dm/impls/plex/transform/interface/plextransform.c:1027
[0]PETSC ERROR: #2 DMPlexTransformGetCone_Internal() at /Users/brad/software/unix/petsc-dev/src/dm/impls/plex/transform/interface/plextransform.c:1492
[0]PETSC ERROR: #3 DMPlexTransformSetCones() at /Users/brad/software/unix/petsc-dev/src/dm/impls/plex/transform/interface/plextransform.c:1553
[0]PETSC ERROR: #4 DMPlexTransformApply() at /Users/brad/software/unix/petsc-dev/src/dm/impls/plex/transform/interface/plextransform.c:2443
[0]PETSC ERROR: #5 pylith::topology::Mesh *pylith::faults::FaultCohesive::transformTopology(pylith::topology::Mesh *const)() at /Users/brad/work/src/cig/pylith/libsrc/pylith/faults/FaultCohesive.cc:344

To Reproduce

PETSc branch: knepley/fix-plex-orient-label
PyLith fork: https://github.com/baagaard-usgs/pylith.git
PyLith branch: feature-initialize-phases-petsc-main

Steps to reproduce the behavior.

cd $SRC/tests/fullscale/cornercases/faults-2d
pylith twoblocks.cfg twoblocks_tri.cfg —nodes=3

[knepley]

@baagaard-usgs Okay, I think the problem is that the surfaceLabel is wrong. On process 0, triangle 0 has type 700, but that is for split triangles on the fault. The type should be 600 for an impinging triangle. Tracking down the numbering now.

[knepley]

@baagaard-usgs Okay, surfaceLabel is wrong for triangle 0 on process 0, which has value 2. It should be 102, as they are on process 1.

[knepley]

@baagaard-usgs The problem is in DMPlexLabelCohesiveComplete(). I am looking at it...

[knepley]

@baagaard-usgs Okay, if you pull my branch and your branch, I think it is fixed.

[baagaard-usgs]

@knepley The "replica number" bug seems to be fixed. I am getting other errors now:

Topology-related error

pylith twoblocks.cfg twoblocks_tri.cfg --nodes=3

[1]PETSC ERROR: PETSc has generated inconsistent data
[1]PETSC ERROR: Size of cone for edge 0 must be 2, not 3
-- snip --
[1]PETSC ERROR: #1 DMPlexOrientCells_Internal() at /software/baagaard/petsc-dev/src/dm/impls/plex/plexorient.c:814
[1]PETSC ERROR: #2 DMPlexOrientLabel() at /software/baagaard/petsc-dev/src/dm/impls/plex/plexorient.c:734
[1]PETSC ERROR: #3 static void pylith::faults::TopologyOps::updateCohesiveLabel(const pylith::topology::Mesh*, const char*, int)() at /home/baagaard/src/cig/pylith/libsrc/pylith/faults/TopologyOps.cc:471

Communication error

RESOLVED: Process 3 did not have any cells after distribution. I added a check to catch this and throw an appropriate exception.

pylith twoblocks.cfg twoblocks_quad.cfg --nodes=5

[0]PETSC ERROR: PETSc has generated inconsistent data
[0]PETSC ERROR: MPIU_Allreduce() called in different locations on different processes:
                On process 0, /software/baagaard/petsc-dev/src/sys/utils/psplit.c:78
                On process 1, /software/baagaard/petsc-dev/src/sys/utils/psplit.c:78
                On process 2, /software/baagaard/petsc-dev/src/sys/utils/psplit.c:78
                On process 3, /software/baagaard/petsc-dev/src/ts/interface/ts.c:2198
                On process 4, /software/baagaard/petsc-dev/src/sys/utils/psplit.c:78

[knepley]

@baagaard-usgs Did you change the mesh? I am now getting a different mesh.

[baagaard-usgs]

@knepley No. https://github.com/baagaard-usgs/pylith/commits/feature-initialize-phases-petsc-main/
Are you on PyLith branch feature-initialize-phases-petsc-main?

[knepley]

Put info in GChat

[knepley]

@baagaard-usgs I pushed a potential fix to the PETSc branch. I think the topology is correct now, but the solve does not go through.

[baagaard-usgs]

It looks like the fault is not being split correctly or the variable point blocks are not grouped correctly. The KSP solve converged but the SNES residual hardly decreased.

    0 SNES Function norm 9.152953038406e+00
      0 KSP Residual norm 2.744573374487e+00
      1 KSP Residual norm 1.683271476075e-15
    1 SNES Function norm 6.440492537399e+00

[knepley]

When I look at the matrix

-ksp_view_mat draw -draw_pause -1 -geometry 0,0,1024,1024

it does not appear to be permuted, so vpbjacobi will fail.