Refined level fails to get periodic patch ghosts values #311
Description
The following describes what appears to be a bug in SAMRAI.
In a (50,50) cells tagged hierarchy with max number of levels =2 and 2 MPI Ranks and fully periodic domain I end up with 1 refined level in the following configuration:
Level 0:
[Patch: box( Box([0, 25],[49, 49])), id(p0#0),
Patch: box( Box([0, 0],[49, 24])), id(p1#1)]
Level 1:
Patch: box( Box([0, 48],[99, 79])), id(p0#0),
Patch: box( Box([0, 16],[99, 47])), id(p1#1)
dashed lines are the L1 patches.
L0 patches are separated in the middle vertically.
Thus Level 1 has 2 patches, each touching the X left and right periodic boundaries.
However, when filling ghost values on L1 data, left and right borders are obtained sometimes obtained from refinement from L0 instead of periodic overlaps.
The trigger seems to be the smallest_patch_size parameters. With (3,3) we do get ghosts from periodic L1 values, whereas for (5,5) we do not. Our patch Data has (2,2) ghosts nodes.
The issue only appears in parallel as far as I can tell. L1 gets its ghosts correctly from periodic L1 overlap in sequential runs although L1 boxes are the same.
Debugging deep into SAMRAI's code reveals that differences appear already in makeFinerLevel>findRefinementBoxes>bridgeWithNesting where the tag_to_tag connector that is passed has a width that differs from (6,6) to (7,7). d_tag_to_cluster_width also differs. This results in things differing in visible_west_nabrs and in the end neighbor periodic box images are not taken into account when the ghost schedule is created and it does not call caculateOverlap()...
While I understand that changing the smallest patch size may changes some internal calculations, I do not think this should change the fact that L1 patches touching periodic borders get or not values from periodic overlaps while the L1 patch boxes etc. are exactly the same.