Results depend on number of MPI ranks for RANS channel flow test case

[trevilo]

I'm observing that different numbers of MPI ranks lead to slight (but significant) changes in results for the RANS channel flow test case. @clarkpede have you seen anything like this before?

I'll post detailed instructions to recreate asap, along with sample results, but basically, just run the channel flow test case to convergence with, for example, mpirun -np 1 and mpirun -np 4 and observe the difference.

Doing the same thing with the flat plate test case does not lead to any significant discrepancy as far as I can tell. So, it appears channel flow specific, which makes the forcing or periodic BCs the most likely culprits.

[clarkpede]

Do you mean significant changes in the transient residuals, or significant changes in the converged (mean) flow results?

[trevilo]

Converged results. I think I have a handle on this now. It has to do with the fact that, for the channel case, the steady equations only determine the pressure up to a constant. The constant is fixed (indirectly) by the initial conditions. But, different time marching (and hence difference temporal errors) lead to different values for that constant.

I noticed the problem in comparing serial vs. parallel b/c the linear solver tolerance was set pretty loose, such that different numbers of cores were getting different time histories. Cranking down on the linear solver tolerance eliminates the serial vs. parallel issue. The underlying issue is still there of course, but I haven't decided what to do (if anything) about it yet.