[FOUR_C_WARNING] Warning messages

[abhi-satheesh]

Description

Similar to FOUR_C_THROW, it would be nice to have a FOUR_C_WARNING macro instead of using if statements with std::cout.

Furthermore, if necessary, all warnings could be collected and printed at the end of the simulation, instead of having to search through the entire simulation log.

Example: /src/adapter/4C_adapter_coupling_nonlin_mortar.cpp
if (abs((*diag)[i]) < 1e-12) { std::cout << "WARNING: Diagonal entry of D matrix is skipped because it is less than 1e-12!!!" << std::endl; (*diag)[i] = 1.0; }
changes to
FOUR_C_WARNING(condition, message )

Possible Solution and Definition of Done

Interested Parties

[sebproell]

Thanks for bringing in this idea, @abhi-satheesh! I migrated this to a discussion, so we can freely explore ideas, and generate an issue with some concrete tasks if they should arise.

[sebproell]

@abhi-satheesh Some random thoughts:

• I totally agree that printouts can appear more or less random, are not that helpful, and are intermixed with too much other output.
• Informal status output always via std::cout always has to do the rank == 0 check.
• What you are describing reminds me about logging and I thought about introducing this a couple of times already. Note that we have a kind of log stream with Core::IO::cout but it is not used consistently.
• If we go for any kind of logging, we should consistently apply it in the whole code base.

[mayrmt]

Side note: we currently write to screen via std::cout, IO::cout, and printf. In parallel, this can lead to the wrong ordering of output, that cannot even fixed by placing barriers of the MPI communicator, since printf is "faster" in actually writing to screen than std::cout (https://stackoverflow.com/a/20238349).

@sbrandstaeter has observed this at the end of the 4C simulations, when every processor reports its finalization and proc 0 writes the total time for the calculation. These are then often intermixed, so that you cannot safely extract the total time of the calculation via a script. Barriers do not fix that: they separate the initiation of writing output, but it appears that the different performance/speed of the individual output mechanism leads to intermixed output to the screen anyways.

[sbrandstaeter]

To give an example of the phenomenon described by @mayrmt above.

Sometimes, the output logs do not conform to the expected structure. Specifically, the summary line reporting the total wall time may become distorted, making it difficult to extract it via tools like grep.
Example:

Total wall time forprocessor 3 finished normally     
processor 6 finished normally
processor 2 finished normally                        
processor 4 finished normally
processor 1 finished normally
processor 5 finished normally                        
processor 7 finished normally
 CALCULATION:  2.619e+01 sec

In a study of 4C simulations, approximately 0.5% of runs failed to produce a greppable total wall time due to unstructured or interleaved output. While rare, this causes significant issues in multi-query analyses and highlights the need for ensuring structured screen output.

[amgebauer]

We should add an option to write these important metrics into a json/csv file so that we don't need to grep for it, which is anyway very error prone.

[abhi-satheesh]

@sebproell thanks for the comments.

1. I like the idea. Structured logging (e.g., levels like INFO, WARNING, ERROR) would allow us to filter and aggregate output more easily, even across ranks. If we define some basic logging macros, we can later plug them into different backends or output streams.

2. Yes, the rank == 0 checks everywhere are tedious and error-prone. A logging system could abstract that away automatically.

[mayrmt]

Furthermore, if necessary, all warnings could be collected and printed at the end of the simulation, instead of having to search through the entire simulation log.

I prefer to have a warning printed, when it occurs. Otherwise, there will be just a bunch of warnings at the end and one does not know, when it occurred.

[sebproell]

Yes, but a better filtering or standard string you search for would be nice. Also, something like x Warnings occurred during the simulation run could be interesting (but only if x is usually zero, or nobody will care).

[abhi-satheesh]

That's a good point. In my opinion, the warning should be retraceable (code part/time step,...)

[mayrmt]

Instead of an explicit FOUR_C_WARNING macro (and probably some more macros for other types of information), we could use a centralized output stream, that takes different "verbosity levels", which can be chosen by the user at run time.

MueLu for example uses the following verbosity levels: https://github.com/trilinos/Trilinos/blob/master/packages/muelu/src/MueCentral/MueLu_VerbosityLevel.hpp