Add timing information to CodeChecker runs

[gamesh411]

Add timing feature via wrapping the analyis runs with /usr/bin/time.

[Xazax-hun]

What do you want to do exactly? Getting the time of each analyzer invocation?

I think there is a better way to do that. Relying on /usr/bin/time is fragile. It might not work on non-Unix systems making future porting harder.

In case we invoke the analyzer with the command -Xclang -analyzer-stats it will emit more detailed timing information such as:

===-------------------------------------------------------------------------===
                                Analyzer timers
===-------------------------------------------------------------------------===
  Total Execution Time: 0.0011 seconds (0.0356 wall clock)

   ---User Time---   --System Time--   --User+System--   ---Wall Time---  --- Name ---
   0.0003 ( 41.0%)   0.0002 ( 41.1%)   0.0005 ( 41.0%)   0.0204 ( 57.3%)  Syntax-based analysis time
   0.0004 ( 58.9%)   0.0002 ( 58.9%)   0.0007 ( 58.9%)   0.0152 ( 42.7%)  Path exploration time
   0.0000 (  0.1%)   0.0000 (  0.0%)   0.0000 (  0.1%)   0.0000 (  0.0%)  Path-sensitive report post-processing time
   0.0007 (100.0%)   0.0004 (100.0%)   0.0011 (100.0%)   0.0356 (100.0%)  Total

I think we should rely on Analyzer's timers rather than on OS-specific solutions. Moreover, we should already collect duration information. In case such information is not collected, maybe the regexes are out of date in https://github.com/Xazax-hun/csa-testbench/blob/master/summarize_sa_stats.py#L85

If measuring the individual TU times (you can also get a histogram of them I believe) is not what you want, maybe it would be better to just measure the time of the whole analysis command within python?

[gamesh411]

What do you want to do exactly? Getting the time of each analyzer invocation?

I think there is a better way to do that. Relying on /usr/bin/time is fragile. It might not work on non-Unix systems making future porting harder.

In case we invoke the analyzer with the command -Xclang -analyzer-stats it will emit more detailed timing information such as:

===-------------------------------------------------------------------------===
                                Analyzer timers
===-------------------------------------------------------------------------===
  Total Execution Time: 0.0011 seconds (0.0356 wall clock)

   ---User Time---   --System Time--   --User+System--   ---Wall Time---  --- Name ---
   0.0003 ( 41.0%)   0.0002 ( 41.1%)   0.0005 ( 41.0%)   0.0204 ( 57.3%)  Syntax-based analysis time
   0.0004 ( 58.9%)   0.0002 ( 58.9%)   0.0007 ( 58.9%)   0.0152 ( 42.7%)  Path exploration time
   0.0000 (  0.1%)   0.0000 (  0.0%)   0.0000 (  0.1%)   0.0000 (  0.0%)  Path-sensitive report post-processing time
   0.0007 (100.0%)   0.0004 (100.0%)   0.0011 (100.0%)   0.0356 (100.0%)  Total

I think we should rely on Analyzer's timers rather than on OS-specific solutions. Moreover, we should already collect duration information. In case such information is not collected, maybe the regexes are out of date in https://github.com/Xazax-hun/csa-testbench/blob/master/summarize_sa_stats.py#L85

If measuring the individual TU times (you can also get a histogram of them I believe) is not what you want, maybe it would be better to just measure the time of the whole analysis command within python?

I think you might be right.

First, I will check the regexes, but I am also aware that they might just be correct, as I only get statistics information if LLVM is built with -DLLVM_FORCE_ENABLE_STATS=On. It is defined here, but is not seems to be documented. If not building in debug mode it is disabled by default. There is also a command-line option.

Second, I think I used /usr/bin/time because it also had max-resident-memory information, but that may not be the most portable solution for that purpose either. I will continue the memory measurement discussion in its relevant issue thread.

[Xazax-hun]

I see.

I think the whole command duration should already be logged and if it does not it is a bug.
We do not measure anything about the memory yet though.
In this case, I think the time command might be really valuable.

I am ok with using time to measure memory if no better alternative is found. But we should make it to fail gracefully, i.e. if the time binary is not available, it should just continue normally without measuring memory.

And the running time should never be measured using time. That measurement should definitely be universal and cross-platform.