[llvm-cov][MC/DC] `mcdc_records` in JSON output insufficient

[Swatinem]

The mcdc_records field of the JSON output (llvm-cov export -format=text) is insufficient to reconstruct the "Executed MC/DC Test Vectors" output of llvm-cov show.

Consider this example:

llvm-cov show output:

    2|      4|    if (a || b) && (c || d) {
                                  ^3   ^2
  ------------------
  |  Branch (2:9): [True: 0, False: 4]
  |  Branch (2:14): [True: 3, False: 1]
  |  Branch (2:21): [True: 1, False: 2]
  |  Branch (2:26): [True: 1, False: 1]
  ------------------
  |---> MC/DC Decision Region (2:8) to (2:28)
  |
  |  Number of Conditions: 4
  |     Condition C1 --> (2:9)
  |     Condition C2 --> (2:14)
  |     Condition C3 --> (2:21)
  |     Condition C4 --> (2:26)
  |
  |  Executed MC/DC Test Vectors:
  |
  |     C1, C2, C3, C4    Result
  |  1 { F,  F,  -,  -  = F      }
  |  2 { F,  T,  F,  F  = F      }
  |  3 { F,  T,  T,  -  = T      }
  |  4 { F,  T,  F,  T  = T      }
  |
  |  C1-Pair: not covered
  |  C2-Pair: covered: (1,3)
  |  C3-Pair: covered: (2,3)
  |  C4-Pair: covered: (2,4)
  |  MC/DC Coverage for Decision: 75.00%
  |
  ------------------

llvm-cov export output:

{
  "branches": [
    [2, 9, 2, 10, 0, 4, 0, 0, 6],
    [2, 14, 2, 15, 3, 1, 0, 0, 6],
    [2, 21, 2, 22, 1, 2, 0, 0, 6],
    [2, 26, 2, 27, 1, 1, 0, 0, 6],
  ],
  "mcdc_records": [
    [2, 8, 2, 28, 0, 5, [false, true, true, true]],
  ]
}

Using the JSON export, I can reconstruct:

• the "Number of Conditions" by collecting all the branches that have a source range within the mcdc_record
• the "Coverage for Decision" and CX-Pair as that is quite literally the trailing array in the mcdc_record

However the information in the JSON output is not enough to reconstruct the "Executed MC/DC Test Vectors" table, or infer which table row was covering each condition pair.

Ideally, the JSON output should encode the whole table of relevant test vectors, and flag the ones that were executed.

For example, this is how it could look like for the example above:

"test_vectors": [
{"conditions": [false, false, null, null], "result": false, "executed": true},
{"conditions": [false, true, false, false], "result": false, "executed": true},
{"conditions": [false, true, true, null], "result": true, "executed": true},
{"conditions": [false, true, false, true], "result": true, "executed": true},
]

The output table of llvm-cov show only includes the executed vectors, however it would also be nice to show and include in the JSON all relevant vectors which were not executed as well.

[AlfredAn]

I think outputting spans for each condition would also be useful, unless I'm missing some way to infer that information?

Matching them up by checking for branches with a span inside the decision's span could work, but I think that for cases like this:

if (a || b) && if c && d { true } else { false } {
    //
}

Multiple nested decisions would make that harder to implement correctly. (this is a contrived example but I hope you get my point)

[kuzmich321]

I agree. The lack of executed test vectors in llvm-cov export is frustrating. Due to the fact that you could have several independent tests that store its llvm-cov export result somewhere and then you'd like to calculate mc/dc coverage by yourself.

[evodius96]

Implemented via #159119 (and #105511)

[evodius96]

Feel free to file a new issue in order to provide an option to display all relevant and unexecuted test vectors as well

[Swatinem]

That would be #105679