test suites and test cases

[pinventado]

I have been using test suites and test cases. Is it possible for grading to consider the cases too?

I often use it to distinguish test cases for a particular class like this:

TEST(Cup, DefaultConstructor) {
...
}
TEST(Cup, drink){
...
}

I think the current code uses the test suite name and only gives points when there are 0 failures.

rubricscore/rubricscore.cpp

Line 86 in b675194

auto testsuites = tree.get_child("testsuites");

rubricscore/rubricscore.cpp

Line 249 in b675194

bool worked = (0 == suite.failures());

Not sure what the best way to go would be, but maybe have grades associated with test cases instead, and check if the value of result is completed? Might need to investigate this further though.

[kevinwortman]

Yes, initially to keep things simple, in my unit tests each suite has exactly one case with the same name. I'm open to allowing multiple cases per suite but it opens up the question of how the scoring should work when there are two tiers of correctness. Would you want points counted per case instead of per suite as it is now?

[pinventado]

Yeah, that's what I was thinking. What are your thoughts on that?

[kevinwortman]

I'm happy to have it work that way.

There's a technical problem that'd need to be overcome, which is why I didn't do it that way to begin with. When you do
TEST(Cup, DefaultConstructor)
googletest creates a class Cup with Cup::DefaultConstructor member function. So DefaultConstructor is not in the toplevel namespace, and googletest allows name collisions like
TEST(Cup, DefaultConstructor)
...
TEST(Bowl, DefaultConstructor)
However in the rubric.json file , currently there is one string for the test name, so it would be ambiguous which case "DefaultConstructor" refers to.

One solution could be to change how the strings in rubric.json are interpreted, so you have to write
"ProductPositive::ProductPositive" : 3,
This wouldn't be too hard to implement, but it would break existing rubric.json files. So I'd prefer to do it after spring semester ends.

What are your thoughts?

[pinventado]

Interesting. Another thought was to have something like a hierarchical setup like ...

{
  Cup : {
    DefaultConstructor : 3
  }
}

Would you prefer the version you suggested?

Namanh is working on a development branch and I think it's ok for us to use that for testing. Then when we come up with a working version and you want to merge to master we can do that whenever you are ready :)