Judy - Gson15-f: Mutant killed but still appears as live mutant in GUI

[catarinaopires]

Mutant in line 543 ('JIR_Ifgt') is killed by test testFlush(), that is when executing defects4j test with the mutation manually inserted in the JsonWriter.java file, the test fails. However, in the GUI, it still appears alive.

Line 543 of JsonWriter.java file

    if (stackSize == 0) {

Line 543 with manually inserted JIR_Ifgt mutation JsonWriter.java file

    if (stackSize > 0) {

StudentTest.java

@Test
    public void testFlush() throws IOException {
        StringWriter stringWriter = new StringWriter();
        JsonWriter writer = new JsonWriter(stringWriter);
        writer.value(1);
        try {
            writer.flush();
        } catch (IllegalStateException e) {
            fail();
        }
    }

JsonWriter.java

/**
   * Ensures all buffered data is written to the underlying {@link Writer}
   * and flushes that writer.
   */
  public void flush() throws IOException {
    if (stackSize == 0) {
      throw new IllegalStateException("JsonWriter is closed.");
    }
    out.flush();
  }

[jose]

@PedroTav, any chance you could take a look at this as soon as possible?

[PedroTav]

I tried running defects4j test after manually changing the code and I'm getting 0 tests failed with the StudentTest.java included in the package.

[jose]

@PedroTav, in your run, the StudentTest.java file was in the com.google.gson.stream package, right?

[catarinaopires]

[PedroTav]

@jose, correct. However, I tested with test suites from previous years that reported having killed the same mutant and I'm also getting 0 tests failed. I'm trying to figure out if this is a problem with Judy because the log file generated from the analyzer tool shows the mutant as being alive after running the test suite.

[jose]

@PedroTav I've just confirmed that the testFlush test developed by @catarinaopires does detect the mutation, i.e., in other words, it does fail when the mutant is injected.

@PedroTav, how would one execute Judy from the command line?

[jose]

@PedroTav, how would one execute Judy from the command line?

Taking judy.sh as example, I guess it would be something like

judy -p target/classes/com/google/gson/stream/JsonWriter.class \
  -t target/test-classes/com/google/gson/stream/JsonWriterTest.class \
  --result-path "$(pwd)/judy-result.json" 2>&1 | tee judy.log

(note: I added @catarinaopires' test

@Test
public void testFlush() throws IOException {
    StringWriter stringWriter = new StringWriter();
    JsonWriter writer = new JsonWriter(stringWriter);
    writer.value(1);
    try {
        writer.flush();
    } catch (IllegalStateException e) {
        fail();
    }
}

to the JsonWriterTest test suite.)

which generates the following

• judy.log
• judy-result.json

and as we can see, according to Judy

    {
      "name": "com.google.gson.stream.JsonWriter",
      "score": 0.74642855,
      "mutantsCount": 280,
      "mutantsKilledCount": 209,
      "notKilledMutant": [
        {
          "operators": [
            "JIR_Ifgt"
          ],
          "points": [
            17
          ],
          "lines": [
            543
          ]
        },

the mutant is not killed.

@PedroTav, where can I find Judy's 3.0.0-M1 documentation? This documentation seem out of date, e.g., your judy.sh script uses -p to specify the class to be mutated and the official documentation says

-p, --properties PATH - the absolute path to a special file with additional settings for Judy.
The file is based on the standard format of a Java properties file. It contains
key=value pairs, where value can span many lines and contain a list of values
separated by ;. Lines beginning with # are treated as comments and ignored.

[jose]

@PedroTav, where can I find Judy's 3.0.0-M1 documentation?

Right here.

[PedroTav]

@catarinaopires I was in a meeting with Profª. Ana and we found out that there are some issues related to your student test class and needed to be fixed:

• The StudentTest class must extend TestCase public final class StudentTest extends TestCase
• Do not use @test annotation before the test methods

After making these changes you should be able to kill an additional mutant, but perhaps not the one you were expecting.

[PedroTav]

@jose thank you for the link, I was not aware of the more recent documentation for Judy. The judy.sh script was not implemented by me so I'm not sure how it's currently working. I'll check the new documentation and see if there may be any required changes.

[jose]

we found out that there are some issues related to your student test class and needed to be fixed

I failed to understand that. First, Gson-15 supports JUnit 4.12. Second, here is my small/minimal JUnit 3 test class (JsonWriterTest) that reveals the mutant when I manually injected. Judy reports, for the same test class, that the mutant isn't killed.

package com.google.gson.stream;

import junit.framework.TestCase;

import java.io.IOException;
import java.io.StringWriter;
import java.math.BigDecimal;
import java.math.BigInteger;

@SuppressWarnings("resource")
public final class JsonWriterTest extends TestCase {

  public void testFlush() throws IOException {
        StringWriter stringWriter = new StringWriter();
        JsonWriter writer = new JsonWriter(stringWriter);
        writer.value(1);
        try {
            writer.flush();
        } catch (IllegalStateException e) {
            fail();
        }
    }
}

After making these changes you should be able to kill an additional mutant, but perhaps not the one you were expecting.

I also failed to understand that. Were you or were you not able to run the provided test case with Judy and kill the mutant?

[PedroTav]

By using the JUnit 3 test class, the report showed that the mutant id = 9307f622 (JIR_Iflt) was killed by the test method.

There are two mutants in line 543; the one that is killed is not the operator JIR_lfgt,

and as we can see, according to Judy

    {
      "name": "com.google.gson.stream.JsonWriter",
      "score": 0.74642855,
      "mutantsCount": 280,
      "mutantsKilledCount": 209,
      "notKilledMutant": [
        {
          "operators": [
            "JIR_Ifgt"
          ],
          "points": [
            17
          ],
          "lines": [
            543
          ]
        },

the mutant is not killed.

but is instead JIR_lflt, which no longer shows up after running with the StudentTest

    {
      "name": "com.google.gson.stream.JsonWriter",
      "score": 0.8056537,
      "mutantsCount": 283,
      "mutantsKilledCount": 228,
      "notKilledMutant": [
       {
          "operators": [
            "JIR_Iflt"
          ],
          "points": [
            17
          ],
          "lines": [
            543
          ]
        },

I also agree that it doesn't make much sense, because as the Professor mentioned Gson is supposed to support JUnit 4.12.

[catarinaopires]

The mutant JIR_lflt was always killed. I too was able to kill this mutant, and the GUI detects that.
The problem is JIR_lfgt.

[jose]

Exactly!

[PedroTav]

I'm sorry, I misunderstood which mutant was having the issue. Is this the only mutant that Judy has not reported as being killed so far? I know Judy does mutation on byte code level and can sometimes mutate in a way we don't predict, however, I also don't see how else you could interpret JIR_lfgt in that particular method.