ArrayIndexOutOfBoundsException on TestDetector

[jose]

Hi @andre15silva,

I've just tried flacoco on Csv-10 from the Defects4J benchmark and it fails on my machine. Here's the command to reproduce this issue:

java -cp target/flacoco-1.0.6-SNAPSHOT-jar-with-dependencies.jar fr.spoonlabs.flacoco.cli.FlacocoMain \
  --binJavaDir="/tmp/Csv-10b/target/classes" \
  --binTestDir="/tmp/Csv-10b/target/test-classes" \
  --classpath="$D4J_HOME/framework/projects/Csv/lib/junit/junit/4.11/junit-4.11.jar:$D4J_HOME/framework/projects/Csv/lib/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar:$D4J_HOME/framework/projects/Csv/lib/commons-io/commons-io/2.2/commons-io-2.2.jar:$D4J_HOME/framework/projects/Csv/lib/com/h2database/h2/1.3.168/h2-1.3.168.jar:/tmp/Csv-10b/target/classes:/tmp/Csv-10b/target/test-classes:$D4J_HOME/framework/projects/lib/junit-4.11.jar:$D4J_HOME/framework/projects/Csv/lib/com/csvreader/1.0/javacsv.jar:$D4J_HOME/framework/projects/Csv/lib/com/generationjava/io/1.0/gj-csv-1.0.jar:$D4J_HOME/framework/projects/Csv/lib/com/h2database/h2/1.4.180/h2-1.4.180.jar:$D4J_HOME/framework/projects/Csv/lib/com/h2database/h2/1.4.181/h2-1.4.181.jar:$D4J_HOME/framework/projects/Csv/lib/com/h2database/h2/1.4.182/h2-1.4.182.jar:$D4J_HOME/framework/projects/Csv/lib/com/h2database/h2/1.4.196/h2-1.4.196.jar:$D4J_HOME/framework/projects/Csv/lib/com/h2database/h2/1.4.198/h2-1.4.198.jar:$D4J_HOME/framework/projects/Csv/lib/com/opencsv/4.0/opencsv-4.0.jar:$D4J_HOME/framework/projects/Csv/lib/commons-io/commons-io/2.4/commons-io-2.4.jar:$D4J_HOME/framework/projects/Csv/lib/commons-io/commons-io/2.5/commons-io-2.5.jar:$D4J_HOME/framework/projects/Csv/lib/commons-io/commons-io/2.6/commons-io-2.6.jar:$D4J_HOME/framework/projects/Csv/lib/junit/junit/3.8.1/junit-3.8.1.jar:$D4J_HOME/framework/projects/Csv/lib/junit/junit/4.10/junit-4.10.jar:$D4J_HOME/framework/projects/Csv/lib/junit/junit/4.12/junit-4.12.jar:$D4J_HOME/framework/projects/Csv/lib/org/apache/commons/commons-lang3/3.4/commons-lang3-3.4.jar:$D4J_HOME/framework/projects/Csv/lib/org/apache/commons/commons-lang3/3.6/commons-lang3-3.6.jar:$D4J_HOME/framework/projects/Csv/lib/org/apache/commons/commons-lang3/3.7/commons-lang3-3.7.jar:$D4J_HOME/framework/projects/Csv/lib/org/apache/commons/commons-lang3/3.8.1/commons-lang3-3.8.1.jar:$D4J_HOME/framework/projects/Csv/lib/org/hamcrest/hamcrest-core/1.1/hamcrest-core-1.1.jar:$D4J_HOME/framework/projects/Csv/lib/org/mockito/mockito-all/1.10.19/mockito-all-1.10.19.jar:$D4J_HOME/framework/projects/Csv/lib/org/mockito/mockito-all/1.9.5/mockito-all-1.9.5.jar:$D4J_HOME/framework/projects/Csv/lib/org/openjdk/jmh/1.21/jmh-core-1.21.jar:$D4J_HOME/framework/projects/Csv/lib/org/skife/csv/1.0/csv-1.0.jar:$D4J_HOME/framework/projects/Csv/lib/org/supercsv/2.4.0/super-csv-2.4.0.jar" \
  --format=CSV \
  --includeZeros \
  --jacocoIncludes="org.apache.commons.csv.Assertions:org.apache.commons.csv.Constants:org.apache.commons.csv.CSVFormat:org.apache.commons.csv.CSVPrinter:org.apache.commons.csv.Quote" \
  --junitClasspath="$D4J_HOME/framework/projects/lib/junit-4.11.jar" \
  --output="/tmp/Csv-10b/flacoco.csv" \
  --projectpath="/tmp/Csv-10b" \
  --srcJavaDir="/tmp/Csv-10b/src/main/java" \
  --srcTestDir="/tmp/Csv-10b/src/test/java" \
  --testDetectionStrategy=CLASSLOADER \
  --testRunnerVerbose \
  --junit4tests="org.apache.commons.csv.CSVFormatTest:org.apache.commons.csv.CSVParserTest:org.apache.commons.csv.CSVPrinterTest:org.apache.commons.csv.CSVRecordTest" \
  -v

which throws the following exception:

[0] INFO Flacoco - Running Flacoco...
[5] DEBUG SpectrumRunner - Running spectrum-based fault localization...
[5] DEBUG SpectrumRunner - FlacocoConfig{workspace='/tmp/flacoco/.', projectPath='/tmp/Csv-10b', srcJavaDir=[/tmp/Csv-10b/src/main/java], srcTestDir=[/tmp/Csv-10b/src/test/java], binJavaDir=[/tmp/Csv-10b/target/classes], binTestDir=[/tmp/Csv-10b/target/test-classes], classpath='/tmp/defects4j/framework/projects/Csv/lib/junit/junit/4.11/junit-4.11.jar:/tmp/defects4j/framework/projects/Csv/lib/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar:/tmp/defects4j/framework/projects/Csv/lib/commons-io/commons-io/2.2/commons-io-2.2.jar:/tmp/defects4j/framework/projects/Csv/lib/com/h2database/h2/1.3.168/h2-1.3.168.jar:/tmp/Csv-10b/target/classes:/tmp/Csv-10b/target/test-classes:/tmp/defects4j/framework/projects/lib/junit-4.11.jar:/tmp/defects4j/framework/projects/Csv/lib/com/csvreader/1.0/javacsv.jar:/tmp/defects4j/framework/projects/Csv/lib/com/generationjava/io/1.0/gj-csv-1.0.jar:/tmp/defects4j/framework/projects/Csv/lib/com/h2database/h2/1.4.180/h2-1.4.180.jar:/tmp/defects4j/framework/projects/Csv/lib/com/h2database/h2/1.4.181/h2-1.4.181.jar:/tmp/defects4j/framework/projects/Csv/lib/com/h2database/h2/1.4.182/h2-1.4.182.jar:/tmp/defects4j/framework/projects/Csv/lib/com/h2database/h2/1.4.196/h2-1.4.196.jar:/tmp/defects4j/framework/projects/Csv/lib/com/h2database/h2/1.4.198/h2-1.4.198.jar:/tmp/defects4j/framework/projects/Csv/lib/com/opencsv/4.0/opencsv-4.0.jar:/tmp/defects4j/framework/projects/Csv/lib/commons-io/commons-io/2.4/commons-io-2.4.jar:/tmp/defects4j/framework/projects/Csv/lib/commons-io/commons-io/2.5/commons-io-2.5.jar:/tmp/defects4j/framework/projects/Csv/lib/commons-io/commons-io/2.6/commons-io-2.6.jar:/tmp/defects4j/framework/projects/Csv/lib/junit/junit/3.8.1/junit-3.8.1.jar:/tmp/defects4j/framework/projects/Csv/lib/junit/junit/4.10/junit-4.10.jar:/tmp/defects4j/framework/projects/Csv/lib/junit/junit/4.12/junit-4.12.jar:/tmp/defects4j/framework/projects/Csv/lib/org/apache/commons/commons-lang3/3.4/commons-lang3-3.4.jar:/tmp/defects4j/framework/projects/Csv/lib/org/apache/commons/commons-lang3/3.6/commons-lang3-3.6.jar:/tmp/defects4j/framework/projects/Csv/lib/org/apache/commons/commons-lang3/3.7/commons-lang3-3.7.jar:/tmp/defects4j/framework/projects/Csv/lib/org/apache/commons/commons-lang3/3.8.1/commons-lang3-3.8.1.jar:/tmp/defects4j/framework/projects/Csv/lib/org/hamcrest/hamcrest-core/1.1/hamcrest-core-1.1.jar:/tmp/defects4j/framework/projects/Csv/lib/org/mockito/mockito-all/1.10.19/mockito-all-1.10.19.jar:/tmp/defects4j/framework/projects/Csv/lib/org/mockito/mockito-all/1.9.5/mockito-all-1.9.5.jar:/tmp/defects4j/framework/projects/Csv/lib/org/openjdk/jmh/1.21/jmh-core-1.21.jar:/tmp/defects4j/framework/projects/Csv/lib/org/skife/csv/1.0/csv-1.0.jar:/tmp/defects4j/framework/projects/Csv/lib/org/supercsv/2.4.0/super-csv-2.4.0.jar', customJUnitClasspath='/tmp/defects4j/framework/projects/lib/junit-4.11.jar', customJacocoClasspath='null', mavenHome='/Users/jose/.m2/repository/', coverTests=false, testRunnerVerbose=true, testRunnerTimeoutInMs=1000000, testRunnerJVMArgs='null', threshold=0.0, includeZeros=true, complianceLevel=8, testDetectionStrategy=CLASSLOADER, ignoredTests=[], jUnit4Tests=[org.apache.commons.csv.CSVFormatTest:org.apache.commons.csv.CSVParserTest:org.apache.commons.csv.CSVPrinterTest:org.apache.commons.csv.CSVRecordTest], jUnit5Tests=[], jacocoIncludes=[org.apache.commons.csv.Assertions:org.apache.commons.csv.Constants:org.apache.commons.csv.CSVFormat:org.apache.commons.csv.CSVPrinter:org.apache.commons.csv.Quote], jacocoExcludes=[], family=SPECTRUM_BASED, spectrumFormula=OCHIAI, computeSpoonResults=false}
[7] DEBUG TestDetector - Computing tests from config.
java.lang.ArrayIndexOutOfBoundsException: 1
	at fr.spoonlabs.flacoco.core.test.TestDetector.lambda$computeTests$0(TestDetector.java:56)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at java.util.Iterator.forEachRemaining(Iterator.java:116)
	at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)
	at fr.spoonlabs.flacoco.core.test.TestDetector.computeTests(TestDetector.java:58)
	at fr.spoonlabs.flacoco.core.test.TestDetector.getTests(TestDetector.java:39)
	at fr.spoonlabs.flacoco.localization.spectrum.SpectrumRunner.computeCoverageMatrix(SpectrumRunner.java:64)
	at fr.spoonlabs.flacoco.localization.spectrum.SpectrumRunner.run(SpectrumRunner.java:42)
	at fr.spoonlabs.flacoco.api.Flacoco.run(Flacoco.java:31)
	at fr.spoonlabs.flacoco.cli.FlacocoMain.call(FlacocoMain.java:157)
	at fr.spoonlabs.flacoco.cli.FlacocoMain.call(FlacocoMain.java:27)
	at picocli.CommandLine.executeUserObject(CommandLine.java:2041)
	at picocli.CommandLine.access$1500(CommandLine.java:148)
	at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2453)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2415)
	at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273)
	at picocli.CommandLine$RunLast.execute(CommandLine.java:2417)
	at picocli.CommandLine.execute(CommandLine.java:2170)
	at fr.spoonlabs.flacoco.cli.FlacocoMain.main(FlacocoMain.java:148)

$ javac -version
javac 1.8.0_292

$ java -version
openjdk version "1.8.0_292"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_292-b10)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.292-b10, mixed mode)

[andre15silva]

Hi @jose ,

I see two issues with that command:

1. The junit4tests only accept qualified test method names (e.g., org.apache.commons.csv.CSVFormatTest#testEquals)
2. flacoco's cli is not handling lists of arguments correctly (even though we test for that). I'm updating the way flacoco handles lists of arguments through the cli (see fix cli list handling #241). You will now be able to use either whitespaces, :, or , as the separator.

Let me know if you need anything else :)

[jose]

Thanks @andre15silva. Regarding the junit4tests argument, how would you set that argument to run flacoco on a D4J bug? I am asking this because D4J only provides the qualified test class names (known as relevant tests) and not the qualified test method names.

[andre15silva]

Hi @jose ,

Sorry for the late response, this got buried after summer.

You are correct, I had forgotten that we actually computed these for the experiment.

They are retrieved from a standard Defects4J run, which generates two files: all_tests and failing_tests. These files contain the method names.

Script to compute fully qualified test method names:
https://github.com/UPCArtifacts/flacoco_experiment/blob/main/scripts/runAllBugs4TestBaseline.py

Results:
https://github.com/UPCArtifacts/flacoco_experiment/tree/main/scripts/test_baselines

We then use these baselines to set the tests method names for both Flacoco and Gzoltar through Astor.
Script to run all FL through Astor:
https://github.com/UPCArtifacts/flacoco_experiment/blob/4edd17d1884359097dcfe3636979fc3a39d48d09/scripts/runAllBugs.py#L399-L413