what is required for a class to be found by console launcher?

[maxandersen]

I'm trying to figure out why a jar with code like this:

public class genTest {

    // Define each Unit test here and run them separately in the IDE
    @Test
    public void testgen() {
            assertEquals(1,2, "You should add some testing code for gen here!");
    }   

is not found by ConsoleLauncher scan classpath when using it with jbang

While it is picked up if I manually do this:

 final LauncherDiscoveryRequest request =
                LauncherDiscoveryRequestBuilder.request()
                        .selectors(selectClass(genTest.class))
                        .build();

but i would really just like junit console launcher do it so i dont have to have to carry around the junit test code.

Steps to reproduce:

Install jbang from https://jbang.dev/download or just run curl -Ls https://sh.jbang.dev | bash -s - app setup

Make a class and a generate junit test suite:
(note: the content of hello.java is not relevant; just there to generate helloTest.java)

jbang init hello.java
jbang init -t junit@jbangdev hello.java

Now you have a hello.java and helloTest.java and you can run the tests using:

jbang helloTest.java

which will give some output like this:

[jbang] Resolving dependencies...
[jbang]    org.junit.jupiter:junit-jupiter-api:5.10.0
[jbang]    org.junit.jupiter:junit-jupiter-engine:5.10.0
[jbang]    org.junit.platform:junit-platform-launcher:1.10.0
[jbang] Dependencies resolved
[jbang] Building jar for helloTest.java...
TestPlan Execution Started: org.junit.platform.launcher.TestPlan@887af79
Execution Started: JUnit Jupiter - [engine:junit-jupiter]
Execution Started: helloTest - [engine:junit-jupiter]/[class:helloTest]
Execution Started: testhello() - [engine:junit-jupiter]/[class:helloTest]/[method:testhello()]
Execution Finished: testhello() - [engine:junit-jupiter]/[class:helloTest]/[method:testhello()] - TestExecutionResult [status = FAILED, throwable = org.opentest4j.AssertionFailedError: You should add some testing code for hello here! ==> expected: <1> but was: <2>]
org.opentest4j.AssertionFailedError: You should add some testing code for hello here! ==> expected: <1> but was: <2>
	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
	at<REDACTED FOR BREVITY>

Showing that the test is runnable - but I really would prefer to not have that main method in the helloTest.java

public static void main(final String... args) {
        final LauncherDiscoveryRequest request =
                LauncherDiscoveryRequestBuilder.request()
                        .selectors(selectClass(helloTest.class))
                        .build();
        final Launcher launcher = LauncherFactory.create();
        final LoggingListener logListener = LoggingListener.forBiConsumer((t,m) -> {
            System.out.println(m.get());
            if(t!=null) {
                t.printStackTrace();
            };
        });
        final SummaryGeneratingListener execListener = new SummaryGeneratingListener();
        launcher.registerTestExecutionListeners(execListener, logListener);
        launcher.execute(request);
        execListener.getSummary().printTo(new java.io.PrintWriter(out));
    }

but rather just run using junit console laucher:

jbang app install --name=junit cli@junit-team

and then just run:

junit --scan-classpath=`jbang info classpath helloTest.java`

The jbang info classpath helloTest.java returns the classpath - you can also just get the jar with no dependencies jbang info jar helloTest.java but it seem to make no diff.

Also tried adding them in a package (instead of default package)

But consoleLauncher never locates the helloTest.java class and I haven't been able to locate why it is not found.

Any ideas/suggestions on why they are not found? Whats the magic needed? :)

[maxandersen]

note, one observation is that if I run:

junit execute -cp=`jbang info classpath genTest.java` --scan-class-path

where I basically take the full classpath of genTest.java which includes some of junits own test examples I get:

💚 Thanks for using JUnit! Support its development at https://junit.org/sponsoring

╷
├─ JUnit Platform Suite ✔
├─ JUnit Jupiter ✔
└─ JUnit Vintage ✔
   ├─ IPTest ✔
   │  ├─ IPTest ✔
   │  └─ IPTest ✔
   └─ AllTests ✔
      └─ org.bouncycastle.util.IPTest ✔
         ├─ IPTest ✔
         └─ IPTest ✔

Test run finished after 21 ms
[         6 containers found      ]
[         0 containers skipped    ]
[         6 containers started    ]
[         0 containers aborted    ]
[         6 containers successful ]
[         0 containers failed     ]
[         4 tests found           ]
[         0 tests skipped         ]
[         4 tests started         ]
[         0 tests aborted         ]
[         4 tests successful      ]
[         0 tests failed          ]

So it is able to discover test classes...just not the one I want it to discover ;)

[marcphilipp]

This works for me:

$ junit execute --classpath=`jbang info classpath helloTest.java` --scan-classpath

💚 Thanks for using JUnit! Support its development at https://junit.org/sponsoring

╷
├─ JUnit Platform Suite ✔
├─ JUnit Jupiter ✔
│  └─ helloTest ✔
│     └─ testhello() ✘ You should add some testing code for hello here! ==> expected: <1> but was: <2>
└─ JUnit Vintage ✔

Failures (1):
  JUnit Jupiter:helloTest:testhello()
    MethodSource [className = 'helloTest', methodName = 'testhello', methodParameterTypes = '']
    => org.opentest4j.AssertionFailedError: You should add some testing code for hello here! ==> expected: <1> but was: <2>
       org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
       org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
       org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
       org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:150)
       org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:563)
       helloTest.testhello(helloTest.java:27)
       java.base/java.lang.reflect.Method.invoke(Method.java:580)
       java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
       java.base/java.util.ArrayList.forEach(ArrayList.java:1596)

Test run finished after 40 ms
[         4 containers found      ]
[         0 containers skipped    ]
[         4 containers started    ]
[         0 containers aborted    ]
[         4 containers successful ]
[         0 containers failed     ]
[         1 tests found           ]
[         0 tests skipped         ]
[         1 tests started         ]
[         0 tests aborted         ]
[         0 tests successful      ]
[         1 tests failed          ]

You need to pass --classpath/-cp and --scan-classpath. The former adds roots to the classpath, the second instructs JUnit to scan the entire classpath.

Docs for --scan-classpath state:

Explicit classpath roots that are not on the classpath will be silently ignored.

After making changes to helloTest.java, you first need to run jbang build helloTest.java. Otherwise the helloTest.jar referenced by jbang info classpath helloTest.java does not exist.

[maxandersen]

I swear I tried that and failed to pick up tests.

Away from laptop atm but definitely will try and if that works add some way to force jbang to ensure there is a build

[sormuras]

helloTest.java nor genTest.java match the default name pattern, don't they?

From https://docs.junit.org/current/user-guide/#running-tests-console-launcher-options-discovering-tests

[...]
 FILTERS

  -n, --include-classname=PATTERN
                             Provide a regular expression to include only classes whose fully
                               qualified names match. To avoid loading classes unnecessarily,
                               the default pattern only includes class names that begin with
                               "Test" or end with "Test" or "Tests". When this option is
                               repeated, all patterns will be combined using OR semantics.
                               Default: ^(Test.*|.+[.$]Test.*|.*Tests?)$
[...]

[sormuras]

The jbang info classpath helloTest.java returns the classpath [...]

Ah, okay. Not related, methinks.

[maxandersen]

'helloTest' is matching, right?

[sormuras]

What is the exact output of jbang info classpath genTest.java?

[marcphilipp]

For me, it prints the following:

/home/marc/.jbang/cache/jars/helloTest.java.d7753cc0892c054adfd3d10c644639a362723c617a6bbc18bb87ba19283167f3/helloTest.jar:/home/marc/.m2/repository/org/junit/jupiter/junit-jupiter-api/5.13.3/junit-jupiter-api-5.13.3.jar:/home/marc/.m2/repository/org/opentest4j/opentest4j/1.3.0/opentest4j-1.3.0.jar:/home/marc/.m2/repository/org/junit/platform/junit-platform-commons/1.13.3/junit-platform-commons-1.13.3.jar:/home/marc/.m2/repository/org/apiguardian/apiguardian-api/1.1.2/apiguardian-api-1.1.2.jar:/home/marc/.m2/repository/org/junit/jupiter/junit-jupiter-engine/5.13.3/junit-jupiter-engine-5.13.3.jar:/home/marc/.m2/repository/org/junit/platform/junit-platform-engine/1.13.3/junit-platform-engine-1.13.3.jar:/home/marc/.m2/repository/org/junit/platform/junit-platform-launcher/1.13.3/junit-platform-launcher-1.13.3.jar

/home/marc/.jbang/cache/jars/helloTest.java.d7753cc0892c054adfd3d10c644639a362723c617a6bbc18bb87ba19283167f3/helloTest.jar does not exist, though, so it's filtered out during discovery.

Edit: Never mind, that was because it changed the dependency versions in helloTest.jar to 5.13.3 to rule out misalignment as a potential issue.

[maxandersen]

Also - if some can make a jar with a test in it that get found by console launcher that would be useful - does not need to have jbang involved i can figure out the delta from there :)

[marcphilipp]

I don't quite understand what you're after. Could you please rephrase/elaborate?

[maxandersen]

just trying to state that I didn't need an example made jbang - anything "handmade" would be fine too.

[maxandersen]

also - and this is what I don't understand:

-c genTest or -c test.genTest when trying to use package fails with:

Failures (3):
  JUnit Platform Suite
    => org.junit.platform.launcher.core.DiscoveryIssueException: TestEngine with ID 'junit-platform-suite' encountered a critical issue during test discovery:

(1) [ERROR] ClassSelector [className = 'test.genTest', classLoader = null] resolution failed
    Source: ClassSource [className = 'test.genTest', filePosition = null]
            at test.genTest.<no-method>(SourceFile:0)
    Cause: org.junit.platform.commons.PreconditionViolationException: Could not load class with name: test.genTest

aso its not even able to find it when excplictly requested - which is what i find weird as its on the classpath with a jar that has the classes (as far as I can see)

[maxandersen]

I did and I can reproduce it with the helloTest.java i made but it still fails with example using more dependencies.

I'm wondering if some classloading issues hitting that makes consoleLauncher not find the classes. any logging possible to activate to debug or need a debugger?

[marcphilipp]

If there's a classloading issue, you should get the full stacktrace when selecting the class explicitly (as you've done above) via -c/--select-class. That's usually not very helpful except to verify there's a classloading issue, though. Could you please share your reproducer?

[maxandersen]

I don't know what to say - whatever I tries now it works (and fails) correctly.

junit execute --classpath=`jbang info classpath <jbang-scriptref>` --scan-classpath

where <jbang-scriptref> can be a jar, maven coordinate or a jbang .java/.kt/.groovy script...and it picks up tests from that.

Only explanation I have I somehow must not have had a proper built .jar archive.

Is there a way to get the junit cli to not be "silent" about missing/bad classpath roots? that could save me for a lot of anguish

[marcphilipp]

There isn't currently but I think we should at least add logging. Could you please open an issue?

[maxandersen]

done #4772

[sormuras]

I'm wondering if some classloading issues hitting that makes consoleLauncher not find the classes. any logging possible to activate to debug or need a debugger?

You wrote that it works with a main method within the test class, right? I guess, that's due to the main (and therefore the test class) is put on the application class loader of the outer java process. The --class-path feature of the console launcher uses a custom URL class loader to enable loading classes (and finding tests there).

What happens, if you move all the --class-path entries from the junit to the java launcher?

[sormuras]

In addition, the junit console launcher changes to use an internal class loader when --class-path is given. See ConsoleTestExecutor.java#L162-L171 and CustomContextClassLoaderExecutor.java for details.

Note, that JUnit'S launcher class loading relies on context class loader of the thread itself its executed on. Is jbang also relying on the Thread.currentThread().setContextClassLoader(...)?

[sormuras]

The API documentation of ClasspathRootSelector has:

A DiscoverySelector that selects a classpath root so that TestEngines can search for class files or resources within the physical classpath — for example, to scan for test classes.
Since engines are not expected to modify the classpath, the classpath root represented by this selector must be on the classpath of the context class loader of the thread that uses this selector.

[maxandersen]

@sormuras jbang does not load any classes - it just generate a java command to run.

[sormuras]

Understood.

What happens, if you move all the --class-path entries from the junit to the java launcher?

❓

[maxandersen]

see #4764 (reply in thread) ...i'm stomped why i for days couldn't get it to work - I'll assume its my own incompetence - thanks for showing me the way :)

[maxandersen]

fyi, made this PR to make this clearer and have a cleaner junit jbang catalog: junit-team/jbang-catalog#1

[maxandersen]

ok so related to this I hit this I still don't grok:

Given:

///usr/bin/env jbang "$0" "$@" ; exit $?
package wonka;

//DEPS org.junit.jupiter:junit-jupiter-engine:5.12.2
//DEPS org.junit.platform:junit-platform-console:1.12.2
//DEPS org.assertj:assertj-core:3.27.3


import static org.assertj.core.api.Assertions.assertThat;

import org.junit.jupiter.api.Test;
import org.junit.platform.console.ConsoleLauncher;

public class consoleTests {

    // Define each Unit test here and run them separately in the IDE
    @Test
    public void testPass() {
        assertThat(2+2).isEqualTo(4);
    }   

    @Test
    public void testFail() {
        assertThat(2+2).isEqualTo(42);
    }   

    // Run all Unit tests with JBang with ./calculator.java
    public static void main(final String... args) {
        // this finds 2 tests as expected
        // commented out because it otherwise calls system.exit(0)
        //ConsoleLauncher.main( "discover", "--select-class", "consoleTests");

        // package discovery works too
        //ConsoleLauncher.main( "discover", "--select-package", "wonka");

        // this finds 0 tests - why?
        ConsoleLauncher.main( "discover", "--scan-class-path");
    }
}

why does the explicit selection work but scan class path does not?

• the naming pattern matches (given package scan is working)
• the test is obviously findable with select-class.

why is scan-class-path not finding it?

[marcphilipp]

As documented in junit discover --help, the --scan-classpath option only scans directories, not jar files:

--scan-classpath, --scan-class-path[=PATH]
   Scan all directories on the classpath or explicit classpath roots. Without arguments, only directories on
     the system classpath as well as additional classpath entries supplied via -cp (directories and JAR
     files) are scanned. Explicit classpath roots that are not on the classpath will be silently ignored.
     This option can be repeated.

This works but is rather verbose:

ConsoleLauncher.main( "discover", "--scan-class-path", consoleTests.class.getProtectionDomain().getCodeSource().getLocation().toString())