Skip to content

Conversation

@wadoon
Copy link
Member

@wadoon wadoon commented Jan 19, 2024

Intended Change

Make test case generation usable again, especially from the CLI in the new verification template.

The test case generation was not used for a long time. It was buggy and unstable, and could only be controlled by the UI. This PR tries to revive it a little bit:

Changes

  • Add a CLI + Facade for easy access from the CLI or to integrate it into build tools, e.g., Gradle or Maven.
  • Get rid of the old project layout, using shell scripts and manually set paths to JAR files.
    The new layout uses the Maven folder layout, and provides a build.gradle.kts and a pom.xml for building and testing
  • Add the test case from the book.
  • Robust generation of Java program code using javapoet. Nobody understands pages of StringBuilder foo.
  • Using picocli for the Command Line interface, also for the existing interfaces in key.ui, keyext.proofmanagement, ...
  • Clean up of old Java (introduction of records, sealed hierarchies, NPE checking, ...)

This is rather an MVP, more options and features can be thought of but first get it running.

Type of pull request

  • X Refactoring (behaviour should not change or only minimally change)
  • X New feature (non-breaking change which adds functionality)
  • X Breaking change (fix or feature that would cause existing functionality to change)
  • X There are changes to the (Java) code
  • X Renovations

@wadoon wadoon added 🐞 Bug Test Case Generator 🛠 Maintenance Code quality and related things w/o functional changes labels Jan 19, 2024
@wadoon wadoon added this to the v2.14.0 milestone Jan 19, 2024
@wadoon wadoon self-assigned this Jan 19, 2024
@wadoon wadoon force-pushed the weigl/testgen branch 2 times, most recently from 87f9f32 to 25a0200 Compare February 2, 2024 15:52
@wadoon wadoon force-pushed the weigl/testgen branch 2 times, most recently from 7abaaf2 to ddab633 Compare February 18, 2024 19:12
@KeYProject KeYProject deleted a comment from codecov bot May 23, 2024
@sonarqubecloud
Copy link

@KeYProject KeYProject deleted a comment from codecov bot Jul 20, 2024
@wadoon wadoon force-pushed the weigl/testgen branch 3 times, most recently from 2cedebc to 21573b2 Compare June 8, 2025 14:18
@wadoon wadoon force-pushed the weigl/testgen branch 3 times, most recently from 64bb3c5 to 82a27ce Compare July 14, 2025 00:37
@wadoon wadoon marked this pull request as ready for review July 14, 2025 00:37
@wadoon
Copy link
Member Author

wadoon commented Jul 14, 2025

@unp1 @mattulbrich Whoever has more experience with the hell of testgen.

@wadoon wadoon requested a review from unp1 July 14, 2025 00:38
@unp1
Copy link
Member

unp1 commented Sep 19, 2025

addition

One first observatuion "./gradlew :key.ui:run" terminates without starting KeY

same for using the shadowJar very brief window shown and it terminates then with

SLF4J(W): Class path contains multiple SLF4J providers.
SLF4J(W): Found provider [ch.qos.logback.classic.spi.LogbackServiceProvider@5d5eef3d]
SLF4J(W): Found provider [org.slf4j.simple.SimpleServiceProvider@56f4468b]
SLF4J(W): See https://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J(I): Actual provider is of type [ch.qos.logback.classic.spi.LogbackServiceProvider@5d5eef3d]

Copy link
Member

@unp1 unp1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I approve the PR once the problem with starting the version is resolved and I tested it on some small examples. Thanks a lot for the work.

@wadoon
Copy link
Member Author

wadoon commented Sep 20, 2025

SLF4J(W): Class path contains multiple SLF4J providers. SLF4J(W): Found provider [ch.qos.logback.classic.spi.LogbackServiceProvider@5d5eef3d] SLF4J(W): Found provider [org.slf4j.simple.SimpleServiceProvider@56f4468b] SLF4J(W): See https://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J(I): Actual provider is of type [ch.qos.logback.classic.spi.LogbackServiceProvider@5d5eef3d]

I have not updated to the recent main. There, it should be fixed by #3639 which was introduced by Isabelle deps.

I check it after rebase.

UPDATE: Not appeared after rebase.

@wadoon wadoon force-pushed the weigl/testgen branch 3 times, most recently from 4370fdf to 6028a2e Compare September 20, 2025 21:53
@wadoon
Copy link
Member Author

wadoon commented Sep 20, 2025

I approve the PR once the problem with starting the version is resolved and I tested it on some small examples. Thanks a lot for the work.

Starting is resolved: Calling System.exit at the end of main does not work b/c the Swing instantiation is non-blocking.

For fostering the command line, the exit code is a good indicator of failure. We should reconsider the Main logic to respect this at some point.

@unp1
Copy link
Member

unp1 commented Sep 22, 2025

Thanks. I will complete the review later today

@unp1
Copy link
Member

unp1 commented Sep 22, 2025

Thanks I can open the UI now!
The following observations:

  • claims to have written test but cannot be found
Screenshot 2025-09-22 at 13 43 18 * selecting a folder where to store not possible via the magnifying class as folder cannot be selected and saved (only opened) * The result was different when not starting any proof and instead clicking the checkbox "Apply symbolic execution rules"

not implying that this PR should solve all these issues (in particular 3 might have been not working since longer and we can start the prover separately)

@unp1
Copy link
Member

unp1 commented Sep 22, 2025

Worked on CMdLine and I could now also find the directory. It was in my home directory (so it seems the GUI info is ignored?)

Maybe we should place it in the directory where it is called by default?

Anyways, I think the issues were present before and now at least the generation can be demoed an dI'll approve the change.

But please check if the mentioned issues should be dhave been resolved or if they fall under known issues.

Thanks!

Copy link
Member

@unp1 unp1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Makes it demoable but see my previous comment about known issues

@wadoon
Copy link
Member Author

wadoon commented Sep 22, 2025

  • claims to have written test but cannot be found

The test case from the book is here:
https://github.com/KeYProject/key/pull/3388/files#diff-3e5bc1e4d3a7fbee0d470dcd7d0b03e6f272fdd0c5ab1c8c8641023c7fd578e6R27

I tried to get binary search working, but was not so successful.

@wadoon
Copy link
Member Author

wadoon commented Sep 22, 2025

Thanks I can open the UI now! The following observations:

I avoided to go into the GUI at all. My user story would rather be the generation from inside of the project management / command line.

Clear is Objenesis and JML are resolved using Maven Central via pom.xml/build.xml/build.gradle.

I am also not sure, how the user story should be for the UI. Because, after the generation you need to leave KeY and go to the terminal.

wadoon and others added 5 commits September 25, 2025 21:35
# Conflicts:
#	key.core.testgen/src/main/java/de/uka/ilkd/key/testgen/TestCaseGenerator.java
#	key.core.testgen/src/test/java/de/uka/ilkd/key/testcase/smt/ce/TestCE.java
@wadoon
Copy link
Member Author

wadoon commented Sep 25, 2025

Rebased for better history. GUI issues are tracked in #3665

@wadoon wadoon added this pull request to the merge queue Sep 25, 2025
Merged via the queue into main with commit 6505edd Sep 25, 2025
35 checks passed
@wadoon wadoon deleted the weigl/testgen branch September 25, 2025 21:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Exciting New Feature 🐞 Bug 🛠 Maintenance Code quality and related things w/o functional changes Test Case Generator

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants