Skip to content

Commit ff5dd2e

Browse files
Dwight Guthrv-jenkins
andauthored
Improve test harness and test coverage for proof hints (#1046)
Previously the tests for proof hints were rather anemic, not to mention somewhat non-standard. We make the following changes in order to canonify a technique for testing proof hints, and also expand the test coverage of this code path: 1. We remove the kore-proof-hint-test binary which appears to be written with the intent of being used in one specific integration test. This seems to be duplicated logic with the test_proof_hint.py python tests, and I don't see the need to maintain a specific C++ binary just for one test. We also remove the test associated with this binary since it appears to be redundant. 2. We clean up the infrastructure for how we test proof hints. We no longer test proof hints without passing --expand-terms, since this exposes information that is not particularly meaningful and makes the tests flaky, like the number of bytes in each kore term in the trace, which may be subject to change. 3. We codify a set of techniques for testing proof hints. Any test which uses `%interpreter` and `%check-diff` can be extended by adding the lines `%proof-interpreter` and `%check-proof-out` to the exact same kore definition. This allows us to reuse kore files and input files and test the same tests both with and without the proof hint. 4. We add the proof hint testing harness to a small selection of more complex tests in the tests directory. If regressions occur in the future, we can extend the number of tests we do this with, and we can selectively choose to enable this test coverage on new integration tests that we create very easily. We do not enable this on all tests because it does not scale well to long execution traces due to the fact that this would require very large expected output files to be stored in source control. --------- Co-authored-by: rv-jenkins <[email protected]>
1 parent 07652ca commit ff5dd2e

21 files changed

+104478
-17304
lines changed

bin/llvm-kompile-clang

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -256,7 +256,7 @@ if $link; then
256256
mkdir -p "$python_path"
257257

258258
script_name="$(basename "$output_file").py"
259-
cp "$LIBDIR/lldb/klldb.py" "$python_path/$script_name"
259+
cp -f "$LIBDIR/lldb/klldb.py" "$python_path/$script_name"
260260
fi
261261
fi
262262
fi

test/defn/imp-slow-proof.kore renamed to test/defn/imp-sum-slow.kore

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,5 @@
11
// RUN: %proof-slow-interpreter
22
// RUN: %check-proof-out
3-
// RUN: %check-expanded-proof-out
43
[topCellInitializer{}(LblinitGeneratedTopCell{}()), org'Stop'kframework'Stop'attributes'Stop'Source{}("Source(/home/robertorosmaninho/rv/k/llvm-backend/src/main/native/llvm-backend/test/defn/k-files/imp.md)")]
54

65
module BASIC-K

test/defn/imp-proof.kore renamed to test/defn/imp-sum.kore

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,5 @@
11
// RUN: %proof-interpreter
22
// RUN: %check-proof-out
3-
// RUN: %check-expanded-proof-out
43
[topCellInitializer{}(LblinitGeneratedTopCell{}()), org'Stop'kframework'Stop'attributes'Stop'Source{}("Source(/home/robertorosmaninho/rv/k/llvm-backend/src/main/native/llvm-backend/test/defn/k-files/imp.md)")]
54

65
module BASIC-K

test/defn/imp.kore

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
// RUN: %interpreter
22
// RUN: %check-grep
3+
// RUN: %proof-interpreter
4+
// RUN: %check-proof-out
35
[topCellInitializer{}(LblinitGeneratedTopCell{}()), org'Stop'kframework'Stop'attributes'Stop'Source{}("Source(/home/robertorosmaninho/rv/k/llvm-backend/src/main/native/llvm-backend/test/defn/k-files/imp.md)")]
46

57
module BASIC-K

test/defn/kool-static.kore

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
// RUN: %interpreter
22
// RUN: %check-diff
3+
// RUN: %proof-interpreter
4+
// RUN: %check-proof-out
35
[topCellInitializer{}(LblinitGeneratedTopCell{}()), org'Stop'kframework'Stop'attributes'Stop'Source{}("Source(/home/dwightguth/kframework-5.0.0/k-distribution/pl-tutorial/2_languages/2_kool/2_typed/2_static/kool-typed-static.md)")]
46

57
module BASIC-K

test/defn/test-proof-trace-slow.kore

Lines changed: 0 additions & 1400 deletions
This file was deleted.

test/defn/test-proof-trace.kore

Lines changed: 0 additions & 1400 deletions
This file was deleted.
File renamed without changes.
File renamed without changes.

test/lit.cfg.py

Lines changed: 2 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -151,17 +151,7 @@ def one_line(s):
151151

152152
('%check-proof-out', one_line('''
153153
%run-proof-out
154-
%kore-proof-trace --verbose %t.out.bin | diff - %test-proof-diff-out
155-
result="$?"
156-
if [ "$result" -ne 0 ]; then
157-
echo "kore-proof-trace error while parsing proof hint trace"
158-
exit 1
159-
fi
160-
''')),
161-
162-
('%check-expanded-proof-out', one_line('''
163-
%run-proof-out
164-
%kore-proof-trace --verbose --expand-terms %t.out.bin | diff - %test-proof-expanded-diff-out
154+
%kore-proof-trace --verbose --expand-terms %t.out.bin | diff - %test-proof-diff-out
165155
result="$?"
166156
if [ "$result" -ne 0 ]; then
167157
echo "kore-proof-trace error while parsing proof hint trace with expanded kore terms"
@@ -185,16 +175,14 @@ def one_line(s):
185175
('%test-diff-out', os.path.join('%output-dir', '%test-basename.out.diff')),
186176
('%test-dir-out', os.path.join('%output-dir', '%test-basename')),
187177
('%test-dir-in', os.path.join('%input-dir', '%test-basename')),
188-
('%test-proof-diff-out', os.path.join('%output-dir', '%test-basename', '%test-basename.out.diff')),
189-
('%test-proof-expanded-diff-out', os.path.join('%output-dir', '%test-basename', '%test-basename.expanded.out.diff')),
178+
('%test-proof-diff-out', os.path.join('%output-dir', '%test-basename.proof.out.diff')),
190179
('%test-basename', '`basename %s .kore`'),
191180

192181
('%allow-pipefail', 'set +o pipefail'),
193182

194183
('%kore-convert', 'kore-convert'),
195184

196185
('%kore-proof-trace', 'kore-proof-trace'),
197-
('%kore-proof-trace-test', 'kore-proof-trace-test'),
198186
])
199187

200188
config.recursiveExpansionLimit = 10

0 commit comments

Comments
 (0)