-
Notifications
You must be signed in to change notification settings - Fork 14.9k
[Utils] Add support for split-file to diff_test_updater #157765
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,37 +1,135 @@ | ||
import shutil | ||
import os | ||
|
||
""" | ||
This file provides the `diff_test_updater` function, which is invoked on failed RUN lines when lit is executed with --update-tests. | ||
It checks whether the failed command is `diff` and, if so, uses heuristics to determine which file is the checked-in reference file and which file is output from the test case. | ||
The heuristics are currently as follows: | ||
- if exactly one file originates from the `split-file` command, that file is the reference file and the other is the output file | ||
- if exactly one file ends with ".expected" (common pattern in LLVM), that file is the reference file and the other is the output file | ||
- if exactly one file path contains ".tmp" (e.g. because it contains the expansion of "%t"), that file is the reference file and the other is the output file | ||
If the command matches one of these patterns the output file content is copied to the reference file to make the test pass. | ||
If the reference file originated in `split-file`, the output file content is instead copied to the corresponding slice of the test file. | ||
Otherwise the test is ignored. | ||
|
||
Possible improvements: | ||
- Support stdin patterns like "my_binary %s | diff expected.txt" | ||
- Scan RUN lines to see if a file is the source of output from a previous command. | ||
- Scan RUN lines to see if a file is the source of output from a previous command (other than `split-file`). | ||
If it is then it is not a reference file that can be copied to, regardless of name, since the test will overwrite it anyways. | ||
- Only update the parts that need updating (based on the diff output). Could help avoid noisy updates when e.g. whitespace changes are ignored. | ||
""" | ||
|
||
|
||
def get_source_and_target(a, b): | ||
class NormalFileTarget: | ||
def __init__(self, target): | ||
self.target = target | ||
|
||
def copyFrom(self, source): | ||
shutil.copy(source, self.target) | ||
|
||
def __str__(self): | ||
return self.target | ||
|
||
|
||
class SplitFileTarget: | ||
def __init__(self, slice_start_idx, test_path, lines): | ||
self.slice_start_idx = slice_start_idx | ||
self.test_path = test_path | ||
self.lines = lines | ||
|
||
def copyFrom(self, source): | ||
lines_before = self.lines[: self.slice_start_idx + 1] | ||
self.lines = self.lines[self.slice_start_idx + 1 :] | ||
slice_end_idx = None | ||
for i, l in enumerate(self.lines): | ||
if SplitFileTarget._get_split_line_path(l) != None: | ||
slice_end_idx = i | ||
break | ||
if slice_end_idx is not None: | ||
lines_after = self.lines[slice_end_idx:] | ||
else: | ||
lines_after = [] | ||
with open(source, "r") as f: | ||
new_lines = lines_before + f.readlines() + lines_after | ||
with open(self.test_path, "w") as f: | ||
for l in new_lines: | ||
f.write(l) | ||
|
||
def __str__(self): | ||
return f"slice in {self.test_path}" | ||
|
||
@staticmethod | ||
def get_target_dir(commands, test_path): | ||
for cmd in commands: | ||
split = cmd.split(" ") | ||
if "split-file" not in split: | ||
continue | ||
start_idx = split.index("split-file") | ||
split = split[start_idx:] | ||
if len(split) < 3: | ||
continue | ||
if split[1].strip() != test_path: | ||
continue | ||
return split[2].strip() | ||
return None | ||
|
||
@staticmethod | ||
def create(path, commands, test_path, target_dir): | ||
filename = path.replace(target_dir, "") | ||
if filename.startswith(os.sep): | ||
filename = filename[len(os.sep) :] | ||
with open(test_path, "r") as f: | ||
lines = f.readlines() | ||
for i, l in enumerate(lines): | ||
p = SplitFileTarget._get_split_line_path(l) | ||
if p == filename: | ||
idx = i | ||
break | ||
else: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. for-else is so cool. love it. |
||
return None | ||
return SplitFileTarget(idx, test_path, lines) | ||
|
||
@staticmethod | ||
def _get_split_line_path(l): | ||
if len(l) < 6: | ||
return None | ||
if l.startswith("//"): | ||
l = l[2:] | ||
else: | ||
l = l[1:] | ||
if l.startswith("--- "): | ||
l = l[4:] | ||
else: | ||
return None | ||
return l.rstrip() | ||
|
||
|
||
def get_source_and_target(a, b, test_path, commands): | ||
""" | ||
Try to figure out which file is the test output and which is the reference. | ||
""" | ||
split_target_dir = SplitFileTarget.get_target_dir(commands, test_path) | ||
if split_target_dir: | ||
a_target = SplitFileTarget.create(a, commands, test_path, split_target_dir) | ||
b_target = SplitFileTarget.create(b, commands, test_path, split_target_dir) | ||
if a_target and b_target: | ||
return None | ||
if a_target: | ||
return b, a_target | ||
if b_target: | ||
return a, b_target | ||
|
||
expected_suffix = ".expected" | ||
if a.endswith(expected_suffix) and not b.endswith(expected_suffix): | ||
return b, a | ||
return b, NormalFileTarget(a) | ||
if b.endswith(expected_suffix) and not a.endswith(expected_suffix): | ||
return a, b | ||
return a, NormalFileTarget(b) | ||
|
||
tmp_substr = ".tmp" | ||
if tmp_substr in a and not tmp_substr in b: | ||
return a, b | ||
return a, NormalFileTarget(b) | ||
if tmp_substr in b and not tmp_substr in a: | ||
return b, a | ||
return b, NormalFileTarget(a) | ||
|
||
return None | ||
|
||
|
@@ -40,16 +138,16 @@ def filter_flags(args): | |
return [arg for arg in args if not arg.startswith("-")] | ||
|
||
|
||
def diff_test_updater(result, test): | ||
def diff_test_updater(result, test, commands): | ||
args = filter_flags(result.command.args) | ||
if len(args) != 3: | ||
return None | ||
[cmd, a, b] = args | ||
if cmd != "diff": | ||
return None | ||
res = get_source_and_target(a, b) | ||
res = get_source_and_target(a, b, test.getFilePath(), commands) | ||
if not res: | ||
return f"update-diff-test: could not deduce source and target from {a} and {b}" | ||
source, target = res | ||
shutil.copy(source, target) | ||
target.copyFrom(source) | ||
return f"update-diff-test: copied {source} to {target}" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,2 +1,9 @@ | ||
; diff-tmp-dir.test clobbers this file | ||
empty.txt | ||
; these test cases are clobbered when run, so they're recreated each time | ||
single-split-file.test | ||
single-split-file-populated.test | ||
multiple-split-file.test | ||
multiple-split-file-populated.test | ||
single-split-file-no-expected.test | ||
split-c-comments.test |
17 changes: 17 additions & 0 deletions
17
llvm/utils/lit/tests/Inputs/diff-test-update/multiple-split-file-populated.in
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,17 @@ | ||
# RUN: split-file %s %t | ||
# RUN: cp %S/1.in %t/out.txt | ||
# RUN: diff %t/test3.expected %t/out.txt | ||
|
||
#--- test1.expected | ||
unrelated | ||
#--- test2.expected | ||
#--- test3.expected | ||
BAR | ||
|
||
BAZ | ||
|
||
#--- test4.expected | ||
filler | ||
#--- test5.expected | ||
|
||
|
13 changes: 13 additions & 0 deletions
13
llvm/utils/lit/tests/Inputs/diff-test-update/multiple-split-file.in
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,13 @@ | ||
# RUN: split-file %s %t | ||
# RUN: cp %S/1.in %t/out.txt | ||
# RUN: diff %t/test3.expected %t/out.txt | ||
|
||
#--- test1.expected | ||
unrelated | ||
#--- test2.expected | ||
#--- test3.expected | ||
#--- test4.expected | ||
filler | ||
#--- test5.expected | ||
|
||
|
14 changes: 14 additions & 0 deletions
14
llvm/utils/lit/tests/Inputs/diff-test-update/multiple-split-file.out
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
# RUN: split-file %s %t | ||
# RUN: cp %S/1.in %t/out.txt | ||
# RUN: diff %t/test3.expected %t/out.txt | ||
|
||
#--- test1.expected | ||
unrelated | ||
#--- test2.expected | ||
#--- test3.expected | ||
FOO | ||
#--- test4.expected | ||
filler | ||
#--- test5.expected | ||
|
||
|
6 changes: 6 additions & 0 deletions
6
llvm/utils/lit/tests/Inputs/diff-test-update/single-split-file-no-expected.in
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
# RUN: split-file %s %t | ||
# RUN: cp %S/1.in %t/out.txt | ||
# RUN: diff %t/test.txt %t/out.txt | ||
|
||
#--- test.txt | ||
|
6 changes: 6 additions & 0 deletions
6
llvm/utils/lit/tests/Inputs/diff-test-update/single-split-file-no-expected.out
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
# RUN: split-file %s %t | ||
# RUN: cp %S/1.in %t/out.txt | ||
# RUN: diff %t/test.txt %t/out.txt | ||
|
||
#--- test.txt | ||
FOO |
7 changes: 7 additions & 0 deletions
7
llvm/utils/lit/tests/Inputs/diff-test-update/single-split-file-populated.in
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
# RUN: split-file %s %t | ||
# RUN: cp %S/1.in %t/out.txt | ||
# RUN: diff %t/test.expected %t/out.txt | ||
|
||
#--- test.expected | ||
BAR | ||
|
5 changes: 5 additions & 0 deletions
5
llvm/utils/lit/tests/Inputs/diff-test-update/single-split-file.in
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
# RUN: split-file %s %t | ||
# RUN: cp %S/1.in %t/out.txt | ||
# RUN: diff %t/test.expected %t/out.txt | ||
|
||
#--- test.expected |
6 changes: 6 additions & 0 deletions
6
llvm/utils/lit/tests/Inputs/diff-test-update/single-split-file.out
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
# RUN: split-file %s %t | ||
# RUN: cp %S/1.in %t/out.txt | ||
# RUN: diff %t/test.expected %t/out.txt | ||
|
||
#--- test.expected | ||
FOO |
11 changes: 11 additions & 0 deletions
11
llvm/utils/lit/tests/Inputs/diff-test-update/split-both.test
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,11 @@ | ||
# RUN: split-file %s %t | ||
# RUN: diff %t/split-both.expected %t/split-both.out | ||
|
||
# ignore the fact that it's called ".expected" | ||
# when comparing two files originating in split-file | ||
|
||
#--- split-both.expected | ||
FOO | ||
#--- split-both.out | ||
BAR | ||
|
6 changes: 6 additions & 0 deletions
6
llvm/utils/lit/tests/Inputs/diff-test-update/split-c-comments.in
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
// RUN: split-file %s %t | ||
// RUN: cp %S/1.in %t/out.txt | ||
// RUN: diff %t/test.txt %t/out.txt | ||
// | ||
//--- test.txt | ||
|
6 changes: 6 additions & 0 deletions
6
llvm/utils/lit/tests/Inputs/diff-test-update/split-c-comments.out
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
// RUN: split-file %s %t | ||
// RUN: cp %S/1.in %t/out.txt | ||
// RUN: diff %t/test.txt %t/out.txt | ||
// | ||
//--- test.txt | ||
FOO |
11 changes: 11 additions & 0 deletions
11
llvm/utils/lit/tests/Inputs/diff-test-update/unrelated-split.test
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,11 @@ | ||
# the fact that this test runs split-file is unrelated | ||
# to the diffed files | ||
|
||
# RUN: mkdir %t | ||
# RUN: split-file %s %t | ||
# RUN: cp %S/1.in %t/unrelated-split.expected | ||
# RUN: cp %S/2.in %t/unrelated-split.txt | ||
# RUN: diff %t/unrelated-split.expected %t/unrelated-split.txt | ||
|
||
#--- distraction.txt | ||
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,2 +1,2 @@ | ||
def should_not_run(foo, bar): | ||
def should_not_run(foo, bar, baz): | ||
raise Exception("this test updater should only run on failure") |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,10 +1,27 @@ | ||
# RUN: cp %S/Inputs/diff-test-update/single-split-file.in %S/Inputs/diff-test-update/single-split-file.test | ||
# RUN: cp %S/Inputs/diff-test-update/single-split-file-populated.in %S/Inputs/diff-test-update/single-split-file-populated.test | ||
# RUN: cp %S/Inputs/diff-test-update/multiple-split-file.in %S/Inputs/diff-test-update/multiple-split-file.test | ||
# RUN: cp %S/Inputs/diff-test-update/multiple-split-file-populated.in %S/Inputs/diff-test-update/multiple-split-file-populated.test | ||
# RUN: cp %S/Inputs/diff-test-update/single-split-file-no-expected.in %S/Inputs/diff-test-update/single-split-file-no-expected.test | ||
# RUN: cp %S/Inputs/diff-test-update/split-c-comments.in %S/Inputs/diff-test-update/split-c-comments.test | ||
|
||
# RUN: not %{lit} --update-tests -v %S/Inputs/diff-test-update | FileCheck %s | ||
|
||
# RUN: diff %S/Inputs/diff-test-update/single-split-file.out %S/Inputs/diff-test-update/single-split-file.test | ||
# RUN: diff %S/Inputs/diff-test-update/single-split-file.out %S/Inputs/diff-test-update/single-split-file-populated.test | ||
# RUN: diff %S/Inputs/diff-test-update/multiple-split-file.out %S/Inputs/diff-test-update/multiple-split-file.test | ||
# RUN: diff %S/Inputs/diff-test-update/multiple-split-file.out %S/Inputs/diff-test-update/multiple-split-file-populated.test | ||
# RUN: diff %S/Inputs/diff-test-update/single-split-file-no-expected.out %S/Inputs/diff-test-update/single-split-file-no-expected.test | ||
# RUN: diff %S/Inputs/diff-test-update/split-c-comments.out %S/Inputs/diff-test-update/split-c-comments.test | ||
|
||
|
||
# CHECK: # update-diff-test: could not deduce source and target from {{.*}}1.in and {{.*}}2.in | ||
# CHECK: # update-diff-test: could not deduce source and target from {{.*}}1.txt and {{.*}}2.txt | ||
# CHECK: # update-diff-test: copied {{.*}}my-file.txt to {{.*}}my-file.expected | ||
# CHECK: # update-diff-test: copied {{.*}}1.txt to {{.*}}empty.txt | ||
# CHECK: # update-diff-test: copied {{.*}}diff-tmp.test.tmp.txt to {{.*}}diff-t-out.txt | ||
# CHECK: # update-diff-test: could not deduce source and target from {{.*}}split-both.expected and {{.*}}split-both.out | ||
# CHECK: # update-diff-test: copied {{.*}}unrelated-split.txt to {{.*}}unrelated-split.expected | ||
|
||
|
||
# CHECK: Failed: 5 (100.00%) | ||
# CHECK: Failed: 13 (100.00%) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.