@@ -30,9 +30,9 @@ LLVM Testing Infrastructure Organization
30
30
========================================
31
31
32
32
The LLVM testing infrastructure contains three major categories of tests:
33
- unit tests, regression tests and whole programs. The unit tests and regression
33
+ unit tests, regression tests, and whole programs. The unit tests and regression
34
34
tests are contained inside the LLVM repository itself under ``llvm/unittests ``
35
- and ``llvm/test `` respectively and are expected to always pass -- they should be
35
+ and ``llvm/test `` respectively and are expected to always pass. They should be
36
36
run before every commit.
37
37
38
38
The whole programs tests are referred to as the "LLVM test suite" (or
@@ -48,7 +48,7 @@ Unit tests
48
48
Unit tests are written using `Google Test <https://github.com/google/googletest/blob/master/docs/primer.md >`_
49
49
and `Google Mock <https://github.com/google/googletest/blob/master/docs/gmock_for_dummies.md >`_
50
50
and are located in the ``llvm/unittests `` directory.
51
- In general unit tests are reserved for targeting the support library and other
51
+ In general, unit tests are reserved for targeting the support library and other
52
52
generic data structure, we prefer relying on regression tests for testing
53
53
transformations and analysis on the IR.
54
54
@@ -61,7 +61,7 @@ written in depends on the part of LLVM being tested. These tests are driven by
61
61
the :doc: `Lit <CommandGuide/lit >` testing tool (which is part of LLVM), and
62
62
are located in the ``llvm/test `` directory.
63
63
64
- Typically when a bug is found in LLVM, a regression test containing just
64
+ Typically, when a bug is found in LLVM, a regression test containing just
65
65
enough code to reproduce the problem should be written and placed
66
66
somewhere underneath this directory. For example, it can be a small
67
67
piece of LLVM IR distilled from an actual application or benchmark.
@@ -82,10 +82,10 @@ for an example of such test.
82
82
83
83
The test suite contains whole programs, which are pieces of code which
84
84
can be compiled and linked into a stand-alone program that can be
85
- executed. These programs are generally written in high level languages
86
- such as C or C++.
85
+ executed. These programs are generally written in high- level languages,
86
+ such as C and C++.
87
87
88
- These programs are compiled using a user specified compiler and set of
88
+ These programs are compiled using a user- specified compiler and set of
89
89
flags, and then executed to capture the program output and timing
90
90
information. The output of these programs is compared to a reference
91
91
output to ensure that the program is being compiled correctly.
@@ -103,11 +103,11 @@ See the :doc:`TestSuiteGuide` for details.
103
103
Debugging Information tests
104
104
---------------------------
105
105
106
- The test suite contains tests to check quality of debugging information.
107
- The test are written in C based languages or in LLVM assembly language.
106
+ The test suite contains tests to check the quality of debugging information.
107
+ The tests are written in C based languages or in LLVM assembly language.
108
108
109
109
These tests are compiled and run under a debugger. The debugger output
110
- is checked to validate of debugging information. See README.txt in the
110
+ is checked to validate the debugging information. See `` README.txt `` in the
111
111
test suite for more information. This test suite is located in the
112
112
``cross-project-tests/debuginfo-tests `` directory.
113
113
@@ -126,13 +126,13 @@ and C++ programs. See the :doc:`TestSuiteGuide` for details.
126
126
Unit and Regression tests
127
127
-------------------------
128
128
129
- To run all of the LLVM unit tests use the check-llvm-unit target:
129
+ To run all of the LLVM unit tests, use the `` check-llvm-unit `` target:
130
130
131
131
.. code-block :: bash
132
132
133
133
% make check-llvm-unit
134
134
135
- To run all of the LLVM regression tests use the check-llvm target:
135
+ To run all of the LLVM regression tests, use the `` check-llvm `` target:
136
136
137
137
.. code-block :: bash
138
138
@@ -163,7 +163,7 @@ to enable testing with valgrind and with leak checking enabled.
163
163
164
164
To run individual tests or subsets of tests, you can use the ``llvm-lit ``
165
165
script which is built as part of LLVM. For example, to run the
166
- ``Integer/BitPacked.ll `` test by itself you can run:
166
+ ``Integer/BitPacked.ll `` test by itself, you can run:
167
167
168
168
.. code-block :: bash
169
169
@@ -224,55 +224,55 @@ only directories does not need the ``lit.local.cfg`` file. Read the :doc:`Lit
224
224
documentation <CommandGuide/lit>` for more information.
225
225
226
226
Each test file must contain lines starting with "RUN:" that tell :program: `lit `
227
- how to run it. If there are no RUN lines, :program: `lit ` will issue an error
227
+ how to run it. If there are no `` RUN `` lines, :program: `lit ` will issue an error
228
228
while running a test.
229
229
230
- RUN lines are specified in the comments of the test program using the
230
+ `` RUN `` lines are specified in the comments of the test program using the
231
231
keyword ``RUN `` followed by a colon, and lastly the command (pipeline)
232
232
to execute. Together, these lines form the "script" that :program: `lit `
233
- executes to run the test case. The syntax of the RUN lines is similar to a
233
+ executes to run the test case. The syntax of the `` RUN `` lines is similar to a
234
234
shell's syntax for pipelines including I/O redirection and variable
235
235
substitution. However, even though these lines may *look * like a shell
236
- script, they are not. RUN lines are interpreted by :program: `lit `.
236
+ script, they are not. `` RUN `` lines are interpreted by :program: `lit `.
237
237
Consequently, the syntax differs from shell in a few ways. You can specify
238
- as many RUN lines as needed.
238
+ as many `` RUN `` lines as needed.
239
239
240
- :program: `lit ` performs substitution on each RUN line to replace LLVM tool names
240
+ :program: `lit ` performs substitution on each `` RUN `` line to replace LLVM tool names
241
241
with the full paths to the executable built for each tool (in
242
242
``$(LLVM_OBJ_ROOT)/bin ``). This ensures that :program: `lit ` does
243
243
not invoke any stray LLVM tools in the user's path during testing.
244
244
245
- Each RUN line is executed on its own, distinct from other lines unless
246
- its last character is ``\ ``. This continuation character causes the RUN
247
- line to be concatenated with the next one. In this way you can build up
245
+ Each `` RUN `` line is executed on its own, distinct from other lines unless
246
+ its last character is ``\ ``. This continuation character causes the `` RUN ``
247
+ line to be concatenated with the next one. In this way, you can build up
248
248
long pipelines of commands without making huge line lengths. The lines
249
- ending in ``\ `` are concatenated until a RUN line that doesn't end in
250
- ``\ `` is found. This concatenated set of RUN lines then constitutes one
249
+ ending in ``\ `` are concatenated until a `` RUN `` line that doesn't end in
250
+ ``\ `` is found. This concatenated set of `` RUN `` lines then constitutes one
251
251
execution. :program: `lit ` will substitute variables and arrange for the pipeline
252
252
to be executed. If any process in the pipeline fails, the entire line (and
253
253
test case) fails too.
254
254
255
- Below is an example of legal RUN lines in a ``.ll `` file:
255
+ Below is an example of legal `` RUN `` lines in a ``.ll `` file:
256
256
257
257
.. code-block :: llvm
258
258
259
259
; RUN: llvm-as < %s | llvm-dis > %t1
260
260
; RUN: llvm-dis < %s.bc-13 > %t2
261
261
; RUN: diff %t1 %t2
262
262
263
- As with a Unix shell, the RUN lines permit pipelines and I/O
263
+ As with a Unix shell, the `` RUN `` lines permit pipelines and I/O
264
264
redirection to be used.
265
265
266
266
There are some quoting rules that you must pay attention to when writing
267
- your RUN lines. In general nothing needs to be quoted. :program: `lit ` won't
268
- strip off any quote characters so they will get passed to the invoked program.
267
+ your `` RUN `` lines. In general, nothing needs to be quoted. :program: `lit ` won't
268
+ strip off any quote characters, so they will get passed to the invoked program.
269
269
To avoid this use curly braces to tell :program: `lit ` that it should treat
270
270
everything enclosed as one value.
271
271
272
- In general, you should strive to keep your RUN lines as simple as possible,
272
+ In general, you should strive to keep your `` RUN `` lines as simple as possible,
273
273
using them only to run tools that generate textual output you can then examine.
274
274
The recommended way to examine output to figure out if the test passes is using
275
- the :doc: `FileCheck tool <CommandGuide/FileCheck >`. *[The usage of grep in RUN
275
+ the :doc: `FileCheck tool <CommandGuide/FileCheck >`. *[The usage of grep in `` RUN``
276
276
lines is deprecated - please do not send or commit patches that use it.] *
277
277
278
278
Put related tests into a single file rather than having a separate file per
@@ -283,11 +283,11 @@ Generating assertions in regression tests
283
283
-----------------------------------------
284
284
285
285
Some regression test cases are very large and complex to write/update by hand.
286
- In that case to reduce the human work we can use the scripts available in
287
- llvm/utils/ to generate the assertions.
286
+ In that case, to reduce the manual work, we can use the scripts available in
287
+ `` llvm/utils/ `` to generate the assertions.
288
288
289
- For example to generate assertions in an :program: `llc `-based test, after
290
- adding one or more RUN lines use:
289
+ For example, to generate assertions in an :program: `llc `-based test, after
290
+ adding one or more `` RUN `` lines, use:
291
291
292
292
.. code-block :: bash
293
293
@@ -368,7 +368,7 @@ Best practices for regression tests
368
368
Extra files
369
369
-----------
370
370
371
- If your test requires extra files besides the file containing the ``RUN: `` lines
371
+ If your test requires extra files besides the file containing the ``RUN: `` lines,
372
372
and the extra files are small, consider specifying them in the same file and
373
373
using ``split-file `` to extract them. For example,
374
374
@@ -442,7 +442,7 @@ Elaborated tests
442
442
443
443
Generally, IR and assembly test files benefit from being cleaned to remove
444
444
unnecessary details. However, for tests requiring elaborate IR or assembly
445
- files where cleanup is less practical (e.g., large amount of debug information
445
+ files where cleanup is less practical (e.g., a large amount of debug information
446
446
output from Clang), you can include generation instructions within
447
447
``split-file `` part called ``gen ``. Then, run
448
448
``llvm/utils/update_test_body.py `` on the test file to generate the needed
@@ -472,7 +472,7 @@ then rewrite the part after ``gen`` with its stdout.
472
472
473
473
For convenience, if the test needs one single assembly file, you can also wrap
474
474
``gen `` and its required files with ``.ifdef `` and ``.endif ``. Then you can
475
- skip ``split-file `` in RUN lines.
475
+ skip ``split-file `` in `` RUN `` lines.
476
476
477
477
.. code-block :: none
478
478
@@ -521,7 +521,7 @@ utilize ``split-file`` in ``RUN`` lines.
521
521
Fragile tests
522
522
-------------
523
523
524
- It is easy to write a fragile test that would fail spuriously if the tool being
524
+ It is easy to write a fragile test that could fail spuriously if the tool being
525
525
tested outputs a full path to the input file. For example, :program: `opt ` by
526
526
default outputs a ``ModuleID ``:
527
527
@@ -552,29 +552,29 @@ default outputs a ``ModuleID``:
552
552
553
553
This test will fail if placed into a ``download `` directory.
554
554
555
- To make your tests robust, always use ``opt ... < %s `` in the RUN line.
555
+ To make your tests robust, always use ``opt ... < %s `` in the `` RUN `` line.
556
556
:program: `opt ` does not output a ``ModuleID `` when input comes from stdin.
557
557
558
558
Platform-Specific Tests
559
559
-----------------------
560
560
561
561
Whenever adding tests that require the knowledge of a specific platform,
562
562
either related to code generated, specific output or back-end features,
563
- you must make sure to isolate the features, so that buildbots that
563
+ you must isolate the features, so that buildbots that
564
564
run on different architectures (and don't even compile all back-ends),
565
565
don't fail.
566
566
567
567
The first problem is to check for target-specific output, for example sizes
568
568
of structures, paths and architecture names, for example:
569
569
570
- * Tests containing Windows paths will fail on Linux and vice- versa.
570
+ * Tests containing Windows paths will fail on Linux and vice versa.
571
571
* Tests that check for ``x86_64 `` somewhere in the text will fail anywhere else.
572
572
* Tests where the debug information calculates the size of types and structures.
573
573
574
- Also, if the test rely on any behaviour that is coded in any back-end, it must
574
+ Also, if the test relies on any behaviour that is coded in any back-end, it must
575
575
go in its own directory. So, for instance, code generator tests for ARM go
576
576
into ``test/CodeGen/ARM `` and so on. Those directories contain a special
577
- ``lit `` configuration file that ensure all tests in that directory will
577
+ ``lit `` configuration file that ensures all tests in that directory will
578
578
only run if a specific back-end is compiled and available.
579
579
580
580
For instance, on ``test/CodeGen/ARM ``, the ``lit.local.cfg `` is:
@@ -622,7 +622,7 @@ with debug builds or on particular platforms. Use ``REQUIRES``
622
622
and ``UNSUPPORTED `` to control when the test is enabled.
623
623
624
624
Some tests are expected to fail. For example, there may be a known bug
625
- that the test detect . Use ``XFAIL `` to mark a test as an expected failure.
625
+ that the test detects . Use ``XFAIL `` to mark a test as an expected failure.
626
626
An ``XFAIL `` test will be successful if its execution fails, and
627
627
will be a failure if its execution succeeds.
628
628
@@ -645,7 +645,7 @@ list of boolean expressions. The values in each expression may be:
645
645
expressions can appear inside an identifier, so for example ``he{{l+}}o `` would match
646
646
``helo ``, ``hello ``, ``helllo ``, and so on.
647
647
- The default target triple, preceded by the string ``target= `` (for example,
648
- ``target=x86_64-pc-windows-msvc ``). Typically regular expressions are used
648
+ ``target=x86_64-pc-windows-msvc ``). Typically, regular expressions are used
649
649
to match parts of the triple (for example, ``target={{.*}}-windows{{.*}} ``
650
650
to match any Windows target triple).
651
651
@@ -684,7 +684,7 @@ have different effects. ``UNSUPPORTED`` causes the test to be skipped;
684
684
this saves execution time, but then you'll never know whether the test
685
685
actually would start working. Conversely, ``XFAIL `` actually runs the test
686
686
but expects a failure output, taking extra execution time but alerting you
687
- if/when the test begins to behave correctly (an XPASS test result). You
687
+ if/when the test begins to behave correctly (an `` XPASS `` test result). You
688
688
need to decide which is more appropriate in each case.
689
689
690
690
**Using ``target=...`` **
@@ -698,7 +698,7 @@ and it's generally a good idea to use a trailing wildcard to allow for
698
698
unexpected suffixes.
699
699
700
700
Also, it's generally better to write regular expressions that use entire
701
- triple components, than to do something clever to shorten them. For
701
+ triple components than to do something clever to shorten them. For
702
702
example, to match both freebsd and netbsd in an expression, you could write
703
703
``target={{.*(free|net)bsd.*}} `` and that would work. However, it would
704
704
prevent a ``grep freebsd `` from finding this test. Better to use:
@@ -708,8 +708,8 @@ prevent a ``grep freebsd`` from finding this test. Better to use:
708
708
Substitutions
709
709
-------------
710
710
711
- Besides replacing LLVM tool names the following substitutions are performed in
712
- RUN lines:
711
+ Besides replacing LLVM tool names, the following substitutions are performed in
712
+ `` RUN `` lines:
713
713
714
714
``%% ``
715
715
Replaced by a single ``% ``. This allows escaping other substitutions.
@@ -726,7 +726,7 @@ RUN lines:
726
726
Example: ``/home/user/llvm/test/MC/ELF ``
727
727
728
728
``%t ``
729
- File path to a temporary file name that could be used for this test case.
729
+ File path to a temporary file name that can be used for this test case.
730
730
The file name won't conflict with other test cases. You can append to it
731
731
if you need multiple temporaries. This is useful as the destination of
732
732
some redirected output.
@@ -811,7 +811,7 @@ RUN lines:
811
811
optional integer offset. These expand only if they appear
812
812
immediately in ``RUN: ``, ``DEFINE: ``, and ``REDEFINE: `` directives.
813
813
Occurrences in substitutions defined elsewhere are never expanded.
814
- For example, this can be used in tests with multiple RUN lines,
814
+ For example, this can be used in tests with multiple `` RUN `` lines,
815
815
which reference the test file's line numbers.
816
816
817
817
**LLVM-specific substitutions: **
@@ -988,7 +988,7 @@ directives:
988
988
- **Substitution value **: The value includes all text from the first
989
989
non-whitespace character after ``= `` to the last non-whitespace character. If
990
990
there is no non-whitespace character after ``= ``, the value is the empty
991
- string. Escape sequences that can appear in python ``re.sub `` replacement
991
+ string. Escape sequences that can appear in Python ``re.sub `` replacement
992
992
strings are treated as plain text in the value.
993
993
- **Line continuations **: If the last non-whitespace character on the line after
994
994
``: `` is ``\ ``, then the next directive must use the same directive keyword
@@ -1057,7 +1057,7 @@ producing incorrect output.
1057
1057
Options
1058
1058
-------
1059
1059
1060
- The llvm lit configuration allows to customize some things with user options:
1060
+ The llvm lit configuration allows some things to be customized with user options:
1061
1061
1062
1062
``llc ``, ``opt ``, ...
1063
1063
Substitute the respective llvm tool name with a custom command line. This
@@ -1076,8 +1076,8 @@ The llvm lit configuration allows to customize some things with user options:
1076
1076
Other Features
1077
1077
--------------
1078
1078
1079
- To make RUN line writing easier, there are several helper programs. These
1080
- helpers are in the PATH when running tests, so you can just call them using
1079
+ To make `` RUN `` line writing easier, several helper programs are available . These
1080
+ helpers are in the `` PATH `` when running tests, so you can just call them using
1081
1081
their name. For example:
1082
1082
1083
1083
``not ``
0 commit comments