Skip to content

Commit aeb9ca5

Browse files
authored
Merge PR #39 from tudasc/devel
For new TypeART 1.5 release tag, see #39 for gist of changes
2 parents 2ac6605 + 146fbaf commit aeb9ca5

File tree

162 files changed

+7378
-3497
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

162 files changed

+7378
-3497
lines changed

.github/workflows/basic-ci.yml

Lines changed: 36 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -7,64 +7,74 @@ on:
77

88
jobs:
99
build-and-run-test:
10-
runs-on: ubuntu-18.04
10+
runs-on: ubuntu-20.04
1111
if: "!contains(github.event.head_commit.message, '[ci skip]')"
1212
steps:
1313
- uses: actions/checkout@v2
1414

15-
- name: Setup LLVM repository
16-
run: |
17-
wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key 2>/dev/null | sudo apt-key add -
18-
sudo add-apt-repository 'deb http://apt.llvm.org/bionic/ llvm-toolchain-bionic-10 main' -y
19-
sudo apt-get update -q
20-
2115
- name: Install LLVM
2216
run: sudo apt-get install libllvm10 llvm-10 llvm-10-dev
2317

2418
- name: Install Clang
25-
run: sudo apt-get install clang-10
19+
run: sudo apt-get install clang-10 clang-tidy-10
2620

2721
- name: Install OpenMPI
2822
run: sudo apt-get install libopenmpi-dev openmpi-bin
2923

24+
- name: Install lcov
25+
run: sudo apt-get install lcov
26+
3027
- name: Setup env
3128
run: |
3229
sudo ln -f -s /usr/bin/clang-10 /usr/bin/clang
3330
sudo ln -f -s /usr/bin/clang++-10 /usr/bin/clang++
3431
sudo ln -f -s /usr/bin/opt-10 /usr/bin/opt
3532
sudo ln -f -s /usr/bin/FileCheck-10 /usr/bin/FileCheck
3633
sudo ln -f -s /usr/bin/llc-10 /usr/bin/llc
37-
echo "::set-env name=CC::clang-10"
38-
echo "::set-env name=CXX::clang++-10"
39-
echo "::set-env name=EXTERNAL_LIT::/usr/lib/llvm-10/build/utils/lit/lit.py"
34+
sudo ln -f -s /usr/bin/clang-tidy-10 /usr/bin/clang-tidy
35+
echo "CC=clang-10" >> $GITHUB_ENV
36+
echo "CXX=clang++-10" >> $GITHUB_ENV
37+
echo "EXTERNAL_LIT=/usr/lib/llvm-10/build/utils/lit/lit.py" >> $GITHUB_ENV
4038
4139
- name: Build TypeART
4240
run: |
43-
cmake -B build -DTEST_CONFIG=ON -DLLVM_EXTERNAL_LIT=${EXTERNAL_LIT}
41+
cmake -B build -DTEST_CONFIG=ON -DENABLE_CODE_COVERAGE=ON -DLLVM_EXTERNAL_LIT=${EXTERNAL_LIT}
4442
cmake --build build --parallel
4543
46-
- name: Test TypeART
47-
run: cmake --build build --target test -- ARGS=-VV
44+
- name: Test TypeART with coverage (exludes lulesh)
45+
run: |
46+
cmake --build build --target lcov-clean
47+
cmake --build build --target test -- ARGS=-VV
48+
49+
- name: Build coverage report
50+
run: cmake --build build --target lcov-html
4851

49-
- name: Build TypeART Release
52+
- name: Build TypeART release
5053
run: |
5154
cmake -B build_lulesh -DCMAKE_BUILD_TYPE=Release -DMPI_INTERCEPT_LIB=ON -DSHOW_STATS=ON
5255
cmake --build build_lulesh --parallel
5356
54-
- name: Test TypeART on lulesh
57+
- name: Test TypeART release on lulesh
5558
working-directory: build_lulesh
5659
run: ctest -V -R lulesh -O lulesh2.0_build.log
57-
58-
- name: Prepare lulesh artifact
60+
61+
- name: Prepare artifact
5962
run: |
60-
mkdir artifact_lulesh
61-
mv build_lulesh/lulesh2.0_build.log artifact_lulesh/
62-
mv test/lulesh/lulesh2.0_out.log artifact_lulesh/
63-
mv test/lulesh/types.yaml artifact_lulesh/lulesh2.0_types.yaml
63+
mkdir -p artifact/lulesh
64+
mkdir -p artifact/coverage
65+
mv build_lulesh/lulesh2.0_build.log artifact/lulesh/
66+
mv test/lulesh/lulesh2.0_out.log artifact/lulesh/
67+
mv test/lulesh/types.yaml artifact/lulesh/lulesh2.0_types.yaml
68+
mv build/profiles/ artifact/coverage
6469
6570
- name: Upload lulesh test artifact
66-
uses: actions/upload-artifact@v1
71+
uses: actions/upload-artifact@v2
72+
with:
73+
name: typeart-ci
74+
path: artifact
75+
76+
- name: Coveralls
77+
uses: coverallsapp/github-action@master
6778
with:
68-
name: typeart-lulesh-archive
69-
path: artifact_lulesh
70-
79+
github-token: ${{ secrets.GITHUB_TOKEN }}
80+
path-to-lcov: build/typeart.coverage

.github/workflows/ext-ci.yml

Lines changed: 33 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -2,13 +2,12 @@ name: TypeART-CI-ext
22

33
on:
44
push:
5-
branches:
6-
- master
5+
branches: [ master, devel ]
76
pull_request:
87

98
jobs:
109
build-and-run-testbench:
11-
runs-on: ubuntu-18.04
10+
runs-on: ubuntu-20.04
1211
if: "!contains(github.event.head_commit.message, '[ci skip]') || !contains(github.event.head_commit.message, '[ci ext skip]')"
1312
steps:
1413
- uses: actions/checkout@v2
@@ -20,17 +19,19 @@ jobs:
2019
ssh-key: ${{ secrets.AUTH_SSH_CI_EXT }}
2120
path: test-bench
2221

23-
- name: Setup LLVM repository
24-
run: |
25-
wget -O - https://apt.llvm.org/llvm-snapshot.gpg.key 2>/dev/null | sudo apt-key add -
26-
sudo add-apt-repository 'deb http://apt.llvm.org/bionic/ llvm-toolchain-bionic-10 main' -y
27-
sudo apt-get update -q
22+
- name: Checkout AD test-bench
23+
uses: actions/checkout@v2
24+
with:
25+
repository: ahueck/typeart-ad-benchmarks
26+
ssh-key: ${{ secrets.AUTH_SSH_CI_EXT_AD }}
27+
ref: feat/ci
28+
path: ad-test-bench
2829

2930
- name: Install LLVM
3031
run: sudo apt-get install libllvm10 llvm-10 llvm-10-dev
3132

3233
- name: Install Clang
33-
run: sudo apt-get install clang-10
34+
run: sudo apt-get install clang-10 clang-tidy-10
3435

3536
- name: Install OpenMPI
3637
run: sudo apt-get install libopenmpi-dev openmpi-bin
@@ -41,14 +42,16 @@ jobs:
4142
sudo ln -f -s /usr/bin/clang++-10 /usr/bin/clang++
4243
sudo ln -f -s /usr/bin/opt-10 /usr/bin/opt
4344
sudo ln -f -s /usr/bin/llc-10 /usr/bin/llc
44-
echo "::set-env name=CC::clang-10"
45-
echo "::set-env name=CXX::clang++-10"
45+
sudo ln -f -s /usr/bin/clang-tidy-10 /usr/bin/clang-tidy
46+
echo "CC=clang-10" >> $GITHUB_ENV
47+
echo "CXX=clang++-10" >> $GITHUB_ENV
48+
echo "EXTERNAL_LIT=/usr/lib/llvm-10/build/utils/lit/lit.py" >> $GITHUB_ENV
4649
4750
- name: Build & install TypeART
4851
run: |
4952
cmake -B build -DCMAKE_BUILD_TYPE=Release -DMPI_INTERCEPT_LIB=ON -DSHOW_STATS=ON
5053
cmake --build build --parallel --target install
51-
echo "::set-env name=TYPEART_PATH::${GITHUB_WORKSPACE}/install/typeart"
54+
echo "TYPEART_PATH=${GITHUB_WORKSPACE}/install/typeart" >> $GITHUB_ENV
5255
5356
- name: Setup tests
5457
working-directory: test-bench
@@ -70,16 +73,27 @@ jobs:
7073
working-directory: test-bench/build
7174
run: ctest -V -R amg2013 -O amg2013_build.log
7275

73-
- name: Prepare test-bench artifact
74-
working-directory: test-bench
76+
- name: Setup AD tests
77+
working-directory: ad-test-bench
78+
run: cmake -B build -DLOG_PATH=${GITHUB_WORKSPACE}/ad-test-bench/artifact
79+
80+
- name: Run AD lulesh
81+
working-directory: ad-test-bench/build
82+
run: ctest -V -R lulesh -O ad-lulesh2.0_build.log
83+
84+
- name: Prepare artifact
7585
run: |
76-
mkdir -p artifact
77-
mv build/*_build.log artifact
86+
mkdir -p artifact/bench
87+
mkdir -p artifact/ad-bench
88+
mv test-bench/build/*_build.log artifact/bench
89+
mv test-bench/artifact/* artifact/bench
90+
mv ad-test-bench/build/*_build.log artifact/ad-bench
91+
mv ad-test-bench/artifact/* artifact/ad-bench
7892
7993
- name: Upload test-bench artifact
80-
uses: actions/upload-artifact@v1
94+
uses: actions/upload-artifact@v2
8195
with:
82-
name: typeart-bench-archive
83-
path: test-bench/artifact
96+
name: typeart-ci-ext
97+
path: artifact
8498

8599

CMakeLists.txt

Lines changed: 14 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,11 @@
1-
cmake_minimum_required(VERSION 3.12)
1+
cmake_minimum_required(VERSION 3.14)
22

33
project(typeart
4-
VERSION 2.0
4+
VERSION 1.5.0
55
)
66

7+
set(TYPEART_BINARY_DIR ${CMAKE_CURRENT_BINARY_DIR})
8+
79
set(CMAKE_EXPORT_COMPILE_COMMANDS ON)
810
set(CMAKE_VERBOSE_MAKEFILE ON)
911

@@ -20,18 +22,19 @@ include(CMakePackageConfigHelpers)
2022

2123
add_format_target(format-sources
2224
"Formats project source files"
23-
TARGETS lib/*.cpp
24-
lib/*.h
25-
runtime/*.cpp
26-
runtime/*.h
27-
typelib/*.cpp
28-
typelib/*.h
29-
)
25+
TARGETS
26+
lib/passes/*.cpp
27+
lib/passes/*.h
28+
lib/runtime/*.cpp
29+
lib/runtime/*.h
30+
lib/typelib/*.cpp
31+
lib/typelib/*.h
32+
lib/support/*.cpp
33+
lib/support/*.h
34+
)
3035

3136
add_subdirectory(externals)
3237
add_subdirectory(lib)
33-
add_subdirectory(typelib)
34-
add_subdirectory(runtime)
3538
add_subdirectory(scripts)
3639
enable_testing()
3740
add_subdirectory(test)

README.md

Lines changed: 66 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,19 @@
1-
# TypeART   [![License](https://img.shields.io/badge/License-BSD%203--Clause-blue.svg)](https://opensource.org/licenses/BSD-3-Clause)   ![](https://github.com/tudasc/TypeART/workflows/TypeART-CI/badge.svg?branch=master)   ![](https://github.com/tudasc/TypeART/workflows/TypeART-CI-ext/badge.svg?branch=master)
1+
# TypeART · [![License](https://img.shields.io/badge/License-BSD%203--Clause-blue.svg)](https://opensource.org/licenses/BSD-3-Clause) ![](https://github.com/tudasc/TypeART/workflows/TypeART-CI/badge.svg?branch=master) ![](https://github.com/tudasc/TypeART/workflows/TypeART-CI-ext/badge.svg?branch=master) [![Coverage Status](https://coveralls.io/repos/github/tudasc/TypeART/badge.svg?branch=master)](https://coveralls.io/github/tudasc/TypeART)
22

3-
TypeART \[[TA18](#ref-typeart-2018)\] is a type and memory allocation tracking sanitizer.
4-
It consists of an LLVM compiler pass and a corresponding runtime to track relevant memory allocation information during the execution of a target program.
5-
It instruments heap, stack and global variable allocations with a callback to our runtime.
6-
The callback consists of the runtime memory pointer value, what type (built-ins, user-defined structs etc.) and extent of the value.
7-
This allows users of our runtime to query detailed type information behind arbritary memory locations, as long as they are mapped.
3+
TypeART \[[TA18](#ref-typeart-2018); [TA20](#ref-typeart-2020)\] is a type and memory allocation tracking sanitizer. It
4+
consists of an LLVM compiler pass and a corresponding runtime to track relevant memory allocation information during the
5+
execution of a target program. It instruments heap, stack and global variable allocations with a callback to our
6+
runtime. The callback consists of the runtime memory pointer value, what type (built-ins, user-defined structs etc.) and
7+
extent of the value. This allows users of our runtime to query detailed type information behind arbitrary memory
8+
locations, as long as they are mapped.
89

910
### Use Case: MUST - A dynamic MPI correctness checker
1011

11-
TypeART is used in conjunction with MUST \[[MU13](#ref-must-2013)\] to track memory (de-)allocation relevant to MPI communication.
12-
Thus, MUST can check for type compatibility between the type-less communication buffer and the declared MPI datatype at all phases of the MPI communication, namely message assembly, message transfer and message disassembly into the receiving buffer.
13-
A brief summary is given in a subsequent section and more information can be found in our publication:
12+
TypeART is used in conjunction with MUST \[[MU13](#ref-must-2013)\] to track memory (de-)allocation relevant to MPI
13+
communication. Thus, MUST can check for type compatibility between the type-less communication buffer and the declared
14+
MPI datatype at all phases of the MPI communication, namely message assembly, message transfer and message disassembly
15+
into the receiving buffer. A brief summary is given in a subsequent section and more information can be found in our
16+
publication:
1417

1518
#### References
1619

@@ -23,6 +26,14 @@ A brief summary is given in a subsequent section and more information can be fou
2326
In <i>2nd International Workshop on Software Correctness for HPC Applications (Correctness)</i>,
2427
pages 51–58. IEEE, 2018.</td>
2528
</tr>
29+
<tr>
30+
<td valign="top"><a name="ref-typeart-2020"></a>[TA20]</td>
31+
<td>Hück, Alexander and Protze, Joachim and Lehr, Jan-Patrick and Terboven, Christian and Bischof, Christian and Müller, Matthias S.
32+
<a href=https://doi.org/10.1109/Correctness51934.2020.00010>
33+
Towards compiler-aided correctness checking of adjoint MPI applications</a>.
34+
In <i>4th International Workshop on Software Correctness for HPC Applications (Correctness)</i>,
35+
pages 40–48. IEEE/ACM, 2020.</td>
36+
</tr>
2637
<tr>
2738
<td valign="top"><a name="ref-must-2013"></a>[MU13]</td>
2839
<td>Hilbrich, Tobias and Protze, Joachim and Schulz, Martin and de Supinski, Bronis R. and Müller, Matthias S.
@@ -35,12 +46,14 @@ A brief summary is given in a subsequent section and more information can be fou
3546

3647
## Software dependencies
3748

38-
TypeART requires [LLVM](https://llvm.org) version 10 and CMake version >= 3.12.
49+
TypeART requires [LLVM](https://llvm.org) version 10 and CMake version >= 3.14.
3950

4051
#### Building TypeART
4152

4253
TypeART uses CMake to build, cf. [GitHub CI build file](.github/workflows/basic-ci.yml) for a complete recipe to build.
43-
Example build recipe (debug build, installs to default prefix)
54+
Example build recipe (debug build, installs to default prefix
55+
`${typeart_SOURCE_DIR}/install/typeart`)
56+
4457
```{.sh}
4558
$> git clone https://github.com/tudasc/TypeART
4659
$> cd TypeART
@@ -50,54 +63,78 @@ $> cmake --build build --target install --parallel
5063

5164
#### CMake Configuration: Options for users
5265

66+
##### Runtime
67+
5368
- `SOFTCOUNTERS` (default: **off**) : Enable runtime tracking of #tracked addrs. / #distinct checks / etc.
54-
- `USE_BTREE` (default: **on**) : Enable usage of btree-backed map instead of std::map for the runtime, typically resulting in higher performance.
69+
- `USE_ABSL` (default: **on**) : Enable usage of btree-backed map of the abseil project instead of std::map for the
70+
runtime.
71+
- `USE_BTREE` (default: **off**) : Enable usage of a btree-backed map (alternative to abseil) instead of std::map for
72+
the runtime.
73+
74+
##### Logging and Passes
75+
76+
- `SHOW_STATS` (default: **on**) : Passes show the statistics w.r.t. allocations etc.
77+
- `MPI_LOGGER` (default: **on**) : Enable better logging support in MPI execution context
78+
- `MPI_INTERCEPT_LIB` (default: **on**) : Library can be used by preloading to intercept MPI calls and check whether
79+
TypeART tracks the buffer pointer
80+
- `LOG_LEVEL_` and `LOG_LEVEL_RT` (default **0**) : Granularity of logger. 3 ist most verbose, 0 is least.
81+
82+
##### Testing
83+
84+
- `TEST_CONFIG` (default: **off**) : Set (force) logging levels to appropriate levels for test runner to succeed
85+
- `ENABLE_CODE_COVERAGE` (default: **off**) : Enable code coverage statistics using LCOV 1.14 and genhtml (gcovr
86+
optional)
87+
- `ENABLE_LLVM_CODE_COVERAGE` (default: **off**) : Enable llvm-cov code coverage statistics (llvm-cov and llvm-profdata
88+
required)
5589

5690
## Using TypeART
5791

58-
Making use of TypeART consists of two phases:
92+
Making use of TypeART consists of two phases:
5993

60-
1. Compile and instrument the target code with our LLVM passes, and,
61-
2. execute the target program with a runtime library (based on the TypeART runtime) to accept the callbacks from the instrumented code and actually do some useful analysis.
94+
1. Compile and instrument the target code with our LLVM passes, and,
95+
2. execute the target program with a runtime library (based on the TypeART runtime) to accept the callbacks from the
96+
instrumented code and actually do some useful analysis.
6297

63-
To that end, the interface [RuntimeInterface.h](runtime/RuntimeInterface.h) can be used to query type information during the target code execution.
98+
To that end, the interface [RuntimeInterface.h](runtime/RuntimeInterface.h) can be used to query type information during
99+
the target code execution.
64100

65101
#### Example: MPI Demo
66102

67-
The folder [demo](demo) contains an example of MPI related type errors that can be detected using TypeART.
68-
The code is compiled with our instrumentation, and executed by preloading the MPI related check library implemented in [tool.c](demo/tool.c), which is linked against the TypeART runtime and uses the aforementioned query interface.
69-
It overloads the required MPI calls and checks that the passed `void* buffer` is correct.
103+
The folder [demo](demo) contains an example of MPI related type errors that can be detected using TypeART. The code is
104+
compiled with our instrumentation, and executed by preloading the MPI related check library implemented
105+
in [tool.c](demo/tool.c), which is linked against the TypeART runtime and uses the aforementioned query interface. It
106+
overloads the required MPI calls and checks that the passed `void* buffer` is correct.
70107

71108
## LLVM pass
72109

73-
The necessary allocation sites and type information are extracted in LLVM passes.
74-
TypeART analyzes:
110+
The necessary allocation sites and type information are extracted in LLVM passes. TypeART analyzes:
75111

76112
- Calls to ```malloc``` and ```free``` to keep track of active pointers referring to objects allocated on the heap,
77-
- relevant stack space allocations, i.e., allocations that cannot be proven to never lead to ```MPI``` functions,
113+
- relevant stack space allocations, i.e., allocations that cannot be proven to never lead to ```MPI``` functions,
78114
- built-in as well as user-defined types to retrieve type size and the size of the allocation, e.g., for arrays.
79115

80-
The type information is necessary to correlate the type of the buffer passed to an MPI call with the MPI datatype the user declared.
81-
In this prototype we restrict ourselves to:
116+
The type information is necessary to correlate the type of the buffer passed to an MPI call with the MPI datatype the
117+
user declared. In this prototype we restrict ourselves to:
82118

83119
+ primitive types (int, float, long, double, char, unsigned int, unsigned long)
84120
+ arrays of primitive types
85121
+ structs which contain only primitive types
86122

87123
### Example of Instrumentation: Handling malloc
88124

89-
To instrument relevant allocations and extract the necessary type information, the LLVM pass searches for specific patterns, e.g., how calls to ```malloc``` look like in LLVM IR.
90-
Calls to the ```malloc``` function are typically call instructions followed by a ```bitcast``` instruction to cast the returned pointer to the desired type.
125+
To instrument relevant allocations and extract the necessary type information, the LLVM pass searches for specific
126+
patterns, e.g., how calls to ```malloc``` look like in LLVM IR. Calls to the ```malloc``` function are typically call
127+
instructions followed by a ```bitcast``` instruction to cast the returned pointer to the desired type.
91128

92129
~~~{.ll}
93130
; %0 == n * sizeof(float)
94131
%1 = tail call i8* @malloc(i64 %0)
95132
%2 = bitcast i8* %1 to float *
96133
~~~
97134

98-
The patterns has all the information we require for our instrumentation.
99-
Our transformation first detects the type that the returned pointer is casted to, then it computes the extent of the allocation.
100-
The information is passed to our instrumentation function.
135+
The patterns has all the information we require for our instrumentation. Our transformation first detects the type that
136+
the returned pointer is casted to, then it computes the extent of the allocation. The information is passed to our
137+
instrumentation function.
101138

102139
~~~{.ll}
103140
; %0 == n * sizeof(float)

0 commit comments

Comments
 (0)