Skip to content

Commit 36dc126

Browse files
wyliwyli
authored
meta tensor basic profiling (#4223)
* Meta tensor basic profiling Signed-off-by: Wenqi Li <[email protected]> * adds cprofile results Signed-off-by: Wenqi Li <[email protected]>
1 parent 3583ea0 commit 36dc126

File tree

6 files changed

+216
-0
lines changed

6 files changed

+216
-0
lines changed

‎.gitignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -131,10 +131,14 @@ tests/testing_data/MedNIST*
131131
tests/testing_data/*Hippocampus*
132132
tests/testing_data/*.tiff
133133
tests/testing_data/schema.json
134+
*.svg
134135

135136
# clang format tool
136137
.clang-format-bin/
137138

138139
# VSCode
139140
.vscode/
140141
*.zip
142+
143+
# profiling results
144+
*.prof

‎tests/profile_subclass/README.md‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
# Profiling the performance of subclassing/`__torch_function__` in MONAI
2+
3+
## Requirements
4+
```bash
5+
pip install py-spy
6+
pip install snakeviz # for viewing the cProfile results
7+
```
8+
9+
## Commands
10+
11+
### Install MONAI
12+
```
13+
./runtests.sh --build # from monai's root directory
14+
```
15+
or follow the installation guide (https://docs.monai.io/en/latest/installation.html)
16+
17+
### Profiling the task of adding two MetaTensors
18+
```bash
19+
python profiling.py
20+
```
21+
22+
### Profiling using `py-spy`
23+
```bash
24+
py-spy record -o Tensor.svg -- python pyspy_profiling.py Tensor
25+
py-spy record -o SubTensor.svg -- python pyspy_profiling.py SubTensor
26+
py-spy record -o SubWithTorchFunc.svg -- python pyspy_profiling.py SubWithTorchFunc
27+
py-spy record -o MetaTensor.svg -- python pyspy_profiling.py MetaTensor
28+
```
29+
30+
### Profiling using `cProfile` and `SNAKEVIZ`
31+
32+
```bash
33+
python cprofile_profiling.py
34+
snakeviz out_200.prof
35+
```
36+
37+
---
38+
These tests are based on the following code:
39+
https://github.com/pytorch/pytorch/tree/v1.11.0/benchmarks/overrides_benchmark
40+
41+
- Overhead for torch functions when run on `torch.Tensor` objects is on the order of 2 microseconds.
42+
- `__torch_function__` should add zero overhead for `torch.Tensor` inputs, a small overhead for subclasses of `torch.Tensor`, and an order of microseconds for `MeatTensor`.
43+
- Changing the dispatching mechanism may result in changes that are on the order of 100 ns, which are hard to detect due to noise, but important.

‎tests/profile_subclass/cprofile_profiling.py‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
# Copyright (c) MONAI Consortium
2+
# Licensed under the Apache License, Version 2.0 (the "License");
3+
# you may not use this file except in compliance with the License.
4+
# You may obtain a copy of the License at
5+
# http://www.apache.org/licenses/LICENSE-2.0
6+
# Unless required by applicable law or agreed to in writing, software
7+
# distributed under the License is distributed on an "AS IS" BASIS,
8+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
9+
# See the License for the specific language governing permissions and
10+
# limitations under the License.
11+
12+
"""
13+
Profiling MetaTensor
14+
"""
15+
16+
import cProfile
17+
18+
import torch
19+
20+
from monai.data.meta_tensor import MetaTensor
21+
22+
if __name__ == "__main__":
23+
n_chan = 3
24+
for hwd in (10, 200):
25+
shape = (n_chan, hwd, hwd, hwd)
26+
a = MetaTensor(torch.rand(shape), meta={"affine": torch.eye(4) * 2, "fname": "something1"})
27+
b = MetaTensor(torch.rand(shape), meta={"affine": torch.eye(4) * 3, "fname": "something2"})
28+
cProfile.run("c = a + b", filename=f"out_{hwd}.prof")

‎tests/profile_subclass/min_classes.py‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
# Copyright (c) MONAI Consortium
2+
# Licensed under the Apache License, Version 2.0 (the "License");
3+
# you may not use this file except in compliance with the License.
4+
# You may obtain a copy of the License at
5+
# http://www.apache.org/licenses/LICENSE-2.0
6+
# Unless required by applicable law or agreed to in writing, software
7+
# distributed under the License is distributed on an "AS IS" BASIS,
8+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
9+
# See the License for the specific language governing permissions and
10+
# limitations under the License.
11+
12+
13+
"""
14+
Minimal subclassing as baselines
15+
Adapted from https://github.com/pytorch/pytorch/tree/v1.11.0/benchmarks/overrides_benchmark
16+
"""
17+
18+
import torch
19+
20+
__all__ = ["SubTensor", "SubWithTorchFunc"]
21+
22+
23+
class SubTensor(torch.Tensor):
24+
pass
25+
26+
27+
class SubWithTorchFunc(torch.Tensor):
28+
def __torch_function__(self, func, types, args=(), kwargs=None):
29+
return super().__torch_function__(func, types, args, {} if kwargs is None else kwargs)

‎tests/profile_subclass/profiling.py‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
# Copyright (c) MONAI Consortium
2+
# Licensed under the Apache License, Version 2.0 (the "License");
3+
# you may not use this file except in compliance with the License.
4+
# You may obtain a copy of the License at
5+
# http://www.apache.org/licenses/LICENSE-2.0
6+
# Unless required by applicable law or agreed to in writing, software
7+
# distributed under the License is distributed on an "AS IS" BASIS,
8+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
9+
# See the License for the specific language governing permissions and
10+
# limitations under the License.
11+
12+
"""
13+
Comparing torch.Tensor, SubTensor, SubWithTorchFunc, MetaTensor
14+
Adapted from https://github.com/pytorch/pytorch/tree/v1.11.0/benchmarks/overrides_benchmark
15+
"""
16+
import argparse
17+
18+
import torch
19+
from min_classes import SubTensor, SubWithTorchFunc
20+
21+
from monai.data import MetaTensor
22+
from monai.utils.profiling import PerfContext
23+
24+
NUM_REPEATS = 1000
25+
NUM_REPEAT_OF_REPEATS = 1000
26+
27+
28+
def bench(t1, t2):
29+
bench_times = []
30+
for _ in range(NUM_REPEAT_OF_REPEATS):
31+
with PerfContext() as pc:
32+
for _ in range(NUM_REPEATS):
33+
torch.add(t1, t2)
34+
bench_times.append(pc.total_time)
35+
36+
bench_time_min = float(torch.min(torch.Tensor(bench_times))) / NUM_REPEATS
37+
bench_time_avg = float(torch.sum(torch.Tensor(bench_times))) / (NUM_REPEATS * NUM_REPEAT_OF_REPEATS)
38+
bench_time_med = float(torch.median(torch.Tensor(bench_times))) / NUM_REPEATS
39+
bench_std = float(torch.std(torch.Tensor(bench_times))) / NUM_REPEATS
40+
return bench_time_min, bench_time_avg, bench_time_med, bench_std
41+
42+
43+
def main():
44+
global NUM_REPEATS
45+
global NUM_REPEAT_OF_REPEATS
46+
47+
parser = argparse.ArgumentParser(description="Run the __torch_function__ benchmarks.")
48+
parser.add_argument(
49+
"--nreps", "-n", type=int, default=NUM_REPEATS, help="The number of repeats for one measurement."
50+
)
51+
parser.add_argument("--nrepreps", "-m", type=int, default=NUM_REPEAT_OF_REPEATS, help="The number of measurements.")
52+
args = parser.parse_args()
53+
54+
NUM_REPEATS = args.nreps
55+
NUM_REPEAT_OF_REPEATS = args.nrepreps
56+
57+
types = torch.Tensor, SubTensor, SubWithTorchFunc, MetaTensor
58+
59+
for t in types:
60+
tensor_1 = t(1)
61+
tensor_2 = t(2)
62+
63+
b_min, b_avg, b_med, b_std = bench(tensor_1, tensor_2)
64+
print(
65+
"Type {} time (microseconds): min: {}, avg: {}, median: {}, and std {}.".format(
66+
t.__name__, (10**6 * b_min), (10**6) * b_avg, (10**6) * b_med, (10**6) * b_std
67+
)
68+
)
69+
70+
71+
if __name__ == "__main__":
72+
main()

‎tests/profile_subclass/pyspy_profiling.py‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
# Copyright (c) MONAI Consortium
2+
# Licensed under the Apache License, Version 2.0 (the "License");
3+
# you may not use this file except in compliance with the License.
4+
# You may obtain a copy of the License at
5+
# http://www.apache.org/licenses/LICENSE-2.0
6+
# Unless required by applicable law or agreed to in writing, software
7+
# distributed under the License is distributed on an "AS IS" BASIS,
8+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
9+
# See the License for the specific language governing permissions and
10+
# limitations under the License.
11+
12+
"""
13+
To be used with py-spy, comparing torch.Tensor, SubTensor, SubWithTorchFunc, MetaTensor
14+
Adapted from https://github.com/pytorch/pytorch/tree/v1.11.0/benchmarks/overrides_benchmark
15+
"""
16+
import argparse
17+
18+
import torch
19+
from min_classes import SubTensor, SubWithTorchFunc # noqa: F401
20+
21+
from monai.data import MetaTensor # noqa: F401
22+
23+
Tensor = torch.Tensor
24+
25+
NUM_REPEATS = 1000000
26+
27+
if __name__ == "__main__":
28+
parser = argparse.ArgumentParser(description="Run the torch.add for a given class a given number of times.")
29+
parser.add_argument("tensor_class", metavar="TensorClass", type=str, help="The class to benchmark.")
30+
parser.add_argument("--nreps", "-n", type=int, default=NUM_REPEATS, help="The number of repeats.")
31+
args = parser.parse_args()
32+
33+
TensorClass = globals()[args.tensor_class]
34+
NUM_REPEATS = args.nreps
35+
36+
t1 = TensorClass(1)
37+
t2 = TensorClass(2)
38+
39+
for _ in range(NUM_REPEATS):
40+
torch.add(t1, t2)

0 commit comments

Comments
 (0)