Skip to content
This repository was archived by the owner on Jan 9, 2020. It is now read-only.

Commit 7c72662

Browse files
committed
[SPARK-22043][PYTHON] Improves error message for show_profiles and dump_profiles
## What changes were proposed in this pull request? This PR proposes to improve error message from: ``` >>> sc.show_profiles() Traceback (most recent call last): File "<stdin>", line 1, in <module> File ".../spark/python/pyspark/context.py", line 1000, in show_profiles self.profiler_collector.show_profiles() AttributeError: 'NoneType' object has no attribute 'show_profiles' >>> sc.dump_profiles("/tmp/abc") Traceback (most recent call last): File "<stdin>", line 1, in <module> File ".../spark/python/pyspark/context.py", line 1005, in dump_profiles self.profiler_collector.dump_profiles(path) AttributeError: 'NoneType' object has no attribute 'dump_profiles' ``` to ``` >>> sc.show_profiles() Traceback (most recent call last): File "<stdin>", line 1, in <module> File ".../spark/python/pyspark/context.py", line 1003, in show_profiles raise RuntimeError("'spark.python.profile' configuration must be set " RuntimeError: 'spark.python.profile' configuration must be set to 'true' to enable Python profile. >>> sc.dump_profiles("/tmp/abc") Traceback (most recent call last): File "<stdin>", line 1, in <module> File ".../spark/python/pyspark/context.py", line 1012, in dump_profiles raise RuntimeError("'spark.python.profile' configuration must be set " RuntimeError: 'spark.python.profile' configuration must be set to 'true' to enable Python profile. ``` ## How was this patch tested? Unit tests added in `python/pyspark/tests.py` and manual tests. Author: hyukjinkwon <[email protected]> Closes apache#19260 from HyukjinKwon/profile-errors.
1 parent 6308c65 commit 7c72662

File tree

2 files changed

+26
-2
lines changed

2 files changed

+26
-2
lines changed

python/pyspark/context.py

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -997,12 +997,20 @@ def runJob(self, rdd, partitionFunc, partitions=None, allowLocal=False):
997997

998998
def show_profiles(self):
999999
""" Print the profile stats to stdout """
1000-
self.profiler_collector.show_profiles()
1000+
if self.profiler_collector is not None:
1001+
self.profiler_collector.show_profiles()
1002+
else:
1003+
raise RuntimeError("'spark.python.profile' configuration must be set "
1004+
"to 'true' to enable Python profile.")
10011005

10021006
def dump_profiles(self, path):
10031007
""" Dump the profile stats into directory `path`
10041008
"""
1005-
self.profiler_collector.dump_profiles(path)
1009+
if self.profiler_collector is not None:
1010+
self.profiler_collector.dump_profiles(path)
1011+
else:
1012+
raise RuntimeError("'spark.python.profile' configuration must be set "
1013+
"to 'true' to enable Python profile.")
10061014

10071015
def getConf(self):
10081016
conf = SparkConf()

python/pyspark/tests.py

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1296,6 +1296,22 @@ def heavy_foo(x):
12961296
rdd.foreach(heavy_foo)
12971297

12981298

1299+
class ProfilerTests2(unittest.TestCase):
1300+
def test_profiler_disabled(self):
1301+
sc = SparkContext(conf=SparkConf().set("spark.python.profile", "false"))
1302+
try:
1303+
self.assertRaisesRegexp(
1304+
RuntimeError,
1305+
"'spark.python.profile' configuration must be set",
1306+
lambda: sc.show_profiles())
1307+
self.assertRaisesRegexp(
1308+
RuntimeError,
1309+
"'spark.python.profile' configuration must be set",
1310+
lambda: sc.dump_profiles("/tmp/abc"))
1311+
finally:
1312+
sc.stop()
1313+
1314+
12991315
class InputFormatTests(ReusedPySparkTestCase):
13001316

13011317
@classmethod

0 commit comments

Comments
 (0)