Skip to content

Commit 9bcc6bc

Browse files
author
Calvin Wang
committed
Hotfix for spark session shutdown for hanging processes
1 parent 2449f60 commit 9bcc6bc

File tree

1 file changed

+9
-0
lines changed

1 file changed

+9
-0
lines changed

README.md

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -146,6 +146,15 @@ result_metrep_df = repository.load() \
146146
.getSuccessMetricsAsDataFrame()
147147
```
148148

149+
### Wrapping up
150+
151+
After you've ran your jobs with PyDeequ, be sure to shut down your Spark session to prevent any hanging processes.
152+
153+
```python
154+
spark.sparkContext._gateway.shutdown_callback_server()
155+
spark.stop()
156+
```
157+
149158
## [Contributing](https://github.com/awslabs/python-deequ/blob/master/CONTRIBUTING.md)
150159
Please refer to the [contributing doc](https://github.com/awslabs/python-deequ/blob/master/CONTRIBUTING.md) for how to contribute to PyDeequ.
151160

0 commit comments

Comments
 (0)