Commit f7b20da
committed
[SPARK-54842][PYTHON][TESTS] Fix
### What changes were proposed in this pull request?
attempt to fix https://github.com/apache/spark/actions/runs/20495264978/job/58904792835,
```
Caused by: org.apache.spark.api.python.PythonException: Traceback (most recent call last):
File "/Users/runner/work/spark/spark/python/lib/pyspark.zip/pyspark/worker.py", line 3511, in main
process()
File "/Users/runner/work/spark/spark/python/lib/pyspark.zip/pyspark/worker.py", line 3502, in process
serializer.dump_stream(out_iter, outfile)
File "/Users/runner/work/spark/spark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py", line 781, in dump_stream
return ArrowStreamSerializer.dump_stream(self, wrap_and_init_stream(), stream)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/runner/work/spark/spark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py", line 120, in dump_stream
for batch in iterator:
File "/Users/runner/work/spark/spark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py", line 765, in wrap_and_init_stream
for packed in iterator:
File "/Users/runner/work/spark/spark/python/lib/pyspark.zip/pyspark/worker.py", line 2954, in func
for result_batch, result_type in result_iter:
File "/Users/runner/work/spark/spark/python/pyspark/sql/tests/arrow/test_arrow_udf_scalar.py", line 930, in <lambda>
lambda it: map(lambda x: pa.compute.subtract(x, 1), it),
^^^^^^^^^^
AttributeError: module 'pyarrow' has no attribute 'compute'
```
This test passed before on macos26;
The parity test on spark connect pass;
### Why are the changes needed?
I suspect there is a cloudpickle pitfall when dealing with complicated nested lambdas, I remember I resolved a similar issue by changing the import.
### Does this PR introduce _any_ user-facing change?
no, test-only
### How was this patch tested?
cannot reproduce this issue locally, will monitor the CI
### Was this patch authored or co-authored using generative AI tooling?
no
Closes #53607 from zhengruifeng/fix_test_arrow_udf_chained_iii.
Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>test_arrow_udf_chained_iii in Python-Only MacOS261 parent 3cc3cc1 commit f7b20da
1 file changed
+7
-6
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
916 | 916 | | |
917 | 917 | | |
918 | 918 | | |
| 919 | + | |
919 | 920 | | |
920 | | - | |
921 | | - | |
922 | | - | |
| 921 | + | |
| 922 | + | |
| 923 | + | |
923 | 924 | | |
924 | 925 | | |
925 | | - | |
| 926 | + | |
926 | 927 | | |
927 | 928 | | |
928 | 929 | | |
929 | 930 | | |
930 | | - | |
| 931 | + | |
931 | 932 | | |
932 | 933 | | |
933 | 934 | | |
934 | 935 | | |
935 | 936 | | |
936 | 937 | | |
937 | 938 | | |
938 | | - | |
| 939 | + | |
939 | 940 | | |
940 | 941 | | |
941 | 942 | | |
| |||
0 commit comments