Skip to content

Commit dd359c3

Browse files
Update python/docs/source/migration_guide/pyspark_upgrade.rst
Co-authored-by: Allison Wang <allison.wang@databricks.com>
1 parent 64ca2b1 commit dd359c3

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

python/docs/source/migration_guide/pyspark_upgrade.rst

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ Upgrading from PySpark 4.0 to 4.1
2424

2525
* In Spark 4.1, Arrow-optimized Python UDF supports UDT input / output instead of falling back to the regular UDF. To restore the legacy behavior, set ``spark.sql.execution.pythonUDF.arrow.legacy.fallbackOnUDT`` to ``true``.
2626

27-
* In Spark 4.1, unnecessary conversion to pandas instances is removed when ``spark.sql.execution.pythonUDTF.arrow.enabled`` is enabled. As a result, the type coercion changes when the produced output has a schema different from the specified schema. To restore the previous behavior, ``enable spark.sql.legacy.execution.pythonUDTF.pandas.conversion.enabled``.
27+
* In Spark 4.1, unnecessary conversion to pandas instances is removed when ``spark.sql.execution.pythonUDTF.arrow.enabled`` is enabled. As a result, the type coercion changes when the produced output has a schema different from the specified schema. To restore the previous behavior, enable ``spark.sql.legacy.execution.pythonUDTF.pandas.conversion.enabled``.
2828

2929

3030
Upgrading from PySpark 3.5 to 4.0

0 commit comments

Comments
 (0)