diff --git a/python/docs/source/migration_guide/pyspark_upgrade.rst b/python/docs/source/migration_guide/pyspark_upgrade.rst index 976aef7cb68b6..893f7e09f3748 100644 --- a/python/docs/source/migration_guide/pyspark_upgrade.rst +++ b/python/docs/source/migration_guide/pyspark_upgrade.rst @@ -77,6 +77,7 @@ Upgrading from PySpark 3.5 to 4.0 * In Spark 4.0, items other than functions (e.g. ``DataFrame``, ``Column``, ``StructType``) have been removed from the wildcard import ``from pyspark.sql.functions import *``, you should import these items from proper modules (e.g. ``from pyspark.sql import DataFrame, Column``, ``from pyspark.sql.types import StructType``). * In Spark 4.0, ``spark.sql.execution.pythonUDF.arrow.enabled`` is enabled by default. If users have PyArrow and pandas installed in their local and Spark Cluster, it automatically optimizes the regular Python UDFs with Arrow. To turn off the Arrow optimization, set ``spark.sql.execution.pythonUDF.arrow.enabled`` to ``false``. * In Spark 4.0, ``spark.sql.execution.arrow.pyspark.enabled`` is enabled by default. If users have PyArrow and pandas installed in their local and Spark Cluster, it automatically makes use of Apache Arrow for columnar data transfers in PySpark. This optimization applies to ``pyspark.sql.DataFrame.toPandas`` and ``pyspark.sql.SparkSession.createDataFrame`` when its input is a Pandas DataFrame or a NumPy ndarray. To turn off the Arrow optimization, set ``spark.sql.execution.arrow.pyspark.enabled`` to ``false``. +* In Spark 4.0, ``spark.sql.execution.pythonUDTF.arrow.enabled`` is enabled by default. If users have PyArrow and pandas installed in their local and Spark Cluster, it automatically optimizes the regular Python UDFTs with Arrow. To turn off the Arrow optimization, set ``spark.sql.execution.pythonUDTF.arrow.enabled`` to ``false``. Upgrading from PySpark 3.3 to 3.4 diff --git a/python/docs/source/user_guide/sql/python_udtf.rst b/python/docs/source/user_guide/sql/python_udtf.rst index ac394d9e5360f..817c47d0567d2 100644 --- a/python/docs/source/user_guide/sql/python_udtf.rst +++ b/python/docs/source/user_guide/sql/python_udtf.rst @@ -417,8 +417,9 @@ data between Java and Python processes. Apache Arrow is disabled by default for Arrow can improve performance when each input row generates a large result table from the UDTF. -To enable Arrow optimization, set the ``spark.sql.execution.pythonUDTF.arrow.enabled`` -configuration to ``true``. You can also enable it by specifying the ``useArrow`` parameter +This is enabled by default with ``spark.sql.execution.pythonUDTF.arrow.enabled`` set ``true``. +To disable Arrow optimization, set the ``spark.sql.execution.pythonUDTF.arrow.enabled`` +configuration to ``false``. You can also disable it by specifying the ``useArrow`` parameter when declaring the UDTF. .. literalinclude:: ../../../../../examples/src/main/python/sql/udtf.py diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala index d5e641287385d..64adab073aaad 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala @@ -3509,7 +3509,7 @@ object SQLConf { .doc("Enable Arrow optimization for Python UDTFs.") .version("3.5.0") .booleanConf - .createWithDefault(false) + .createWithDefault(true) val PYTHON_PLANNER_EXEC_MEMORY = buildConf("spark.sql.planner.pythonExecution.memory")