Skip to content

Commit 9d45e67

Browse files
viiryacloud-fan
authored andcommitted
[SPARK-22541][SQL] Explicitly claim that Python udfs can't be conditionally executed with short-curcuit evaluation
## What changes were proposed in this pull request? Besides conditional expressions such as `when` and `if`, users may want to conditionally execute python udfs by short-curcuit evaluation. We should also explicitly note that python udfs don't support this kind of conditional execution too. ## How was this patch tested? N/A, just document change. Author: Liang-Chi Hsieh <[email protected]> Closes #19787 from viirya/SPARK-22541.
1 parent 41c6f36 commit 9d45e67

File tree

1 file changed

+6
-12
lines changed

1 file changed

+6
-12
lines changed

python/pyspark/sql/functions.py

Lines changed: 6 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -2079,12 +2079,9 @@ def udf(f=None, returnType=StringType()):
20792079
duplicate invocations may be eliminated or the function may even be invoked more times than
20802080
it is present in the query.
20812081
2082-
.. note:: The user-defined functions do not support conditional execution by using them with
2083-
SQL conditional expressions such as `when` or `if`. The functions still apply on all rows no
2084-
matter the conditions are met or not. So the output is correct if the functions can be
2085-
correctly run on all rows without failure. If the functions can cause runtime failure on the
2086-
rows that do not satisfy the conditions, the suggested workaround is to incorporate the
2087-
condition logic into the functions.
2082+
.. note:: The user-defined functions do not support conditional expressions or short curcuiting
2083+
in boolean expressions and it ends up with being executed all internally. If the functions
2084+
can fail on special rows, the workaround is to incorporate the condition into the functions.
20882085
20892086
:param f: python function if used as a standalone function
20902087
:param returnType: a :class:`pyspark.sql.types.DataType` object
@@ -2194,12 +2191,9 @@ def pandas_udf(f=None, returnType=None, functionType=None):
21942191
21952192
.. note:: The user-defined function must be deterministic.
21962193
2197-
.. note:: The user-defined functions do not support conditional execution by using them with
2198-
SQL conditional expressions such as `when` or `if`. The functions still apply on all rows no
2199-
matter the conditions are met or not. So the output is correct if the functions can be
2200-
correctly run on all rows without failure. If the functions can cause runtime failure on the
2201-
rows that do not satisfy the conditions, the suggested workaround is to incorporate the
2202-
condition logic into the functions.
2194+
.. note:: The user-defined functions do not support conditional expressions or short curcuiting
2195+
in boolean expressions and it ends up with being executed all internally. If the functions
2196+
can fail on special rows, the workaround is to incorporate the condition into the functions.
22032197
"""
22042198
# decorator @pandas_udf(returnType, functionType)
22052199
is_decorator = f is None or isinstance(f, (str, DataType))

0 commit comments

Comments
 (0)