Skip to content

Commit 90a2d24

Browse files
committed
restructured udf user guide
1 parent 72f7b62 commit 90a2d24

File tree

1 file changed

+88
-167
lines changed

1 file changed

+88
-167
lines changed

doc/source/user_guide/user_defined_functions.rst

Lines changed: 88 additions & 167 deletions
Original file line numberDiff line numberDiff line change
@@ -13,24 +13,28 @@ flexibility when built-in methods are not sufficient. These functions can be
1313
applied at different levels: element-wise, row-wise, column-wise, or group-wise,
1414
and change the data differently, depending on the method used.
1515

16-
Why Use User-Defined Functions?
17-
-------------------------------
16+
Why Not To Use User-Defined Functions
17+
-----------------------------------------
1818

19-
Pandas is designed for high-performance data processing, but sometimes your specific
20-
needs go beyond standard aggregation, transformation, or filtering. User-defined functions allow you to:
19+
While UDFs provide flexibility, they come with significant drawbacks, primarily
20+
related to performance. Unlike vectorized pandas operations, UDFs are slower because pandas lacks
21+
insight into what they are computing, making it difficult to apply efficient handling or optimization
22+
techniques. As a result, pandas resorts to less efficient processing methods that significantly
23+
slow down computations. Additionally, relying on UDFs often sacrifices the benefits
24+
of pandas’ built-in, optimized methods, limiting compatibility and overall performance.
2125

22-
* **Customize Computations**: Implement logic tailored to your dataset, such as complex
23-
transformations, domain-specific calculations, or conditional modifications.
24-
* **Improve Code Readability**: Encapsulate logic into functions rather than writing long,
25-
complex expressions.
26-
* **Handle Complex Grouped Operations**: Perform operations on grouped data that standard
27-
methods do not support.
28-
* **Extend pandas' Functionality**: Apply external libraries or advanced calculations that
29-
are not natively available.
26+
.. note::
27+
In general, most tasks can and should be accomplished using pandas’ built-in methods or vectorized operations.
28+
29+
Despite their drawbacks, UDFs can be helpful when:
3030

31+
* **Custom Computations Are Needed**: Implementing complex logic or domain-specific calculations that pandas'
32+
built-in methods cannot handle.
33+
* **Extending pandas' Functionality**: Applying external libraries or specialized algorithms unavailable in pandas.
34+
* **Handling Complex Grouped Operations**: Performing operations on grouped data that standard methods do not support.
3135

32-
What functions support User-Defined Functions
33-
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
36+
Methods that support User-Defined Functions
37+
-------------------------------------------
3438

3539
User-Defined Functions can be applied across various pandas methods:
3640

@@ -47,124 +51,66 @@ User-Defined Functions can be applied across various pandas methods:
4751
Series in a clean, readable manner.
4852

4953
All of these pandas methods can be used with both Series and DataFrame objects, providing versatile
50-
ways to apply user-defined functions across different pandas data structures.
51-
54+
ways to apply UDFs across different pandas data structures.
55+
56+
57+
Choosing the Right Method
58+
-------------------------
59+
When applying UDFs in pandas, it is essential to select the appropriate method based
60+
on your specific task. Each method has its strengths and is designed for different use
61+
cases. Understanding the purpose and behavior of each method will help you make informed
62+
decisions, ensuring more efficient and maintainable code.
63+
64+
Below is a table overview of all methods that accept UDFs:
65+
66+
+------------------+--------------------------------------+---------------------------+--------------------+---------------------------+------------------------------------------+
67+
| Method | Purpose | Supports UDFs | Keeps Shape | Performance | Recommended Use Case |
68+
+==================+======================================+===========================+====================+===========================+==========================================+
69+
| :meth:`apply` | General-purpose function | Yes | Yes (when axis=1) | Slow | Custom row-wise or column-wise operations|
70+
+------------------+--------------------------------------+---------------------------+--------------------+---------------------------+------------------------------------------+
71+
| :meth:`agg` | Aggregation | Yes | No | Fast (if using built-ins) | Custom aggregation logic |
72+
+------------------+--------------------------------------+---------------------------+--------------------+---------------------------+------------------------------------------+
73+
| :meth:`transform`| Transform without reducing dimensions| Yes | Yes | Fast (if vectorized) | Broadcast Element-wise transformations |
74+
+------------------+--------------------------------------+---------------------------+--------------------+---------------------------+------------------------------------------+
75+
| :meth:`map` | Element-wise mapping | Yes | Yes | Moderate | Simple element-wise transformations |
76+
+------------------+--------------------------------------+---------------------------+--------------------+---------------------------+------------------------------------------+
77+
| :meth:`pipe` | Functional chaining | Yes | Yes | Depends on function | Building clean pipelines |
78+
+------------------+--------------------------------------+---------------------------+--------------------+---------------------------+------------------------------------------+
79+
| :meth:`filter` | Row/Column selection | Not directly | Yes | Fast | Subsetting based on conditions |
80+
+------------------+--------------------------------------+---------------------------+--------------------+---------------------------+------------------------------------------+
5281

5382
:meth:`DataFrame.apply`
54-
-----------------------
55-
56-
The :meth:`DataFrame.apply` allows applying a user-defined functions along either axis (rows or columns):
57-
58-
.. ipython:: python
59-
60-
import pandas as pd
83+
~~~~~~~~~~~~~~~~~~~~~~~
6184

62-
# Sample DataFrame
63-
df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})
64-
65-
# User-Defined Function
66-
def add_one(x):
67-
return x + 1
68-
69-
# Apply function
70-
df_applied = df.apply(add_one)
71-
print(df_applied)
72-
73-
# This works with lambda functions too
74-
df_lambda = df.apply(lambda x : x + 1)
75-
print(df_lambda)
85+
The :meth:`DataFrame.apply` allows you to apply UDFs along either rows or columns. While flexible,
86+
it is slower than vectorized operations and should be used only when you need operations
87+
that cannot be achieved with built-in pandas functions.
7688

89+
When to use: :meth:`DataFrame.apply` is suitable when no alternative vectorized method is available, but consider
90+
optimizing performance with vectorized operations wherever possible.
7791

78-
:meth:`DataFrame.apply` also accepts dictionaries of multiple user-defined functions:
79-
80-
.. ipython:: python
81-
82-
# Sample DataFrame
83-
df = pd.DataFrame({'A': [1, 2, 3], 'B': [1, 2, 3]})
84-
85-
# User-Defined Function
86-
def add_one(x):
87-
return x + 1
88-
89-
def add_two(x):
90-
return x + 2
91-
92-
# Apply function
93-
df_applied = df.apply({"A": add_one, "B": add_two})
94-
print(df_applied)
95-
96-
# This works with lambda functions too
97-
df_lambda = df.apply({"A": lambda x : x + 1, "B": lambda x : x + 2})
98-
print(df_lambda)
99-
100-
:meth:`DataFrame.apply` works with Series objects as well:
101-
102-
.. ipython:: python
103-
104-
# Sample Series
105-
s = pd.Series([1, 2, 3])
106-
107-
# User-Defined Function
108-
def add_one(x):
109-
return x + 1
110-
111-
# Apply function
112-
s_applied = s.apply(add_one)
113-
print(s_applied)
114-
115-
# This works with lambda functions too
116-
s_lambda = s.apply(lambda x : x + 1)
117-
print(s_lambda)
92+
Examples of usage can be found at :meth:`DataFrame.apply`
11893

11994
:meth:`DataFrame.agg`
120-
---------------------
121-
122-
The :meth:`DataFrame.agg` allows aggregation with a user-defined function along either axis (rows or columns):
123-
124-
.. ipython:: python
125-
126-
# Sample DataFrame
127-
df = pd.DataFrame({
128-
'Category': ['A', 'A', 'B', 'B'],
129-
'Values': [10, 20, 30, 40]
130-
})
95+
~~~~~~~~~~~~~~~~~~~~~
13196

132-
# Define a function for group operations
133-
def group_mean(group):
134-
return group.mean()
97+
If you need to aggregate data, :meth:`DataFrame.agg` is a better choice than apply because it is
98+
specifically designed for aggregation operations.
13599

136-
# Apply UDF to each group
137-
grouped_result = df.groupby('Category')['Values'].agg(group_mean)
138-
print(grouped_result)
100+
When to use: Use :meth:`DataFrame.agg` for performing aggregations like sum, mean, or custom aggregation
101+
functions across groups.
139102

140-
In terms of the API, :meth:`DataFrame.agg` has similar usage to :meth:`DataFrame.apply`,
141-
but it is primarily used for **aggregation**, applying functions that summarize or reduce data.
142-
Typically, the result of :meth:`DataFrame.agg` reduces the dimensions of data as shown
143-
in the above example. Conversely, :meth:`DataFrame.apply` is more general and allows for both
144-
transformations and custom row-wise or element-wise operations.
103+
Examples of usage can be found at :meth:`DataFrame.agg <api.dataframe.agg>`
145104

146105
:meth:`DataFrame.transform`
147-
---------------------------
106+
~~~~~~~~~~~~~~~~~~~~~~~~~~~
148107

149-
The :meth:`DataFrame.transform` allows transforms a Dataframe, Series or Grouped object
150-
while preserving the original shape of the object.
108+
The transform method is ideal for performing element-wise transformations while preserving the shape of the original DataFrame.
109+
It’s generally faster than apply because it can take advantage of pandas' internal optimizations.
151110

152-
.. ipython:: python
153-
154-
# Sample DataFrame
155-
df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})
156-
157-
# User-Defined Function
158-
def double(x):
159-
return x * 2
111+
When to use: When you need to perform element-wise transformations that retain the original structure of the DataFrame.
160112

161-
# Apply transform
162-
df_transformed = df.transform(double)
163-
print(df_transformed)
164-
165-
# This works with lambda functions too
166-
df_lambda = df.transform(lambda x: x * 2)
167-
print(df_lambda)
113+
Documentation: DataFrame.transform
168114

169115
Attempting to use common aggregation functions such as ``mean`` or ``sum`` will result in
170116
values being broadcasted to the original dimensions:
@@ -187,15 +133,17 @@ values being broadcasted to the original dimensions:
187133
print(df)
188134
189135
:meth:`DataFrame.filter`
190-
------------------------
136+
~~~~~~~~~~~~~~~~~~~~~~~~
191137

192138
The :meth:`DataFrame.filter` method is used to select subsets of the DataFrame’s
193139
columns or row. It is useful when you want to extract specific columns or rows that
194140
match particular conditions.
195141

142+
When to use: Use :meth:`DataFrame.filter` when you want to use a UDF to create a subset of a DataFrame or Series
143+
196144
.. note::
197-
:meth:`DataFrame.filter` does not accept user-defined functions, but can accept
198-
list comprehensions that have user-defined functions applied to them.
145+
:meth:`DataFrame.filter` does not accept UDFs, but can accept
146+
list comprehensions that have UDFs applied to them.
199147

200148
.. ipython:: python
201149
@@ -207,72 +155,45 @@ match particular conditions.
207155
'D': [10, 11, 12]
208156
})
209157
158+
# Define a function that filters out columns where the name is longer than 1 character
210159
def is_long_name(column_name):
211160
return len(column_name) > 1
212161
213-
# Define a function that filters out columns where the name is longer than 1 character
214162
df_filtered = df[[col for col in df.columns if is_long_name(col)]]
215163
print(df_filtered)
216164
217165
:meth:`DataFrame.map`
218-
---------------------
219-
220-
The :meth:`DataFrame.map` method is used to apply a function element-wise to a pandas Series
221-
or Dataframe. It is particularly useful for substituting values or transforming data.
222-
223-
.. ipython:: python
224-
225-
# Sample DataFrame
226-
df = pd.DataFrame({ 'A': ['cat', 'dog', 'bird'], 'B': ['pig', 'cow', 'lamb'] })
166+
~~~~~~~~~~~~~~~~~~~~~
227167

228-
# Using map with a user-defined function
229-
def animal_to_length(animal):
230-
return len(animal)
168+
:meth:`DataFrame.map` is used specifically to apply element-wise UDFs and is better
169+
for this purpose compared to :meth:`DataFrame.apply` because of its better performance.
231170

232-
df_mapped = df.map(animal_to_length)
233-
print(df_mapped)
171+
When to use: Use map for applying element-wise UDFs to DataFrames or Series.
234172

235-
# This works with lambda functions too
236-
df_lambda = df.map(lambda x: x.upper())
237-
print(df_lambda)
173+
Documentation: DataFrame.map
238174

239175
:meth:`DataFrame.pipe`
240-
----------------------
241-
242-
The :meth:`DataFrame.pipe` method allows you to apply a function or a series of functions to a
243-
DataFrame in a clean and readable way. This is especially useful for building data processing pipelines.
244-
245-
.. ipython:: python
246-
247-
# Sample DataFrame
248-
df = pd.DataFrame({ 'A': [1, 2, 3], 'B': [4, 5, 6] })
249-
250-
# User-defined functions for transformation
251-
def add_one(df):
252-
return df + 1
176+
~~~~~~~~~~~~~~~~~~~~~~
253177

254-
def square(df):
255-
return df ** 2
178+
The pipe method is useful for chaining operations together into a clean and readable pipeline.
179+
It is a helpful tool for organizing complex data processing workflows.
256180

257-
# Applying functions using pipe
258-
df_piped = df.pipe(add_one).pipe(square)
259-
print(df_piped)
181+
When to use: Use pipe when you need to create a pipeline of transformations and want to keep the code readable and maintainable.
260182

261-
The advantage of using :meth:`DataFrame.pipe` is that it allows you to chain together functions
262-
without nested calls, promoting a cleaner and more readable code style.
183+
Documentation: DataFrame.pipe
263184

264185

265-
Performance Considerations
266-
--------------------------
186+
Best Practices
187+
--------------
267188

268-
While user-defined functions provide flexibility, their use is currently discouraged as they can introduce
189+
While UDFs provide flexibility, their use is currently discouraged as they can introduce
269190
performance issues, especially when written in pure Python. To improve efficiency,
270-
consider using built-in ``NumPy`` or ``pandas`` functions instead of user-defined functions
191+
consider using built-in ``NumPy`` or ``pandas`` functions instead of UDFs
271192
for common operations.
272193

273194
.. note::
274195
If performance is critical, explore **vectorizated operations** before resorting
275-
to user-defined functions.
196+
to UDFs.
276197

277198
Vectorized Operations
278199
~~~~~~~~~~~~~~~~~~~~~
@@ -285,10 +206,10 @@ Below is an example of vectorized operations in pandas:
285206
def calc_ratio(row):
286207
return 100 * (row["one"] / row["two"])
287208
288-
df["new_col2"] = df.apply(calc_ratio, axis=1)
209+
df["new_col"] = df.apply(calc_ratio, axis=1)
289210
290211
# Vectorized Operation
291-
df["new_col"] = 100 * (df["one"] / df["two"])
212+
df["new_col2"] = 100 * (df["one"] / df["two"])
292213
293214
Measuring how long each operation takes:
294215

@@ -298,8 +219,8 @@ Measuring how long each operation takes:
298219
User-defined function: 5.6435 secs
299220
300221
Vectorized operations in pandas are significantly faster than using :meth:`DataFrame.apply`
301-
with user-defined functions because they leverage highly optimized C functions
222+
with UDFs because they leverage highly optimized C functions
302223
via NumPy to process entire arrays at once. This approach avoids the overhead of looping
303224
through rows in Python and making separate function calls for each row, which is slow and
304225
inefficient. Additionally, NumPy arrays benefit from memory efficiency and CPU-level
305-
optimizations, making vectorized operations the preferred choice whenever possible.
226+
optimizations, making vectorized operations the preferred choice whenever possible.

0 commit comments

Comments
 (0)