QST: Why is it necessary to not allow `include_groups=True` in future versions for groupby(...).apply(...) operations?

[mattmills49]

Research

• I have searched the [pandas] tag on StackOverflow for similar questions.

• I have asked my usage related question on StackOverflow.

Link to question on StackOverflow

https://stackoverflow.com/questions/77969964/deprecation-warning-with-groupby-apply

Question about pandas

I find it strange that in the docs it is stated that the include_groups option won't allow True as an option starting in 3.0. I use groupby(...).apply(...) all the time working with tabular data and fully expect the group by columns to be included in the result. I feel like I have to be mis-understanding something about the warning because I can't fathom not having to select all the columns I need from the group by every time I use the operation. This is a very common data pattern I use:

summarize_df = (
  df.
  groupby(['user_id']).
  apply(lambda df: pd.Series(dict(
    num_orders = len(df),
    aov = np.mean(df['amount']),
    first_order = np.min(df['order_date'])
  ))).
  reset_index()
)

Which will give me another dataframe that I can use and has user_id in the column. The stack overflow question I linked seems to suggest the only way to get the user_id as a column in the future will be to 1.) add it to the index first or 2.) selecting the group by column after the apply call, but I would have to copy and re-select every column I created in the apply statement as well. The documentation for DataFrameGroupBy.apply doesn't make it clear how you'd keep the group by column.

Is this the intended behavior of this change? Is it assumed every user who is doing what I am doing is using agg instead of apply? Thanks!

[mattmills49]

Here is maybe a better scenario where I have to use apply, not agg. I am trying to calculate various metrics around model performance. If my dataframe is a tall dataframe with the columns sample, pred, and dv and I want to calculate the model performance for each sample I would create a list of functions, then call .apply to call each function with both the dv and pred as inputs, and then return a series that gets converted to columns within apply.

metrics_list_cont = {
    'mse': mse, 'rmse': rmse, 'mae': mae,
    'r2_rank': r2_rank, 'gini': gini_metric
}

def calc_metrics(df, metrics_list=metrics_list_cont,
                 pred_col='preds', dv_col='click_rate'):
    scores = {name: f(df[dv_col], df[pred_col]) for name, f in metrics_list.items()}
    return pd.Series(scores)
    
scores_df.groupby('sample').apply(calc_metrics).reset_index()

Would the expectation be that sample is no longer included in the output?

[rhshadrach]

Thanks for the report. Can you please post reproducible examples? We cannot reproduce the results you're seeing without the input DataFrames.

I suspect there is a misunderstanding of the change.

df = pd.DataFrame({"a": [1, 1, 2], "b": [3, 4, 5]})
gb = df.groupby("a")

# Prior to pandas 3.0
print(gb.apply(lambda x: x.sum()))
#    a  b
# a
# 1  2  7
# 2  2  5

# pandas 3.0
print(gb.apply(lambda x: x.sum()))
#    b
# a   
# 1  7
# 2  5

The groups are still included in the result, but they are not acted upon by the groupby kernel (sum, in this case).

[mattmills49]

Ok, that definitely makes more sense! Thanks for the clarification, the way I read it I assumed it wouldn't be returned in the result.