Skip to content

Conversation

@IndarKarhana
Copy link
Contributor

@IndarKarhana IndarKarhana commented Jan 29, 2026

Closes #834

This PR replaces custom evaluate_performance functions and direct metric calls (e.g., mse(), mae()) with the standardized utilsforecast.evaluation.evaluate function across documentation notebooks.

Changes

  • getting_Started_complete.ipynb

    • Added NIXTLA_ID_AS_COL=1 env var to ensure unique_id is a column (not index)
    • Replaced custom evaluate_cv implementation with evaluate(df, metrics=[metric], models=models)
  • getting_Started_complete_polars.ipynb

    • Same evaluate() migration (polars already returns unique_id as column)
  • GARCH_tutorial.ipynb

    • Added NIXTLA_ID_AS_COL=1
    • Replaced mae(cv_df, models=models) with evaluate(cv_df, metrics=[mae], models=models)
  • CrossValidation.ipynb

    • Added NIXTLA_ID_AS_COL=1
    • Replaced rmse(cv_df, models=['AutoETS']) with evaluate(cv_df, metrics=[rmse], models=['AutoETS'], agg_fn='mean')

Why NIXTLA_ID_AS_COL?

The cross_validation() and forecast() methods currently return DataFrames with unique_id as the index. The evaluate() function expects unique_id as a column. Setting this env var ensures consistent behavior and prepares for the future default where unique_id will be a column.

Testing

  • getting_Started_complete.ipynb passes nbdev_test
  • Polars notebook unchanged (already compatible)
  • Tutorial notebooks have .notest but changes follow same pattern

Closes Nixtla#834

- Set NIXTLA_ID_AS_COL=1 to ensure unique_id is a column (not index)
- getting_Started_complete.ipynb: use evaluate() for per-series metrics
- getting_Started_complete_polars.ipynb: use evaluate() (polars already has id as col)
- GARCH_tutorial.ipynb: replace mae() direct call with evaluate()
- CrossValidation.ipynb: replace rmse() direct call with evaluate()
@review-notebook-app
Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

@codspeed-hq
Copy link

codspeed-hq bot commented Feb 3, 2026

CodSpeed Performance Report

Merging this PR will not alter performance

Comparing IndarKarhana:issue/834-docs-evaluate-refactor (fe9b1e3) with main (0621049)

Summary

✅ 6 untouched benchmarks

Copy link
Contributor

@nasaul nasaul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@nasaul nasaul merged commit 9ba7bfa into Nixtla:main Feb 4, 2026
57 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Deprecate references to custom evaluate_performace function in favor of utilsforecast.evaluation.evaluate?

2 participants