Evaluation of TimesFM-2.5 and clarification about leakage.

[rajatsen91]

Firstly, thanks a lot for creating this great benchmark.

We had a small clarification about the leakage column for TimesFM-2.5.

According to the leaderboard TiReX has 1% leakage and TimesFM-2.5 has 8% leakage. However according to the model cards of TimesFM-2.5 and of TiREX the datasets used by them are:

1. TimesFM-2.5
   --GiftEvalPretrain
   --Wikimedia Pageviews, cutoff Nov 2023 (see paper for details).
   --Google Trends top queries, cutoff EoY 2022 (see paper for details).
   --Synthetic and augmented data.

2. TiReX
   --chronos_datasets (Subset - Zero Shot Benchmark data is not used for training - details in the paper)
   --GiftEvalPretrain (Subset - details in the paper)
   --Synthetic Data

3. Since TimesFM-2.5 does not use chronos_datasets and GiftEvalPretrain is common to both, we were wondering why there is a difference in leakage percentage. Would you be able to clarify which 7 datasets are there in TimesFM-2.5's pretraining corpus and not in TiReX's? Thanks in advance. This might also have an impact on the evaluation rankings as the leakage ones are replaced by chronos-bolt numbers.

Also thanks for pointing out inference speed related issues here. We have made the inference speed around 7x faster.

[shchur]

Hi @rajatsen91, thanks for opening the issue.

As we could infer from the publications, the following datasets were used both in the training corpus of the models and are present in fev-bench:

• TiRex: ["kdd_cup_2022_10T"]
• TimesFM-2.5: ["favorita_transactions_1D", "fred_md_2025/cee", "fred_md_2025/macro", "proenfo_gfc12", "proenfo_gfc14", "proenfo_gfc17", "kdd_cup_2022_10T", "m5_1D"]

Let me know if any of these were not used for TimesFM-2.5, so that we can update the imputation logic.

As far as I can tell, imputation does not change the conclusions about the relative ranking of the models. Even if we exclude these datasets, the ranking stays consistent.

Details

import pandas as pd
import fev

summaries = pd.read_csv("https://raw.githubusercontent.com/autogluon/fev/refs/heads/main/benchmarks/fev_bench/results/summaries.csv")
tfm_leaky_tasks = ["favorita_transactions_1D", "fred_md_2025/cee", "fred_md_2025/macro", "proenfo_gfc12", "proenfo_gfc14", "proenfo_gfc17", "kdd_cup_2022_10T", "m5_1D"]
filtered = summaries.query("task_name not in @tfm_leaky_tasks")

for metric in ["SQL", "MASE"]:
    lb = fev.leaderboard(
        filtered, 
        metric_column=metric, 
        baseline_model="Seasonal Naive", 
        missing_strategy="impute", 
        leakage_imputation_model="Chronos-Bolt",
    ).round(3)
    print(lb[["win_rate", "skill_score"]])

SQL
                win_rate  skill_score
model_name                           
TiRex              0.866        0.428
TimesFM-2.5        0.832        0.427
Toto-1.0           0.742        0.410
Chronos-Bolt       0.688        0.390
Moirai-2.0         0.687        0.395
TabPFN-TS          0.673        0.398
Stat. Ensemble     0.486        0.199
Sundial-Base       0.477        0.326
AutoARIMA          0.445        0.207
AutoETS            0.349       -0.309
AutoTheta          0.288        0.039
Seasonal Naive     0.220        0.000
Naive              0.150       -0.463
Drift              0.095       -0.469
MASE
                win_rate  skill_score
model_name                           
TiRex              0.809        0.295
TimesFM-2.5        0.809        0.301
Toto-1.0           0.700        0.277
Moirai-2.0         0.648        0.267
Chronos-Bolt       0.645        0.259
TabPFN-TS          0.622        0.272
Sundial-Base       0.558        0.229
Stat. Ensemble     0.514        0.150
AutoARIMA          0.398        0.108
AutoTheta          0.362        0.099
AutoETS            0.342        0.014
Seasonal Naive     0.228        0.000
Naive              0.211       -0.165
Drift              0.156       -0.182

Also, as we discuss in the paper, since the bootstrap confidence intervals for the pairwise comparison of TimesFM-2.5 and TiRex heavily overlap, our conclusion is that these models are essentially on par, and it's easy to get either one to rank as #1 with small changes to the benchmark (adding or removing a few tasks).

As a side note, we decided to include the above datasets in fev-bench even though they are present in GiftEvalPretrain because these are really high-quality datasets vetted by the experts from the respective fields (FRED-MD is the benchmark dataset for macroeconomics, M5 is very relevant for retail, etc).

Also thanks for looking into the inference speed issue - I will rerun the eval with the new version and update the leaderboard accordingly next week.

[abdulfatir]

@rajatsen91 Adding on to what @shchur said:

Details on which exact tasks overlap between the training corpus of a model and fev-bench are available in the evaluation summaries. Please check the trained_on_this_dataset column. For TiRex we referred to Table 6 in the Appendix of the paper which lists the datasets from GIFTEval-Pretrain which were used for training.

[siriuz42]

Hi @shchur and @abdulfatir, thanks for the details! I really like this pairwise comparison approach. Appreciate your work put into this benchmark.

Both TimesFM-2.5 and TiRex may have leakage in Favorita Sales. TimesFM-2.5 might have more leakage according to the fev-bench paper definition, that we have the resampled Favorita Sales, Favorita Transactions and M5 at the weekly level.

And please allow me complain a bit, that the practice of replacing results of in-distribution tasks by Chronos-Bolt results makes it difficult to interpret the pairwise comparison on the aggregated level: Now it is neither about zero-shot comparison when two candidates have different in-distribution overlaps, nor about zero-shot + in-distribution comparison because the relative performance of the two candidates against Chronos-Bolt comes into play.

I was wondering what if we

1. Do not overwrite results, and allow the leakage percentage column to provide the intended discounting effect.
2. Do not overwrite results, and handicap a candidate on an in-distribution task: If candidate A shows better results than candadidate B on a task, we force a tie that

A	B	Conclusion
zero-shot	zero-shot	A wins
zero-shot	in-distribution	A wins
in-distribution	zero-shot	tie
in-distribution	in-distribution	A wins

[shchur]

Hi @siriuz42,

Regarding the Favorita dataset: we use the (store x category) Favorita dataset from this competition with ~1.5K series. It's different from the (store x product) dataset from the original competition with ~140K series, so we did not consider them as leakage.

Thanks for kicking off the discussion - the current version with replacing the leaky scores with Chronos-Bolt scores was our initial solution, and we are happy to take feedback from the community to improve the benchmark. Regarding the options that you described:

1. While the leakage indicator can be somehow connected to the win rate (if X% are leaky, we could subtract X% from the win rate), the leaky datasets really distort the average error / skill score metrics. As an example, here are the relative errors for top-10 datasets where TimesFM-2.5 achieves the largest error reduction w.r.t. TiRex (leaky datasets for TimesFM marked in bold):

Details

import pandas as pd
import fev

summaries = pd.read_csv("https://raw.githubusercontent.com/autogluon/fev/refs/heads/main/benchmarks/fev_bench/results/summaries.csv")
scores = fev.pivot_table(summaries, task_columns=["task_name"])[["TiRex", "TimesFM-2.5", "Chronos-Bolt"]]
rel_scores = scores.divide(scores["TiRex"], axis=0)
print(rel_scores.sort_values("TimesFM-2.5").head(20).round(3).reset_index().to_markdown(index=False))

task_name	TiRex	TimesFM-2.5	Chronos-Bolt
kdd_cup_2022_10T	1	0.172	1.203
proenfo_gfc17	1	0.18	1.012
proenfo_gfc14	1	0.195	1.065
proenfo_gfc12	1	0.207	1.01
bizitobs_l2c_5T	1	0.679	1.115
fred_md_2025/macro	1	0.752	1.101
uk_covid_nation_1W/new	1	0.835	0.915
favorita_transactions_1D	1	0.847	0.945
boomlet_1975	1	0.87	0.934
favorita_transactions_1W	1	0.884	1.032
fred_md_2025/cee	1	0.886	1.341
redset_15T	1	0.89	1.493
bizitobs_l2c_1H	1	0.89	0.934
solar_with_weather_1H	1	0.906	0.906
jena_weather_10T	1	0.917	1.073
redset_5T	1	0.919	1.303
ecdc_ili	1	0.919	1.101
uk_covid_nation_1D/cumulative	1	0.921	1.066
rossmann_1D	1	0.93	0.973
australian_tourism	1	0.932	1.181

The 5-10x error reduction on the small datasets seen during training (kdd_cup_2022_10T, proenfo_gfc12/14/17) looks like memorization here and has a disproportionate effect on the overall score (skill score w/o imputation: 46%, with imputation: 42%). Just reporting the leakage % alongside the overall error / skill score won't solve the problem.

2. I'm not sure if I understood this option, could you clarify this a bit more?

---

For completeness, here is our reasoning for the current setup.

The benchmark is forward-looking, and we want to discourage training on the benchmark datasets. Ideally, we want the scores to reflect the zero-shot performance of models.

Tracking whether the model was trained only on the training split without touching the test portion of the dataset is difficult and error-prone, and incentivizes directly overfitting the benchmark. Thus, we consider using any portion of the dataset as leakage.

This leaves us with the following feasible methods:

1. Remove models that have any leakage / drop them to the bottom of the leaderboard
   • This is a very strict criterion that excludes many existing and potentially future models.
2. Remove all datasets that were used in the corpus of prominent pretrained models to date
   • We exclude high-quality datasets constructed by the experts (M5, FRED-MD, GFC, Favorita, KDD 2022).
   • This does not solve the problem completely - eventually someone will train a model on the datasets used in our benchmark and we'll need to decide how to represent its scores.
3. Report 2 versions of each score (with leakage / without leakage)
   • The results become harder to interpret (leading to strange claims like "model A is SotA with leakage, but model B is SotA without leakage").
   • The results with leakage become trivial to overfit, and we still need some strategy for computing the non-leaky results.
4. Impute the leaky results with some reasonable guess of the model's performance. We have a few options here:
   1. seasonal naive - extremely harsh to models that have any leakage
   2. median of all non-leaky models' scores for this task - changes as new models are added to the benchmark
   3. an a priori fixed zero-shot model.

The last option seemed as a reasonable compromise to us as it simultaneously does not penalize the leaky models too much while also discouraging training on the benchmark datasets.

---

To double check, is this the full list of changes to TimesFM-2.5 that need to be made, or is there something else to consider?

• Re-run with the latest inference code to fix the inference speed issue
• Mark tasks m5_1W and favorita_transactions_1W as seen in the training corpus

[rajatsen91]

Thanks @shchur and @abdulfatir for the discussion.

1. I think what @siriuz42 was saying that if model A has leakage on benchmark i and model B does not; further model A's score is better (or loss is lower) than model B then instead of declaring model A wins we can declare model A is tied with model B.
2. On the other hand I agree that this will not work for the skill score chart. So I actually do not have any better ideas, may be what you have decided is a good compromise. @siriuz42 any thoughts?
3. We are still making some changes to the package (not the model); would it be possible to hold off till the latest package update. If it is easier we can also follow the official process of submitting to the benchmark via a pull request?

[shchur]

@rajatsen91 thanks, regarding 3: sure, we can wait until the inference code is updated. I don't mind re-running the eval myself to ensure that we use the same hardware / GPU - this way the inference times will remain comparable at least for the initial version of the benchmark. Is it okay if I add you as a reviewer on the TimesFM-2.5 model wrapper, and then wait for your green light before re-running the benchmark and merging?

[rajatsen91]

sure sounds good.

[siriuz42]

Thanks @shchur for the clarification! And the change list to TimesFM looks good to me. Our model wrapper should be stable now - let's wait for @rajatsen91's green light.

Agreed with both you and Rajat about the leakage handling and imputation. As Rajat has explained, we could drop imputation for pairwise comparison (either winrate or skill score) if we are OK with a rule that per dataset a leaky model can only tie with or lose to a nonleaky model.

I want to make a point that imputation itself is more confusing than no imputation provided the leakage percentage is already shown to the audience. It is hard to claim "imputing by an a priori fixed zero-shot model does not penalize the leaky models too much" - we actually have no idea how this reflects the counterfactual if the leaky models were trained to be unleaky for that task. Consider a linear regression fit on all fev bench training splits - this model is likely garbage but would appear as good as Chronos-Bolt on fev-bench, which looks wrong to me.

[shchur]

@rajatsen91 please have a look at https://github.com/autogluon/fev/pull/53/files and let me know if the updated code looks good to you.