[BUG] Single k in top_ks metric leads to an error caused by iteration over 0-d tensor

[juriwiens]

Bug description

Using a single k as top_ks parameter to a metric like e.g. RecallAt(top_ks=[4]) leads to a TypeError: iteration over a 0-d tensor when zip(metric, topks[name]) is run in the NextItemPredictionTask.compute_metrics method because metric is a 0-d (torch) tensor in this case.

Steps/Code to reproduce bug

1. Import the torch variant and create a NextItemPredictionTask with a metric using a single value top_ks like e.g.:

from transformers4rec import torch as tr

prediction_task = tr.NextItemPredictionTask(
    hf_format=True,
    weight_tying=True,
    metrics=[tr.ranking_metric.RecallAt(top_ks=[4], labels_onehot=True)]
)

2. Use prediction_task to run a training with a Trainer having compute_metrics=True and using the fit_and_evaluate function from examples_utils:

from transformers4rec.torch.utils.examples_utils import fit_and_evaluate

input_module = tr.TabularSequenceFeatures.from_schema(
    schema,
    max_sequence_length=20,
    aggregation="concat",
    d_output=64,
    masking="clm",
)

transformer_config = tr.XLNetConfig.build(
    d_model=64, n_head=1, n_layer=1, total_seq_length=20
)

model = transformer_config.to_torch_model(input_module, prediction_task)

training_args = tr.trainer.T4RecTrainingArguments(
    output_dir="./tmp",
    max_sequence_length=20,
    data_loader_engine='nvtabular',
    num_train_epochs=10,
    dataloader_drop_last=False,
    per_device_train_batch_size = 384,
    per_device_eval_batch_size = 512,
    learning_rate=0.0005,
    fp16=True,
    report_to = [],
    logging_steps=200
)

recsys_trainer = tr.Trainer(
    model=model,
    args=training_args,
    schema=schema,
    compute_metrics=True
)

# SESSIONS_BY_DAY_PATH = ...

aot_results = fit_and_evaluate(
    recsys_trainer,
    start_time_index=1,
    end_time_index=3,
    input_dir=SESSIONS_BY_DAY_PATH
)

Expected behavior

Metrics evaluation should not lead to an error.

Environment details

• Transformers4Rec version: 0.1.8
• Platform: linux
• Python version: 3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 06:04:10) \n[GCC 10.3.0]
• Huggingface Transformers version: 4.12.0
• PyTorch version (GPU?): 1.12.0a0+bd13bc6
• Tensorflow version (GPU?): /

Additional context

Maybe just metric.unsqueeze(0) if metric.dim() == 0? :)

[EvenOldridge]

@gabrielspmoreira @sararb any thoughts on what might be going on?

[gabrielspmoreira]

Hi @juriwiens . Thanks for placing this PR. In fact, it would be more intuitive for the user if the argument top_ks of top-k metrics (e.g. Recall(top_ks=[5,10]), accept both a list of cut-offs or a single cut-off.
The fix should be simple: as all top-k metrics inherit from RankingMetric, we could do something before this line like

if not isinstance(self.top_ks, (list,tuple)):
    self.top_ks = [self.top_ks]

If you are interested to place a Pull Request with that fix, we would be glad to merge it. :)

[juriwiens]

Hi @gabrielspmoreira. Thanks for the tips! I will create a PR as soon as I've figured out how to set up my local dev environment for this project :)

[juriwiens]

I was not able to reproduce the problem in my local dev environment, that's why I've dug a little deeper to find the cause:

• The RankingMetric inherits from the TorchMetric Metric base class which squeezes the result if it contains a single element since v0.7.0. This feature leads to the 0-d metric tensor and therefore to the error.
• Actually the torchmetrics version is pinned to v0.3.2
• But I'm using the merlin docker image nvcr.io/nvidia/merlin/merlin-pytorch-training:22.05 to run my code which comes with torchmetrics v0.8.2 installed.

How to deal with it? Maybe we should adapt the behaviour to the newer torchmetrics versions, even though these should not be used? Or have I misunderstood something? :)

[rnyak]

@juriwiens hello. can you please use merlin-pytorch:22.11 and pull the latest main branch of transformers4rec, core, nvtabular and dataloader repos, test if you still have the issue? we upgraded torchmetrics version recently.

you can do something like that to pull the latest main branches in the container instance:

cd /transformers4rec
git checkout main && git pull origin main && pip install .

cd /nvtabular
git checkout main && git pull origin main && pip install .

...