[BUG] TopK metrics are wrong when model predicts same value for all items.

[bschifferer]

Bug description

When the model predicts a constant value, the topk metrics scores perfect.

When the model predicts the same value for all items, the sort order doesn't change
https://github.com/NVIDIA-Merlin/models/blob/main/tests/unit/tf/metrics/test_metrics_topk.py#L53

The negative sampler creates the batches by putting the positive item on the first position and the negatives on the next ones:

models/merlin/models/tf/blocks/retrieval/base.py

Line 363 in 7f6170c

predictions = tf.concat([positive_scores, negative_scores], axis=-1)

In that way, we get perfect scores by predicting a constant.

Steps/Code to reproduce bug

Based on the unttest

labels = tf.convert_to_tensor([[0, 1, 0, 1, 0], [1, 0, 0, 1, 0], [0, 0, 0, 0, 1]], tf.float32)
predictions = tf.convert_to_tensor(
    [[10, 10, 10, 10, 10], [5, 5, 5, 5, 5], [6, 6, 6, 6, 6]], tf.float32
)
label_relevant_counts = tf.convert_to_tensor([2, 2, 1], tf.float32)
print(labels)
predictions, labels, label_relevant_counts = extract_topk(5, predictions, labels)
print(labels)

[bschifferer]

If a model predicts a lot of 0s, then this behavior gets worse

[angmc]

There are widely different results when comparing to tfr.keras.metrics.MRRMetric

in the sort_by_score function they are accounting for ties by shuffling. It might be worth implementing something similar.

https://github.com/tensorflow/ranking/blob/de7d7c2a8b8663a9d813a68ef036ef0c7585ab37/tensorflow_ranking/python/utils.py#L110

[rnyak]

current workaround is to add noise to break ties.. The problem itself is the ties. This is a critical issue. @bschifferer can you add the code to this issue as a reference?

[bschifferer]

    def update_state(
        self, y_true: tf.Tensor, y_pred: tf.Tensor, sample_weight: Optional[tf.Tensor] = None
    ):
        # Extractubg sorted top-k prediction scores and labels only ONCE
        # so that sorting does not need to happen for each individual metric
        # (as the top-k metrics have been set with pre_sorted=True in this constructor
        y_pred = y_pred + tf.random.uniform(tf.shape(y_pred))*0.000001

We add random noise to it. This might not solve the issue. If the models collaps (really high learning rate) and the models predicts only tf.nan, then this hack might not work