Issues when calling model.predict(input)

[Veronicacz]

Describe the bug
When attempting to call model.predict(input), a ValueError pops up that I haven't encountered before with SimpleTransformers.

To Reproduce
transformers==4.55.4, simpletransformers==0.70.5

Sample code:

model_args = MultiLabelClassificationArgs(
    num_train_epochs=N_EPOCHS,
    overwrite_output_dir=True,
    train_batch_size=32,
    output_dir=args.output_path,
    use_multiprocessing=False,
    use_multiprocessing_for_evaluation=False,
    save_steps = -1, # set -1 to disable
    save_eval_checkpoints = False, 
    save_model_every_epoch = False
)

logger.info('model args: ')
logger.info(model_args)

model = MultiLabelClassificationModel(
    'roberta',
    'distilroberta-base',
    use_cuda=True,
    args=model_args,
    num_labels=1
)

# Train the model
model.train_model(train)

test_data = list(test['Text'])
logger.info(f'Test len: {test.shape}')   # Test len: (66, 2)

# Evaluate on test set
predictions, raw_outputs = model.predict(test_data)

Screenshots

Predicting: 0%| | 0/1 [00:00<?, ?it/s]
Traceback (most recent call last):
File "train.py", line 135, in
predictions, raw_outputs = model.predict(test_data)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/lib/python3.12/site-packages/simpletransformers/classification/multi_label_classification_model.py", line 406, in predict
return super().predict(to_predict, multi_label=multi_label)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/lib/python3.12/site-packages/simpletransformers/classification/classification_model.py", line 2295, in predict
out_label_ids[start_index:end_index] = (
~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
ValueError: could not broadcast input array from shape (66,) into shape (66,1)

Desktop (please complete the following information):

• Ubuntu

Additional context
My dataset has two columns: text and labels

@ThilinaRajapakse Thank you so much for fixing the previous issues, I really appreciate it!

I ran into this new issue when trying to make predictions with the trained model. I’m not sure if it’s something specific to my environment, but I didn’t encounter this before, and I haven’t changed any code or packages aside from upgrading these two (transformers==4.55.4, simpletransformers==0.70.5).

Happy to share more details if needed!

[ThilinaRajapakse]

I don't seem to be encountering this issue. Here's the script I ran to test.

import logging

import pandas as pd

from simpletransformers.classification import (
    MultiLabelClassificationModel,
    MultiLabelClassificationArgs,
)

logging.basicConfig(level=logging.INFO)
transformers_logger = logging.getLogger("transformers")
transformers_logger.setLevel(logging.WARNING)


# Train and Evaluation data needs to be in a Pandas Dataframe of two columns. The first column is the text with type str, and the second column in the label with type int.
train_data = [["Example sentence 1 for multilabel", [1, 1, 1, 1, 0, 1]]] + [
    ["This thing is entirely different from the other thing. ", [0, 1, 1, 0, 0, 0]]
]
train_df = pd.DataFrame(train_data, columns=["text", "labels"])


eval_data = [
    ["Example sentence belonging to class 1", [1, 1, 1, 1, 0, 1]],
    [
        "This thing should be entirely different from the other thing. ",
        [0, 0, 0, 0, 1, 0],
    ],
]
eval_df = pd.DataFrame(eval_data, columns=["text", "labels"])

model_args = MultiLabelClassificationArgs()
model_args.train_custom_parameters_only = True
model_args.reprocess_input_data = True
model_args.overwrite_output_dir = True
model_args.learning_rate = 3e-3
model_args.custom_parameter_groups = [
    {
        "params": ["classifier.weight"],
        "lr": 1e-3,
    },
    {
        "params": ["classifier.bias"],
        "lr": 1e-3,
        "weight_decay": 0.0,
    },
    {
        "params": ["classifier_1.weight"],
        "lr": 1e-3,
    },
    {
        "params": ["classifier_1.bias"],
        "lr": 1e-3,
        "weight_decay": 0.0,
    },
]

# train_args = {
#     "reprocess_input_data": True,
#     "overwrite_output_dir": True,
#     "evaluate_during_training": True,
#     "evaluate_during_training_verbose": True,
#     "logging_steps": 50,
#     "num_train_epochs": 1,
#     # "evaluate_during_training_steps": 1,
#     "best_model_dir": "TESTING BEST",
#     "wandb_project": "test-new-project",
#     "no_cache": True,
#     "use_early_stopping": True,
#     "max_seq_length": 3,
#     # "sliding_window": True,
#     # "fp16": False,
# }

# Create a MultiLabelClassificationModel
model = MultiLabelClassificationModel(
    "roberta",
    "distilbert/distilroberta-base",
    num_labels=6,
    use_cuda=False,
    args=model_args,
    cuda_device=1,
)

# Train the model
model.train_model(train_df)

# # Evaluate the model
result, model_outputs, wrong_predictions = model.eval_model(eval_df)
print(result)

# print("\nmodel_outputs from model.eval_model()")
# print(model_outputs)

predictions, raw_outputs = model.predict(
    [
        "Example sentence belonging to class 1",
        "This thing should be entirely different from the other thing.",
        "This thing should be entirely different from the other thing.",
    ]
)
print(predictions)
print(raw_outputs)
exit()

predictions, raw_outputs = model.predict(
    [
        "Example sentence belonging to class 1",
        "This thing should be entirely different from the other thing.",
    ]
)

print("\npredictions from model.predict()")
print(predictions)

print("\nraw_outputs from model.predict()")
print(raw_outputs)


model.args["threshold"] = [0.9, 0.6, 0.5, 0.3, 0.2, 0.5]
print("\nAfter variable thresholds.")

predictions, raw_outputs = model.predict(
    [
        "Example sentence belonging to class 1",
        "This thing should be entirely different from the other thing.",
    ]
)

print("\npredictions from model.predict()")
print(predictions)

print("\nraw_outputs from model.predict()")
print(raw_outputs)

This yields:

INFO:simpletransformers.classification.classification_utils: Converting to features started. Cache is not used.
1it [00:00, 26.55it/s]
INFO:simpletransformers.classification.classification_utils: Saving features into cached file cache_dir/cached_train_roberta_128_0_2
Epochs 1/1. Running Loss:    0.6859: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.59it/s]
Epoch 1 of 1: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:01<00:00,  1.35s/it]
INFO:simpletransformers.classification.classification_model: Training of roberta model complete. Saved to outputs/.
INFO:simpletransformers.classification.classification_utils: Converting to features started. Cache is not used.
1it [00:00, 30.46it/s]
INFO:simpletransformers.classification.classification_utils: Saving features into cached file cache_dir/cached_dev_roberta_128_0_2
Running Evaluation: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 21.91it/s]
{'LRAP': 0.5883333333333334, 'eval_loss': 0.6939830780029297}
INFO:simpletransformers.classification.classification_utils: Converting to features started. Cache is not used.
1it [00:00, 30.77it/s]
Predicting: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 18.16it/s]
[[0, 1, 0, 1, 0, 0], [0, 1, 0, 1, 0, 0], [0, 1, 0, 1, 0, 0]]
[[0.4839192  0.5008688  0.49631941 0.52659863 0.4923515  0.48482797]
 [0.48650301 0.50185305 0.49392578 0.52900618 0.49298042 0.48755068]
 [0.48650301 0.50185305 0.49392578 0.52900618 0.49298042 0.48755068]]

[Veronicacz]

Hi @ThilinaRajapakse ,

I really appreciate your response. And sorry for not updating this issue in the past week — I was focused on something else. I’ve now revisited it, resolved the above issue but ran into two blocking problems while training a MultiLabelClassificationModel (roberta, distilroberta-base) with simpletransformers:

• Multiprocessing enabled (use_multiprocessing=True, use_multiprocessing_for_evaluation=True):

  Training hangs indefinitely and does not progress with the below args.

model_args = MultiLabelClassificationArgs(
    num_train_epochs = 5, 
    overwrite_output_dir = True,
    train_batch_size = 32,
    save_steps = -1, # set -1 to disable
    save_eval_checkpoints = False, 
    save_model_every_epoch = False
)

model = MultiLabelClassificationModel(
    'roberta',
    'distilroberta-base', 
    num_labels= 38, 
    use_cuda = True, 
    args= model_args
)

• Multiprocessing disabled (False):

Training starts, but fails with the following error:

Traceback (most recent call last):
File "/train.py", line 181, in
model.train_model( train )
File "/lib/python3.12/site-packages/simpletransformers/classification/multi_label_classification_model.py", line 326, in train_model
return super().train_model(
^^^^^^^^^^^^^^^^^^^^
File "/lib/python3.12/site-packages/simpletransformers/classification/classification_model.py", line 644, in train_model
train_dataset = self.load_and_cache_examples(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/lib/python3.12/site-packages/simpletransformers/classification/multi_label_classification_model.py", line 384, in load_and_cache_examples
return super().load_and_cache_examples(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/lib/python3.12/site-packages/simpletransformers/classification/classification_model.py", line 1890, in load_and_cache_examples
dataset = ClassificationDataset(
^^^^^^^^^^^^^^^^^^^^^^
File "/lib/python3.12/site-packages/simpletransformers/classification/classification_utils.py", line 369, in init
self.examples, self.labels = build_classification_dataset(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/lib/python3.12/site-packages/simpletransformers/classification/classification_utils.py", line 344, in build_classification_dataset
labels = torch.tensor(labels, dtype=torch.long)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: only integer tensors of a single element can be converted to an index

And here are the format of sample labels:

Sample labels:
79218 [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...
414 [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, ...
47377 [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...
46266 [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...
53731 [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ...
Name: labels, dtype: object
Label types:
[<class 'list'>, <class 'list'>, <class 'list'>, <class 'list'>, <class 'list'>]
First label shape/content:
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0]

Before upgrading the package, the same training script worked with the same dataset (as long as multiprocessing was disabled).

Have you seen similar issues before, or could you provide guidance on how to resolve them? In particular:

• Why multiprocessing causes the training process to freeze.

• Why the dataset loader is now misinterpreting the labels tensor after the upgrade.

Thanks again!