multilabel classification: only integer tensors of a single element can be converted to an index when training and predicating models

[Veronicacz]

I've faced the following issues with the multi-label classification functionality when use_multiprocessing=False is set. This affects both training and prediction workflows.

Bug Description

Location: simpletransformers/classification/classification_utils.py, line 344

Issue: The build_classification_dataset function incorrectly uses torch.long dtype for multi-label classification when multiprocessing is disabled:

if output_mode == "classification":
    labels = torch.tensor(labels, dtype=torch.long)  # BUG: Should be torch.float for multi-label
elif output_mode == "regression":
    labels = torch.tensor(labels, dtype=torch.float)

Error Details

Error Message:

TypeError: only integer tensors of a single element can be converted to an index

Stack Trace:

File "simpletransformers/classification/classification_utils.py", line 344, in build_classification_dataset
    labels = torch.tensor(labels, dtype=torch.long)
TypeError: only integer tensors of a single element can be converted to an index

Reproduction Steps

1. Create multi-label training data with labels as lists: [[1,0,1,0], [0,1,0,1], ...]
2. Set model args with use_multiprocessing=False and use_multiprocessing_for_evaluation=False
3. Call model.train_model() or model.predict()
4. Error occurs in build_classification_dataset function

Possible Root Cause

The bug occurs might be due to:

1. Multi-label data requires torch.float dtype, not torch.long
2. The function doesn't distinguish between single-label and multi-label classification
3. When multiprocessing is enabled, the data goes through a different processing pipeline that handles this correctly. But it causes another hanging issues.
4. When multiprocessing is disabled, it hits the broken build_classification_dataset function

Additional Info

• Training fails for multi-label classification with disabled multiprocessing
• Prediction fails even when training succeeds (using HuggingFace datasets workaround)
• Forces users to enable multiprocessing, which can cause deadlocks and hanging issues
• Inconsistent behavior between multiprocessing enabled/disabled modes

Sample code I am using:

import logging

import pandas as pd

from simpletransformers.classification import (
    MultiLabelClassificationModel,
    MultiLabelClassificationArgs,
)

logging.basicConfig(level=logging.INFO)
transformers_logger = logging.getLogger("transformers")
transformers_logger.setLevel(logging.WARNING)


# Train and Evaluation data needs to be in a Pandas Dataframe of two columns. The first column is the text with type str, and the second column in the label with type int.
train_data = [["Example sentence 1 for multilabel", [1, 1, 1, 1, 0, 1]]] + [
    ["This thing is entirely different from the other thing. ", [0, 1, 1, 0, 0, 0]]
]
train_df = pd.DataFrame(train_data, columns=["text", "labels"])


eval_data = [
    ["Example sentence belonging to class 1", [1, 1, 1, 1, 0, 1]],
    [
        "This thing should be entirely different from the other thing. ",
        [0, 0, 0, 0, 1, 0],
    ],
]
eval_df = pd.DataFrame(eval_data, columns=["text", "labels"])
model_args = MultiLabelClassificationArgs(
    num_train_epochs = 1, 
    overwrite_output_dir = True,
    train_batch_size = 8, # 32,
    eval_batch_size=8,    # Smaller eval batch

    use_multiprocessing=False,
    use_multiprocessing_for_evaluation=False,
    dataloader_num_workers=0,
    process_count = 1,
    save_steps = -1, # set -1 to disable
    save_eval_checkpoints = False, 
    save_model_every_epoch = False,
    # Force single-process data handling
    no_cache=True,
    reprocess_input_data=True,
    # fp16 = False
)

# Create a MultiLabelClassificationModel
model = MultiLabelClassificationModel(
    'roberta',
    'distilroberta-base', 
    num_labels= 6, 
    use_cuda = True, 
    args= model_args
)

# Train the model
model.train_model(train_df)

# # Evaluate the model
result, model_outputs, wrong_predictions = model.eval_model(eval_df)
transformers_logger.info(result)

# print("\nmodel_outputs from model.eval_model()")
# print(model_outputs)

predictions, raw_outputs = model.predict(
    [
        "Example sentence belonging to class 1",
        "This thing should be entirely different from the other thing.",
        "This thing should be entirely different from the other thing.",
    ]
)
transformers_logger.info(predictions)
transformers_logger.info(raw_outputs)
exit()

predictions, raw_outputs = model.predict(
    [
        "Example sentence belonging to class 1",
        "This thing should be entirely different from the other thing.",
    ]
)

transformers_logger.info("\npredictions from model.predict()")
transformers_logger.info(predictions)

transformers_logger.info("\nraw_outputs from model.predict()")
transformers_logger.info(raw_outputs)


model.args.threshold = [0.9, 0.6, 0.5, 0.3, 0.2, 0.5]
transformers_logger.info("\nAfter variable thresholds.")

predictions, raw_outputs = model.predict(
    [
        "Example sentence belonging to class 1",
        "This thing should be entirely different from the other thing.",
    ]
)

transformers_logger.info("\npredictions from model.predict()")
transformers_logger.info(predictions)

transformers_logger.info("\nraw_outputs from model.predict()")
transformers_logger.info(raw_outputs)

If I force to use HuggingFace datasets workaround and set use_hf_datasets=True, then model.train() succeeded but it failed with prediction

Traceback (most recent call last):
File "train.py", line 126, in
predictions, raw_outputs = model.predict(
^^^^^^^^^^^^^^
File "lib/python3.12/site-packages/simpletransformers/classification/multi_label_classification_model.py", line 406, in predict
return super().predict(to_predict, multi_label=multi_label)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/lib/python3.12/site-packages/simpletransformers/classification/classification_model.py", line 2160, in predict
eval_dataset = self.load_and_cache_examples(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/lib/python3.12/site-packages/simpletransformers/classification/multi_label_classification_model.py", line 384, in load_and_cache_examples
return super().load_and_cache_examples(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/lib/python3.12/site-packages/simpletransformers/classification/classification_model.py", line 1890, in load_and_cache_examples
dataset = ClassificationDataset(
^^^^^^^^^^^^^^^^^^^^^^
File "/lib/python3.12/site-packages/simpletransformers/classification/classification_utils.py", line 369, in init
self.examples, self.labels = build_classification_dataset(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/lib/python3.12/site-packages/simpletransformers/classification/classification_utils.py", line 344, in build_classification_dataset
labels = torch.tensor(labels, dtype=torch.long)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: only integer tensors of a single element can be converted to an index

Here are the packages that I have on my machine:

    - torch==2.7.1
    - torchvision==0.22.1
    - torchaudio==2.7.1
    - --extra-index-url https://download.pytorch.org/whl/cu126

    - transformers==4.56.0
    - simpletransformers==0.70.5

@ThilinaRajapakse I really appreciate if you could provide any guidance on this. Thank you for maintaining this excellent library! I'm happy to provide additional details or testing if needed.

[Veronicacz]

Hi @ThilinaRajapakse,
Is there a way to force to use HuggingFace datasets for model.predict too to bypass the broken function build_classification_dataset()?

Currently,

• train_model() with use_hf_datasets=True uses load_hf_dataset() (line 585) ✅ Works

• predict() always uses load_and_cache_examples() → ClassificationDataset → build_classification_dataset() ❌ Broken

[Ulrixon]

I am able to fix the code and make the model_train process and it work correctly with use_multiprocessing=False is set by changing the code in simpletransformers/classification/classification_utils.py

if output_mode == "classification":
    labels = torch.tensor(labels, dtype=torch.long)  # BUG: Should be torch.float for multi-label
elif output_mode == "regression":
    labels = torch.tensor(labels, dtype=torch.float)

by using

    labels = np.array(labels)
    labels = torch.tensor(labels, dtype=torch.float)

to replace this line:
labels = torch.tensor(labels, dtype=torch.long)
make sure to import numpy as np on the top of document.

Not really sure if this code will work for other dataset.
Hope this will be helpful to other people.