[BUG] SwitchTransformersConfig creates sparse layer when num_sparse_encoder_layers=0 with single layer model

[harshaljanjani]

System Info

• transformers version: 5.0.0.dev0
• Platform: Linux-5.15.167.4-microsoft-standard-WSL2-x86_64-with-glibc2.39
• Python version: 3.12.3
• huggingface_hub version: 1.3.2
• safetensors version: 0.7.0
• accelerate version: 1.12.0
• Accelerate config: not installed
• DeepSpeed version: not installed
• PyTorch version (accelerator?): 2.9.1+cu128 (CUDA)
• GPU type: NVIDIA GeForce RTX 4060 Laptop GPU

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

from transformers import SwitchTransformersConfig, SwitchTransformersModel

config = SwitchTransformersConfig(
    num_layers=1,
    num_sparse_encoder_layers=0,
    num_decoder_layers=1,
    num_sparse_decoder_layers=0,
    vocab_size=100,
    d_model=64,
    d_ff=128,
    num_heads=4,
    d_kv=16
)
model = SwitchTransformersModel(config)
encoder_sparse_count = sum(
    1 for block in model.encoder.block if block.is_sparse
)
print(f"Encoder sparse layers: {encoder_sparse_count}")

The bug is in configuration_switch_transformers.py (lines 151 and 157). When num_sparse_encoder_layers=0, the code sets encoder_sparse_step = num_layers (marked with a HACK comment). Combined with the modeling logic in line 668 → when num_layers=1, sparse_step=1 triggers the sparse_step==1 condition, and incorrectly creates a sparse layer.

Expected behavior

When num_sparse_encoder_layers=0 is set, zero sparse layers should be created regardless of num_layers value.

cc: @Rocketknight1 @sayakpaul

[Rocketknight1]

Hmn, are there any checkpoints that take this path? Switch Transformers are quite old now, and I don't think many new architectures are being trained.

[harshaljanjani]

@Rocketknight1 Took some time to browse the configs, I see that num_sparse_encoder_layers=0 or num_sparse_decoder_layers=0 isn’t explicitly used in them. The PR still helps for custom instances with num_sparse_{*}_layers=0, ensuring the config behaves as documented.
Said that, I understand if this is low-priority given Switch Transformers’ age and the lack of checkpoints hitting this path -- happy to close, or I can add a docstring note for the behavior instead.
Also, the PR also updates an outdated comment in audio_utils.py; we can keep that and drop the rest if preferred.

[Rocketknight1]

@harshaljanjani I think I'd prefer not to add support for an unused path, since it probably won't get tested properly anyway. How about just a docstring note plus the audio utils change, then rename the PR and ping me to review/merge?

[harshaljanjani]

@Rocketknight1 Made the changes, and the PR is ready for review!