Index error when trying to re-run training

[michelnv26]

I'm working on a training model a former graduate student was trying to create that tracks ferrets in our experiments. We've been having issues with identity swapping between the ferrets, particularly when the ferrets are on top of each other and SLEAP can't see all the nodes. I think it's mainly due to there only being 5 nodes on the skeleton, so when the ferrets are on top of each other, SLEAP can only see 2 nodes and can switch identities easily.

So far, I've been going through the training videos and fixing the tracks when I see the identity switch. I tried to rerun the training with the fixes but I get a index error that says the list index is out of range. I'm not familiar with coding, so I'm not sure how to fix the problem. Does anyone have any suggestions on both the identity switching and the index error?

[gitttt-1234]

Hi @michelnv26 !

Could you please share the full error message for the index error?

For the identity switching: tracking quality relies a lot on how well the keypoints are detected in each frame. and you're right, if the pose predictions are poor or missing nodes, tracking can easily swap identities or create extra tracks. If you’re running the simple or flow-based tracker after inference (i.e. not using an ID model), you don’t need to manually correct track IDs in the labels — those tracks are not used for training, only the keypoints are.

Could you also share:

• Which tracker you’re using (simple / flow-based and the tracker settings)?

• Which SLEAP version you’re running?

Thanks,

Divya

[michelnv26]

Hi,

Here's the full script below. We're using a simple tracker and I've attached the training configurations here (if that's what the tracker settings are). We're using version 1.4.1a2.

Software versions:
SLEAP: 1.4.1a2
TensorFlow: 2.7.0
Numpy: 1.21.6
Python: 3.7.12
OS: Linux-6.8.0-85-generic-x86_64-with-debian-bookworm-sid

Happy SLEAPing! :)
Resetting monitor window.
Polling: /home/lakdawala_lab/Documents/SLEAP/Deb's SLEAP files/models/1758_bu_id251123_174954.multi_class_bottomup.n=4608/viz/validation.*.png
Start training multi_class_bottomup...
['sleap-train', '/tmp/tmpvirlnpwo/251123_174955_training_job.json', "/home/lakdawala_lab/Documents/SLEAP/Deb's SLEAP files/Michelle_labels_20251105.slp", '--zmq', '--controller_port', '9000', '--publish_port', '9001', '--save_viz']
INFO:sleap.nn.training:Versions:
SLEAP: 1.4.1a2
TensorFlow: 2.7.0
Numpy: 1.21.6
Python: 3.7.12
OS: Linux-6.8.0-85-generic-x86_64-with-debian-bookworm-sid
INFO:sleap.nn.training:Training labels file: /home/lakdawala_lab/Documents/SLEAP/Deb's SLEAP files/Michelle_labels_20251105.slp
INFO:sleap.nn.training:Training profile: /tmp/tmpvirlnpwo/251123_174955_training_job.json
INFO:sleap.nn.training:
INFO:sleap.nn.training:Arguments:
INFO:sleap.nn.training:{
"training_job_path": "/tmp/tmpvirlnpwo/251123_174955_training_job.json",
"labels_path": "/home/lakdawala_lab/Documents/SLEAP/Deb's SLEAP files/Michelle_labels_20251105.slp",
"video_paths": [
""
],
"val_labels": null,
"test_labels": null,
"base_checkpoint": null,
"tensorboard": false,
"save_viz": true,
"zmq": true,
"publish_port": 9001,
"controller_port": 9000,
"run_name": "",
"prefix": "",
"suffix": "",
"cpu": false,
"first_gpu": false,
"last_gpu": false,
"gpu": "auto"
}
INFO:sleap.nn.training:
INFO:sleap.nn.training:Training job:
INFO:sleap.nn.training:{
"data": {
"labels": {
"training_labels": null,
"validation_labels": null,
"validation_fraction": 0.1,
"test_labels": null,
"split_by_inds": false,
"training_inds": null,
"validation_inds": null,
"test_inds": null,
"search_path_hints": [],
"skeletons": []
},
"preprocessing": {
"ensure_rgb": false,
"ensure_grayscale": false,
"imagenet_mode": null,
"input_scaling": 1.0,
"pad_to_stride": null,
"resize_and_pad_to_target": true,
"target_height": null,
"target_width": null
},
"instance_cropping": {
"center_on_part": null,
"crop_size": null,
"crop_size_detection_padding": 16
}
},
"model": {
"backbone": {
"leap": null,
"unet": {
"stem_stride": null,
"max_stride": 16,
"output_stride": 2,
"filters": 64,
"filters_rate": 2.0,
"middle_block": true,
"up_interpolate": false,
"stacks": 1
},
"hourglass": null,
"resnet": null,
"pretrained_encoder": null
},
"heads": {
"single_instance": null,
"centroid": null,
"centered_instance": null,
"multi_instance": null,
"multi_class_bottomup": {
"confmaps": {
"part_names": null,
"sigma": 5.0,
"output_stride": 2,
"loss_weight": 1.0,
"offset_refinement": false
},
"class_maps": {
"classes": [
"dark blue",
"green",
"light blue",
"pink",
"red",
"purple",
"yellow"
],
"sigma": 15.0,
"output_stride": 16,
"loss_weight": 2.0
}
},
"multi_class_topdown": null
},
"base_checkpoint": null
},
"optimization": {
"preload_data": true,
"augmentation_config": {
"rotate": false,
"rotation_min_angle": -180.0,
"rotation_max_angle": 180.0,
"translate": false,
"translate_min": -5,
"translate_max": 5,
"scale": false,
"scale_min": 0.9,
"scale_max": 1.1,
"uniform_noise": false,
"uniform_noise_min_val": 0.0,
"uniform_noise_max_val": 10.0,
"gaussian_noise": false,
"gaussian_noise_mean": 5.0,
"gaussian_noise_stddev": 1.0,
"contrast": false,
"contrast_min_gamma": 0.5,
"contrast_max_gamma": 2.0,
"brightness": false,
"brightness_min_val": 0.0,
"brightness_max_val": 10.0,
"random_crop": false,
"random_crop_height": 256,
"random_crop_width": 256,
"random_flip": true,
"flip_horizontal": false
},
"online_shuffling": true,
"shuffle_buffer_size": 128,
"prefetch": true,
"batch_size": 1,
"batches_per_epoch": null,
"min_batches_per_epoch": 200,
"val_batches_per_epoch": null,
"min_val_batches_per_epoch": 10,
"epochs": 10,
"optimizer": "adam",
"initial_learning_rate": 0.0001,
"learning_rate_schedule": {
"reduce_on_plateau": true,
"reduction_factor": 0.5,
"plateau_min_delta": 1e-06,
"plateau_patience": 5,
"plateau_cooldown": 3,
"min_learning_rate": 1e-08
},
"hard_keypoint_mining": {
"online_mining": false,
"hard_to_easy_ratio": 2.0,
"min_hard_keypoints": 2,
"max_hard_keypoints": null,
"loss_scale": 5.0
},
"early_stopping": {
"stop_training_on_plateau": true,
"plateau_min_delta": 1e-06,
"plateau_patience": 10
}
},
"outputs": {
"save_outputs": true,
"run_name": "251123_174954.multi_class_bottomup.n=4608",
"run_name_prefix": "1758_bu_id",
"run_name_suffix": "",
"runs_folder": "/home/lakdawala_lab/Documents/SLEAP/Deb's SLEAP files/models",
"tags": [
""
],
"save_visualizations": true,
"delete_viz_images": true,
"zip_outputs": false,
"log_to_csv": true,
"checkpointing": {
"initial_model": false,
"best_model": true,
"every_epoch": false,
"latest_model": false,
"final_model": false
},
"tensorboard": {
"write_logs": false,
"loss_frequency": "epoch",
"architecture_graph": false,
"profile_graph": false,
"visualizations": true
},
"zmq": {
"subscribe_to_controller": true,
"controller_address": "tcp://127.0.0.1:9000",
"controller_polling_timeout": 10,
"publish_updates": true,
"publish_address": "tcp://127.0.0.1:9001"
}
},
"name": "",
"description": "",
"sleap_version": "1.4.1a2",
"filename": "/tmp/tmpvirlnpwo/251123_174955_training_job.json"
}
INFO:sleap.nn.training:
2025-11-23 17:49:59.739283: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1932] Ignoring visible gpu device (device: 0, name: Quadro P400, pci bus id: 0000:02:00.0, compute capability: 6.1) with core count: 2. The minimum required count is 8. You can adjust this requirement with the env var TF_MIN_GPU_MULTIPROCESSOR_COUNT.
INFO:sleap.nn.training:Auto-selected GPU 1 with 12196 MiB of free memory.
Traceback (most recent call last):
File "/home/lakdawala_lab/miniconda3/envs/sleap/bin/sleap-train", line 33, in
sys.exit(load_entry_point('sleap==1.4.1a2', 'console_scripts', 'sleap-train')())
File "/home/lakdawala_lab/miniconda3/envs/sleap/lib/python3.7/site-packages/sleap/nn/training.py", line 2029, in main
trainer = create_trainer_using_cli(args=args)
File "/home/lakdawala_lab/miniconda3/envs/sleap/lib/python3.7/site-packages/sleap/nn/training.py", line 2004, in create_trainer_using_cli
sleap.nn.system.use_gpu(gpu_ind)
File "/home/lakdawala_lab/miniconda3/envs/sleap/lib/python3.7/site-packages/sleap/nn/system.py", line 72, in use_gpu
tf.config.set_visible_devices(gpus[device_ind], "GPU")
IndexError: list index out of range
Run Path: /home/lakdawala_lab/Documents/SLEAP/Deb's SLEAP files/models/1758_bu_id251123_174954.multi_class_bottomup.n=4608

[gitttt-1234]

Hi @michelnv26 !

You are using an ID model, so the tracking labels are used for training — you’re right about correcting them before training. The index error you’re seeing seems to be related to TensorFlow’s GPU detection. Could you check how many GPUs are recognized by the system and by TensorFlow?

In a terminal,

nvidia-smi

and then:

conda activate sleap
python
>>> import tensorflow as tf
>>> tf.config.list_physical_devices("GPU")

Thanks,

Divya