pose pca loss default bugfix (#381)

Author: themattinthehatt

.github/workflows/lint.yml

@@ -22,7 +22,11 @@ jobs:
           python-version: '3.10'
 
       - name: Install linters
-        run: pip install autopep8 flake8
+        run: pip install autopep8 flake8 isort
+
+      - name: Check import sorting with isort
+        run: isort --check-only --diff lightning_pose tests
+        # Reads config from [tool.isort] in pyproject.toml
 
       - name: Check formatting with autopep8
         run: autopep8 --diff --recursive --exit-code lightning_pose tests
@@ -31,6 +35,10 @@ jobs:
       - name: Lint with flake8 (critical errors only)
         run: flake8 lightning_pose tests --select=E9,F63,F7,F82
         # Reads config from .flake8 file
+        # F9: syntax errors, can't parse file
+        # F63: invalid ** use in `assert` or `raise`
+        # F7: syntax errors in type annotations
+        # F82: undefined names (var or function that was never imported)
 
       - name: Show fix instructions if formatting needed
         if: failure()
@@ -40,6 +48,7 @@ jobs:
           echo ""
           echo "To fix formatting issues locally, run:"
           echo "  autopep8 --in-place --recursive lightning_pose tests"
+          echo "  isort lightning_pose tests"
           echo ""
           echo "To check for flake8 errors locally, run:"
           echo "  flake8 lightning_pose tests --select=E9,F63,F7,F82"

README.md

@@ -10,9 +10,9 @@
 Lightning Pose is an end-to-end toolkit designed for robust multi-view and single-view animal
 pose estimation using advanced transformer architectures. It leverages Multi-View Transformers
 and patch-masking training to learn geometric relationships between views,
-resulting in strong performance on occlusions [Aharon, Lee et al. 2025](https://arxiv.org/abs/2510.09903).
-For single-view datasets  it leverages temporal context and learned plausibility constraints for strong performance
-in challenging scenarios [Biderman, Whiteway et al. 2024, Nature Methods](https://rdcu.be/dLP3z).
+resulting in strong performance on occlusions [Aharon, et al. 2025](https://arxiv.org/abs/2510.09903).
+For single-view datasets it leverages temporal context and learned plausibility constraints for 
+strong performance in challenging scenarios [Biderman, Whiteway et al. 2024, Nature Methods](https://rdcu.be/dLP3z).
 It has a rich GUI that supports the end-to-end workflow: labeling, model management, and evaluation.
 
 
@@ -64,10 +64,20 @@ a simple and performant post-processor that works with any pose estimation packa
 Lightning Pose, DeepLabCut, and SLEAP.
 
 Lightning Pose is primarily maintained by 
-[Karan Sikka](https://github.com/ksikka) (Columbia University),
-[Matt Whiteway](https://themattinthehatt.github.io) (Columbia University),
-and
-[Dan Biderman](https://dan-biderman.netlify.app) (Stanford University). 
+[Karan Sikka](https://github.com/ksikka) (Columbia University) and
+[Matt Whiteway](https://themattinthehatt.github.io) (Columbia University). 
 
 Lightning Pose is under active development and we welcome community contributions.
-Whether you want to implement some of your own ideas or help out with our [development roadmap](docs/roadmap.md), please get in touch with us on Discord (see contributing guidelines [here](CONTRIBUTING.md)). 
+Whether you want to implement some of your own ideas or help out with our [development roadmap](docs/roadmap.md), please get in touch with us on Discord (see contributing guidelines [here](CONTRIBUTING.md)).
+
+## Funding
+
+We are grateful for support from the following:
+* Gatsby Charitable Foundation GAT3708
+* [NIH R50NS145433](https://reporter.nih.gov/search/Hmj4KMmLv0evcYPlPEDa-Q/project-details/11240675)
+* [NIH U19NS123716](https://reporter.nih.gov/search/Hmj4KMmLv0evcYPlPEDa-Q/project-details/11141703)
+* [NSF 1707398](https://ui.adsabs.harvard.edu/abs/2017nsf....1707398A/abstract)
+* [The NSF AI Institute for Artificial and Natural Intelligence](https://ui.adsabs.harvard.edu/abs/2023nsf....2229929Z/abstract)
+* Simons Foundation
+* Wellcome Trust 216324
+* Zuckerman Institute (Columbia University) Team Science

docs/source/directory_structure_reference/model_config_file.rst

@@ -217,7 +217,7 @@ The following parameters relate to model architecture and unsupervised losses.
     * vits_dinov3: Vision Transformer (Small) pretrained on ImageNet with DINOv3
     * vitb_dino: Vision Transformer (Base) pretrained on ImageNet with DINO
     * vitb_dinov2: Vision Transformer (Base) pretrained on ImageNet with DINOv2
-    * vitb_dinov3: Vision Transformer (Base) pretrained on ImageNet with DINOv3
+    * vitb_dinov3: Vision Transformer (Base) pretrained on ImageNet with DINOv3; note this is a gated repo and you will need a Hugging Face account
     * vitb_imagenet: Vision Transformer (Base) pretrained on ImageNet with MAE loss
     * vitb_sam: Segment Anything Model (Vision Transformer Base)
 

docs/source/user_guide_multiview/patch_masking_3d_loss.rst

@@ -33,7 +33,8 @@ To encourage the model to develop this cross-view reasoning during training, we
 space patch masking scheme inspired by the success of masked autoencoders and dropout.
 We use a training curriculum that starts with a short warmup period where no patches are masked
 (controlled by ``training.patch_mask.init_epoch`` in the config file), then increase the ratio of
-masked patches over the course of training (controlled by ``training.patch_mask.init_ratio`` and ``training.patch_mask.final_ratio``).
+masked patches over the course of training
+(controlled by ``training.patch_mask.init_ratio`` and ``training.patch_mask.final_ratio``).
 This technique creates gradients that flow through the attention mechanism and encourage
 cross-view information propagation, which in turn develops internal representations that capture
 statistical relationships between the different views.
@@ -49,8 +50,14 @@ statistical relationships between the different views.
 
 To turn patch masking off, set ``final_ratio: 0.0``.
 
-3D augmentations and losses
-===========================
+3D augmentations and loss
+=========================
+
+.. note::
+
+    As of March 2026, the unsupervised losses introduced in the original Lightning Pose paper have
+    not yet been implemented for the ``multi-view transformer`` model, including the
+    ``pca_multiview`` loss.
 
 The MVT produces a 2D heatmap for each keypoint in each view.
 Without explicit geometric constraints, it is possible for these individual 2D predictions to be
@@ -61,14 +68,14 @@ encourage geometric consistency in the outputs
 formats for camera calibration; note also that bounding box information must be shared if the
 training images are cropped from larger frames).
 
-The 3D losses require geometrically consistent input images, which precludes applying geometric
+The 3D loss requires geometrically consistent input images, which precludes applying geometric
 augmentations like rotation to each view independently.
 Instead, we triangulate the ground truth labels and augment the 3D poses by translating and scaling in 3D space.
 The augmented 3D pose is then projected back to individual 2D views.
 These augmentations do not affect the camera parameters;
 rather, they are equivalent to keeping the cameras fixed and scaling and translating the subject within the scene.
 For each view, we then estimate the affine transformation from the original to augmented 2D keypoints,
-and apply this transformation to the original image
+and apply this transformation to the original image.
 
 To enable 3D augmentations, add the ``imgaug_3d`` field to the ``training`` section of your configuration
 file and set it to `true`:
@@ -79,30 +86,27 @@ file and set it to `true`:
         imgaug: dlc
         imgaug_3d: true
 
-Pairwise projection loss
-------------------------
-To compute the 3D pairwise projection loss, we first take the soft argmax of the 2D heatmaps to get predicted coordinates.
-Then, for each keypoint, and for each pair of views, we triangulate both the ground truth keypoints
-and the predictions, and compute the mean square error between the two.
-The 3D loss is weighted by a hyperparameter, which is set in the ``losses`` section of the
-configuration file:
+To compute the 3D reprojection loss, we:
 
-.. code-block:: yaml
+1. take the soft argmax of the 2D heatmaps to get predicted coordinates.
+2. for each keypoint, and for each pair of views, we triangulate the predictions into 3D
+3. project the predicted 3D points back into 2D coordinates for each view
+4. turn these reprojected coordinates into heatmaps
+5. computes the mean square error between the reprojected and ground truth heatmaps.
 
-    losses:
-        supervised_pairwise_projections:
-            log_weight: 0.5
-
-Reprojected heatmap loss
-------------------------
-An alternative loss projects the predicted 3D points back into 2D coordinates for each view,
-turns these reprojected coordinates into heatmaps, and computes the mean square error between the
-reprojected and ground truth heatmaps.
 The advantage of this loss is that it is on the same scale as the standard supervised heatmap loss,
 which may make for easier hyperparameter tuning.
 
+The default ``log_weight`` value of 1.0 should be a reasonable place to start; if the training curve
+for this loss is unstable (for example it doesn't decrease, or spikes to a large value during training),
+you can _decrease_ the effect of the 3D loss by _increasing_ the log_weight; we recommend a secondary
+value of 1.5.
+
 .. code-block:: yaml
 
     losses:
         supervised_reprojection_heatmap_mse:
-            log_weight: 0.5
+            log_weight: 1.0
+
+To turn this loss off (but, for example, continue to use 3D augmentations), set
+``log_weight: null`` in the config file.