Lightning CLI, PyTorch Profiler, Improved Early Stopping

[Borda]

[1.3.0] - 2021-05-06

Added

• Added support for the EarlyStopping callback to run at the end of the training epoch ([1/2] Add support for early stopping during training epoch end #6944)
• Added synchronization points before and after setup hooks are run ([fix] Add barriers before and after setup hook is run #7202)
• Added a teardown hook to ClusterEnvironment ([fix] Add a cluster environment teardown to clean up environment state #6942)
• Added utils for metrics to scalar conversions (Move metrics_to_scalars to a dedicated utilities file #7180)
• Added utils for NaN/Inf detection for gradients and parameters (Move NaN/Inf detection to a separate utilities file #6834)
• Added more explicit exception message when trying to execute trainer.test() or trainer.validate() with fast_dev_run=True (More explicit exception message when testing with fast_dev_run=True #6667)
• Added LightningCLI class to provide simple reproducibility with minimum boilerplate training CLI (Simple reproducibility with minimum boilerplate train_cli #4492, Changed basic_examples to use LightningCLI #6862, Improve LightningCLI documentation and tests #7156, Example and documentation for LightningCLI linking model and data arguments #7299)
• Added gradient_clip_algorithm argument to Trainer for gradient clipping by value (Add Trainer(gradient_clip_algorithm='value'|'norm') #6123).
• Added a way to print to terminal without breaking up the progress bar (Feature/5275 clean progress bar print #5470)
• Added support to checkpoint after training steps in ModelCheckpoint callback ([feat] Support iteration-based checkpointing in model checkpoint callback #6146)
• Added TrainerStatus.{INITIALIZING,RUNNING,FINISHED,INTERRUPTED} (TrainerState refactor [5/5] #7173)
• Added Trainer.validate() method to perform one evaluation epoch over the validation set (Add Trainer.validate(…) method to run one validation epoch #4948)
• Added LightningEnvironment for Lightning-specific DDP (introduce default cluster environment for lightning-specific ddp #5915)
• Added teardown() hook to LightningDataModule (Add teardown hook to LightningDataModule #4673)
• Added auto_insert_metric_name parameter to ModelCheckpoint (Allow user to disable the automatic formatting of checkpoint file names. #6277)
• Added arg to self.log that enables users to give custom names when dealing with multiple dataloaders (Add possibility for custom naming when using multiple dataloaders #6274)
• Added teardown method to BaseProfiler to enable subclasses defining post-profiling steps outside of __del__ (Add teardown method to BaseProfiler. #6370)
• Added setup method to BaseProfiler to enable subclasses defining pre-profiling steps for every process ([refactor] Add setup to profilers + _run_stage_setup to trainer 2/5 #6633)
• Added no return warning to predict (Remove no return warning from val/test step #6139)
• Added Trainer.predict config validation (Add trainer.predict config validation #6543)
• Added AbstractProfiler interface (Refactor base profilers 3/5 #6621)
• Added support for including module names for forward in the autograd trace of PyTorchProfiler (Refactor PyTorch profiler 4/5 #6349)
• Added support for the PyTorch 1.8.1 autograd profiler (Add PyTorch 1.8 Profiler 5/5 #6618)
• Added outputs parameter to callback's on_validation_epoch_end & on_test_epoch_end hooks (Add outputs param for on_val/test_epoch_end hooks #6120)
• Added configure_sharded_model hook ([Model Parallel] Add configure sharded model hook #6679)
• Added support for precision=64, enabling training with double precision (Feature/double precision #6595)
• Added support for DDP communication hooks (Supporting Adding DDP Communication Hooks #6736)
• Added artifact_location argument to MLFlowLogger which will be passed to the MlflowClient.create_experiment call (Add artifcact_location arg to MLFlow logger #6677)
• Added model parameter to precision plugins' clip_gradients signature (Update clip gradients signature for precision plugins #6764, Add back clip_gradients(model) #7231)
• Added is_last_batch attribute to Trainer (Disable lr_scheduler.step() in manual optimization #6825)
• Added LightningModule.lr_schedulers() for manual optimization ([RFC] Add self.lr_schedulers() to LightningModule for manual optimization #6567)
• Added MpModelWrapper in TPU Spawn (Add MpModelWrapper in TPU Spawn #7045)
• Added max_time Trainer argument to limit training time (Add Trainer max_time argument + Callback #6823)
• Added on_predict_{batch,epoch}_{start,end} hooks (Add on_predict_{batch,epoch}_{start,end} and Callback.on_predict_{start,end} #7141)
• Added new EarlyStopping parameters stopping_threshold and divergence_threshold (more early stopping options (convergence and divergence threshold) #6868)
• Added debug flag to TPU Training Plugins (PT_XLA_DEBUG) (Add debug flag to TPU Training Plugins (PT_XLA_DEBUG) #7219)
• Added new UnrepeatedDistributedSampler and IndexBatchSamplerWrapper for tracking distributed predictions ([feat] Add better support for predict + ddp 2/3 #7215)
• Added trainer.predict(return_predictions=None|False|True) ([feat] Add better support for predict + ddp 2/3 #7215)
• Added BasePredictionWriter callback to implement prediction saving ([feat] Add BasePredictionWriter 3/3 #7127)
• Added trainer.tune(scale_batch_size_kwargs, lr_find_kwargs) arguments to configure the tuning algorithms (Attach data refactor and tuner bugs [4/n] #7258)
• Added tpu_distributed check for TPU Spawn barrier (Add a check for TPU Spawn barrrier #7241)
• Added device updates to TPU Spawn for Pod training (Device updates for TPU Pod #7243)
• Added warning when missing Callback and using resume_from_checkpoint ([warning] Add a warning with missing callback with resume_from_checkpoint #7254)
• DeepSpeed single file saving ([Feat] DeepSpeed single file saving #6900)
• Added Training type Plugins Registry (Add Training Type Plugins Registry #6982, Better approach to register plugins #7063, Update Accelerator Connector for Registry #7214, Add ddp_find_unused_parameters_false to Registry #7224)
• Add ignore param to save_hyperparameters (Add ignore param to save_hyperparameters #6056)

Changed

• Changed LightningModule.truncated_bptt_steps to be property (Deprecatetruncated_bptt_steps flag on Trainer in favor of same setting on the LightningModule #7323)
• Changed EarlyStopping callback from by default running EarlyStopping.on_validation_end if only training is run. Set check_on_train_epoch_end to run the callback at the end of the train epoch instead of at the end of the validation epoch ([2/2] Remove training loop force calling early stopping callback #7069)
• Renamed pytorch_lightning.callbacks.swa to pytorch_lightning.callbacks.stochastic_weight_avg (cleaning SWA #6259)
• Refactor RunningStage and TrainerState usage (Refactor RunningStage usage in advance of implementing Trainer.validate() #4945, TrainerState refactor [5/5] #7173)
  • Added RunningStage.SANITY_CHECKING
  • Added TrainerFn.{FITTING,VALIDATING,TESTING,PREDICTING,TUNING}
  • Changed trainer.evaluating to return True if validating or testing
• Changed setup() and teardown() stage argument to take any of {fit,validate,test,predict} (Pass {fit,validate,test,predict} to setup() and teardown() #6386)
• Changed profilers to save separate report files per state and rank (Refactor base profilers 3/5 #6621)
• The trainer no longer tries to save a checkpoint on exception or run callback's on_train_end functions (Train End Error Handling Fix #6864)
• Changed PyTorchProfiler to use torch.autograd.profiler.record_function to record functions (Refactor PyTorch profiler 4/5 #6349)
• Disabled lr_scheduler.step() in manual optimization (Disable lr_scheduler.step() in manual optimization #6825)
• Changed warnings and recommendations for dataloaders in ddp_spawn (Update DataLoader.persistent_workers warnings in ddp_spawn #6762)
• pl.seed_everything will now also set the seed on the DistributedSampler (Set DistributedSampler seed if seed_everything was called #7024)
• Changed default setting for communication of multi-node training using DDPShardedPlugin (Set smarter default for DDP sharded for performance optimization #6937)
• trainer.tune() now returns the tuning result (Attach data refactor and tuner bugs [4/n] #7258)
• LightningModule.from_datasets() now accepts IterableDataset instances as training datasets. (Prune deprecated utils modules #7503)
• Changed resume_from_checkpoint warning to an error when the checkpoint file does not exist (Changes resume_from_checkpoint warning to error #7075)
• Automatically set sync_batchnorm for training_type_plugin (Automatically set sync_batchnorm for training_type_plugin #6536)
• Allowed training type plugin to delay optimizer creation (Allow training type plugin to delay optimizer creation (FSDP 2/n) #6331)
• Removed ModelSummary validation from train loop on_trainer_init (Remove ModelSummary validation from train loop on_trainer_init #6610)
• Moved save_function to accelerator ([refactor] Move save_function to accelerator 1/n [DeepSpeed] #6689)
• Updated DeepSpeed ZeRO (DeepSpeed ZeRO Update #6546, DeepSpeed ZeRO Docs update #6752, Add specifics around DeepSpeed docs #6142, [fix] Use training type plugin hook when saving (FSDP 1/n) #6321)
• Improved verbose logging for EarlyStopping callback (improve early stopping verbose logging #6811)
• Run ddp_spawn dataloader checks on Windows (Run ddp_spawn dataloader checks on windows #6930)
• Updated mlflow with using resolve_tags (Update mlflow with using resolve_tags #6746)
• Moved save_hyperparameters to its own function (Move save_hyperparameters to its own function #7119)
• Replaced _DataModuleWrapper with __new__ (Replace _DataModuleWrapper with __new__ [1/2] #7289)
• Reset current_fx properties on lightning module in teardown (Reset current_fx properties on lightning module in teardown #7247)
• Auto-set DataLoader.worker_init_fn with seed_everything (Auto-set DataLoader.worker_init_fn with seed_everything #6960)
• Remove model.trainer call inside of dataloading mixin (Remove model.trainer call inside of dataloading mixin #7317)
• Split profilers module (split profilers #6261)
• Ensure accelerator is valid if running interactively (Ensure accelerator is valid if running interactively #5970)
• Disabled batch transfer in DP mode (Disable batch transfer in DP mode #6098)

Deprecated

• Deprecated outputs in both LightningModule.on_train_epoch_end and Callback.on_train_epoch_end hooks ([1/2] Deprecate outputs in on_train_epoch_end hooks #7339)
• Deprecated Trainer.truncated_bptt_steps in favor of LightningModule.truncated_bptt_steps (Deprecatetruncated_bptt_steps flag on Trainer in favor of same setting on the LightningModule #7323)
• Deprecated outputs in both LightningModule.on_train_epoch_end and Callback.on_train_epoch_end hooks ([1/2] Deprecate outputs in on_train_epoch_end hooks #7339)
• Deprecated LightningModule.grad_norm in favor of pytorch_lightning.utilities.grads.grad_norm (Move grad_norm to a dedicated utilities file  #7292)
• Deprecated the save_function property from the ModelCheckpoint callback (Deprecate save_function from model checkpoint callback #7201)
• Deprecated LightningModule.write_predictions and LightningModule.write_predictions_dict (Deprecate write_predictions on the LightningModule #7066)
• Deprecated TrainerLoggingMixin in favor of a separate utilities module for metric handling (Move metrics_to_scalars to a dedicated utilities file #7180)
• Deprecated TrainerTrainingTricksMixin in favor of a separate utilities module for NaN/Inf detection for gradients and parameters (Move NaN/Inf detection to a separate utilities file #6834)
• period has been deprecated in favor of every_n_val_epochs in the ModelCheckpoint callback ([feat] Support iteration-based checkpointing in model checkpoint callback #6146)
• Deprecated trainer.running_sanity_check in favor of trainer.sanity_checking (Refactor RunningStage usage in advance of implementing Trainer.validate() #4945)
• Deprecated Profiler(output_filename) in favor of dirpath and filename (Refactor base profilers 3/5 #6621)
• Deprecated PytorchProfiler(profiled_functions) in favor of record_functions (Refactor PyTorch profiler 4/5 #6349)
• Deprecated @auto_move_data in favor of trainer.predict (Deprecate @auto_move_data in favor of trainer.predict #6993)
• Deprecated Callback.on_load_checkpoint(checkpoint) in favor of Callback.on_load_checkpoint(trainer, pl_module, checkpoint) ([bugfix] Add reloading support using BaseFinetuning #7253)
• Deprecated metrics in favor of torchmetrics (deprecate metrics pkg 1/n #6505, Prune metrics base classes 2/n #6530, prune warning & deprecation wrapper #6540, Prune metric: helpers and inputs 3/n #6547, prune metric: accuracy 4/n #6515, Prune metrics: AUC & ROC 5/n #6572, Prune metrics: precision & recall 6/n #6573, Prune metrics: other classification 7/n #6584, Prune metrics: regression 8/n #6636, Prune metrics: regression 9/n #6637, prune metrics: info retrieval 10/n #6649, Prune metrics: others 11/DoNe #6659, Fix torchmetrics compatibility #7131)
• Deprecated the LightningModule.datamodule getter and setter methods; access them through Trainer.datamodule instead (Deprecate LightningModule.datamodule reference in favor of the trainer one (#6929) #7168)
• Deprecated the use of Trainer(gpus="i") (string) for selecting the i-th GPU; from v1.5 this will set the number of GPUs instead of the index (make gpus=str in Trainer consistent with command line parsing of string #6388)

Removed

• Removed the exp_save_path property from the LightningModule (Remove exp_save_path on the LightningModule #7266)
• Removed training loop explicitly calling EarlyStopping.on_validation_end if no validation is run ([2/2] Remove training loop force calling early stopping callback #7069)
• Removed automatic_optimization as a property from the training loop in favor of LightningModule.automatic_optimization (Enforce Lightning module as source of truth for automatic optimization #7130)
• Removed evaluation loop legacy returns for *_epoch_end hooks (Remove evaluation loop legacy dict returns for *_epoch_end hooks #6973)
• Removed support for passing a bool value to profiler argument of Trainer (prune deprecated profiler as bool #6164)
• Removed no return warning from val/test step (Remove no return warning from val/test step #6139)
• Removed passing a ModelCheckpoint instance to Trainer(checkpoint_callback) (Prune deprecated Trainer(checkpoint_callback=ModelCheckpoint()) #6166)
• Removed deprecated Trainer argument enable_pl_optimizer and automatic_optimization (prune deprecated Trainer arg enable_pl_optimizer #6163)
• Removed deprecated metrics (Prune deprecated metrics for 1.3 #6161)
  • from pytorch_lightning.metrics.functional.classification removed to_onehot, to_categorical, get_num_classes, roc, multiclass_roc, average_precision, precision_recall_curve, multiclass_precision_recall_curve
  • from pytorch_lightning.metrics.functional.reduction removed reduce, class_reduce
• Removed deprecated ModelCheckpoint arguments prefix, mode="auto" (Prune deprecated checkpoint arguments #6162)
• Removed mode='auto' from EarlyStopping (Prune deprecated EarlyStopping(mode='auto') #6167)
• Removed epoch and step arguments from ModelCheckpoint.format_checkpoint_name(), these are now included in the metrics argument (Pass current_epoch/global_step as monitor candidates [1/2] #7344)
• Removed legacy references for magic keys in the Result object (Remove legacy Result parameters #6016)
• Removed deprecated LightningModule hparams setter (Prune deprecated hparams setter #6207)
• Removed legacy code to log or include metrics in the progress bar by returning them in a dict with the "log"/"progress_bar" magic keys. Use self.log instead (Remove legacy support for the magic log/progress_bar keys in dict returns #6734)
• Removed trainer.fit() return value of 1. It has no return now (Remove trainer.fit return value [2/n] #7237)
• Removed logger_connector legacy code (Remove logger_connector legacy code #6733)
• Removed unused mixin attributes (Remove unused mixin attributes #6487)

Fixed

• Fixed NaN errors in progress bars when training with iterable datasets with no length defined ([bugfix] Fix dataloading for iterable datasets and limit_train_batches #7306)
• Fixed attaching train and validation dataloaders when reload_dataloaders_every_epoch=True and num_sanity_val_steps=0 ([fix] Attach train+val dataloaders to trainer in trainer loop #7207)
• Added a barrier in the accelerator teardown to synchronize processes before execution finishes ([fix] Add barrier to accelerator's teardown #6814)
• Fixed multi-node DDP sub-process launch by using local_rank instead of global_rank for main process assertion ([fix] Fix multi-node DDP launch by using local rank instead of global rank for main process #7061)
• Fixed incorrect removal of WORLD_SIZE environment variable in DDP training when launching with torch distributed/torchelastic ([fix] Add a cluster environment teardown to clean up environment state #6942)
• Made the Plugin.reduce method more consistent across all Plugins to reflect a mean-reduction by default (consistent behavior for reduce method across all Plugins #6011)
• Move lightning module to correct device type when using LightningDistributedWrapper (Enable ZeRO tests for CI, fix to/half function calls for LightningDistributedWrapper #6070)
• Do not print top-k verbose log with ModelCheckpoint(monitor=None) (Avoid printing ModelCheckpoint log with monitor=None and verbose=True #6109)
• Fixed ModelCheckpoint(save_top_k=0, save_last=True) not saving the last checkpoint (Fix ModelCheckpoint(monitor=None, save_last=True) not saving checkpoints #6136)
• Fixed .teardown(stage='fit') and .on_fit_{start,end}() getting called during trainer.test (Pass {fit,validate,test,predict} to setup() and teardown() #6386)
• Fixed LightningModule all_gather on cpu tensors ([bug] All_gather support tensor on cpu #6416)
• Fixed torch distributed not available in setup hook for DDP ([Fix] Move init dist connection into the setup function #6506)
• Fixed trainer.tuner.{lr_find,scale_batch_size} not setting the Trainer state properly (Attach data refactor and tuner bugs [4/n] #7258)
• Fixed bug where the learning rate schedulers did not follow the optimizer frequencies (Update LR schedulers only when their corresponding Optimizer is being… #4868)
• Fixed pickle error checker to now check for pickle.PickleError to catch all pickle errors (Use PickleError base class to detect all pickle errors #6917)
• Fixed a bug where the outputs object passed to LightningModule.training_epoch_end was different from the object passed to the on_train_end_epoch hook (Fix inconsistent outputs in on_*_end and *_end #6969)
• Fixed a bug where the outputs passed to train_batch_end would be listed even when using a single optimizer and no truncated backprop through time steps (Fix inconsistent outputs in on_*_end and *_end #6969)
• Fixed bug for trainer error handling which would cause hang for distributed training (Train End Error Handling Fix #6864)
• Fixed self.device not returning the correct device in replicas of data-parallel (fix self.device access in DataParallel #6414)
• Fixed lr_find trying beyond num_training steps and suggesting a too high learning rate (Fix lr_finder suggesting too high learning rates #7076)
• Fixed logger creating incorrect version folder in DDP with repeated Trainer.fit calls (fix logger experiment version in multiple run DDP #7077)
• Fixed metric objects passed directly to self.log not being reset correctly (Correctly reset metric objects in self.log #7055)
• Fixed CombinedLoader in distributed settings for validation / testing ([bugfix] Add support for CombinedLoader in validation with ddp #7102)
• Fixed the save_dir in WandbLogger when the run was initiated externally (Fixed setting of _save_dir when run initiated externally #7106)
• Fixed num_sanity_val_steps affecting reproducibility of training data shuffling (Fixed num_sanity_val_steps affecting reproducibility of training data shuffling #7014)
• Fixed resetting device after fitting/evaluating/predicting ([bug] Set device on LightningModule after fitting #7188)
• Fixed bug where trainer.tuner.scale_batch_size(max_trials=0) would not return the correct batch size result (Code cleaning in preparation for #7258 [3/n] #7262)
• Fixed metrics not being properly logged with precision=16 and manual_optimization ([bugfix] Metric not logged properly in manual optimization #7228)
• Fixed BaseFinetuning properly reloading optimizer_states when using resume_from_checkpoint ([BUG] BaseFinetuning not working with resume_from_checkpoint #6891)
• Fixed parameters_to_ignore not properly set to DDPWrapper ([bug/feat] Support parameters_to_ignore in DDP #7239)
• Fixed parsing of fast_dev_run=True with the built-in ArgumentParser (fix fast_dev_run parsing from cli #7240)
• Fixed handling an IterableDataset that fails to produce a batch at the beginning of an epoch (fix case where an IterableDataset doesn't produce a batch for an epoch #7294)
• Fixed LightningModule.save_hyperparameters() when attempting to save an empty container (fix save_hyperparameters(container) if container is empty  #7268)
• Fixed apex not properly instantiated when running with ddp ([bugfix] Apex never instantiated. #7274)
• Fixed optimizer state not moved to GPU (Fix Adagrad optimizer not working with DDP/GPU #7277)
• Fixed custom init args for WandbLogger (fix(wandb): allow custom init args #6989)
• Fixed a bug where an error would be raised if the train dataloader sometimes produced None for a batch (Fix support for dataloader with None batches #7342)
• Fixed examples (fixing examples #6600, fix: update example autoencoder.py to reflect args #6638, Fixed bug: replaced bce_loss_with_logits with bce_loss #7096, Simplify backbone_image_classifier example #7246, CI: fix examples - patch download MNIST #6357, Hotfix for torchvision #6476, [doc] Improve Manual Optimization Example #6294, Fix manual optimization in pl_example #6373, argparse: Add use_argument_group=True #6088, Call super().__init__() in MilestonesFinetuning example #7398)
• Resolved schedule step bug for PyTorch Profiler ([bugfix] Resolve schedule step bug for PyTorch Profiler #6674, Profilers - do not write when there's no summary #6681)
• Updated logic for checking TPUs availability (Update logic for checking TPUs availability #6767)
• Resolve TPU miss rendezvous ([bugfix] Resolve TPU miss rdz #6781)
• Fixed auto-scaling mode when calling tune method on trainer (Fix auto scaling mode when calling tune method on trainer. #7321)
• Fixed finetuning complex models correctly unfreezes (Fix finetuning complex models correctly unfreezes. #6880)
• Ensure we set the eval/train flag correctly on accelerator model ([Fix] Ensure we set the eval/train flag correctly on accelerator model  #6877)
• Set better defaults for rank_zero_only.rank when training is launched with SLURM and torchelastic ([fix] Better support for rank_zero_only setting for SLURM and torchelastic #6802)
• Fixed matching the number of outputs of backward with forward for AllGatherGrad (Match the number of outputs of backward with forward for AllGatherGrad #6625)
• Fixed the gradient_clip_algorithm has no effect (Fix the gradient_clip_algorithm has no effect issue. #6928)
• Fixed CUDA OOM detection and handling (Bugfix/cuda oom detection and handling #6934)
• Fixed unfreeze_and_add_param_group expects modules rather than module (Correct parameter name in doc string #6822)
• Fixed DPP + SyncBN when move on device (Fix DPP + SyncBN #6838)
• Fixed missing arguments in lr_find call (Fixed missing arguments in lr_find call #6784)
• Fixed set_default_tensor_type to torch.DoubleTensor with precision=64 ([bugfix] Add set_default_tensor_type to torch.DoubleTensor with precision=64 #7108)
• Fixed NeptuneLogger.log_text(step=None) (Fix NeptuneLogger.log_text(step=None) #7194)
• Fixed importing torchtext batch (fix importing torchtext batch #6365, hotfix for PT1.6 and torchtext #6323, apply_func.py: from torchtext.legacy.data import Batch #6211)

Contributors

@akihironitta, @alessiobonfiglio, @amisev, @amogkam, @ananthsub, @ArvinZhuang, @ashleve, @asnorkin, @awaelchli, @BloodAxe, @bmahlbrand, @Borda, @borisdayma, @camruta, @carmocca, @ceshine, @dbonner, @dhkim0225, @EdwardJB, @EliaCereda, @EricCousineau-TRI, @ethanwharris, @FlorianMF, @hemildesai, @ifsheldon, @kaushikb11, @mauvilsa, @maxfrei750, @mesejo, @ramonemiliani93, @rohitgr7, @s-rog, @sadiqj, @scart97, @SeanNaren, @shuyingsunshine21, @SkafteNicki, @SpontaneousDuck, @stllfe, @tchaton, @THasthika, @vballoli

If we forgot someone due to not matching commit email with GitHub account, let us know :]

---

This discussion was created from the release Lightning CLI, PyTorch Profiler, Improved Early Stopping.

[carmocca]

Epic!