Audit of places we profile in Lightning besides hooks

[daniellepintz]

I did an audit of all the places we currently profile in Lightning, besides user-defined hooks which are profiled in the _call_{}_hook methods. This is a good way to see if we have any missing or redundant profiling.

• run_train_evaluation / run_sanity_check_evaluation / run_validate_evaluation / run_test_evaluation / run_predict_evaluation / run_tune_evaluation
  https://github.com/PyTorchLightning/pytorch-lightning/blob/d902b5988601a21db04ffe6316a012581fe152ec/pytorch_lightning/trainer/trainer.py#L1315

• model_forward(Removed in Remove calls to profile model_forward #12032)
  https://github.com/PyTorchLightning/pytorch-lightning/blob/d902b5988601a21db04ffe6316a012581fe152ec/pytorch_lightning/loops/optimization/optimizer_loop.py#L417
  https://github.com/PyTorchLightning/pytorch-lightning/blob/d902b5988601a21db04ffe6316a012581fe152ec/pytorch_lightning/loops/optimization/manual_loop.py#L110

• run_training_batch
  https://github.com/PyTorchLightning/pytorch-lightning/blob/d902b5988601a21db04ffe6316a012581fe152ec/pytorch_lightning/loops/epoch/training_epoch_loop.py#L198

• run_training_epoch
  https://github.com/PyTorchLightning/pytorch-lightning/blob/d902b5988601a21db04ffe6316a012581fe152ec/pytorch_lightning/loops/fit_loop.py#L272

• optimizer_step_without_closure / optimizer_step_with_closure
  https://github.com/PyTorchLightning/pytorch-lightning/blob/d902b5988601a21db04ffe6316a012581fe152ec/pytorch_lightning/core/optimizer.py#L173

[daniellepintz]

For the model_forward profiling, do we feel this is necessary in addition to profiling of training_step a few lines down?
https://github.com/PyTorchLightning/pytorch-lightning/blob/d902b5988601a21db04ffe6316a012581fe152ec/pytorch_lightning/loops/optimization/optimizer_loop.py#L424

https://github.com/PyTorchLightning/pytorch-lightning/blob/d902b5988601a21db04ffe6316a012581fe152ec/pytorch_lightning/loops/optimization/manual_loop.py#L117

There is a lot of overlap between model_forward and training_step, with training_step taking up the majority of model_forward's time.

see lines 11-12 in this example profiler output: https://gist.github.com/daniellepintz/7b37ab4807fab345248bf60c72e46ae8

[rohitgr7]

agreed! we can remove profiling model_forward.

[daniellepintz]

#12032