How to Use Early Stopping in test/evaluate

[yanbing-j]

Hi pytorch-lightning team,

I want to early stop after several iterations in inference in trainer.test(). However, in training, on_train_batch_start() can return -1 to early stop. In test, using on_test_batch_start and return -1 cannot help. Does pytorch-lightning has this function in inference? Could you please share some test cases? Thank you!

[rohitgr7]

EarlyStopping isn't supposed to be used during test/evaluation. Its sole purpose is to stop the training during convergence.

[yanbing-j]

Thank you for your clarification!

In performance profiling, EarlyStopping is equal important in test/evaluation. Will you take this into account?

[rohitgr7]

can you explain more on performance profiling + early stopping ?

[yanbing-j]

Sure. For example, I want to optimize the inference part of the model, that equals to test/evaluation part. Now I can only run one complete epoch to profile, which is to large to store the trace down. Therefore, I want to only record the profiling in several iterations with enough warmup. EarlyStopping can help doing this.

[rohitgr7]

you can limit the batches there using limit_val_batches/limit_test_batches.