Release2 (#2262)

* fix missing arg

* fix missing arg

* fix missing arg

* fix missing arg

* fix missing arg

* fix missing arg

* fix missing arg

Author: williamFalcon

docs/source/apex.rst

@@ -7,6 +7,8 @@
 =================
 Lightning offers 16-bit training for CPUs, GPUs and TPUs.
 
+----------
+
 GPU 16-bit
 ----------
 16 bit precision can cut your memory footprint by half.
@@ -67,6 +69,8 @@ Enable 16-bit
 If you need to configure the apex init for your particular use case or want to use a different way of doing
 16-bit training, override   :meth:`pytorch_lightning.core.LightningModule.configure_apex`.
 
+----------
+
 TPU 16-bit
 ----------
 16-bit on TPus is much simpler. To use 16-bit with TPUs set precision to 16 when using the tpu flag

docs/source/early_stopping.rst

@@ -13,12 +13,16 @@ You can stop an epoch early by overriding :meth:`~pytorch_lightning.core.lightni
 
 If you do this repeatedly, for every epoch you had originally requested, then this will stop your entire run.
 
+----------
+
 Default Epoch End Callback Behavior
 -----------------------------------
 By default early stopping will be enabled if `'val_loss'`
 is found in :meth:`~pytorch_lightning.core.lightning.LightningModule.validation_epoch_end`'s
 return dict. Otherwise training will proceed with early stopping disabled.
 
+----------
+
 Enable Early Stopping using the EarlyStopping Callback
 ------------------------------------------------------
 The
@@ -81,6 +85,8 @@ and change where it is called:
     - :class:`~pytorch_lightning.trainer.trainer.Trainer`
     - :class:`~pytorch_lightning.callbacks.early_stopping.EarlyStopping`
 
+----------
+
 Disable Early Stopping with callbacks on epoch end
 --------------------------------------------------
 To disable early stopping pass ``False`` to the

docs/source/experiment_reporting.rst

@@ -10,6 +10,7 @@ Lightning supports many different experiment loggers. These loggers allow you to
 as training progresses. They usually provide a GUI to visualize and can sometimes even snapshot hyperparameters
 used in each experiment.
 
+----------
 
 Control logging frequency
 ^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -21,6 +22,8 @@ It may slow training down to log every single batch. Trainer has an option to lo
    k = 10
    trainer = Trainer(row_log_interval=k)
 
+----------
+
 Control log writing frequency
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -35,6 +38,8 @@ want to log using this trainer flag.
     k = 100
     trainer = Trainer(log_save_interval=k)
 
+----------
+
 Log metrics
 ^^^^^^^^^^^
 
@@ -84,6 +89,8 @@ For instance, here we log images using tensorboard.
         ...
         return results
 
+----------
+
 Modify progress bar
 ^^^^^^^^^^^^^^^^^^^
 
@@ -102,6 +109,8 @@ Here we show the validation loss in the progress bar
         results = {'progress_bar': logs}
         return results
 
+----------
+
 Snapshot hyperparameters
 ^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -117,6 +126,8 @@ Some loggers also allow logging the hyperparams used in the experiment. For inst
 when using the TestTubeLogger or the TensorBoardLogger, all hyperparams will show
 in the `hparams tab <https://pytorch.org/docs/stable/tensorboard.html#torch.utils.tensorboard.writer.SummaryWriter.add_hparams>`_.
 
+----------
+
 Snapshot code
 ^^^^^^^^^^^^^
 

docs/source/hooks.rst

@@ -30,6 +30,8 @@ Training set-up
 - :meth:`~pytorch_lightning.core.lightning.LightningModule.summarize`
 - :meth:`~pytorch_lightning.trainer.training_io.TrainerIOMixin.restore_weights`
 
+----------
+
 Training loop
 ^^^^^^^^^^^^^
 
@@ -46,6 +48,8 @@ Training loop
 - :meth:`~pytorch_lightning.core.lightning.LightningModule.training_epoch_end`
 - :meth:`~pytorch_lightning.core.hooks.ModelHooks.on_epoch_end`
 
+----------
+
 Validation loop
 ^^^^^^^^^^^^^^^
 
@@ -59,6 +63,8 @@ Validation loop
 - ``torch.set_grad_enabled(True)``
 - :meth:`~pytorch_lightning.core.hooks.ModelHooks.on_post_performance_check`
 
+----------
+
 Test loop
 ^^^^^^^^^
 

docs/source/hyperparameters.rst

@@ -13,6 +13,8 @@ Hyperparameters
 Lightning has utilities to interact seamlessly with the command line ArgumentParser
 and plays well with the hyperparameter optimization framework of your choice.
 
+----------
+
 ArgumentParser
 ^^^^^^^^^^^^^^
 Lightning is designed to augment a lot of the functionality of the built-in Python ArgumentParser
@@ -30,6 +32,7 @@ This allows you to call your program like so:
 
     python trainer.py --layer_1_dim 64
 
+----------
 
 Argparser Best Practices
 ^^^^^^^^^^^^^^^^^^^^^^^^
@@ -100,6 +103,8 @@ Finally, make sure to start the training like so:
     dict_args = vars(args)
     model = LitModel(**dict_args)
 
+----------
+
 LightningModule hyperparameters
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 Often times we train many versions of a model. You might share that model or come back to it a few months later
@@ -188,6 +193,7 @@ In that case, choose only a few
     # this works
     model.hparams.anything
 
+----------
 
 Trainer args
 ^^^^^^^^^^^^
@@ -204,6 +210,7 @@ To recap, add ALL possible trainer flags to the argparser and init the Trainer t
     # or if you need to pass in callbacks
     trainer = Trainer.from_argparse_args(hparams, checkpoint_callback=..., callbacks=[...])
 
+----------
 
 Multiple Lightning Modules
 ^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -284,6 +291,8 @@ and now we can train MNIST or the GAN using the command line interface!
     $ python main.py --model_name gan --encoder_layers 24
     $ python main.py --model_name mnist --layer_1_dim 128
 
+----------
+
 Hyperparameter Optimization
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 Lightning is fully compatible with the hyperparameter optimization libraries!

docs/source/introduction_guide.rst

@@ -256,14 +256,14 @@ under the `train_dataloader` method. This is great because if you run into a pro
 to figure out how they prepare their training data you can just look in the `train_dataloader` method.
 
 Usually though, we want to separate the things that write to disk in data-processing from
-things like transforms which happen in memory.
+things like transforms which happen in memory. This is only relevant in multi-GPU or TPU training.
 
 .. testcode::
 
     class LitMNIST(LightningModule):
 
         def prepare_data(self):
-            # download only
+            # download only (not called on every GPU, just the root GPU per node)
             MNIST(os.getcwd(), train=True, download=True)
 
         def train_dataloader(self):
@@ -302,6 +302,31 @@ In general fill these methods with the following:
             # return a DataLoader
             ...
 
+Models defined by data
+^^^^^^^^^^^^^^^^^^^^^^
+Sometimes a model needs to know about the data to be built (ie: number of classes or vocab size).
+In this case we recommend the following:
+
+1. use `prepare_data` to download and process the dataset.
+2. use `setup` to do splits, and build your model internals
+
+Example::
+
+    class LitMNIST(LightningModule):
+
+        def __init__(self):
+            self.l1 = None
+
+        def prepare_data(self):
+            download_data()
+            tokenize()
+
+        def setup(self, step):
+            # step is either 'fit' or 'test' 90% of the time not relevant
+            data = load_data()
+            num_classes = data.classes
+            self.l1 = nn.Linear(..., num_classes)
+
 Optimizer
 ^^^^^^^^^
 

docs/source/loggers.rst

@@ -3,11 +3,139 @@
 
 Loggers
 ===========
-.. automodule:: pytorch_lightning.loggers
-   :noindex:
-   :exclude-members:
-        _abc_impl,
-        _save_model,
-        on_epoch_end,
-        on_train_end,
-        on_epoch_start,
+Lightning supports the most popular logging frameworks (TensorBoard, Comet, Weights and Biases, etc...).
+To use a logger, simply pass it into the :class:`~pytorch_lightning.trainer.trainer.Trainer`.
+Lightning uses TensorBoard by default.
+
+.. code-block:: python
+
+    from pytorch_lightning import Trainer
+    from pytorch_lightning import loggers
+    tb_logger = loggers.TensorBoardLogger('logs/')
+    trainer = Trainer(logger=tb_logger)
+
+Choose from any of the others such as MLflow, Comet, Neptune, WandB, ...
+
+.. code-block:: python
+
+    comet_logger = loggers.CometLogger(save_dir='logs/')
+    trainer = Trainer(logger=comet_logger)
+
+To use multiple loggers, simply pass in a ``list`` or ``tuple`` of loggers ...
+
+.. code-block:: python
+
+    tb_logger = loggers.TensorBoardLogger('logs/')
+    comet_logger = loggers.CometLogger(save_dir='logs/')
+    trainer = Trainer(logger=[tb_logger, comet_logger])
+
+Note:
+    All loggers log by default to ``os.getcwd()``. To change the path without creating a logger set
+    ``Trainer(default_root_dir='/your/path/to/save/checkpoints')``
+
+----------
+
+Custom Logger
+-------------
+
+You can implement your own logger by writing a class that inherits from
+:class:`LightningLoggerBase`. Use the :func:`~pytorch_lightning.loggers.base.rank_zero_only`
+decorator to make sure that only the first process in DDP training logs data.
+
+.. code-block:: python
+
+    from pytorch_lightning.utilities import rank_zero_only
+    from pytorch_lightning.loggers import LightningLoggerBase
+    class MyLogger(LightningLoggerBase):
+
+        @rank_zero_only
+        def log_hyperparams(self, params):
+            # params is an argparse.Namespace
+            # your code to record hyperparameters goes here
+            pass
+
+        @rank_zero_only
+        def log_metrics(self, metrics, step):
+            # metrics is a dictionary of metric names and values
+            # your code to record metrics goes here
+            pass
+
+        def save(self):
+            # Optional. Any code necessary to save logger data goes here
+            pass
+
+        @rank_zero_only
+        def finalize(self, status):
+            # Optional. Any code that needs to be run after training
+            # finishes goes here
+            pass
+
+If you write a logger that may be useful to others, please send
+a pull request to add it to Lighting!
+
+----------
+
+Using loggers
+-------------
+
+Call the logger anywhere except ``__init__`` in your
+:class:`~pytorch_lightning.core.lightning.LightningModule` by doing:
+
+.. code-block:: python
+
+    from pytorch_lightning import LightningModule
+    class LitModel(LightningModule):
+        def training_step(self, batch, batch_idx):
+            # example
+            self.logger.experiment.whatever_method_summary_writer_supports(...)
+
+            # example if logger is a tensorboard logger
+            self.logger.experiment.add_image('images', grid, 0)
+            self.logger.experiment.add_graph(model, images)
+
+        def any_lightning_module_function_or_hook(self):
+            self.logger.experiment.add_histogram(...)
+
+Read more in the `Experiment Logging use case <./experiment_logging.html>`_.
+
+------
+
+Supported Loggers
+-----------------
+The following are loggers we support
+
+Comet
+^^^^^
+
+.. autoclass:: pytorch_lightning.loggers.comet.CometLogger
+    :noindex:
+
+MLFlow
+^^^^^^
+
+.. autoclass:: pytorch_lightning.loggers.mlflow.MLFlowLogger
+    :noindex:
+
+Neptune
+^^^^^^^
+
+.. autoclass:: pytorch_lightning.loggers.neptune.NeptuneLogger
+    :noindex:
+
+Tensorboard
+^^^^^^^^^^^^
+
+.. autoclass:: pytorch_lightning.loggers.tensorboard.TensorBoardLogger
+    :noindex:
+
+Test-tube
+^^^^^^^^^
+
+.. autoclass:: pytorch_lightning.loggers.test_tube.TestTubeLogger
+    :noindex:
+
+Trains
+^^^^^^
+
+.. autoclass:: pytorch_lightning.loggers.trains.TrainsLogger
+    :noindex:

docs/source/lr_finder.rst

@@ -22,6 +22,8 @@ initial lr.
     For the moment, this feature only works with models having a single optimizer. 
     LR support for DDP is not implemented yet, it is comming soon.
 
+----------
+
 Using Lightning's built-in LR finder
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^