Documentation improvements (#69)

* Add cloud cover landsat8 example.
* Extensive documentation updates.
* Move resume options to train section.
* Remove model intermediate field from network.
* Delete max_tile_offset (didn't work well).
* Hold back usgs package version.

Author: bcoltin

README.md

@@ -1,83 +1,88 @@
 **DELTA** (Deep Earth Learning, Tools, and Analysis) is a framework for deep learning on satellite imagery,
-based on Tensorflow. Use DELTA to train and run neural networks to classify large satellite images. DELTA
-provides pre-trained autoencoders for a variety of satellites to reduce required training data
-and time.
+based on Tensorflow. DELTA classifies large satellite images with neural networks, automatically handling
+tiling large imagery.
 
 DELTA is currently under active development by the
-[NASA Ames Intelligent Robotics Group](https://ti.arc.nasa.gov/tech/asr/groups/intelligent-robotics/). Expect
-frequent changes. It is initially being used to map floods for disaster response, in collaboration with the
+[NASA Ames Intelligent Robotics Group](https://ti.arc.nasa.gov/tech/asr/groups/intelligent-robotics/).
+Initially, it is mapping floods for disaster response, in collaboration with the
 [U.S. Geological Survey](http://www.usgs.gov), [National Geospatial Intelligence Agency](https://www.nga.mil/),
 [National Center for Supercomputing Applications](http://www.ncsa.illinois.edu/), and
-[University of Alabama](https://www.ua.edu/). DELTA is a component of the
-[Crisis Mapping Toolkit](https://github.com/nasa/CrisisMappingToolkit), in addition
-to our previous software for mapping floods with Google Earth Engine.
+[University of Alabama](https://www.ua.edu/).
 
 Installation
 ============
 
-1. Install [python3](https://www.python.org/downloads/), [GDAL](https://gdal.org/download.html), and the [GDAL python bindings](https://pypi.org/project/GDAL/).
-   For Ubuntu Linux, you can run `scripts/setup.sh` from the DELTA repository to install these dependencies.
+1. Install [python3](https://www.python.org/downloads/), [GDAL](https://gdal.org/download.html),
+   and the [GDAL python bindings](https://pypi.org/project/GDAL/). For Ubuntu Linux, you can run
+   `scripts/setup.sh` from the DELTA repository to install these dependencies.
 
-2. Install Tensorflow with pip following the [instructions](https://www.tensorflow.org/install). For
+2. Install Tensorflow following the [instructions](https://www.tensorflow.org/install). For
    GPU support in DELTA (highly recommended) follow the directions in the
    [GPU guide](https://www.tensorflow.org/install/gpu).
 
 3. Checkout the delta repository and install with pip:
 
-   ```
-   git clone http://github.com/nasa/delta
-   python3 -m pip install delta
-   ```
+```bash
+git clone http://github.com/nasa/delta
+python3 -m pip install delta
+```
+
+DELTA is now installed and ready to use!
+
+Documentation
+=============
+DELTA can be used either as a command line tool or as a python library.
+See the python documentation for the master branch [here](https://nasa.github.io/delta/),
+or generate the documentation with `scripts/docs.sh`.
+
+Example
+=======
+
+As a simple example, consider training a neural network to map clouds with Landsat-8 images.
+The script `scripts/example/l8_cloud.sh` trains such a network using DELTA from the
+[USGS SPARCS dataset](https://www.usgs.gov/core-science-systems/nli/landsat/spatial-procedures-automated-removal-cloud-and-shadow-sparcs),
+and shows how DELTA can be used. The steps involved in this, and other, classification processes are:
+
+1. **Collect** training data. The SPARCS dataset contains Landsat-8 imagery with and without clouds.
 
-  This installs DELTA and all dependencies (except for GDAL which must be installed manually in step 1).
+2. **Label** training data. The SPARCS labels classify each pixel according to cloud, land, water and other classes.
 
-Usage
-=====
+3. **Train** the neural network. The script `scripts/example/l8_cloud.sh` invokes the command
 
-As a simple example, consider training a neural network to map water in Worldview imagery.
-You would:
+    ```
+    delta train --config l8_cloud.yaml l8_clouds.h5
+    ```
 
-1. **Collect** training data. Find and save Worldview images with and without water. For a robust
-   classifier, the training data should be as representative as possible of the evaluation data.
+    where `scripts/example/l8_cloud.yaml` is a configuration file specifying the labeled training data and
+    training parameters (learn more about configuration files below). A neural network file
+    `l8_clouds.h5` is output.
 
-2. **Label** training data. Create images matching the training images pixel for pixel, where each pixel
-   in the label is 0 if it is not water and 1 if it is.
+4. **Classify** with the trained network. The script runs
 
-3. **Train** the neural network. Run
-   ```
-   delta train --config wv_water.yaml wv_water.h5
-   ```
-   where `wv_water.yaml` is a configuration file specifying the labeled training data and any
-   training parameters (learn more about configuration files below). The command will output a
-   neural network file `wv_water.h5` which can be
-   used for classification. The neural network operates on the level of *chunks*, inputting
-   and output smaller blocks of the image at a time.
+    ```
+    delta classify --config l8_cloud.yaml --image-dir ./validate --overlap 32 l8_clouds.h5
+    ```
 
-4. **Classify** with the trained network. Run
-   ```
-   delta classify --image image.tiff wv_water.h5
-   ```
-   to classify `image.tiff` using the network `wv_water.h5` learned previously.
-   The file `image_predicted.tiff` will be written to the current directory showing the resulting labels.
+    to classify the images in the `validate` folder using the network `l8_clouds.h5` learned previously.
+    The overlap tiles to ignore border regions when possible to make a more aesthetically pleasing classified
+    image. The command outputs a predicted image and confusion matrix.
 
-Configuration Files
--------------------
+The results could be improved--- with more training, more data, an improved network, or more--- but this
+example shows the basic usage of DETLA.
 
-DELTA is configured with YAML files. Some options can be overwritten with command line options (use
-`delta --help` to see which). [Learn more about DELTA configuration files](./delta/config/README.md).
+Configuration and Extensions
+============================
 
-All available configuration options and their default values are shown [here](./delta/config/delta.yaml).
-We suggest that users create one reusable configuration file to describe the parameters specific
-to each dataset, and separate configuration files to train on or classify that dataset.
+DELTA provides many options for customizing data inputs and training. All options are configured via
+YAML files. Some options can be overwritten with command line options (use
+`delta --help` to see which). See the `delta.config` README to learn about available configuration
+options.
 
-Supported Image Formats
------------------------
-DELTA supports tiff files and a few other formats.
-Users can extend DELTA with their own custom formats. We are looking to expand DELTA to support other
-useful file formats.
+DELTA can be extended to support custom neural network layers, image types, preprocessing operations, metrics, losses,
+and training callbacks. Learn about DELTA extensions in the `delta.config.extensions` documentation.
 
-MLFlow
-------
+Data Management
+=============
 
 DELTA integrates with [MLFlow](http://mlflow.org) to track training. MLFlow options can
 be specified in the corresponding area of the configuration file. By default, training and
@@ -93,18 +98,6 @@ View all the logged training information through mlflow by running::
 and navigating to the printed URL in a browser. This makes it easier to keep track when running
 experiments and adjusting parameters.
 
-Using DELTA from Code
-=====================
-You can also call DELTA as a python library and customize it with your own extensions, for example,
-custom image types. The python API documentation can be generated as HTML. To do so:
-
-```
-  pip install pdoc3
-  ./scripts/docs.sh
-```
-
-Then open `html/delta/index.html` in a web browser.
-
 Contributors
 ============
 We welcome pull requests to contribute to DELTA. However, due to NASA legal restrictions, we must require

delta/config/README.md

@@ -5,9 +5,33 @@ all options, showing all parameters DELTA and their default values, see [delta.y
 
 `delta` accepts multiple config files on the command line. For example, run
 
-    delta train --config dataset.yaml --config train.yaml
+```bash
+delta train --config dataset.yaml --config train.yaml
+```
+
+to train on a dataset specified by `dataset.yaml`:
+
+```yaml
+dataset:
+  images:
+    type: tiff
+    directory: train/
+  labels:
+    type: tiff
+    directory: labels/
+  classes: 2
+```
+
+with training parameters given in `train.yaml`:
+
+```yaml
+train:
+  network:
+    model:
+      yaml_file: networks/convpool.yaml
+  epochs: 10
+```
 
-to train on a dataset specified by `dataset.yaml` with training parameters given in `train.yaml`.
 Parameters can be overriden globally for all runs of `delta` as well, by placing options in
 `$HOME/.config/delta/delta.yaml` on Linux. This is only recommended for global parameters
 such as the cache directory.
@@ -17,79 +41,137 @@ only setting the necessary options.
 Note that some configuration options can be overwritten on the command line: run
 `delta --help` to see which.
 
-The remainder of this document details the available configuration parameters. Note that
-DELTA is still under active development and parts are likely to change in the future.
+The remainder of this document details the available configuration parameters.
 
 Dataset
 -----------------
 Images and labels are specified with the `images` and `labels` fields respectively,
 within `dataset`. Both share the
 same underlying options.
 
- * `type`: Indicates which loader to use, e.g., `tiff` for geotiff.
+ * `type`: Indicates which `delta.imagery.delta_image.DeltaImage` image reader to use, e.g., `tiff` for geotiff.
+   The reader should previously be registered with `delta.config.extensions.register_image_reader`.
  * Files to load must be specified in one of three ways:
-   * `directory` and `extension`: Use all images in the directory ending with the given extension.
-   * `file_list`: Provide a text file with one image file name per line.
-   * `files`: Provide a list of file names in yaml.
- * `preprocess`: Supports limited image preprocessing. We recommend
+    * `directory` and `extension`: Use all images in the directory ending with the given extension.
+    * `file_list`: Provide a text file with one image file name per line.
+    * `files`: Provide a list of file names in yaml.
+ * `preprocess`: Specify a preprocessing chain. We recommend
    scaling input imagery in the range 0.0 to 1.0 for best results with most of our networks.
    DELTA also supports custom preprocessing commands. Default actions include:
-   * `scale` with `factor` argument: Divide all values by amount.
-   * `offset` with `factor` argument: Add `factor` to pixel values.
-   * `clip` with `bounds` argument: clip all pixels to bounds.
+    * `scale` with `factor` argument: Divide all values by amount.
+    * `offset` with `factor` argument: Add `factor` to pixel values.
+    * `clip` with `bounds` argument: clip all pixels to bounds.
+   Preprocessing commands are registered with `delta.config.extensions.register_preprocess`.
+   A full list of defaults (and examples of how to create new ones) can be found in `delta.extensions.preprocess`.
  * `nodata_value`: A pixel value to ignore in the images.
+ * `classes`: Either an integer number of classes or a list of individual classes. If individual classes are specified,
+   each list item should be the pixel value of the class in the label images, and a dictionary with the
+   following potential attributes (see example below):
+    * `name`: Name of the class.
+    * `color`: Integer to use as the RGB representation for some classification options.
+    * `weight`: How much to weight the class during training (useful for underrepresented classes).
 
 As an example:
 
-  ```
-  dataset:
-    images:
-      type: worldview
-      directory: images/
-    labels:
-      type: tiff
-      directory: labels/
-      extension: _label.tiff
-  ```
-
-This configuration will load worldview files ending in `.zip` from the `images/` directory.
+```yaml
+dataset:
+  images:
+    type: tiff
+    directory: images/
+    preprocess:
+      - scale:
+          factor: 256.0
+    nodata_value: 0
+  labels:
+    type: tiff
+    directory: labels/
+    extension: _label.tiff
+    nodata_value: 0
+  classes:
+    - 1:
+        name: Cloud
+        color: 0x0000FF
+        weight: 2.0
+    - 2:
+        name: Not Cloud
+        color: 0xFFFFFF
+        weight: 1.0
+```
+
+This configuration will load tiff files ending in `.tiff` from the `images/` directory.
 It will then find matching tiff files ending in `_label.tiff` from the `labels` directory
-to use as labels.
+to use as labels. The image values will be divied by a factor of 256 before they are used.
+(It is often helpful to scale images to a range of 0-1 before training.) The labels represent two classes:
+clouds and non-clouds. Since there are fewer clouds, these are weighted more havily. The label
+images should contain 0 for nodata, 1 for cloud pixels, and 2 for non-cloud pixels.
 
 Train
 -----
 These options are used in the `delta train` command.
 
- * `network`: The nueral network to train. See the next section for details.
+ * `network`: The nueral network to train. One of `yaml_file` or `layers` must be specified.
+    * `yaml_file`: A path to a yaml file with only the params and layers fields. See `delta/config/networks`
+      for examples.
+    * `params`: A dictionary of parameters to substitute in the `layers` field.
+    * `layers`: A list of layers which compose the network. See the following section for details.
  * `stride`: When collecting training samples, skip every `n` pixels between adjacent blocks. Keep the 
-   default of 1 to use all available training data.
- * `batch_size`: The number of chunks to train on in a group. May affect convergence speed. Larger
-   batches allow higher training data throughput, but may encounter memory limitations.
+   default of ~ or 1 to use all available training data. Not used for fully convolutional networks.
+ * `batch_size`: The number of patches to train on at a time. If running out of memory, reducing
+   batch size may be helpful.
  * `steps`: If specified, stop training for each epoch after the given number of batches.
  * `epochs`: the number of times to iterate through all training data during training.
  * `loss`: [Keras loss function](https://keras.io/losses/). For integer classes, use
-   `sparse_categorical_cross_entropy`.
- * `metrics`: A list of [Keras metrics](https://keras.io/metrics/) to evaluate.
- * `optimizer`: The [Keras optimizer](https://keras.io/optimizers/) to use.
+   `sparse_categorical_cross_entropy`. May be specified either as a string, or as a dictionary
+   with arguments to pass to the loss function constructor. Custom losses registered with
+   `delta.config.extensions.register_loss` may be used.
+ * `metrics`: A list of [Keras metrics](https://keras.io/metrics/) to evaluate. Either the string
+   name or a dictionary with the constructor arguments may be used. Custom metrics registered with
+   `delta.config.extensions.register_metric` or loss functions may also be used.
+ * `optimizer`: The [Keras optimizer](https://keras.io/optimizers/) to use. May be specified as a string or
+   as a dictionary with constructor parameters.
+ * `callbacks`: A list of [Keras callbacks)(https://keras.io/api/callbacks/) to use during training, specified as
+   either a string or as a dictionary with constructor parameters. Custom callbacks registered with
+   `delta.config.extensions.register_metric` may also be used.
  * `validation`: Specify validation data. The validation data is tested after each epoch to evaluate the
    classifier performance. Always use separate training and validation data!
    * `from_training` and `steps`: If `from_training` is true, take the `steps` training batches
      and do not use it for training but for validation instead.
    * `images` and `labels`: Specified using the same format as the input data. Use this imagery as testing data
      if `from_training` is false.
+ * `log_folder` and `resume_cutoff`: If log_folder is specified, store read records of how much of each image
+   has been trained on in this folder. If the number of reads exceeds resume_cutoff, skip the tile when resuming
+   training. This allows resuming training skipping part of an epoch. You should generally not bother using this except
+   on very large training sets (thousands of large images).
 
 ### Network
 
-These options configure the neural network to train with the `delta train` command.
+For the `layers` attribute, any [Keras Layer](https://keras.io/api/layers/) can
+be used, including custom layers registered with `delta.config.extensions.register_layer`.  
 
- * `classes`: The number of classes in the input data. The classes must currently have values
-   0 - n in the label images.
- * `model`: The network structure specification.
-   folder. You can either point to another `yaml_file`, such as the ones in the delta/config/networks
-   directory, or specify one under the `model` field in the same format as these files. The network
-   layers are specified using the [Keras functional layers API](https://keras.io/layers/core/)
-   converted to YAML files.
+Sub-fields of the layer are argument names and values which are passed to the layer's constructor.  
 
+A special sub-field, `inputs`, is a list of the names of layers to pass as inputs to this layer.
+If `inputs` is not specified, the previous layer is used by default. Layer names can be specified `name`.
+
+```yaml
+layers:
+  Input:
+    shape: [~, ~, num_bands]
+    name: input
+  Add:
+    inputs: [input, input]
+```
+
+This simple example takes an input and adds it to itself.
+
+Since this network takes inputs of variable size ((~, ~, `num_bands`) is the input shape) it is a **fully
+convolutional network**. This means that during training and classification, it will be evaluated on entire
+tiles rather than smaller chunks.
+
+A few special parameters are available by default:
+
+ * `num_bands`: The number of bands / channels in an image.
+ * `num_classes`: The number of classes provided in dataset.classes.
 
 MLFlow
 ------
@@ -119,10 +201,18 @@ General
 -------
 
  * `gpus`: The number of GPUs to use, or `-1` for all.
+ * `verbose`: Trigger verbose printing.
+ * `extensions`: List of extensions to load. Add custom modules here and they will be loaded when
+   delta starts.
+
+I/O
+-------
  * `threads`: The number of threads to use for loading images into tensorflow.
- * `tile_size`: The size of a tile to load from an image at a time. For convolutional networks (input size is [~, ~, X],
-   an entire tile is one training sample. For fixed size networks the tile is split into chunks. This parameter affects
-   performance: larger tiles will be faster but take more memory (quadratic with chunk size for fixed size networks).
- * `cache`: Configure cacheing options. The subfield `dir` specifies a directory on disk to store cached files,
-   and `limit` is the number of files to retain in the cache. Used mainly for image types
-   which much be extracted from archive files.
+ * `tile_size`: The size of a tile to load into memory at a time. For fully convolutional networks, the
+   entire tile will be processed at a time, for others it will be chunked.
+ * `interleave_images`: The number of images to interleave between. If this value is three, three images will
+   be opened at a time. Chunks / tiles will be interleaved from the first three tiles until one is completed, then
+   a new image will be opened. Larger interleaves can aid training (but comes at a cost in memory).
+ * `cache`: Options for a cache, which is used by a few image types (currently worldview and landsat).
+    * `dir`: Directory to store the cache. `default` gives a reasonable OS-specific default.
+    * `limit`: Maximum number of items to store in the cache before deleting old entries.