Merge pull request #261 from s22s/docs/fix-248

Add band to concepts, address issue 248 items on catalog doc page

Author: vpipkt

pyrasterframes/src/main/python/docs/concepts.md

@@ -45,6 +45,10 @@ A "NoData" (or N/A) value is a specifically identified value for a cell type use
 
 A scene (or granule) is a discrete instance of EO @ref:[raster data](concepts.md#raster) with a specific extent (region), date-time, and map projection (or CRS).
 
+## Band
+
+A @ref:[scene](concepts.md#scene) frequently defines many different measurements captured a the same date-time, over the same extent, and meant to be processed together. These different measurements are referred to as bands. The name comes from the varying bandwidths of light and electromagnetic radiation measured in many EO datasets. 
+
 ## Coordinate Reference System (CRS)
 
 A [coordinate reference system (or spatial reference system)][CRS] is a set of mathematical constructs used to translate locations on the three-dimensional surface of the earth to the two dimensional raster grid. A CRS typically accompanies any EO data so it can be precisely located. 
@@ -57,10 +61,10 @@ An extent (or bounding box) is a rectangular region specifying the geospatial co
 
 A tile (sometimes called a "chip") is a rectangular subset of a @ref:[scene](concepts.md#scene). As a scene is a raster, a tile is also a raster. A tile can conceptually be thought of as a two-dimensional array. 
 
-Some EO data has many bands or channels. Tiles in this context are conceptually a three-dimensional array, with the extra dimension representing the bands.
+Some EO data has many @ref:[bands](concepts.md#band) or channels. Within RasterFrames, this third dimension is handled across columns of the DataFrame, such that the tiles within DataFrames are all two-dimensional arrays.
 
 Tiles are often square and the dimensions are some power of two, for example 256 by 256.
 
-The tile is the primary discretization unit used in RasterFrames. Each band of a scene is in a separate column. The scene's overall @ref:[extent](concepts.md#extent) is carved up into smaller extents for each tile. Each row of the DataFrame contains a two-dimensional tile per band column.
+The tile is the primary discretization unit used in RasterFrames. The scene's overall @ref:[extent](concepts.md#extent) is carved up into smaller extents and spread across rows. 
 
-[CRS]: https://en.wikipedia.org/wiki/Spatial_reference_system
+[CRS]: https://en.wikipedia.org/wiki/Spatial_reference_system

pyrasterframes/src/main/python/docs/raster-catalogs.pymd

@@ -1,15 +1,15 @@
 # Raster Catalogs
 
-While much interesting processing can be done on a @ref:[single raster file](raster-read.md#single-raster), RasterFrames shines when _Catalogs_ of raster data are to be processed. In its simplest form, a _Catalog_ is a listing of @ref:[URLs referencing raster files](raster-read.md#uri-formats). This listing can be manifested as a DataFrame, CSV file or CSV string. The _Catalog_ is input into the `raster` DataSource, described in the next page.
+While much interesting processing can be done on a @ref:[single raster file](raster-read.md#single-raster), RasterFrames shines when _catalogs_ of raster data are to be processed. In its simplest form, a _catalog_ is a list of @ref:[URLs referencing raster files](raster-read.md#uri-formats). This list can be a Spark DataFrame, Pandas DataFrame, CSV file or CSV string. The _catalog_ is input into the `raster` DataSource, described in the @ref:[next page](raster-read.pymd), which creates tiles from the rasters at the referenced URLs.
 
-A _Catalog_ can have one or two dimensions:
+A _catalog_ can have one or two dimensions:
 
-* One-D: A single column containing one or many URLs across the rows. All referenced rasters represent the same content type. For example, a column of URLs to Landsat 8 NIR rasters covering Europe. Each row represents different places and times.
-* Two-D: Many columns containing raster URLs. Each column references the same content type, and each row represents the same place and time. For example, red-, green-, and blue-band columns for scenes covering Europe. Each row represents a single spatiotemporal location (or scene) with the same dimensions, extent, [_CRS_][CRS], etc across the row.
+* One-D: A single column contains raster URLs across the rows. All referenced rasters represent the same @ref:[band](concepts.md#band). For example, a column of URLs to Landsat 8 near-infrared rasters covering Europe. Each row represents different places and times.
+* Two-D: Many columns containing raster URLs. Each column references the same band, and each row represents the same place and time. For example, red-, green-, and blue-band columns for scenes covering Europe. Each row represents a single @ref:[scene](concepts.md#scene) with the same resolution, extent, [_CRS_][CRS], etc across the row.
 
 ## Creating a Catalog
 
-This section will provide some examples of creating your own _Catalogs_, as well as introduce some experimental _Catalogs_ built into RasterFrames. Reading raster data represented by a _Catalog_ is covered in more detail in the @ref:[next page](raster-read.md).
+This section will provide some examples of creating your own _catalogs_, as well as introduce some experimental _catalogs_ built into RasterFrames. Reading raster data represented by a _catalog_ is covered in more detail in the @ref:[next page](raster-read.md).
 
 ```python, echo=False
 from IPython.display import display
@@ -25,30 +25,37 @@ A single URL is the simplest form of a catalog.
 
 ```python
 from pyspark.sql import Row
-my_cat = "https://modis-pds.s3.amazonaws.com/MCD43A4.006/04/09/2018185/MCD43A4.A2018185.h04v09.006.2018194032851_B01.TIF"
-# or
-my_cat_df = spark.createDataFrame([Row(B01=my_cat)])
+
+file_uri = "/data/raster/myfile.tif"
+# Pandas DF
+my_cat = pd.DataFrame({'B01': [file_uri]})
+
+# equivalent Spark DF
+my_cat = spark.createDataFrame([Row(B01=file_uri)])
+
+#equivalent CSV string
+my_cat = "B01\n{}".format(file_uri)
 ```
 
-A single column represents the same content type with different observations along the rows. In this example it is band 1 of MODIS, which is visible red. In the example the location of the images is the same (h04v09) but the dates differ.
+A single column represents the same content type with different observations along the rows. In this example it is band 1 of MODIS surface reflectance, which is visible red. In the example the location of the images is the same, indicated by the granule identifier `h04v09`, but the dates differ: 2018185 (July 4, 2018) and 2018188 (July 7, 2018).
 
 ```python
 scene1_B01 = "https://modis-pds.s3.amazonaws.com/MCD43A4.006/04/09/2018185/MCD43A4.A2018185.h04v09.006.2018194032851_B01.TIF"
 scene2_B01 = "https://modis-pds.s3.amazonaws.com/MCD43A4.006/04/09/2018188/MCD43A4.A2018188.h04v09.006.2018198232008_B01.TIF"
 
-# As CSV string
-my_cat = '\n'.join(['B01', scene1_B01, scene2_B01])
-# or
-my_cat_df = spark.createDataFrame([Row(B01=scene1_B01), Row(B01=scene2_B01)])
-my_cat_df.printSchema()
-# or
-pandas_cat = pd.DataFrame([{'B01': scene1_B01}, {'B01': scene2_B01}])
-my_cat = spark.createDataFrame(pandas_cat)
+# a pandas DF
+one_d_cat = pd.DataFrame({'B01': [scene1_B01, scene2_B01]})
+
+# equivalent spark DF
+one_d_cat = spark.createDataFrame([Row(B01=scene1_B01), Row(B01=scene2_B01)])
+
+# equivalent CSV string
+one_d_cat = '\n'.join(['B01', scene1_B01, scene2_B01])
 ```
 
 ### Two-D
 
-Example of a multiple columns representing multiple content types (bands) across multiple scenes. In each row, the scene is the same. The first column is band 1 and the second is band 2, near infrared.
+Example of a multiple columns representing multiple content types (bands) across multiple scenes. In each row, the scene is the same: granule id `h04v09` on July 4 or July 7, 2018. The first column is band 1, red,  and the second is band 2, near infrared.
 
 ```python
 scene1_B01 = "https://modis-pds.s3.amazonaws.com/MCD43A4.006/04/09/2018185/MCD43A4.A2018185.h04v09.006.2018194032851_B01.TIF"
@@ -68,36 +75,37 @@ my_cat_df.printSchema()
 
 ## Using External Catalogs
 
-The simplest example of an external _Catalog_ is a DataFrame (or a transformation of) in one of the formats above. Here's an extended example of reading an external CSV file of MODIS scenes and transforming it into a _Catalog_. The metadata describing the content of each URL is an important aspect of processing raster data. This example includes some minimal metadata.
+The concept of a _catalog_ is much more powerful when we consider examples beyond constructing the DataFrame, and instead read the data from an external source. Here's an extended example of reading an cloud-hosted CSV file containing MODIS scene metadata and transforming it into a _catalog_. The metadata describing the content of each URL is an important aspect of processing raster data. 
 
 ```python
 from pyspark import SparkFiles
 from pyspark.sql import functions as F
 
 spark.sparkContext.addFile("https://modis-pds.s3.amazonaws.com/MCD43A4.006/2018-07-04_scenes.txt")
 
-# The scenes list file has index URIs in the download_url column, for example:
-#    https://modis-pds.s3.amazonaws.com/MCD43A4.006/04/09/2018185/index.html
-# Image URIs take the form:
-#    https://modis-pds.s3.amazonaws.com/MCD43A4.006/04/09/2018185/MCD43A4.A2018185.h04v09.006.2018194032851_B01.TIF
-
-modis_catalog = spark.read \
+scene_list = spark.read \
     .format("csv") \
     .option("header", "true") \
-    .load(SparkFiles.get("2018-07-04_scenes.txt")) \
+    .load(SparkFiles.get("2018-07-04_scenes.txt")) 
+scene_list.head(3)
+```
+
+Observe the scenes list file has URIs to `index.html` files in the download_url column. The image URI's are in the same directory. The filenames are of the form `${gid}_B${band}.TIF`. The next code chunk builds these URIs, which completes our catalog.
+
+```python
+modis_catalog = scene_list \
     .withColumn('base_url',
         F.concat(F.regexp_replace('download_url', 'index.html$', ''), 'gid',)
     ) \
     .withColumn('B01' , F.concat('base_url', F.lit("_B01.TIF"))) \
     .withColumn('B02' , F.concat('base_url', F.lit("_B02.TIF"))) \
     .withColumn('B03' , F.concat('base_url', F.lit("_B03.TIF")))
-# ... and so on.
-modis_catalog.printSchema()
+modis_catalog.head(3)
 ```
 
 ## Using Built-in Experimental Catalogs
 
-RasterFrames comes with two experimental catalogs over the AWS PDS [Landsat 8][Landsat] and [MODIS][MODIS] repositories. They are created by downloading the latest scene lists and transforming as in the prior example.
+RasterFrames comes with two experimental catalogs over the AWS PDS [Landsat 8][Landsat] and [MODIS][MODIS] repositories. They are created by downloading the latest scene lists and building up the appropriate band URI columns as in the prior example.
 
 > Note: The first time you run these may take some time, as the catalogs are large. However, they are cached and subsequent invocations should be faster.
 

pyrasterframes/src/main/python/docs/raster-read.pymd

@@ -146,7 +146,7 @@ In the initial examples on this page, we used @ref:[`rf_tile`](reference.md#rf-t
 
 ## Multiband Rasters
 
-A multiband raster represents a three dimensional numeric array. The first two dimensions are spatial, and the third dimension is typically designated as different bands. The bands could represent intensity of different wavelengths of light (or other electromagnetic radiation), or they could represent other phenomena such as measurement time, quality indications, or additional measurements.
+A multiband raster represents a three dimensional numeric array. The first two dimensions are spatial, and the third dimension is typically designated as different @ref:[bands](concepts.md#band). The bands could represent intensity of different wavelengths of light (or other electromagnetic radiation), or they could represent other phenomena such as measurement time, quality indications, or additional measurements.
 
 When reading a multiband raster or a _Catalog_ describing multiband rasters, you will need to know ahead of time which bands you want to read. You will specify the bands to read, indexed from zero, passing a list of integers into the `band_indexes` parameter of the `raster` reader.