You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: pyrasterframes/src/main/python/docs/supervised-learning.pymd
+49-23Lines changed: 49 additions & 23 deletions
Original file line number
Diff line number
Diff line change
@@ -1,12 +1,14 @@
1
1
# Supervised Machine Learning
2
2
3
-
In this example we will demonstrate how to fit and score an unsupervised learning model with a sample of Sentinel-2 data and hand-drawn vector labels over different [land cover](https://en.wikipedia.org/wiki/Land_cover) types.
3
+
In this example we will demonstrate how to fit and score a supervised learning model with a sample of Sentinel-2 data and hand-drawn vector labels over different [land cover](https://en.wikipedia.org/wiki/Land_cover) types.
4
4
5
5
```python, setup, echo=False
6
6
from IPython.core.display import display
7
7
from pyrasterframes.utils import create_rf_spark_session
The first step is to create a Spark DataFrame of our imagery data. To achieve that we will create @ref:[a catalog DataFrame](raster-catalogs.md#creating-a-catalog). In the catalog, each row represents a distinct area and time; and each column is the URI to a band's image product. In this example our catalog just has one row. After reading the catalog, the resulting Spark DataFrame may have many rows per URI, with a column corresponding to each band.
20
22
23
+
The imagery for feature data will come from [eleven bands of 60 meter resolution Sentinel 2](https://earth.esa.int/web/sentinel/user-guides/sentinel-2-msi/resolutions/spatial) imagery. We also will use the [scene classification (SCL)](https://earth.esa.int/web/sentinel/technical-guides/sentinel-2-msi/level-2a/algorithm) data to identify high quality, non-cloudy pixels.
The land classification labels are based on a smalls set of hand drawn polygons in the geojson file [here](https://github.com/locationtech/rasterframes/blob/develop/pyrasterframes/src/test/resources/luray-labels.geojson). The property `id` indicates the type of land cover in each area. For these integer values 1 is forest, 2 is cropland, 3 is developed areas.
62
+
The land classification labels are based on a small set of hand drawn polygons in the geojson file [here](https://github.com/locationtech/rasterframes/blob/develop/pyrasterframes/src/test/resources/luray-labels.geojson). The property `id` indicates the type of land cover in each area. For these integer values 1 is forest, 2 is cropland and 3 is developed areas.
63
+
64
+
We will create a very small Spark DataFrame of the label shapes and then join it to the raster DataFrame. Such joins are typically expensive, but in this case both datasets are quite small. To speed up the join for the small vector DataFrame, we put the `broadcast` hint on it. Spark will put a copy of it on each Spark executor.
59
65
60
-
We will create a very small Spark DataFrame of the label shapes and then join it to the raster DataFrame. Such joins are typically expensive but in this case both datasets are quite small. After the raster and vector data are joined, we will convert the vector shapes into _tiles_ using the @ref:[`rf_rasterize`](reference.md#rf-rasterize) function. This procedure is sometimes called "burning in" a geometry into a raster. The values in the resulting _tiles_ are the `id` property of the geojson; which we will use as labels in our supervised learning task.
66
+
After the raster and vector data are joined, we will convert the vector shapes into _tiles_ using the @ref:[`rf_rasterize`](reference.md#rf-rasterize) function. This procedure is sometimes called "burning in" a geometry into a raster. The values in the resulting _tile_ cells are the `id` property of the geojson; which we will use as labels in our supervised learning task. In areas where the geometry does not intersect, the cells will contain a NoData.
To filter only for good quality pixels, we follow the same procedure as demonstrated in the @ref:[quality masking](nodata-handling.md#masking) section of the chapter on NoData. Instead of actually masking we will just sort on the mask cell values later in the process.
88
+
To filter only for good quality pixels, we follow roughly the same procedure as demonstrated in the @ref:[quality masking](nodata-handling.md#masking) section of the chapter on NoData. Instead of actually setting NoData values in the unwanted cells of any of the imagery bands, we will just on filter out the mask cell values later in the process.
@@ -113,23 +124,22 @@ from pyspark.ml.evaluation import MulticlassClassificationEvaluator
113
124
from pyspark.ml import Pipeline
114
125
```
115
126
116
-
SparkML requires that each observation be in its own row, and those observations be packed into a single [`Vector`](https://spark.apache.org/docs/latest/api/python/pyspark.ml.html#module-pyspark.ml.linalg) object. The first step is to "explode" the tiles into a single row per cell/pixel with the `TileExploder` (see also @ref:[`rf_explode_tiles`](reference.md#rf_explode_tiles)). Then we filter out any rows that have `NoData` values (which will cause an error during training). Finally we use the SparkML `VectorAssembler` to create that `Vector`.
127
+
SparkML requires that each observation be in its own row, and those observations be packed into a single [`Vector`](https://spark.apache.org/docs/latest/api/python/pyspark.ml.html#module-pyspark.ml.linalg) object. The first step is to "explode" the tiles into a single row per cell or pixel with the `TileExploder` (see also @ref:[`rf_explode_tiles`](reference.md#rf_explode_tiles)). If a _tile_ cell contains a NoData it will become a null value after the exploder stage. Then we filter out any rows that missing or null values, which will cause an error during training. Finally we use the SparkML `VectorAssembler` to create that `Vector`.
117
128
118
-
It is worth discussing a couple of interesting things about the `NoDataFilter`. First we filter by the mask column. This achieves the filtering of observations only to the good pixels, without having to explicitly mask and assign NoData in all eleven columns of raster data. The other column specified is the `label` column. When it is time to score the model, the pipeline will ignore the fact that there is no `label` column on the input DataFrame.
129
+
It is worth discussing a couple of interesting things about the `NoDataFilter`. First, we filter out missing values in the mask column. Recall above we set undesirable pixelsto NoData, so they will be removed at this stage. The other column for the `NoDataFilter` is the `label` column. When it is time to score the model, the pipeline will ignore the fact that there is no `label` column on the input DataFrame.
119
130
120
131
```python, transformers
121
132
exploder = TileExploder()
122
133
123
134
noDataFilter = NoDataFilter() \
124
-
.setInputCols(cols + ['label', 'mask'])
135
+
.setInputCols(['label', 'mask'])
125
136
126
137
assembler = VectorAssembler() \
127
138
.setInputCols(bands) \
128
139
.setOutputCol("features")
129
140
```
130
141
131
-
We are going to use a decision tree for classification. You can swap out one of the other multi-class
132
-
classification algorithms if you like. With the algorithm selected we can assemble our modeling pipeline.
142
+
We are going to use a decision tree for classification. You can swap out one of the other multi-class classification algorithms if you like. With the algorithm selected we can assemble our modeling pipeline.
133
143
134
144
```python, pipeline
135
145
classifier = DecisionTreeClassifier() \
@@ -144,15 +154,15 @@ pipeline.getStages()
144
154
145
155
## Train the Model
146
156
147
-
Push the "go button"! This will actually run each step of the Pipeline we created including fitting the decision tree model.
157
+
Push the "go button"! This will actually run each step of the Pipeline we created including fitting the decision tree model. We filter the dataframe for only tiles intersecting the label raster, because the label shapes are relatively sparse over the imagery. It would be logically equivalent to either include or exclude this, but it is more efficient to do this filter because it will mean less data going into the pipeline.
148
158
149
159
```python, train
150
160
model = pipeline.fit(df_mask.filter(rf_tile_sum('label') > 0).cache())
151
161
```
152
162
153
163
## Model Evaluation
154
164
155
-
To view the model's performance, we first call the pipeline's `transform` method on the training dataset. This transformed dataset will have the model's prediction included in each row. We next construct an evaluator and pass it the transformed dataset to easily compute the performance metric. We could also use a variety of DataFrame or SQL transformations to compute our metric if we like.
165
+
To view the model's performance, we first call the pipeline's `transform` method on the training dataset. This transformed dataset will have the model's prediction included in each row. We next construct an evaluator and pass it the transformed dataset to easily compute the performance metric. We can also create custom metrics using a variety of DataFrame or SQL transformations.
As an example of using the flexibility provided by DataFrames, the code below computes and displays the confusion matrix.
181
+
As an example of using the flexibility provided by DataFrames, the code below computes and displays the confusion matrix. The categories down the rows are the predictions, and the truth labels are across the columns.
Because the pipeline included a `TileExploder`, we will recreate the tiled data structure. The explosion transformation includes metadata enabling us to recreate the _tiles_. See the @ref:[`rf_assemble_tile`](reference.md#rf-assemble-tile) function documentation for more details. In this case, the pipeline is scoring on all areas, regardless of whether they intersect the label polygons. This is simply done by removing the `label` column, as @ref:[discussed above](supervised-learning.md#create-ml-pipeline).
SELECT extent, crs, rf_assemble_tile(column_index, row_index, prediction, 128, 128) as prediction
199
+
SELECT extent, crs,
200
+
rf_assemble_tile(column_index, row_index, prediction, 128, 128) as prediction,
201
+
rf_assemble_tile(column_index, row_index, B04, 128, 128) as red,
202
+
rf_assemble_tile(column_index, row_index, B03, 128, 128) as grn,
203
+
rf_assemble_tile(column_index, row_index, B02, 128, 128) as blu,
189
204
FROM scored
190
205
GROUP BY extent, crs
191
206
""")
192
207
193
208
retiled.printSchema()
194
209
```
195
210
196
-
Take a look at a sample of the resulting output. Recall the label coding: 1 is forest (purple), 2 is cropland (green) and 3 is developed areas(yellow).
211
+
Take a look at a sample of the resulting output and the corresponding area's red-green-blue composite image.
0 commit comments